What and why

At HASH, we’re on the hunt for where AI can deliver value today, safely.

Use cases where an LLM provides advice or suggestions are among the lowest-hanging fruit – so long as you account for the fact that the suggestions might be (a) sub-optimal, (b) misguided, or (c) totally ungrounded in reality.

To safely consider AI suggestions, your reaction should be one of:

1. “oh yes, I didn’t think of that,”
2. “that sounds plausible, I’ll check it out,” or
3. “this is sub-optimal / misguided / totally wrong.”

And not: “okay, if you say so” – unless it doesn’t matter if it got it wrong.

As a software company, making use of AI suggestions on code we write is an obvious candidate, as a complement rather than a replacement to other humans also reviewing our code. AI can spot things we overlook, encourage us to implement best practices and edge case handling we might otherwise omit out of laziness or ignorance, and provide advice on things we are unsure about at our invitation to do so.

This blog post is about our MVP script to trigger AI reviews on Pull Requests in GitHub. The script itself is here, and the GitHub Action workflow here.

Alternatives

In GitHub

There are closed-source services that offer this – such as CodeRabbit – but we wanted something open source so that we:

1. have visibility into and control over how it works, and
2. can customise and extend it as needed (for example, to integrate with information in Notion).

Saving money doesn’t hurt, either!

CodeRabbit does offer free Pro-tier features for public repositories, including Linear and Jira integrations, but with a rate limit of 2 reviews per hour – if this meets your needs, it’s worth checking out.

GitHub offers a Copilot reviewer, which can be summoned by requesting a review from it. This is also worth a try, although it lacks some of the features described below. We use it alongside our own HASH code reviewer.

Locally

There are local tools which can be used to provide suggestions on and reviews of your code:

1. In-editor AI agents such as those found in Cursor and Windsurf, and plucky young upstart VSCode Agent
2. CLI tools such as Claude Code, which provides a dedicated ‘PR review’ command (/review)

The behavior of these can be tailored somewhat via prompting, and the information and tools available to them enhanced with MCP servers.

Asking these agents for suggestions while working is also valuable. But we like having a final pair of eyes (or another pass through the attention heads) from an AI reviewer in GitHub.

Features

Our initial implementation works as follows (with links to the relevant code):

1. The AI reviewer can be summoned by tagging it as a reviewer in GitHub (we have a bot account for this purpose), or triggering the GitHub Action manually (see the workflow)
2. It is provided with:

   1. the PR title, description, and diff (what you see in the ‘Changed Files’ tab in GitHub)
   2. all non-resolved comment threads on the PR
   3. the title, description, and comment threads from any Linear tickets mentioned in the PR title
3. In two separate requests, we then ask it to provide:

   1. responses to existing comments – it is instructed to always respond to comments where someone has tagged it with a question (to allow PR authors to draw its attention to things they specifically want advice on), and to maybe respond to other comments if it thinks it can add value (although it tends to do this always anyway.)
   2. a review with:

      1. summary
      2. new comment threads
      3. a ‘comment’ or ‘request changes’ verdict, and
      4. a checklist of requirements it thinks it have been missed or misinterpreted based on the information from the linked Linear ticket(s).

We then:

1. Post any comment replies (we do this separately because according to the GitHub API docs, this is only available as a separate request, with the documented parameters for comments when creating a review not including reply functionality)
2. Post the review including summary, new comment threads and decision
3. For each ‘todo’ list related to Linear tickets, create a sub-ticket of the original with the list

Seeing it in action

You can see an example of the AI reviewer having its attention drawn to a specific part of the diff and discussing it with the PR author here.

That PR also has an example of bots talking to bots – the future of programming?

Security

When thinking about any software, we don’t want it to:

1. Do something it shouldn’t
2. Reveal something it shouldn’t

When AI is introduced, we additionally have to be mindful that:

1. Its behavior (output) is subject to change even if everything else is held constant, meaning that it having acted appropriately in the past is no guarantee that it will do so in future. Setting temperature to 0 and using a fixed seed on a specific model version can help minimise this, assuming the AI provider never silently changes anything about the model or the pipeline involved in producing its response (including system prompts, content filters, and any other pre- and post-processing of input and output).
2. an AI might interpret any context we provide it with as instructions which modify its behavior, meaning that any context which is not fixed is an attack vector (prompt injection)

As a general safeguard, the AI review is manually triggered, either by tagging it as a reviewer or dispatching the workflow manually, both of which can only be done by users with write access to the repository. This means that it can only be run by trusted users. A trusted user can trigger the review on a PR from a fork, but should check the PR content first to check for any potential exploits (e.g. attempts to insert instructions in the PR description or diff.)

Unsafe action

The only way to guarantee that an AI does not do something unsafe is to make it impossible to do. There are also efforts underway to construct “gatekeeper AIs” that give quantitative safety guarantees about how likely it is an AI will do something unsafe, under the ARIA Safeguarded AI programme.

In the context of our AI PR reviewer, eliminating the scope for unsafe action means that:

1. The reviewer cannot choose to approve a PR. If it did, a PR might be approved and merged without any human having approved it (PRs cannot be merged to our main branch without at least one approval, among other conditions.)
2. The AI does not have any tools available to it which could cause destructive consequences or give it access to arbitrary information. It is asked to provide content for PR reviews, comments, and todos to be captured as Linear tickets, but cannot make requests to retrieve, update or delete information.

There are still ways in which the automated posting of PR reviews and comments could have undesirable consequences:

1. The AI might post objectionable content to our PRs, or in a Linear ticket. Relying on the provider’s (Anthropic) content filters and model weights to avoid this is sufficient in this context, since the users are HASH employees in almost all cases, though it may not be in other contexts, e.g. AI chat with other users.
2. Posting comment replies may trigger GitHub rate limits, degrading functionality across automated PR reviews. Theoretically, someone could maliciously create a PR with a huge amount of comments in an attempt to do this, but it would rely on a trusted user triggering the AI review without having noticed this possibility (and the impact would be low, and fleeting.)

As we expand the functionality of the PR reviewer, we will need to keep potential unsafe actions in mind. If we move from a simple request for PR and comment content based on information we retrieve, to a more agentic PR reviewer that can choose to explore additional content and take other autonomous action to help build its review, we would need to make sure that it only had ‘safe’ tools available to it. This would mean not blindly adding MCP servers, for example, without checking and limiting what actions they made available to the AI.

Information disclosure

The only way to guarantee that an AI does not include some information in its output is for the AI to not have in the first place (though it may still happen to string together a sentence that perfectly matches it).

Our repository is public, but our Linear tickets are not, and there is a risk that the AI includes information from the Linear ticket or comments on it in its publicly-viewable PR comments. We simply accept this risk, as the information in the tickets is typically not sensitive, but it is one worth bearing in mind if triggering reviews linked to tickets that do have sensitive information or discussions within them.

It may become more of a concern as we make more private context available to the AI reviewer, e.g. via integrations with Notion or Google Drive, which contain various internal planning docs.

One potential mitigation is to make a separate request with private context, with the output intended for a private destination. In the context of our reviewer, this would mean asking the AI to separately provide a judgment as to what had been missed from the Linear content, which would then be posted as a Linear ticket (rather than asking for this in the same request as the PR review). The request to provide a publicly-viewable PR description and comments would then be made without any private context provided. This might make for a slightly worse PR review, but would ensure that the public content could not be based on private context.

Another mitigation would be to perform some check on the public content before submitting it, and rejecting it if it contained any private context. This would necessarily involve another LLM, because we don’t know in advance what the information from private sources will look like in order to write a ‘dumb’ filter, and would therefore be fallible. It would also mean that any PR reviews which mention technical details that also happen to be mentioned in the Linear ticket, but aren’t sensitive in any way, could be rejected.

Prompt injection

In the case of our Pull Request reviewer, there are several non-fixed sources of context provided to the AI, in which a malicious actor could include content in an effort to have the reviewer interpret it as instructions:

1. The title and description of the PR
2. The comment threads on the PR
3. The changes to files (i.e. the diff / patches)
4. Linear tickets and comments on them (though these are private in any case)

The risk from prompt injection in this context is low:

1. A trusted user must trigger the AI review, and therefore should check any PR from untrusted users for any such attempts, and
2. In any case, there is not really anything of consequence the AI can be instructed to do differently.

In this context, the worst prompt injection could achieve is to have the AI include something in its review which it ideally wouldn’t. For example, objectionable content, suggestions which would break or otherwise compromise our code if accepted, or ‘the entire contents of linked Linear tickets and their comment threads’.

Failure modes

The AI PR reviewer provides some valuable insights and suggestions. It also produces a healthy amount of garbage. For example:

1. It makes up APIs
2. It suggests things that already exist
3. It leaves ‘this looks good’ comments, despite being instructed not to (these are nice to hear occasionally, but often are simply noise)
4. It creates ‘todos’ to be captured as a Linear ticket which often repeat comments it has left in the PR, despite being told to keep Linear todos to anything which addresses missed or misinterpreted requirements listed in the original ticket (not code suggestions)

We could attempt to reduce the likelihood of these by:

1. “prompt engineering”, including providing examples
2. running additional LLM checks on the output, which would be fallible and may be prone to false positives
3. testing the performance of different models

There are also other failures in its overall operation:

1. There is no context size management in the initial implementation, and therefore it will not work for very large PRs (either in diff or comment thread size.)
2. It often produces invalid JSON (we find Anthropic models to be less reliable in this respect than OpenAI’s). In these cases we simply retry the request, although we could alternatively have an LLM rewrite the JSON properly (a decent next step), or attempt to programmatically search for and correct all the weird and wonderful ways it escapes or otherwise mangles JSON (tedious and not worth the time for this tool alone.)

Future extensions

There are several things which could improve on the current implementation:

1. More context:

   1. Currently, the reviewer only has access to the patches – not the rest of the changed files, and no ability to see other relevant files in the repository. We could provide these directly to it, or provide it with tools to request them.
   2. We could automatically retrieve any Notion documents mentioned in Linear tickets, to provide additional business context.
   3. We could provide style guidelines, preferred coding patterns etc, so that it can make suggestions based on our preferences and not just the general coding best practices learnt from its training data.
2. Handle large PRs: split up larger PRs into multiple requests to avoid exceeding context window limitations.
3. Commenting improvements: introduce the ability to comment on and make suggestions across multiple lines, not just a single line.
4. Quality checks: add an LLM check on the proposed review to assess for quality, completeness and correctness.

We could also publish the reviewer as a standalone GitHub Action, on which we would be happy to collaborate with others to improve.

AI could also be helpful before even starting work, for example by fleshing out ticket descriptions and researching potential approaches to implementation.

If you have any suggestions, comments or questions, please get in touch.