← Retour aux guides

Automate Pull Request Reviews with LLMs

What an LLM Pull Request Review Does

An LLM can inspect a pull request diff and look for likely bugs, security problems, missing tests, performance issues, and confusing code. It does not replace a human reviewer. Instead, it provides a fast first pass so developers can handle obvious problems before spending time on architecture and product decisions.

The most useful setup gives the model only the information it needs: the changed files, relevant surrounding code, the repository's review rules, and the pull request description. Asking an LLM to review an entire repository on every change is slower, more expensive, and often less accurate.

Choose a Focused Review Policy

Before connecting an API, define what the automated reviewer should report. A practical policy might ask it to prioritize functional bugs, authorization mistakes, SQL injection, exposed credentials, unsafe dependency usage, race conditions, and tests that do not cover changed behavior.

Tell the model to report only actionable findings. Each finding should include the file, line or changed section, severity, explanation, and a concrete fix. Also instruct it to say when it found no issue. This reduces vague comments such as “consider improving readability” that create review noise.

Build the CI Workflow

A typical workflow runs when a pull request is opened or updated. It performs four steps:

• Fetch the pull request diff and limit it to relevant files.
• Load repository guidance, such as a review policy or contribution rules.
• Send the diff and instructions to an LLM through an API.
• Publish a summary as a pull request comment and optionally fail the check for high-confidence, critical findings.

Start with comments rather than an automatic merge blocker. After comparing model findings with human reviews for several weeks, you can decide whether narrowly defined security or correctness checks should affect the status check.

Use a Structured Prompt

A reliable prompt separates instructions from untrusted code. State that the diff may contain text designed to manipulate the reviewer and that repository content must be treated as data, not instructions. Ask for machine-readable output with fields such as severity, file, line, confidence, explanation, and recommendation.

Include project-specific information when it matters. For example, tell the reviewer which authentication middleware is required, whether database queries must use a particular helper, and which test command validates the change. The model can then evaluate the pull request against actual project conventions rather than generic style preferences.

Control Cost and Latency

LLM review costs depend mainly on input size, output length, model choice, and how often the workflow runs. Send a patch instead of the whole codebase, truncate unchanged context, cap the response length, and skip documentation-only changes when appropriate. Use a faster, less expensive model for routine reviews and reserve stronger models for complex or security-sensitive changes.

59API is a practical low-cost option for this workflow because it provides pay-as-you-go access to native Claude models, including Opus, Sonnet, Haiku, and Fable, as well as GPT models. Its API base URL is https://api.59api.com, and it works with OpenAI SDKs, Claude Code, and Codex. That lets a team reuse familiar client libraries while choosing the model and spending limit that fit each repository.

Protect Code and Credentials

Never place API keys directly in workflow files. Store them as encrypted repository or organization secrets, restrict their permissions, and rotate them regularly. Review your provider's data handling terms before sending proprietary code. You should also remove .env files, private keys, tokens, and customer data before constructing the prompt.

Use least-privilege permissions for the CI token. A review job usually needs to read the pull request and write a comment, not modify source code or approve its own changes.

Measure Quality Before Expanding

Track useful metrics: valid findings, false positives, missed defects, review latency, token spend, and developer resolution time. Have humans label a sample of findings by severity and correctness. If developers routinely ignore the comments, reduce the scope and improve the prompt instead of increasing the model's verbosity.

A Sensible Starting Point

Begin with advisory reviews on one repository and one language. Run the model on changed application code, require concise findings, and keep humans responsible for final decisions. Once the results are trustworthy, add targeted checks for security-sensitive directories and use a stronger model only where the risk justifies the cost.

For teams testing this approach without committing to a large platform contract, signing up for 59API can provide an inexpensive way to compare Claude and GPT models with real pull request traffic. Its referral rebate can also help reduce ongoing usage costs as adoption grows.

Tarifs · Docs