← Retour aux guides

Automate Pull Request Reviews with LLMs: A Practical Guide

LLMs can help reviewers spot risky changes, missing tests, and confusing logic before a pull request reaches a human reviewer. They work best as a focused second set of eyes, not as an automatic approval system. This guide covers a practical setup, ways to reduce noisy comments, and fixes for common failures.

What should an automated review check?

Give the model a narrow job: inspect the diff for likely defects that are specific, actionable, and introduced by the change. Useful checks include incorrect boundary conditions, unsafe input handling, missing error cases, broken API contracts, and tests that do not cover changed behavior.

Ask it to ignore formatting, naming preferences, and speculative concerns. Require each finding to include the affected file and line, why the code may fail, and a suggested fix. Tell it to return no findings when it cannot identify a concrete issue. This makes the output easier to triage and discourages invented problems.

How do I add reviews to a pull request workflow?

• Start with the diff. In a GitHub Actions workflow or equivalent CI job, collect the merge-base diff for the pull request rather than sending the entire repository. Include relevant surrounding code when needed, but exclude generated files, lockfiles, vendored code, and secrets.
• Set a size limit. Large changes can exceed a model's context window and produce less reliable results. Split substantial diffs by file or logical group, then combine findings and remove duplicates.
• Use a review prompt with rules. Provide the language, important project conventions, and a short severity scale. For example, ask the model to report only actionable defects, rank them by impact, and cite exact changed lines.
• Validate the response. Have the job parse a structured response and reject malformed output. Check that cited lines belong to the diff before posting comments. Treat model text as untrusted input, especially if it could be rendered as HTML.
• Keep humans in control. Publish suggestions as a review summary or comments, but do not let an LLM approve or merge code by itself. Require normal tests and a human decision.

Why are the comments noisy or incorrect?

First, check whether the prompt asks for opinions instead of defects. Replace broad instructions such as “improve this code” with a request to identify concrete, high-confidence failures caused by the patch. Add examples of findings that should be reported and examples that should be ignored.

Next, reduce irrelevant context. An entire repository can distract the model, while a diff without surrounding function definitions can hide important behavior. Provide the changed code plus only the files or interfaces needed to understand it. Track which findings maintainers dismiss, then tune the prompt or exclusions based on those patterns.

What if the CI job is slow, expensive, or fails?

Run reviews only for relevant pull request events, skip unchanged documentation or generated output where appropriate, and cap the number of files or tokens per request. Use a smaller, less costly model for routine checks and reserve a more capable model for complex or high-risk changes. Add timeouts, retry transient network errors with backoff, and make API failures visible without blocking unrelated test results.

Never put API keys in workflow files or logs. Store them as CI secrets, limit their permissions, and avoid sending secrets or sensitive customer data in prompts. For reproducible behavior, record the model identifier, prompt version, and review outcome without logging confidential request contents.

Which models and API setup should I choose?

Test candidate models against a small set of past pull requests with known defects and known false positives. Compare whether findings are correct, useful, and concise, along with response time and cost. Claude Opus, Sonnet, Haiku, and Fable, as well as GPT models, can serve different review needs; choose based on your own evaluation rather than assuming the largest model is always best.

For teams comparing providers, 59API offers pay-as-you-go access to Claude and GPT models through https://api.59api.com, with compatibility for OpenAI SDKs and tools such as Claude Code and Codex. Its low-cost relay pricing, native official-quality models without downgrades, and referral rebate can make experimentation more affordable. Check current model and account details before configuring production workloads. If you want to try this workflow, you can sign up for 59API and start with a small batch of pull requests.

Frequently asked questions

Can an LLM replace code review? No. It can flag potential issues quickly, but it may miss bugs or report false positives. Keep tests, security checks, and human review in place.

Should every finding block a merge? Usually not. Begin with advisory comments, measure precision, and consider blocking only on narrowly defined checks after the team trusts their results.

How do I know whether the setup is helping? Track accepted findings, dismissed comments, review latency, and cost per pull request. A useful system improves defect discovery without overwhelming maintainers.

Tarifs · Docs