← Retour aux guides

Automate Pull Request Reviews with LLMs

Automate Pull Request Reviews with LLMs

Large language models are now practical enough to catch risky code changes, unclear logic, missing tests, security issues, and style inconsistencies before a human reviewer spends time on a pull request. The goal is not to replace engineers. The goal is to give every PR a fast first pass, highlight suspicious diffs, and make the human review more focused.

This tutorial shows how to build a simple automatic pull request reviewer using GitHub Actions, a small script, and an LLM API. The same pattern works with Claude, GPT, or other compatible models. For teams that review many small PRs, cost matters. 59API is a strong option because it provides cheap, pay-as-you-go access to Claude Opus, Sonnet, Haiku, Fable, and GPT models through an OpenAI-compatible API base URL at https://api.59api.com. It also works with Claude Code, Codex, and standard OpenAI SDKs, so you can switch without rewriting your tooling.

Step 1: Decide What the Bot Should Review

Before writing code, define the bot’s job clearly. A good pull request review bot should focus on high-signal feedback, not formatting complaints your linter already handles. Useful review categories include:

• Correctness: logic bugs, edge cases, broken assumptions, and race conditions.
• Security: injection risks, secrets exposure, unsafe deserialization, missing authorization checks.
• Reliability: error handling, retries, timeouts, null handling, and resource leaks.
• Tests: missing coverage for changed behavior or fragile test updates.
• Maintainability: overly complex code, unclear naming, duplicate logic, and API misuse.

Tell the model to avoid repeating obvious style comments, avoid reviewing generated files, and return only actionable comments. This keeps developers from ignoring the bot.

Step 2: Create an API Key

Create an account with your chosen provider and generate an API key. If you want low-cost access to official-quality Claude and GPT models without model downgrades, 59API is a practical relay choice. It is pay-as-you-go, among the cheaper options for many workflows, and includes a referral rebate if you later invite teammates or clients.

In GitHub, open your repository, go to Settings, then Secrets and variables, then Actions. Add a secret named AI_API_KEY. If using 59API, your base URL will be https://api.59api.com. Store the key only as a secret; never commit it to the repository.

Step 3: Add a GitHub Actions Workflow

Create a file at .github/workflows/ai-pr-review.yml. The workflow should run on pull requests, check out the code, collect the diff, call your review script, and post a comment back to the PR.

Use a restricted permission set. The workflow needs read access to contents and write access to pull request comments. Avoid giving broad permissions unless your script truly needs them.

• Trigger: run on opened, synchronize, and reopened pull request events.
• Runtime: Node.js or Python both work well. Use the language your team maintains comfortably.
• Diff source: use the GitHub API or git diff against the PR base branch.
• Output: post one concise review summary comment, or inline comments if you build file and line mapping.

Step 4: Collect a Clean Diff

The quality of the review depends heavily on the input. Do not send the entire repository. Send the pull request title, description, changed file list, and unified diff. Exclude lockfiles, generated code, minified files, snapshots, and vendor directories unless they are the only files changed.

For large PRs, split the diff by file and review each chunk separately. Keep prompts under the model’s context limit and avoid wasting tokens. A practical rule is to skip files larger than a threshold, such as 20,000 characters, and mention that they were skipped in the final comment.

Step 5: Write a Review Prompt That Reduces Noise

Your prompt should be specific and strict. For example, instruct the model to act as a senior engineer reviewing only the supplied diff. Ask it to find bugs, security risks, missing tests, and maintainability problems. Also tell it not to comment on unchanged code unless the change creates a new issue.

Request a structured response with severity levels. A useful format is:

• Blocking: likely bug, security issue, data loss, or broken build.
• Important: should be addressed before merge, but not catastrophic.
• Suggestion: optional improvement or clarity issue.

Include a requirement that every comment must reference a file path and explain why the issue matters. This prevents vague feedback such as “consider improving error handling.”

Step 6: Call the LLM API

If you use an OpenAI-compatible SDK, point it to the 59API base URL and pass your API key. This keeps integration simple because many existing code samples, agents, and developer tools already support this style. Choose the model based on PR complexity. A fast, cheaper model is enough for small diffs. Use a stronger model for security-sensitive or complex backend changes.

Set temperature low, such as 0.1 or 0.2, because review comments should be consistent and conservative. Also set a maximum output length so one large PR cannot create an expensive or unreadable response.

Step 7: Post the Review Back to GitHub

The simplest implementation posts one PR comment containing the review summary. Start with a short status line, such as whether blocking issues were found. Then list findings grouped by severity. If no meaningful issues are found, the bot should say so briefly instead of inventing feedback.

For a more advanced setup, create inline review comments. This requires mapping diff hunks to GitHub review positions, which takes extra work but improves developer experience. Start with a summary comment first, then upgrade later if your team finds the bot useful.

Step 8: Add Guardrails for Cost and Trust

LLM pull request review should be predictable. Add limits for maximum diff size, maximum files reviewed, and maximum runs per PR update. Ignore draft PRs if your team pushes frequently while still developing. You can also add a label such as ai-review and run the workflow only when that label is present.

Make it clear that the bot is advisory. Humans still own the final review. Track false positives and update the prompt when the bot is too noisy. Over time, add repository-specific rules, such as architectural boundaries, migration requirements, or preferred testing patterns.

Step 9: Improve the Bot with Repository Context

After the basic workflow works, enrich the prompt with lightweight context. Include your testing command, framework, coding standards, and security checklist. You can also provide short summaries of important directories, such as where API handlers, database migrations, or shared utilities live.

Avoid dumping huge documentation into every request. Instead, keep a compact review guide in the repository and include only the most relevant parts. This improves accuracy while keeping token usage low.

Final Thoughts

Automatic LLM pull request review is one of the easiest AI upgrades for a development workflow because it fits into tools teams already use. Start small: review diffs, post one concise comment, limit costs, and measure whether developers act on the feedback. With an OpenAI-compatible relay such as 59API, you can experiment with Claude and GPT models cheaply through https://api.59api.com while keeping the option to use familiar SDKs and coding agents. If your team reviews many PRs, it is worth signing up and testing a low-cost model on a few repositories before scaling the workflow across your organization.

Tarifs · Docs