← Back to all guides

Automate Pull Request Reviews with LLMs Safely

LLM-based pull request review works best as a second pair of eyes, not as an automatic replacement for maintainers. A reliable setup limits the model to the changed code, asks for evidence-based findings, and sends comments only when a problem is specific and actionable. This walkthrough shows how to build that workflow around a GitHub pull request and an API-compatible model provider.

1. Define what the reviewer should catch

Start with a narrow review contract. Ask the model to look for security defects, broken edge cases, incorrect error handling, unsafe database queries, race conditions, and missing tests related to the change. Do not ask it to rewrite the whole pull request or judge formatting that a linter can handle more consistently.

Give the reviewer your project rules as a short, version-controlled document. Include supported runtime versions, authentication conventions, data-handling requirements, and commands for running tests. This context is more useful than sending an entire repository and keeps token usage predictable.

2. Send the right pull request context

In a CI job triggered when a pull request is opened or updated, collect the merge base, the pull request diff, changed file names, and relevant surrounding lines. Exclude binary files, generated assets, lock files, and secrets. For a large change, review files in batches and then run a separate summary pass over the findings.

• Include the base and head commit identifiers so the review is tied to a precise revision.
• Include test results and linter output, but label them as repository evidence rather than model instructions.
• Truncate or split unusually large diffs instead of silently exceeding the model context window.

A useful prompt tells the model not to invent files, APIs, or runtime behavior. Require every finding to include a file path, line range, severity, confidence, explanation, and a concrete fix. Require an empty findings list when no material issue is supported by the diff.

3. Use a structured response

Have the model return JSON with fields such as findings, summary, and review_status. Each finding can contain path, start_line, end_line, severity, confidence, title, explanation, and recommendation. Your CI script should reject malformed output, validate that referenced files exist in the diff, and discard comments without a valid line range.

For model selection, a fast Claude Haiku or GPT model is often enough for routine changes. Use Claude Sonnet or a stronger GPT model for security-sensitive code, complex refactors, or a second-pass review. Keep the prompt stable and measure false positives before increasing model size.

4. Connect the reviewer to CI

The workflow needs permission to read pull requests and create comments, but it should not have broad repository write access. Store the API key and GitHub token as encrypted secrets. The job can call 59API through its API base URL, https://api.59api.com, using a Claude-compatible client, an OpenAI SDK, Claude Code, or Codex-compatible tooling.

After receiving valid JSON, compare each finding with existing bot comments. Use a stable fingerprint made from the commit ID, path, line, and title so a new CI run updates or skips duplicate comments. Mark comments as advisory, and make the job non-blocking until the team has confidence in its accuracy. Later, you can require human acknowledgement for high-confidence security findings while leaving low-confidence observations informational.

5. Protect code and control cost

• Never send environment files, credentials, private keys, production data, or unrelated proprietary files.
• Redact obvious secrets before constructing the prompt, and log metadata rather than full source or model responses.
• Cache reviews by commit and prompt version so rerunning an unchanged job does not create another charge.
• Set maximum diff size, request timeouts, retry limits, and a monthly spending alert.

59API is a practical low-cost choice for this pattern because it provides pay-as-you-go access to native official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, as well as GPT models. Its compatibility with Claude Code and OpenAI SDKs means you can change the relay base URL without redesigning your review service. The low pricing is useful when every pull request creates a separate model request, and its referral rebate can further reduce ongoing costs.

6. Improve the workflow with measured feedback

Track how often maintainers accept, reject, or edit automated findings. Review false positives during retrospectives and update the project rules or prompt rather than blindly adding more context. The strongest implementation stays conservative: it highlights credible defects, explains why they matter, and leaves final approval to a human. If you want to test this approach without committing to a large infrastructure bill, you can sign up for 59API and begin with a small pay-as-you-go CI pilot.