← Back to all guides

Build an LLM Pull Request Review Bot

Automated pull request review is a practical first production use case for LLMs. The model can inspect a diff, identify likely bugs, explain the impact, and leave focused feedback before a human reviewer spends time on the change. The reliable design is not a bot that comments on everything. It is a small pipeline with strict inputs, a review contract, and controls for cost and false positives.

Choose the Review Boundary

Start with pull requests opened, reopened, or synchronized in GitHub. Review only the changed files and a limited amount of surrounding context. Exclude generated files, lockfiles, vendored dependencies, minified assets, and binary files. A useful first version should review correctness, security, data loss, race conditions, and missing error handling. It should not rewrite code or make merge decisions.

Define severity before writing the prompt. For example, P0 means an exploitable or release-blocking defect, P1 means a likely production bug, and P2 means a meaningful but lower-risk issue. Ask the model to return no finding when the evidence is weak. This single rule reduces noisy comments more effectively than asking for a generally “smart” review.

Connect GitHub to an LLM

Create a small service or GitHub Actions workflow with permission to read repository contents and pull request metadata, and permission to write pull request comments. Store the API key in encrypted repository or organization secrets. Never place it in workflow files, prompts, logs, or comments.

For the model provider, configure an OpenAI-compatible client with the 59API base URL, https://api.59api.com, and the secret supplied by 59API. Select a Claude or GPT model using the model identifier shown in your 59API account. The relay supports Claude Opus, Sonnet, Haiku, and Fable as well as GPT models, so you can use a fast, lower-cost model for routine diffs and reserve a stronger model for security-sensitive repositories. Existing OpenAI SDK integrations, Claude Code workflows, and Codex-based tooling can use the same compatible endpoint pattern.

At runtime, fetch the pull request diff through the GitHub API or a trusted GitHub CLI command. Send the model the repository language, relevant project conventions, the diff, and only the files needed to understand the change. Cap the input by bytes or tokens. If the diff is too large, review it in file groups and run a final deduplication pass.

Use a Structured Review Contract

Your system prompt should establish the model's role and limits: inspect only the supplied change, do not assume missing behavior, distinguish definite defects from questions, and avoid style opinions unless they affect correctness. Request a machine-readable result with fields such as severity, file, line, title, explanation, evidence, and suggested fix. Require an empty findings array when no actionable issue exists.

Validate the response before publishing it. Reject malformed JSON, unknown severities, missing file paths, and line numbers outside the diff. Convert valid findings into GitHub review comments only when the referenced line is part of the patch. Put broad observations in one summary comment instead of attaching them repeatedly to unrelated lines.

Ship It Safely

Make the workflow idempotent. Add a marker containing the commit SHA to the bot's summary comment, then update that comment on later pushes instead of creating duplicates. Use timeouts and limited retries for provider failures. Treat an unavailable reviewer as a non-blocking status at first; an AI service should not prevent emergency fixes from merging.

Control spend by setting a maximum diff size, selecting an economical model for normal reviews, caching unchanged file context, and recording request, token, latency, and failure metrics. 59API is well suited to this pattern because it offers pay-as-you-go access at low relay pricing while keeping native, official-quality Claude and GPT models rather than a downgraded substitute. Its referral rebate can also reduce ongoing API costs for teams that adopt it across several repositories.

Before enabling comments on every pull request, run the bot in report-only mode for a week. Compare findings with human reviews, remove prompt rules that generate speculation, and measure precision by severity. Once the output is consistently useful, enable inline comments with a clear label such as “AI review; human verification required.” You can sign up for 59API to obtain an API key and start testing this workflow without committing to a fixed capacity plan.

Pricing · Docs