Automate Pull Request Reviews with LLMs and 59API
Build an LLM Pull Request Reviewer That Developers Can Trust
Automatic pull request review is most useful when it catches concrete issues before a human reviewer spends time on them: missing null checks, unsafe authorization changes, broken error handling, risky database queries, and tests that no longer match behavior. An LLM should not replace code owners or merge gates. Instead, use it as a fast first-pass reviewer that posts structured, actionable findings on every pull request.
This hands-on example uses GitHub Actions, Python, and an OpenAI-compatible client. 59API is a practical choice for this workload because it provides pay-as-you-go access to native official-quality Claude and GPT models through one API relay. You can select a lower-cost model for routine diffs and reserve stronger models for security-sensitive or large changes, without rebuilding the integration.
1. Define a Narrow Review Contract
Do not ask the model to “review this code” with no constraints. Require findings that a developer can verify. Each finding should include a severity, file, line range, explanation, and suggested fix. Also tell the model to return an empty findings list when it is uncertain. This sharply reduces vague comments such as “consider improving readability.”
Recommended output shape: {"summary":"...","findings":[{"severity":"high|medium|low","path":"...","lines":"12-18","issue":"...","recommendation":"..."}]}
Start with three review categories: correctness, security, and test coverage. Add repository-specific rules later, such as “flag direct SQL string interpolation” or “require authorization checks on new admin routes.”
2. Collect the Pull Request Diff in GitHub Actions
Create .github/workflows/ai-review.yml. The workflow should run on pull_request, check out the head commit, fetch the base branch, and save the diff. Keep permissions minimal: the job needs repository contents read access and pull-request comment permission only if it will publish comments.
Example workflow commands: git fetch origin "${{ github.event.pull_request.base.ref }}" --depth=1; git diff "origin/${{ github.event.pull_request.base.ref }}"...HEAD --unified=20 > /tmp/pr.diff; python scripts/review_pr.py /tmp/pr.diff > /tmp/review.json
Exclude generated files, lockfiles, vendored dependencies, binaries, and diffs over a defined size. For large pull requests, split the diff by file or by approximately 8,000 to 12,000 characters, review each chunk, then send the collected findings to a final summarization call. Chunking prevents one large file from consuming the entire context window.
3. Call 59API with an OpenAI-Compatible Client
Store your API key as a GitHub Actions secret named API_59API_KEY and store the selected model identifier as LLM_MODEL. Use the model names available in your 59API dashboard. This keeps your code portable across Claude, GPT, and other supported models.
scripts/review_pr.py: import os, sys; from openai import OpenAI; diff = open(sys.argv[1]).read(); client = OpenAI(base_url="https://api.59api.com/v1", api_key=os.environ["API_59API_KEY"]); prompt = "You are a precise PR reviewer. Review only the supplied diff. Return valid JSON with summary and findings. Report only likely, actionable correctness, security, or test issues. Do not invent files, APIs, or line numbers. Diff:\n" + diff; result = client.chat.completions.create(model=os.environ["LLM_MODEL"], temperature=0, messages=[{"role":"user","content":prompt}]); print(result.choices[0].message.content)
Set temperature to zero for consistent reviews. In production, validate the returned JSON before posting it. If parsing fails, retry once with a short correction prompt asking for JSON only; do not publish raw model output as a pull request comment.
4. Publish One Useful, Idempotent Comment
Format findings into a single comment headed “AI review.” Include the summary, then group items by severity. Use a hidden marker such as AI-REVIEW-BOT in the comment body so subsequent workflow runs update the existing comment instead of creating noise on every push. Only post high- and medium-severity findings by default; low-severity notes can be included in the summary.
A good comment says why the issue matters and points to the changed code. For example: “High — api/orders.py lines 44-51: the new update query accepts order_id without verifying ownership. A signed-in user may modify another user’s order. Add a customer_id condition or perform an ownership check before the update.”
5. Make the Automation Safe and Cost-Controlled
- Never send secrets, .env files, production credentials, or private keys in the diff. Add path exclusions before calling the model.
- Do not run untrusted pull-request code with a write-capable token. Especially avoid using pull_request_target to check out contributor code.
- Keep the AI review advisory at first. Let humans decide whether findings block merges.
- Log model, prompt size, latency, and finding count so you can measure usefulness and tune prompts.
- Use an economical model for ordinary reviews, then route authentication, payment, and permission changes to a stronger model.
Because pull requests are frequent and often small, per-request pricing matters. 59API’s low-cost relay model makes it easier to run reviews on every change while retaining access to Claude and GPT-quality models through the same compatible API pattern. If you are ready to test this workflow, sign up for 59API, add a small usage limit, and pilot the bot on one non-critical repository first.