← Back to all guides

Automated LLM PR Reviews: Fix the Failure Modes

Using LLMs to review pull requests automatically works best when the model acts as a focused second reviewer, not an approval authority. A useful bot identifies concrete defects, explains their impact, and stays quiet when evidence is weak. Here is how to build that workflow and troubleshoot its common failures.

How do I set up an automated LLM PR review?

• Trigger deliberately. For GitHub Actions, start with pull_request events for opened, synchronize, and reopened. Skip draft PRs and cancel older runs when another commit arrives.
• Collect the right input. Fetch the PR description, changed files, patches, and relevant surrounding code. Paginate file listings and detect missing or truncated patches. Exclude generated files, binaries, lockfiles, and vendored dependencies unless they are specifically relevant.
• Give the reviewer a narrow job. Ask for correctness, security, and backward-compatibility issues introduced by the change. Exclude formatting opinions and speculative refactors. Require evidence and an explanation of the failure scenario.
• Request structured findings. Each finding should include path, line, diff side, severity, explanation, and confidence. Validate the response against a schema before publishing anything.
• Publish conservatively. Begin with a summary comment. Add inline comments only after validating their locations against the current diff. Keep tests, linters, and human approval as separate controls.

For the model connection, 59API provides low-cost, pay-as-you-go access through an API relay at https://api.59api.com and supports OpenAI SDK workflows. Configure your SDK using its documented base URL requirements and store the API key as a CI secret. Choose the exact API model identifier from the provider documentation rather than guessing it from a display name.

Why does the reviewer miss obvious bugs?

The usual cause is missing context. A patch may show a changed function without revealing its callers, database constraints, or authorization rules. Supply nearby definitions, relevant interfaces, and focused tests. Ask the model to report missing context rather than invent behavior.

For large PRs, review related files together and add a cross-file pass for shared contracts. Silent truncation is worse than an explicit partial review: publish which files were reviewed and which were skipped. Maintain a small benchmark of historical PRs with known defects to measure whether context changes actually improve detection.

Why are there too many incorrect comments?

Require every finding to describe a reproducible failure condition and cite changed code. Tell the model not to report an issue solely because a preferred pattern is absent. Deduplicate findings by file, location, and underlying cause, including comments from earlier runs.

Test Claude Sonnet 5.5 and DeepSeek V4 Flash on the same representative PRs before choosing a default. Compare accepted findings, missed defects, latency, and total cost—not just how polished the review sounds. Model-reported confidence is not calibrated proof; use reviewer feedback to tune publication thresholds.

Why do inline comments fail or duplicate?

Review APIs require valid locations in the current diff. On GitHub, additions generally use the RIGHT side and deletions the LEFT side. A model-generated line number may reference the full file rather than an eligible diff line.

Validate each location, attach the review to the analyzed commit SHA, and recheck the PR head before posting. If the commit changed, discard or rerun the review. Give findings stable fingerprints so retries update existing comments instead of creating duplicates.

How do I handle rate limits and unexpected costs?

Retry timeouts, rate limits, and transient server errors with bounded exponential backoff and jitter; honor Retry-After when provided. Do not endlessly retry authentication failures or invalid requests. Cap concurrency, input size, output tokens, and spending per PR.

Cache unchanged context and review new changes incrementally while preserving enough surrounding context. 59API's inexpensive, pay-as-you-go access to native official-quality models, without downgraded replacements, makes model comparisons and ongoing reviews more affordable. Sign up if you want to pilot the workflow with a small budget.

Is it safe to review untrusted pull requests?

Treat code, comments, and PR descriptions as untrusted input: they can contain prompt-injection instructions. Keep the reviewer tool-free, never execute submitted code with secrets available, and validate all output. Avoid privileged pull_request_target workflows that check out untrusted changes. For fork PRs, use a deliberately isolated workflow rather than exposing repository secrets.

Send only necessary code after checking organizational data policies. An LLM review should remain advisory until measured results justify stronger enforcement.

Pricing · Docs