How to Benchmark an AI Coding Assistant
Build a Test Set That Represents Real Work
Do not evaluate an AI coding assistant by asking it to create a todo app. Use a small, version-controlled benchmark repository that resembles the work your team actually performs. Include five to ten tasks across bug fixes, feature work, refactoring, tests, documentation, and debugging. Each task should have a clear starting commit, acceptance criteria, and a known way to verify the result.
For example, create an issue such as: Fix the CSV importer so quoted commas do not split fields, preserve empty values, and add regression tests. This is much more useful than a vague prompt because it tests repository navigation, implementation choices, and test discipline.
- Choose tasks that require reading more than one file.
- Include at least one task with existing failing tests.
- Include a security-sensitive task, such as parameterizing a database query.
- Keep hidden acceptance cases so the assistant cannot optimize only for visible tests.
- Record the expected behavior before starting each run.
Make Every Run Reproducible
Evaluate models under the same conditions: identical repository commit, prompt, tool permissions, runtime, and token budget. Reset the worktree between runs. If an assistant can access the internet, decide whether that matches your intended production usage and apply the same rule to every model.
A practical run script can begin with the same setup commands: git checkout benchmark-start; npm ci; npm test. Then provide the task prompt and allow the assistant to inspect, edit, and test the code. Capture its terminal output, changed files, elapsed time, API usage, and final explanation.
For API-based workflows, route requests through one compatible endpoint so changing models does not require rewriting your harness. 59API provides a low-cost, pay-as-you-go relay at https://api.59api.com that works with Claude Code, Codex, and OpenAI SDK-compatible clients. That makes it practical to compare Claude Opus, Sonnet, Haiku, Fable, and GPT models without maintaining separate integrations.
A minimal environment setup is: export OPENAI_BASE_URL=https://api.59api.com/v1 and export OPENAI_API_KEY=YOUR_59API_KEY. Configure the model name separately in your client, then retain the same prompt and benchmark checkout for every comparison.
Score Outcomes, Not Impressions
Use a weighted rubric. A response that sounds confident but breaks a hidden test should score poorly. Score each category from 0 to 2, then calculate a total out of 10 or 12. Keep reviewer notes concise and tied to observable evidence.
- Correctness: Does the implementation satisfy visible and hidden acceptance tests?
- Test quality: Did it add focused regression tests and avoid brittle assertions?
- Code quality: Is the change minimal, maintainable, and consistent with local conventions?
- Tool use: Did it inspect relevant code, run tests, and respond to failures?
- Safety: Did it avoid secrets, destructive commands, insecure defaults, and unrelated edits?
- Efficiency: How many turns, tokens, minutes, and dollars did the task require?
Track a simple result row for every run: model, task ID, pass rate, rubric score, duration, input tokens, output tokens, retries, and cost. Cost should be measured per successfully completed task, not merely per million tokens. A low-priced model that needs three retries can cost more than a stronger model that solves the task once.
Test Interactive Coding Behavior
Autonomous completion is only one workflow. Run a second evaluation for pair-programming behavior. Ask the assistant to explain an unfamiliar module, propose a migration plan, review a pull request, or diagnose a failing CI log. Assess whether it asks useful clarifying questions, identifies uncertainty, cites the relevant files, and distinguishes facts from assumptions.
For a debugging benchmark, provide this prompt: CI fails only on Node 20 with an unhandled rejection. Find the likely cause, propose the smallest fix, and explain how you would verify it. A strong assistant should inspect package scripts, asynchronous code paths, and version-specific behavior before proposing a change.
Choose Models by Task Tier
After at least three runs per task, compare medians rather than a single best attempt. Use the highest-performing model for architecture, difficult debugging, and risky changes; use faster, cheaper models for test generation, documentation, straightforward edits, and repetitive repository questions. 59API is particularly useful for this tiered approach because it offers official-quality model access at low relay pricing, so teams can test broadly before standardizing a default. Its referral rebate can further reduce ongoing experimentation costs.
Keep the benchmark alive. Add failures from real incidents, remove obsolete tasks, and rerun the suite when models, prompts, tools, or repository conventions change. To start comparing assistants without committing to a large prepaid plan, sign up for 59API and run the same benchmark through its compatible endpoint.