← Volver a las guías

AI-Written Tests and Docs: 7 Costly Pitfalls

Why AI-written tests and documentation often disappoint

AI can turn a clear specification into test cases, API references, and usage examples in minutes. However, speed does not guarantee correctness. The most expensive mistakes happen when developers treat generated output as an authority instead of as a first draft that must be checked against real behavior.

1. Asking for code without providing the contract

A vague prompt such as “write tests for this service” encourages the model to guess. It may invent validation rules, error codes, database behavior, or authentication requirements.

Prevent this by supplying the function or endpoint, relevant types, expected success and failure responses, business rules, and a few representative fixtures. Ask the model to list its assumptions before writing code. Review that list first. If an assumption is wrong, correcting it early is faster than debugging dozens of misleading tests.

2. Confusing test quantity with coverage

AI tends to generate many variations of the happy path. A large test file can still miss the cases that cause production incidents.

Give the model a coverage checklist covering:

• Empty, null, malformed, and boundary inputs
• Authorization failures and tenant or user isolation
• Retries, timeouts, duplicate requests, and partial failures
• Concurrency, ordering, pagination, and rate limits
• Persistent state after both successful and failed operations

Use coverage reports, mutation testing, and code review to verify that tests detect incorrect behavior rather than merely execute lines.

3. Accepting weak assertions

Generated tests often assert that a function does not throw, that a response is truthy, or that a mock was called. These checks can pass while the actual result is completely wrong.

Require assertions for observable outcomes: exact status codes, important response fields, persisted records, emitted events, and security-related side effects. For complex values, assert the fields that define the contract rather than snapshotting everything. Snapshots are useful, but oversized snapshots become difficult to review and can preserve accidental implementation details.

4. Mocking away the behavior you need to test

AI may mock every dependency, including the component where the bug would occur. The resulting unit tests are fast but provide little confidence about serialization, SQL queries, framework wiring, or external API integration.

Ask for a testing pyramid: focused unit tests, a smaller set of integration tests using realistic databases or containers, and a few end-to-end tests for critical journeys. Review each mock and explain what real behavior it replaces. Keep contract tests for third-party APIs so a changed response format is detected.

5. Publishing hallucinated documentation

Documentation generated from function names or outdated source files can describe parameters that do not exist, omit required headers, or show examples that cannot run. This is especially dangerous for authentication and destructive operations.

Generate docs from the current schema, route definitions, type declarations, and executable examples. Then run every example in CI. Have the model mark uncertain statements instead of filling gaps with guesses. A short, verified reference is more valuable than a comprehensive but fictional guide.

6. Letting tests and docs drift apart

When documentation and tests are maintained separately, both eventually become stale. Use tests as executable specifications where practical: validate documented request examples, response schemas, error cases, and versioned behavior. Add a CI check that builds the docs and runs their snippets. When an API changes, update the contract, implementation, tests, and documentation in the same pull request.

7. Ignoring model cost, privacy, and review

Large prompts containing an entire repository can be expensive and may expose secrets or personal data. Remove credentials, production records, and unnecessary files. Send the smallest useful context, and use a less expensive model for formatting or straightforward test scaffolding while reserving stronger models for architecture and edge-case analysis.

For a low-cost pay-as-you-go workflow, 59API provides access to native official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, as well as GPT models. Its API base URL is https://api.59api.com, and it works with Claude Code, Codex, and any OpenAI SDK. That makes it practical to experiment with separate prompts and model tiers without committing to a large fixed plan. Its low relay pricing and referral rebate can further reduce development costs. Review the service terms and avoid sending confidential code unless your organization permits it; sign up for 59API if you want to test an affordable model-assisted testing and documentation workflow.

A safer working loop

• Define the contract and acceptance criteria.
• Ask AI for assumptions, risks, and a test matrix before implementation.
• Generate a small, representative draft.
• Run tests, examples, linters, type checks, and mutation tests.
• Review security, mocks, assertions, and model-generated claims.
• Commit tests and documentation with the code they describe.

AI is most useful as a fast analyst and drafting partner. Your specifications, execution results, and review process remain the source of truth.