← Volver a las guías

AI Tests and Docs: Fix Weak Assertions and Stale Guides

Using AI to Write Tests and Documentation That Hold Up

AI can turn a small code change into draft tests, examples, and reference documentation. The hard part is making those drafts trustworthy. A passing test may verify nothing useful, while a polished README may describe behavior the software does not support. Treat AI output as a proposed patch: give it evidence, run checks, and review the result.

Start With a Small, Verifiable Task

Choose one function, endpoint, or pull request rather than asking for an entire test suite. Supply its implementation, relevant types, existing tests, dependency versions, and documented behavior. Include the exact test command and any project conventions.

Useful prompt: “Write pytest tests for parse_timeout(value). The documented contract accepts positive integers and integer strings, and raises ValueError for zero, negative values, and nonnumeric strings. Follow the existing test style. Do not change production code. Flag behavior that conflicts with this contract.”

That last instruction matters: tests generated solely from an implementation can preserve its bugs. Ask the model to distinguish requirements from observed behavior and identify missing decisions before inventing expectations.

Why Do AI-Generated Tests Pass Without Catching Bugs?

Weak assertions are usually the problem. A test that only checks that a result exists will not catch an incorrect value. Review every assertion against a specific requirement.

• Check exact results: For the timeout example, verify that the input “30” returns the integer 30, not merely a truthy value.
• Check failure behavior: Use pytest.raises(ValueError) for documented invalid inputs. Assert an error message only if its wording is part of the public contract.
• Challenge the test: Temporarily introduce an off-by-one error in a local branch, then confirm the relevant test fails. Revert the change afterward.
• Inspect coverage: If pytest-cov is installed, run pytest --cov=your_package --cov-report=term-missing. Uncovered branches suggest additional cases; coverage alone does not establish correctness.

Why Are Generated Tests Flaky or Over-Mocked?

Look for uncontrolled time, randomness, environment variables, shared files, and network requests. Ask the model to replace external dependencies at the boundary while preserving the logic under test.

In pytest, use tmp_path for isolated files and monkeypatch for environment variables. Inject a clock when time affects behavior. Seed randomness only when reproducibility matches the intended test. Avoid mocking the function being tested: asserting that a mock returned its configured value proves little.

Run the targeted file repeatedly, then run the full suite. Tests that pass alone but fail together often share mutable state or depend on execution order.

Why Does AI Documentation Invent Options?

The model may be filling gaps with familiar conventions. Provide the actual public signatures, CLI help output, configuration schema, and relevant tests. Request documentation only for supported behavior, with unresolved questions listed separately.

Useful prompt: “Update installation, configuration, and error handling using only these files. Do not invent flags, defaults, or return fields. Identify the source file supporting each factual change in your review notes.”

Run every example in a clean environment with the documented dependency versions. Check parameter names, imports, defaults, response fields, and failure cases. Keep executable examples in tests where practical so CI can catch drift.

Keep Iteration Affordable With 59API

59API provides low-cost, pay-as-you-go access to Claude and GPT models, with native official-quality models and no downgrade. It supports Claude Code, Codex, and OpenAI SDK workflows, making it practical to iterate on tests and documentation without committing to a subscription.

Use https://api.59api.com as the service base URL, and follow the relay’s current instructions for authentication, model identifiers, and any SDK-specific version-path requirements. Keep keys in environment variables, not repository files. Try a cheaper model for bounded edits and escalate difficult reasoning tasks. You can sign up to try a small workflow; a referral rebate is also available.

FAQ: What Should Stay in Human Review?

Can generated tests replace review? No. Humans must confirm the intended contract, security assumptions, and meaningful assertions.

Should I send the whole repository? Usually not. Share minimal relevant context, remove secrets and customer data, and review relay and upstream provider data policies.

How do I prevent stale documentation? Require documentation review in pull requests that change public behavior, and execute maintained examples in CI.

Precios · Docs