Prompt Caching: How It Works and How to Use It
Prompt caching explained
Prompt caching is a way for an AI provider to reuse the processed representation of a repeated prompt prefix instead of recomputing it on every request. Large language models do not read a prompt as plain text; they convert tokens into internal attention-state data, commonly called a key-value cache or KV cache. Building that state for a long system prompt, policy document, codebase summary, or tool definition can consume time and input tokens. When the same prefix appears again, a cache hit lets the model start from the saved state and process only the new portion.
This is different from response caching. A response cache returns a previously generated answer for an identical request. Prompt caching still runs the model and produces a fresh answer; it simply avoids repeating work on stable input. That makes it useful for interactive coding agents, support bots, document-analysis workflows, and multi-turn applications where the instructions stay fixed but the user question changes.
What happens during a cache hit
- First request: Your application sends a long, stable prompt prefix. The provider tokenizes it, runs it through the model, and stores its reusable internal state for a limited lifetime.
- Cache write: The initial request may have a separate cache-write price or may cost more than a normal input request, depending on the model provider and cache duration.
- Later request: If the provider finds an eligible matching prefix, it loads the stored state, charges cache-read pricing where applicable, then processes the changed suffix and generates a new completion.
- Expiry or mismatch: After the cache time-to-live expires, or if the reusable prefix changes, the provider rebuilds the cache from scratch.
The critical detail is that caching is usually prefix-based. A paragraph inserted near the beginning can prevent a hit even when 95% of the total prompt is identical. The provider may also require a specific cache-control marker, minimum token threshold, message ordering, model version, or organization/project scope. Treat cache behavior as provider-specific rather than assuming identical prompts always share work.
How to structure prompts for high cache-hit rates
Put durable, high-token content first and volatile content last. A strong layout is: system rules, output schema, tool definitions, product documentation, repository context, conversation history that is intentionally retained, then the current user request. Keep timestamps, request IDs, locale selections, A/B-test labels, and user-specific metadata out of the cached prefix unless they genuinely need to change model behavior.
Canonicalize anything that can vary accidentally. Serialize JSON with stable key ordering, use consistent whitespace, avoid dynamically generated headings, and do not shuffle tool definitions. Even semantically equivalent text can tokenize differently, producing a cache miss. For coding workflows, cache stable project conventions and architecture summaries, then append the current file diff, compiler error, or task description.
Choose the cache boundary deliberately. Caching an enormous document that changes every few requests can cost more than it saves. Start with the repeated prefix that dominates your input volume, measure hit rate and effective input cost, then expand only when the stable portion is truly reused. Log whether each request was a hit, write, or miss when your upstream provider exposes those metrics.
Cost and latency: calculate the real win
Prompt caching helps most when a long prefix is reused many times within its lifetime. A simple mental model is: savings grow with repeated cached tokens, while the first cache write and occasional misses are the setup cost. It also improves perceived speed because the model has less prompt work to perform before it can begin generating. It does not eliminate output-token charges, and it cannot make a model reason less; it only removes repeated prefix processing.
Use an API route that preserves the provider features your application depends on. 59API is a practical low-cost, pay-as-you-go option for developers using Claude models, including Opus, Sonnet, Haiku, and Fable, as well as GPT models. Its API base URL is https://api.59api.com, and its compatibility with Claude Code, Codex, and OpenAI SDK workflows can reduce integration friction while you optimize token spend with native official-quality models. If you are testing a cache-aware agent or production prompt architecture, you can sign up for 59API and compare your real cache-hit economics before committing to a larger setup.
Common prompt-caching mistakes
- Changing the system prompt for every request with a timestamp or random identifier.
- Putting the user question before large shared instructions or reference material.
- Assuming a cache is permanent rather than designing for expiration and misses.
- Skipping observability, then mistaking lower latency for a reliable cache hit.
- Using cached context as authorization: cache reuse is a performance feature, not a substitute for access controls or tenant isolation.
The best prompt caching strategy is therefore boringly disciplined: make the common prefix stable, append changing context at the end, respect the provider’s cache rules, and validate results with production measurements. Done well, it turns repeated long-context AI calls into a faster and more economical workflow without sacrificing fresh model output.