Prompt caching: 8 best practices for Claude, OpenAI and Gemini

What does prompt caching store?

A language model remembers nothing between requests. Send the same system instructions, the same document or the same code context a hundred times, and the model processes it a hundred times and you pay a hundred times. Prompt caching changes that: the provider processes the unchanging start of the request once and stores it; later requests that begin the same way read it from the cache.

What gets stored is not the text itself but the intermediate states the model computes while processing it; we unpack this in the next section. That is why the match has to be exact: if the prefix is not identical byte for byte, it cannot be read from the cache.

Three ideas often get mixed up:

Source: the documentation of OpenAI, Anthropic, vLLM and GPTCache, 15 October 2026.

We deal with this every day: when diffox reviews a pull request, many specialist agents read the same context in separate calls (how that works is covered in another post).

How the cache works

Before a language model starts generating a response, it reads the whole input; this phase is called prefill. Along the way it computes intermediate states for every token, called keys and values. OpenAI’s documentation calls them KV states: the model uses them to look back at earlier tokens while it generates the response. That is what prompt caching stores: not the text itself, but the tensors the model computed while processing it.

That is where the gain comes from. If the prefix is in the cache, the model skips computing that part and processes only the new tokens. vLLM’s documentation also says where the gain stops: the cache speeds up processing the input, not generating new tokens. So the work that gains most from caching has a long prefix and a short response. The Prompt Cache paper measured time-to-first-token improvements of up to 8× on GPU and 60× on CPU.

This structure also explains why the prefix must be identical byte for byte. vLLM’s design document identifies each block in the cache by two things: the block’s own tokens and the tokens of the prefix before it. So the same sentence is a different block when different text comes before it. Add a single date at the start and every block after it becomes a new block; the cache only helps up to the point where something changed.

These states are not small, either. The PagedAttention paper notes that the KV cache memory of each request is huge and grows and shrinks over the course of the request. That is why providers keep caches for only minutes: according to Google Cloud’s documentation, in-memory cache is limited and entries are evicted as capacity is reclaimed. Anthropic also says these representations are held in memory only and are not stored at rest.

How to enable prompt caching at four providers, and what it saves

All four providers support caching, but they differ in how you turn it on, how long the prefix must be and what it costs. We took the table from each provider’s own documentation on 15 October 2026.

Source: the providers’ documentation, 15 October 2026. Multipliers are relative to the uncached input price. The Google Cloud row covers Gemini 2.5 and later.

On Anthropic and on OpenAI’s GPT-5.6 and later models, writing to the cache costs more than sending the same tokens uncached, and reading costs far less. So a cache that writes at 1.25× pays for itself on the first read, and Anthropic’s one-hour cache, which writes at 2×, on the second. On Gemini’s explicit cache a write costs the standard price, but a storage fee is charged for every hour the cache is kept; whether it pays off depends on how many reads you get and how long you keep the cache.

In the Claude API, when many calls read the same context, put the breakpoint on the shared block and each call’s own instruction after it. ephemeral is currently the only supported cache type; the model name comes from the documentation’s own example:

{
  "model": "claude-opus-5-5",
  "max_tokens": 1024,
  "system": [
    {
      "type": "text",
      "text": "<shared context: the same text on every call>",
      "cache_control": {"type": "ephemeral"}
    },
    {
      "type": "text",
      "text": "<this call’s own instruction>"
    }
  ],
  "messages": [
    {"role": "user", "content": "<this call’s question>"}
  ]
}

How long does the cache stay warm?

An entry in the cache does not last forever: it has a lifetime, its TTL (time to live). Claude Code’s documentation describes the state in two words: while the entry is within its TTL the cache is warm, and once the TTL has run out it is cold. On a warm cache the next request reads the prefix cheaply; on a cold one it processes the prefix from scratch and pays the write price.

Source: the providers’ documentation, 15 October 2026. The documentation does not state the lifetime of Gemini’s implicit cache; on Google Cloud it is at most 24 hours, and most entries are evicted sooner.

The TTL works like a timer, and two details matter. On Anthropic the timer runs from the start of the request, not from the end of the response: if a response took 4 minutes to stream, the next request that reads the same prefix must start within about 1 minute. And on Anthropic, OpenAI and Bedrock every hit restarts the timer, so steady traffic keeps the cache warm on its own.

Most providers have no endpoint that tells you whether the cache is still warm. The exception is Gemini’s explicit cache: cachedContents.get always returns the expiry time (expireTime), though not the content. On Anthropic, OpenAI and Bedrock the signal is the next response: if the number of tokens read from the cache has dropped to zero, either the prefix changed or the entry went cold. Anthropic’s cache diagnostics tell the two apart: if the requests are identical and the read is zero, the entry has expired.

In Claude Code, the TTL depends on how you pay. According to the documentation, on a Claude subscription the main conversation uses a 1-hour cache as long as you stay within your plan’s included usage. Once you move to usage credits, or if you use an API key or a cloud provider, it uses a 5-minute cache. Subagents get 5 minutes even on a subscription. You can change the TTL with the promptCacheTtl setting or the CLAUDE_CODE_PROMPT_CACHE_TTL variable.

You can see the state in three places. /usage says whether the cache is warm or cold, along with the TTL in effect. In the JSON the status line receives, the prompt_cache object has the fields warm, ttl and expires_at; the last one is the moment the cache goes cold. And the cache clock in the VS Code extension counts down from a 5-minute or 1-hour lifetime and shows the minutes left; on a 1-hour cache it starts at 60 minutes.

Three things to watch:

• The gap between requests must stay under the TTL. If requests come less often than every 5 minutes, each one writes the prefix again. Anthropic recommends staying on the 5-minute cache if the prefix is used more often than every 5 minutes, because refreshing it is free.
• A longer TTL has a price. On Anthropic a 1-hour write costs 2×, and it does not pay for itself until the prefix has been read twice; Gemini’s explicit cache charges storage for every hour.
• The first request after a long gap is expensive. On a cold cache the first request writes the whole prefix. You can warm the cache before the real request arrives: on Anthropic with max_tokens: 0, and on OpenAI’s GPT-5.6 and later models with prompt_cache_options.prewarm.

In a ten-call example, the prefix costs 78.5% less

The numbers are round and hypothetical; they only illustrate the documented multipliers. Say 10 calls read a shared prefix of 50,000 tokens. The unit is one token billed at the base input price; we take a write as 1.25× and a read as 0.1×. Anthropic’s 5-minute cache and most of OpenAI’s GPT-5.6 and later models use these multipliers.

uncached       10 × 50,000 × 1      = 500,000

cached          1 × 50,000 × 1.25   =  62,500   first call writes
              + 9 × 50,000 × 0.1    =  45,000   nine calls read
                                    = 107,500

Ten calls started one after another pay the equivalent of 107,500 tokens for the prefix: 78.5% less than the 500,000 they pay without the cache. On a model whose read multiplier is 0.05× the total falls to 85,000. On the 1-hour cache the write rises to 100,000 and the total to 145,000.

The third bar is the worst case: the same ten calls all start at the same moment. None of them can read what another wrote, so if each one writes the prefix itself, the total is 625,000. That is 25% more than not turning the cache on at all; the section on parallel calls below explains why.

8 rules for a high cache hit rate

The four providers give the same advice on cache hits: each rule below appears in the documentation of at least two of them.

The second rule is the one most often skipped. One line in Claude Code’s release notes shows why it matters: moving the date out of the system prompt raised the hit rate. The date changes every day, and if it sits at the start of the prefix, the first request of each new day rewrites the whole prefix.

Two rules are specific to one provider. On OpenAI’s models before GPT-5.6, you give requests that share a prefix a stable prompt_cache_key. It helps route them to the same cache, and the documentation recommends about 15 requests per minute per key. Anthropic asks for a stable order of JSON keys in tool calls: according to its documentation, some languages such as Swift and Go randomise key order, so the same content goes out as different bytes.

What invalidates the cache

On Anthropic and Bedrock the cache follows a hierarchy: tools, system, messages. A change at one level invalidates that level and every level after it. Here are six of the cases Anthropic’s documentation lists:

Turning citations on or off has the same effect as web search, and adding or removing images in the prompt has the same effect as tool_choice. The fourth card is what the documentation calls a “common mistake”: if the breakpoint sits on a block that contains a timestamp or the user’s message, every request writes a new entry and no request can read one. The system looks back at most 20 blocks, but it is looking for an entry an earlier request wrote, not for unchanged content. The cache is written only at the breakpoint. So the breakpoint goes on the last block that stays identical across requests.

Caching with parallel calls

It is tempting to start all the calls that read the same prefix at once. Anthropic’s documentation is clear on this: for concurrent requests, a cache entry only becomes available after the first response begins; if you need parallel requests to read from the cache, wait for the first response before sending the rest. OpenAI also notes that its cache lives on individual machines, and that more than 15 requests a minute can be routed to other machines.

The first response does not have to finish, only to begin. The documentation offers three ways to do this:

• Send the first call on its own. Claude Code does this itself when a workflow starts several agents with the same prefix at once: it holds all but the first until the first agent’s response begins, for at most 5,000 milliseconds by default.
• Warm the cache in advance. A request sent with max_tokens: 0 writes the cache at the breakpoint and returns without generating output; no output tokens are billed. The breakpoint must sit on the shared block, and the thinking setting and the output_config.effort value must match the real calls. The request is rejected if it comes with settings that need output (stream: true, for example).
• Use the 1-hour cache for batch work. In the Batch API, requests are processed in any order and a read is not guaranteed. The documentation recommends sending the single request that carries the shared prefix first, on the 1-hour cache, and the rest once it completes.

How to measure your hit rate

Don’t assume the cache is working; check the response. All four providers report the tokens read from the cache in a separate field of the response’s usage data:

Source: the providers’ documentation, 15 October 2026.

The hit rate is the share of total input that was read from the cache. OpenAI defines it this way and recommends aggregating it by user, workspace or day. Getting total input right matters: on Anthropic and Bedrock the input field does not count the cached tokens, so the total is the sum of three fields; on Gemini, promptTokenCount already includes the tokens read from the cache.

Total input is the sum of three fields. input_tokens alone does not show all of the input; it counts only what comes after the last breakpoint. When everything works, the write field on the first call equals the length of the prefix, and on later calls the same number shows up in the read field. If both fields are zero, the prompt was not cached; the most likely reason is a prefix under the minimum length. If the write field stays high on every call, something in the prefix is changing.

We record each call’s three fields separately. If a call comes back without usage data, we count it as unmeasured, not as zero: a gap recorded as zero looks like a free call.

When caching doesn’t pay

Caching is not worth it for every workload. Before you turn it on, check for these four cases:

• The prefix is read only once. If a write costs 1.25× and no second call reads the prefix, you pay 25% more.
• The gap between calls is longer than the TTL. Every call finds the cache cold and writes the prefix again.
• The prefix is under the minimum length. The request works, the cache does not, and no error comes back.
• The prefix changes on every request. A date written at the start turns every request into a new write.

The gain is not only in price: with long prefixes, the first token also arrives sooner. Providers differ on rate limits: according to Anthropic and Bedrock, on Claude the tokens read from the cache do not count against the rate limit, while on OpenAI they count towards the tokens-per-minute limit.

In diffox, the specialist agents in a review read the same shared context: the change’s diff and the file context. That context sits in a single block at the start of the request, and that block carries the cache breakpoint. The first call goes out on its own, and the rest start in parallel after it. We record the tokens each call writes and reads separately.