Ask most teams why they turned on prompt caching and you get some version of the same answer: it makes the model faster and it saves money on the graphics processing unit (GPU). Both things are true, and both things hide the part that actually decides whether caching pays off. Prompt caching lives or dies on where you put the cache. That is a storage question, and once you frame it that way, a lot of confusing cost behavior starts to make sense. 

Here is what is really happening, because the mechanics point directly at the answer. 

What You Are Caching 

When a model processes a prompt, the first thing it does is read every token of that prompt and build an internal representation of it. In a transformer, that representation is the key and value tensors for each layer, usually called the KV cache. This step is the prefill. It is compute heavy and it scales with the length of the prompt. Once prefill is done, the model can start generating output tokens, and each new token reuses the KV cache that prefill built. 

Here is the useful observation. If the same prompt prefix shows up again, whether that is a system prompt, a long retrieval-augmented generation (RAG) context, a few-shot example block, or the running history of an agent, you have already done the expensive prefill work once. You have two choices. Do it again from scratch on the GPU or hold onto the KV cache from last time and load it back. Prompt caching is just the second option with a name. 

The decision is not really about the GPU feature. It comes down to whether keeping that KV cache around costs less than regenerating it. 

The Tradeoff in Thirteen Words 

Recomputing prefill spends GPU time. Caching spends storage. That is the whole tradeoff. You are trading seconds of a GPU you are paying for against rent on a piece of storage that holds the cached state until the next hit. 

The interesting question is where those two costs cross. And there is a small piece of math that makes the answer cleaner than you might expect. 

The size of a KV cache grows with the number of tokens in the prompt. The cost of recomputing prefill also grows with the number of tokens. When you set the two costs equal to find the break-even, the token count appears on both sides and cancels out. What you are left with is not a number of tokens. It is a length of time. 

In plain terms: the break-even for caching is a reuse window. The question is never “is this prompt long enough to cache.” The question is “will I see this prompt again before the storage rent adds up to the cost of one recompute.” A 2,000-token system prompt and a 100,000-token document have the same break-even window on the same hardware and the same storage. That surprises people, and it is worth sitting with, because it means you can reason about caching policy without getting lost in token accounting. 

Where the Break-Even Lands 

Now put real storage tiers into that reuse window and the picture gets practical. Think of storage as a ladder, from the fast expensive tier near the GPU down to the slow cheap tier far from it. 

At the top is GPU memory itself, the high-bandwidth memory (HBM) on the card. It is the fastest place to keep a KV cache, and it is by far the most expensive real estate you own. After the model weights are loaded, a single card only has room for a few hundred thousand tokens of cache, and every gigabyte you spend holding old context is a gigabyte you are not spending on serving live requests. On typical numbers, holding a cache in HBM only beats recomputing it if you reuse that cache within about a minute. That is not a cost you tune. It falls out of how much cache fits in memory and how fast the GPU can refill it. Keep KV in HBM for the genuinely hot stuff and let everything else move down the ladder quickly. 

The middle of the ladder is host memory and local NVMe (non-volatile memory express) storage. This is the warm tier. It is much larger and much cheaper per gigabyte than HBM, and it still sits inside the same server, so reload is quick. Here the break-even stretches from minutes into a few hours depending on the tier and the model. This is a comfortable home for context you expect to touch again during a session. 

The bottom of the ladder is where the economics really open up. Shared file storage over NFS (network file system), or object storage like S3, is cold, pooled, and cheap. It usually runs one to two orders of magnitude less per gigabyte than the HBM it displaces. Drop the storage rent that far and the break-even window stretches from seconds all the way out to many hours, and often to a couple of days. On rented cloud GPUs, where you are paying by the second at a premium, caching a 70B model’s context to object storage pays off whenever you reuse the prompt within roughly two days. Almost every real workload reuses prompts far sooner than that. Conversation turns come back in seconds. RAG chunks and system prompts repeat constantly. Agent loops revisit the same context over and over. 

That is the heart of it. Cheap shared storage moves the break-even from “only cache the hottest prompts for a minute” to “cache almost everything and keep it for a day.” The tier you choose changes the answer by a factor of a thousand. 

Why Shared Storage Is the Lever, Not Just a Cheaper Shelf 

It would be easy to read the above as “cheaper storage is better, film at eleven.” The reason shared storage matters is more specific than price, and it comes down to three things working together. 

The first is the obvious one. Lower cost per gigabyte directly buys you a longer reuse window, and a longer window means a higher share of your traffic clears the bar for caching. Price is the lever that moves the break-even. 

The second is pooling, and this is the part that gets undersold. In a multi-GPU box, or across a cluster, you do not want each card keeping its own private copy of the same cached prefix. That wastes capacity and it wastes hits, because a request that lands on card 3 cannot benefit from a cache that only card 7 has. A centralized KV store fixes this. Tools like NVIDIA Dynamo manage exactly this kind of tiered, shared KV cache, letting the cache spill from HBM down to host memory, to local disk, and out to networked storage, with any GPU able to reach any cached prefix. The working set is stored once and shared, instead of duplicated per card. On a big, shared context, that difference decides whether your cache hit rate is any good at all. 

The third is that cheap does not have to mean slow enough to matter. There is a bandwidth threshold hiding in this problem. Reloading a cached prefix is only worth it if the read is faster than regenerating the prefix on the GPU. Work out that threshold and it lands at a few hundred megabytes per second per stream for a large model. Modern NFS, NVMe, and object tiers clear that comfortably.  The cheap tier is usually also fast enough to win on latency, not just on cost. You get a better time to first token and a lower bill from the same move. 

Put those three together and shared storage stops being a place you dump cold data. It becomes the thing that makes fleet-wide caching economical in the first place. 

The Cloud Versus On-Prem Twist 

There is one more wrinkle that changes the recommendation depending on where you run, and it is worth being honest about. 

When you rent GPUs in the cloud, you pay for every second the card is busy, and the rate is high. Recomputing prefill is genuinely expensive because those GPU seconds have a clear price tag. Caching to shared storage almost always wins, and it wins over a wide reuse window. Turn it on, tier aggressively, and move on. 

On owned hardware the story flips if your GPUs are not saturated. If you bought the cluster and it has idle headroom, the marginal cost of a recompute is close to just the power it draws, which is nearly nothing. In that world, recompute is cheap and the break-even windows collapse to minutes, so caching only pays for fast reuse or when it lands on the very cheapest shared tier. The moment those same owned GPUs get busy enough that a recompute displaces paying work, the calculus swings right back toward caching, because now the GPU second has an opportunity cost again. On-prem, the honest answer is that your caching policy should track your utilization. Idle boxes favor recompute. Busy boxes favor cache. Cloud is simply the busy case with the meter always running. 

What to Take Away 

If you remember one thing, make it this. Prompt caching is not a switch you flip on the GPU to get free savings. It is a decision about where the KV cache lives, and the tier you pick sets your break-even by a factor of a thousand. Keep the hottest context in GPU memory for the seconds it stays hot. Let everything else fall to a shared, pooled, reasonably fast storage layer that any GPU in the fleet can read. Price the recompute path against the storage path using a reuse window rather than a token count and remember that your own GPU utilization changes the number. 

Do that and the default inverts. Instead of recomputing prefill and caching only the obvious wins, you cache almost everything and recompute only the rare cases that sit unused past the break-even. The savings are real, they scale with your traffic, and they come from a storage architecture decision more than a model one. That is usually the cheapest kind of win to get, because it does not ask you to buy another GPU. It asks you to put the cache in the right place.