Your serving engine runs out of KV cache memory at the context length and concurrency your application needs. Lowering those limits would make the deployment fit, but it would also change what users can do. The next step is choosing a memory optimization that addresses the actual constraint.

Quantization, token eviction, paging and offloading change different parts of memory use, with different costs in accuracy, latency and complexity. This guide compares those tradeoffs and explains how cross-request reuse fits alongside them. A smaller cache, a better-managed allocation, and less repeated computation solve different problems.

Why KV cache memory grows so fast

In the basic full-attention case, the cache holds key and value tensors for each token at each layer until the request finishes. So it grows with every token the model reads and writes, and it grows again with every concurrent user, against a memory pool your model weights already occupy.

Three of the terms in that growth are fixed the moment you pick a model: how many layers it has, how many key-value heads each layer carries, and how wide each head is. You choose them when you choose the checkpoint, which is why architecture is a lever even though it doesn't appear in a config file you control.

Batch size, context length and precision are configurable. Batch size multiplies the cache across independent requests, so it trades against concurrency and you feel it immediately. Context length multiplies it within a single request, and it's the one that hurts, because a long-context workload can exhaust GPU memory for one user. Precision is the third and scales everything linearly.

None of that needs a number to be actionable, and the numbers exist elsewhere. The mechanism and a per-model memory table live in our KV cache guide; the formula, the config fields it reads and worked examples live in our KV cache calculator. If your problem is instead that context you already paid to compute gets thrown away and recomputed next request, that is a different failure, and it is the one our Amnesia Tax post is about.

Four levers, and the one that gets pulled first

Set architecture aside, since you rarely get to revisit it, and four things remain that you can do to a cache that doesn't fit: throw tokens away, store each token in fewer bits, waste less of the memory you allocated, or move the cache off the GPU. Reuse is the fifth and sits in a category of its own because it doesn't shrink the cache at all.

Precision is the cleanest of the four. Storing K and V in eight bits instead of sixteen halves the footprint by construction, the engines all support it behind a flag, and the entire difficulty is what it does to accuracy, which is model-specific, format-specific and calibration-specific. We give that argument its own piece. It's usually the first lever worth trying, and it isn't the one people get wrong.

The one people get wrong is eviction.

Eviction is the obvious lever, and the one that disappoints

Attention does not weight all past tokens equally. H2O's authors found that a small subset contributes most of the attention score, called those tokens Heavy Hitters, and designed an eviction policy that retains a balance of recent tokens and heavy hitters.

StreamingLLM's authors found that keeping only the most recent tokens fails once the sequence exceeds the cache window. Retaining the initial tokens alongside that window largely recovers performance. Those initial tokens act as attention sinks: they attract strong attention scores even when their semantic content is unimportant.

‍

Our KV cache guide introduces block allocation. In the example described by NVIDIA Research, each physical block holds KV data for about 16 tokens and can be freed only when it is completely empty. Eviction policies operate on tokens. Allocators operate on blocks.

Work the case NVIDIA Research works and the problem is obvious. Evict ninety percent of a long sequence's tokens by importance score and the survivors are, by construction, scattered across the whole sequence, because importance is not correlated with position. Almost every allocated block still holds at least one survivor, so almost every block stays pinned and the allocator hands back close to nothing, even though your eviction metric says you dropped nine tokens in ten.

Eviction policies differ in how effectively they free allocated blocks. A sliding window drops a contiguous range of positions, so the blocks it empties are contiguous and empty completely. A heavy-hitter policy drops tokens by score regardless of position, which is exactly the pattern that leaves one survivor per block. Dropping tokens therefore does not always free physical blocks.

So read every compression result as a token-count reduction until proven otherwise, and measure reclaimed GPU memory rather than evicted token count.

The headline numbers deserve the same skepticism. H2O reports throughput improvements of up to 29x, 29x, and 3x against DeepSpeed Zero-Inference, Hugging Face Accelerate, and FlexGen, respectively, at twenty percent heavy hitters, on OPT-6.7B and OPT-30B. The gain varies substantially with the baseline. StreamingLLM's 22.2x is a speedup over a sliding-window recomputation baseline in streaming settings, not a memory saving and not a throughput result against a modern stack.

Fragmentation is a different problem from size

Before compressing the cache, check how much of its allocation holds usable token state. The PagedAttention paper identifies fragmentation and redundant duplication as sources of wasted KV memory that limit batch size. These are allocation problems, not changes to the tensor size itself.

That is why paging belongs in a different category from every technique above. It doesn't make your cache smaller. It shrinks the gap between your cache and your allocation, which for most people is the larger of the two problems, and it also enables cache sharing: sequences with identical prompt prefixes can point at the same physical blocks, with a copy-on-write duplication of the single affected block when one diverges.

RadixAttention often gets listed next to PagedAttention, and the two aren't the same kind of thing. SGLang's paper describes it as a KV cache reuse mechanism. Paging is an allocation strategy that fights fragmentation. Radix-tree matching is a reuse strategy that fights recomputation. SGLang's headline 6.4x throughput figure belongs to the whole system, which combines RadixAttention with compressed finite state machines and a frontend language, not to the reuse mechanism alone.

Block size affects both reclamation and reuse. In NVIDIA's example, an 80-token cache stored in 64-token blocks can reuse the first 64 tokens but must recompute the remaining 16. With 16-token blocks, all 80 tokens fit into five complete blocks. Smaller blocks can therefore improve reuse granularity and memory reclamation.

The granularity argument that makes eviction disappoint also makes reuse leak.

Where the cache goes when it leaves the GPU

Offloading is the least ambiguous lever. The cache doesn't get smaller, it moves, and you pay in transfer time instead of memory. The transfer path between those tiers determines the cost.

LMCache can offload KV state to CPU RAM or local storage on the inference machine, or to remote backends such as Redis, Mooncake, Valkey and Infinistore. CPU RAM can hold the most recently used subset of the cache backed by disk or remote storage, while distributed coordination extends access across servers.

CPU RAM also serves as an intermediate buffer for transfers between the GPU and local or remote storage. The pinned CPU allocation must therefore be non-zero even when you do not intend to retain cached data in host memory.

In the illustrated offload path, host RAM stages transfers between GPU memory and filesystem-backed storage.

The defaults are asymmetric. CPU offload is on out of the box; disk offload is not, and its maximum size defaults to zero, because LMCache can operate without disk storage. They allocate differently too: the pinned CPU buffer is claimed greedily up front, while the disk backend creates one file per cache chunk as chunks arrive and evicts least-recently-used when it fills. Disk and remote writes are asynchronous to keep their I/O latency off the inference path, while reads block, which tells you where offload latency shows up: on the fetch path, not the store path.

Prefetching can reduce fetch latency by loading cached tokens into pinned CPU RAM before a request arrives. This is useful when upcoming context is predictable, as in structured or agentic workflows.

LMCache's documented examples, both on a single L4 with 23 GB of GPU memory and a 15,376-token prompt on Llama-3.1-8B-Instruct, show a warm second request at 44.5x faster time to first token via CPU RAM and 42.6x via local disk. Those are documentation examples rather than benchmarks, and the disk run deliberately disables CPU offload to isolate disk latency. On that setup, the latency difference between RAM and disk was small; measure it on your own hardware.

Whether offloading helps depends on transfer bandwidth as well as storage capacity, a point we examine in Why LLM Inference Is a Data Problem. Decode already tends to be memory-bandwidth-bound; offloading introduces another, slower transfer path to measure.

Reuse changes the bill, not the bytes per token

Every technique so far acts on the cache within a request, with offload as a partial exception. LMCache can retain KV state beyond individual calls and even process restarts when backed by disk or external storage.

Reuse goes further, because it operates between requests, and it's the only one of the five that doesn't reduce a single byte of KV cache memory. Prefix caching does not cut the bytes you spend per cached token. In a runtime with a fixed KV pool it does not cut the pool either, since vLLM allocates the whole block pool when the cache manager starts up; what caching changes is which blocks stay resident and reusable rather than how large the pool is.

What changes on the invoice is how often you pay to build the cache from nothing.

This is the distinction sizing discussions lose. Gigabytes are a capacity constraint and decide whether you can serve the request at all. Recomputation is a cost constraint and decides what the request costs you. A workload can be comfortable on the first and ruinous on the second; a classic example is forty users reading the same document, every conversation carrying the same system prompt, or an agent replaying its whole context each loop iteration.

Prefix caching reuses the KV blocks of an identical prefix, avoiding repeated computation without changing model outputs. That reuse requires an exact prefix match. Any change before the reuse point invalidates everything after it, which is why agent loops that mutate their context, or RAG pipelines that shuffle retrieved chunks, can lose reuse after the first changed block.

CacheBlend extends reuse beyond matching prefixes. It can reuse cached chunks at shifted positions using re-RoPE and partial recomputation. Check our compatibility matrix to confirm non-prefix caching support for your dense model before deployment.

Tensormesh Platform provides non-prefix caching for supported dense models. To use the plugin image from our private registry, request an access token.

In our own Agent Skills measurements, CacheBlend reached cache hit rates of 63.6 to 85 percent on skill-related content. That is our own measurement, on one agent workload, with no hardware or trial count published. It's a strong result on a narrow workload, not a general one.

Tensormesh Platform supports vLLM, with prefill/decode disaggregation so those phases can run on separate GPU pools, and peer-to-peer KV cache sharing over RDMA where available, falling back to TCP.

Tensormesh Platform uses three cache tiers: L0 in GPU HBM, L1 in host RAM, and L2 in filesystem-backed storage. The economic argument is the same no matter which product you use. Our separate Serverless Inference service does not charge for cached tokens, so the reuse rate drives your bill rather than the byte count of your cache. If you want the version of this that survives a server restart, owning the cache lifetime is the next thing to read.

Matching the lever to the workload shape

The Dell survey maps optimization techniques to seven deployment scenarios, including long-context requests, high-throughput datacenter serving, edge devices, multi-turn conversations and accuracy-critical reasoning. The table below groups those tradeoffs into four common workload patterns.

An out-of-memory error can reflect a large cache or wasted allocation. Repeated prefill adds a separate computation cost. Identify the constraint before choosing a technique.

And these levers aren't alternatives. Precision and paging compose with each other and with everything else. Offload composes with reuse, and it is most worth it when the displaced cache gets fetched again, within a long-running request, after a preemption, or by a later request. Without that, it's not worthless: LMCache documents offload as a way to free GPU memory in its own right, pushing overflow KV to host DRAM to relieve pressure on the card. Reuse is what turns it from a capacity trick into a cost one.

Eviction is the one that composes badly, because it competes with reuse for the same tokens: anything you evict is something you cannot reuse later, and on most production workloads reuse is worth more.

Conclusion: choosing a KV cache strategy

Size the cache first, because you can't choose a lever without knowing which constraint binds. If your allocation is much larger than your live cache, that is memory fragmentation and paging fixes it. If the cache itself is the problem, precision is the cheapest real reduction and the one to try before anything exotic.

Treat eviction results as token-count claims until you have measured reclaimed blocks, and prefer policies whose eviction pattern is contiguous in position over policies that scatter survivors, because a block frees only when it empties completely. Offload does buy capacity, and LMCache documents it as a way to free GPU memory outright, but what you pay for it is movement. Treat it as much a bandwidth decision as a capacity one, and measure your own interconnect rather than assuming the tier hierarchy is steep.

And keep the two questions separate. “How many gigabytes do I need?” is a capacity question with an arithmetic answer. “How often do I rebuild that state?” is a cost question, and on most production workloads it has the larger number attached. The cheapest token is the one you never recompute.

Contact our team to discuss a KV cache strategy for your deployment.