A model's weights fit on your GPU, but increasing the context length or serving more users can still exhaust its memory. Each active request also needs space for the attention state the model accumulates as it processes and generates tokens.

That state is the KV cache. Keeping it avoids repeated computation, but its memory requirements can limit context length and concurrency. Understanding how it grows lets you size a deployment before provisioning hardware and choose which memory-management techniques are worth testing.

What is KV cache?

A KV cache, also written kv-cache, stores the key and value tensors produced by a model's attention layers for tokens it has already processed, then reuses them on later steps rather than recomputing them. It exists because autoregressive generation would otherwise redo that work on every token.

It's kept per layer, each layer holding a pair of tensors shaped [batch_size, num_heads, seq_len, head_dim], where on a grouped-query model that second dimension counts KV heads, not query heads. It's inference-time only and even Hugging Face's docs warn that enabling it during training may cause unexpected errors.

How KV caching works

Caching works because of causal masking. In a decoder-only model a token can't attend to anything after it, so once a token is processed its key and value representations never change. That immutability is the license to cache.

The loop: process the prompt, keep K and V for every token at every layer, then for each new token compute its query, key and value, append the rows, and attend against the whole cached set. You don't build it by hand. The flag is on by default:

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-1.7B")
model = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-1.7B").cuda()

inputs = tok("The red cat was", return_tensors="pt").to("cuda")
out = model(**inputs, use_cache=True)   # use_cache=True is the default
past = out.past_key_values              # per-layer K and V for every prompt token

# next step: feed one token, reuse everything already computed
next_id = out.logits[:, -1:].argmax(-1)          # shape [batch, 1]
out = model(next_id, past_key_values=past, use_cache=True)

past_key_values is the cache, and generate() threads it through the loop for you, which is why most people never see it.

One property matters later: a token's cached KV depends on every token before it, so the same token at a different position produces different KV, which is why conventional prefix caching can only reuse an identical prefix. Systems built for non-prefix reuse, CacheBlend among them, match any repeated chunk and recompute a small fraction at the boundaries.

During decode, earlier keys and values are reused while each step computes and appends new state.

Why KV cache exists: the prefill and decode problem

Inference runs in two phases, and your GPU behaves like different hardware in each. Prefill processes the whole prompt at once, which NVIDIA describes as a highly parallelized matrix-matrix operation that effectively saturates GPU utilization. Your GPU is good at this.

Decode is the opposite. At small batch sizes, generating one token at a time resembles a matrix-vector operation, with latency dominated by moving weights, keys, values, and activations out of memory rather than by arithmetic. NVIDIA calls that memory-bound. It's where a request spends most of its latency.

The gap shows up even on small hardware. A community article on Hugging Face benchmarked generate() on a T4 with and without caching, up to 300 new tokens from SmolLM2-1.7B: 11.7 seconds cached against 1 minute 1 second uncached, roughly 5.2x. One model on one old GPU: an illustration, not a planning number. Nearly all that uncached time is decode recomputing keys and values it already had.

The cache exists because prefill is expensive and decode shouldn't pay for it twice, trading arithmetic for memory capacity in the phase already starved for bandwidth. Our LLM inference guide covers how both phases shape a deployment. The tradeoff is the memory needed to retain that state.

What KV cache actually costs you

Calculate the memory requirement from the model's architecture and cache precision. The arithmetic is in the NVIDIA guide linked above, generalized to bytes_per_element in place of a hardcoded sizeof(FP16):

KV cache per token (bytes) = 2 * num_layers * (num_heads * dim_head) * precision_in_bytesTotal KV cache (bytes) = batch_size * seq_length * 2 * num_layers * hidden_size * bytes_per_element

The leading 2 is K and V; num_heads * dim_head is usually the model's hidden size. Llama 2 7B at FP16, 4,096-token context, batch size 1 comes to roughly 2 GB.

First, check which head count the formula uses. That formula assumes every attention head carries its own K and V, which doesn't hold under grouped-query attention. On a GQA model, count KV heads, not query heads, or the answer comes out several times too large: 2 x layers x KV heads x head dim x bytes per element.

Across real context windows, the problem changes shape. Every column past 4K is a memory projection, not a native limit, since Llama 2 shipped a 4,096-token window.

Per-token values: Figure 3 of arXiv:2603.20397, Llama-2-family models at FP16. Totals from the exact per-token byte counts at batch size 1, 1K = 1,024 tokens.

Two things jump out. A 7B at 128K needs about 64 GB of KV cache for one user, and the same paper marks where that cache alone overruns a 24 GB RTX 4090 at around 48K tokens. The 70B has a smaller per-token footprint than the 7B, because 8 KV heads against 32 outweighs 80 layers against 32. Parameter count is a bad proxy for cache size.

Without eviction or offloading, cache growth can exhaust GPU memory: NVIDIA Research, cited below, documented Qwen3-32B with 4-bit weights hitting an out-of-memory error on a 24 GB GPU after roughly 24,000 generated tokens. Then multiply by batch. Forty GB per user scales linearly against a pool model weights already occupy, and that squeeze is what turns up on the invoice, near the top of our guide to what drives LLM inference cost.

Scaling KV cache beyond a single GPU

Once the cache stops fitting, you have four options, none free. The Dell survey behind that table above concludes, for all four, that no single technique dominates and the right choice depends on context length, hardware, and workload.

Manage the memory better. PagedAttention, the algorithm behind vLLM, borrows paging from operating systems because fragmentation and duplication waste KV cache memory and cap batch size. Profiling of pre-PagedAttention systems found only 20.4% to 38.2% of KV cache memory holding real token state, because they pre-allocated a contiguous chunk per request sized to its maximum length. Fixing that produced 2-4x throughput at equal latency. If you're not on a paged engine, start by measuring the benefit of paging.

Shrink each entry. Quantizing to FP8 significantly reduces the footprint: halving precision_in_bytes above is a straight 2x on every row of that table. The effect isn't uniform. vLLM warns that layer types such as sliding-window are more sensitive, and ships a flag to leave those at native precision. Treat any blanket accuracy claim about FP8 KV cache as unsupported until someone shows the evaluation.

Move it somewhere else. LMCache's docs list exactly two on-machine options: CPU RAM and local disk. Past that, you're into distributed KV cache across nodes. The dominant cost is moving the bytes. NVIDIA measured offload to CPU memory improving time to first token by up to 14x on x86-based H100 and 28x on GH200, without isolating how much of that gap is interconnect. The platforms do differ: NVLink-C2C runs at 900 GB/s, seven times PCIe Gen 5 by NVIDIA's reckoning. On PCIe, you're trading recompute against transfer, so measure first.

Throw tokens away. Eviction-based compression is what people reach for first, and it's the one we'd reach for last. vLLM's paged cache stores tokens in configurable blocks, and a block frees only once it's completely empty. NVIDIA Research works the common case of about 16 tokens per block: evicting 14,400 of 16,000 tokens scatters the survivors across nearly every block allocated, and the allocator reclaims almost nothing. A compression paper's results table is not your dashboard.

Moving KV state beyond GPU memory trades access cost for capacity and retention. Whether reuse pays depends on the transfer path and workload.

KV cache vs. prompt caching, and other terms people confuse

Three things sit next to the KV cache and get mistaken for it, plus one naming collision.

OpenAI's documentation describes the second row. Prompt caching preserves the intermediate KV state for a reusable prefix so a later request skips reprocessing those tokens, and it specifies that the cache stores KV tensors, not the tokens themselves. Reused input tokens are billed at up to a 90% discount. The catch is exactness: any change before the breakpoint invalidates everything after it. The engine-side name for the same idea is prefix caching.

Redis is the naming collision. Its "key" is a lookup string and its "value" is the object that comes back; neither touches the per-token tensors an attention layer produces. The confusion sticks because Redis does turn up in serving stacks, as a remote backend the cache gets written to. It holds the cache. It isn't the cache.

How Tensormesh and LMCache approach KV cache

Keeping KV state beyond one request allows later requests to reuse it. The same document gets read by forty users, the same system prompt fronts every conversation, the same agent replays its context every loop.

LMCache, the open-source project we build on, keeps KV cache as reusable state rather than temporary state, persisted and reused across serving engines. LMCache's CPU-offload docs above show the payoff on one setup: a 15,376-token prompt on Llama-3.1-8B-Instruct, cold TTFT 6.537 seconds against warm 0.147, 44.5x. That is an LMCache documentation example on one configuration rather than a product measurement, and we have not published an end-to-end Platform benchmark.

We run it as a self-hosted layer in Tensormesh Platform; our separate Serverless Inference service does not charge for cached tokens. The Platform manages that as a three-tier hierarchy: L0 in GPU HBM, L1 in host RAM, and L2 in filesystem-backed storage.

For more on managing KV state beyond individual requests, see Stop Calling It KV Cache.

Where to go from here

Treat the KV cache as the artifact your GPU spent its expensive compute-bound phase producing, and consider how long you can reuse it. Nothing keeps it once it falls out of a node's memory unless you make that happen. Size it with the formula, then work out how much your workload regenerates from scratch every day. If that number looks bad, our case for owning the cache lifetime is next.

Contact our team to discuss KV cache management for your deployment.