To generate the next word in an answer, a transformer needs information about the tokens that came before it. Recomputing that information from scratch at every step would repeat much of the same work as the response grows.

KV caching avoids part of that repetition by retaining the keys and values produced by attention. The reason those tensors can be reused comes from causal masking. Walking through the attention calculation shows what the cache saves, what still has to be computed, and why longer sequences need more memory.

What attention actually computes

Attention maps a query against a set of key-value pairs and returns a weighted sum of the values. Every token in a sequence is projected into all three: a query, a key, and a value. Later tokens read the keys and values, and a KV cache stores them.

In the original transformer formulation, attention produces a weighted sum of values. Each weight reflects how well the query matches the corresponding key.

The redundancy problem: why naive decoding recomputes everything

Generation is autoregressive: one token at a time, each conditioned on everything before it. Causal masking makes that exploitable. A token attends only backward, so once processed its representation never changes with respect to the tokens that follow, and its keys and values are fixed from then on.

A naive implementation recomputes them anyway. Every step re-runs the forward pass over the whole sequence, the projections for the entire prefix and the attention for every position in it, then throws it away and repeats. It is re-reading the document before writing each sentence.

Without a cache, each step recomputes earlier K/V state. With one, each step computes new state and reads the retained prefix.

How the KV cache fixes it

It stores the keys and values, splitting generation into two phases. Prefill processes the whole prompt in one step and fills the cache. Decode then emits one token per step, computing K and V for the new token alone and reading the rest from the cache.

For standard full attention, Hugging Face describes per-step attention cost as quadratic in sequence length without caching and linear with caching.

Linear, not constant. What goes away is the earlier positions' projections and attention rows. The full QK^T calculation reduces to q_c K^T for the current query, as shown in Hugging Face's optimization guide. What is left is the new token attending over a cache that grows by one entry each step. That residue is the linear term.

The memory trade-off

The memory cost grows with sequence length, KV head count, head dimension, and layer count, with a factor of two for keys and values. Hugging Face's scaling expression assumes ordinary multi-head attention; grouped-query and multi-query models require counting KV heads instead of query heads. Batch size also multiplies the total: the cache tensors have shape [batch_size, num_heads, seq_len, head_dim].

For deployment sizing, use elements = 2 × batch_size × layers × KV_heads × sequence_length × head_dimension, times bytes per element. Omitting a multiplier changes the estimate proportionally. Run the four-term version at batch 32, and you are under by 32x, and you find out as an out-of-memory error in production.

Long context therefore increases both storage requirements and the amount of cached state that must move. Our CTO examines the bandwidth between GPU memory, DRAM, NVMe, and distributed storage in our analysis of inference as a data problem. Our serverless pricing lists cached tokens at $0.

Contact our team to discuss managing KV cache memory in your deployment.

‍