An LLM response can be slow in two different ways: the user waits for the first token, or the answer starts promptly but streams too slowly. A single end-to-end latency number hides that distinction and makes it harder to choose the right optimization.

Prefill processes the input prompt; decode generates the response one token at a time. Their compute and memory demands differ, so improving one phase may do little for the other. Understanding both helps you interpret latency metrics and choose between batching, caching and separating the phases across GPUs.

What is the difference between prefill and decode?

Prefill and decode are the two phases of a single LLM inference request. Prefill reads the entire input prompt in one parallel forward pass, builds the KV cache, and emits the first output token. Decode then generates each remaining token one at a time, reading from and appending to that cache. Prefill is typically compute-bound; decode is often memory-bandwidth-bound.

That's AWS's own formulation, and it's why we argue the KV cache deserves first-class treatment rather than an afterthought.

The prefill phase: reading the prompt and building the KV cache

Prefill processes the prompt's tokens together and produces the first response token in one step, as described in the DistServe paper on separating prefill and decode.

Prefill also produces the state needed for decode. As it runs, prefill stores the key and value tensors for every position in the prompt. That's the KV cache, and decode never recomputes them.

Why prefill is compute-bound

Processing the whole prompt together turns prefill into a highly parallel matrix-matrix operation that can make extensive use of GPU compute.

Prompt length matters here. DistServe found a 13B model prefilling 512 tokens on an A100 was close to compute-bound. Brief prompts may not provide enough parallel work to saturate the GPU.

What drives time to first token

DistServe uses prefill duration as its TTFT measure. NVIDIA measures TTFT end to end, from submitting a query to receiving its first token, including queueing and tokenization. Use the end-to-end measure when evaluating user-visible latency.

TTFT therefore grows with prompt length, faster than linearly: DistServe describes a superlinear increase in prefill computation as more tokens are processed in parallel. A user who pastes a 30,000-token document waits noticeably longer, which is what we mean by the business cost of slow time-to-first-token.

Prefill builds prompt state; decode extends it token by token. End-to-end time to first token also includes waiting and setup.

The decode phase: generating tokens one at a time

Decode is autoregressive. The model produces one token, appends its keys and values to the cache, and feeds it back in to produce the next. It can't be parallelized along the sequence: token n+1 doesn't exist until token n is sampled.

Why decode is memory-bandwidth-bound, not compute-bound

Each decode step does little arithmetic relative to the data it moves. It reads the model's full weight set, plus every request's KV cache, out of HBM. As NVIDIA explains, decode makes less use of GPU compute than prefill and is generally limited by memory transfer speed.

DistServe likewise finds that decode moves a similar amount of data to prefill while processing just one new token, making memory bandwidth a constraint. At production batch sizes those matmuls are a skinny GEMM, not a true matrix-vector product. At small to moderate batch sizes, decode usually has low arithmetic intensity and is therefore memory-bandwidth-bound. Larger batches can raise arithmetic intensity, so you must measure the limiting resource on the target model and hardware.

Why running prefill and decode together causes problems

When prefill and decode share a GPU batch, each can slow the other. DistServe measured increases in both TTFT and time per output token after adding a prefill job to a decode batch, with longer prompts causing greater interference.

A long prompt can stall token generation for concurrent requests sharing the GPU, as AWS describes. SGLang calls this prefill interruption.

Two ways out, and we'd try them in this order. The first is chunked prefill, from the SARATHI paper: split a prefill into chunks and fill each batch's remaining slots with decode steps that run alongside that computation. SARATHI reported up to 10x higher decode throughput for LLaMA-13B on an A6000, with end-to-end throughput on that configuration up to 1.33x. In vLLM V1 chunked prefill is enabled by default when supported, with the scheduler prioritizing decode over pending prefills. So the cheap fix is probably already on in your cluster. Check that before you build anything.

The second is to stop sharing the GPU at all. Both get cheaper when the prompt doesn't have to be computed from scratch. In document-heavy workloads, it often doesn't have to be, yet the same contracts and knowledge bases get reprocessed anyway, which we cover in the document reprocessing problem.

Prefill/decode disaggregation: separating the two phases

Prefill/decode disaggregation, also written PD disaggregation or DPD, runs prefill on one pool of GPUs and decode on another. The prefiller computes the KV cache and ships it to the decoder, which generates without recomputing the prompt. We cover standing one up separately.

Each pool can then be sized, batched, and parallelized for its own bottleneck. vLLM describes disaggregation as a way to tune TTFT and inter-token latency independently and control tail inter-token latency more reliably than adjusting chunk size alone.

DistServe reports serving up to 7.4x as many requests or meeting up to 12.6x tighter latency SLOs than its baselines. The quantity is goodput, the request rate served within both TTFT and TPOT constraints at over 90% SLO attainment. Those are two readings of one tradeoff curve, not raw throughput.

Disaggregation separates prompt processing from token generation, adding a KV transfer whose cost must be justified by workload and hardware.

When disaggregation is worth it, and when it isn't

Disaggregation adds costs. vLLM marks disaggregated prefilling as experimental and warns that it does not improve raw throughput. AWS reports a modest TTFT increase from transferring KV state over EFA RDMA; its router skips the prefiller for prompts under 4,096 tokens.

It pays off when prompts are long, concurrency is high, and TTFT and ITL have separate SLOs you're missing in different directions. For short prompts, low concurrency, or offline batch work, colocation is right.

The real bottleneck: moving the KV cache between phases

Separating the phases turns a scheduling problem into a data-movement problem. The KV cache for a long prompt is large, and it crosses the network between prefill finishing and decode starting. Every millisecond in transit lands on TTFT.

The KV-transfer layer connects the two GPU pools. vLLM documents a list of connectors there and still labels the feature experimental. LMCache is the one AWS and Ray both document, an open-source KV cache management layer out of systems research at the University of Chicago.

LMCache lets KV state persist beyond an individual request and be reused across serving engines. It is vendor-neutral and remains an open-source project. Tensormesh was founded by its creators and builds on that project.

AWS's SageMaker HyperPod implementation uses the vLLM Production Stack router and LMCache to transfer KV state over NIXL and EFA. Ray Serve supports LMCacheConnectorV1 as one of two KV transfer backends for disaggregated serving, recommending it for advanced caching or multiple storage backends and NIXLConnector for simpler deployments.

The transfer problem generalizes past disaggregation. If a prefix has been computed anywhere in the fleet, recomputing it is waste. LMCache's peer-to-peer cache-sharing architecture lets engine pods read KV state directly from each other rather than each holding an isolated cache.

Our work with Tencent reported 4x better TTFT and 5x better total query completion time on long-document QA. Without published hardware or model details, those results cannot establish expected gains for another deployment. They are not an end-to-end Tensormesh Platform benchmark.

Tensormesh Platform provides prefill/decode disaggregation, so those phases can run on separate GPU pools sized for their workloads. Peer-to-peer KV cache sharing complements it, fetching warm state from a peer instead of recomputing it, over RDMA where the network supports it and over ordinary TCP where it does not.

Tensormesh Platform supports vLLM. Our compatibility matrix provides model and configuration details.

How to tell which phase is your bottleneck

If the first token is slow but streaming is fast, prefill is your bottleneck. Check prompt length distribution and queue depth before the GPU. When prompts are long and repetitive, prefix reuse pays better than more compute. Our own agent-workload numbers show the difference a higher cache hit rate makes, and it compounds with every repeated system prompt and retrieved document.

If the first token arrives quickly but streaming stutters, decode is your bottleneck. Check memory bandwidth utilization and batch size rather than FLOPs. ITL that degrades as concurrency rises means contention for bandwidth and cache capacity.

If both look fine at p50 and fall apart at p99, suspect interference before capacity. Long prefills are landing in decode batches, and chunked prefill is the cheap fix where disaggregation is the structural one.

One last caveat. Tools disagree on whether ITL includes the first token, so compare numbers within one harness.

Key takeaways

Prefill and decode are one request and two workloads with opposite hardware profiles. Prefill wants FLOPs, decode wants bandwidth, and batching them costs TTFT or ITL.

The KV cache bridges them, which makes it the thing worth optimizing. There's also a question upstream of the architecture: how often you reuse a cache you already built instead of paying to compute it twice. Measure your prompt-length distribution and prefix hit rate before you buy a second GPU pool.

Contact our team to discuss reducing repeated prefill work.