A chat endpoint may respond quickly until a long document request arrives alongside a batch job. The requests now compete for memory and processing time, even though interactive users still expect tokens to arrive without long pauses.

LLM serving determines how those requests share the hardware. Scheduling, KV cache allocation, batching, and model parallelism all affect the throughput you can sustain within a latency target. This guide explains those mechanisms, the frameworks that implement them, and how to identify bottlenecks under production load.

What is LLM serving?

LLM serving is the practice of running a trained large language model as a production service: accepting requests, scheduling them across GPUs, managing the memory each one consumes while it generates, and returning tokens under a latency target. It's the operational layer around inference, not the model itself.

LLM serving vs. training vs. fine-tuning

Training builds the model from scratch, and fine-tuning adapts an existing one, both by running backward passes and updating weights. Serving does neither: the weights are frozen, and every request is a forward pass. The practical difference is the bill. Training is a bounded project with an end date, while serving runs for as long as the model is live and scales with every user you add.

Why LLM serving breaks the rules of classical ML serving

Classical model serving is a request-response problem. An image classifier takes an input, runs one forward pass, returns a label, and forgets everything. Requests are independent, latency is roughly constant, and batching is trivial because every input costs the same.

Autoregressive generation breaks all three. A single request isn't one forward pass, it's one pass over the prompt plus one more per output token, and the model carries state between them. Response length is unknown when the request arrives. And in the ordinary case, that carried state, the KV cache, occupies GPU memory that grows with every token generated and isn't freed until the request finishes.

So a serving stack built for classifiers schedules the wrong thing. It optimizes for compute throughput on independent short jobs, while LLM serving also requires careful memory management and scheduling.

The practical shape of that: two requests arrive in the same second, one a 200-token summary and one a 4,000-token report, and they neither cost the same nor finish together. A 4,000-token prompt uses roughly twenty times the KV cache memory of a 200-token prompt for the same model and precision, before accounting for generated tokens.

How long that allocation stays resident is a separate question, set by output length, scheduler behavior, and eviction. Under a naive scheduler the long request sits on a slot the short one could have used and released many times over. Nearly every technique below exists to stop that from happening.

The prefill and decode phases

Every request runs in two phases with opposite hardware profiles. Treat them as one workload, and you will tune for the wrong bottleneck.

Prefill processes the entire prompt at once. Because the whole input is known up front, the work is a matrix-matrix operation that parallelizes well. That parallelism can make extensive use of GPU compute. Prefill is compute-bound, and it scales with prompt length.

Decode generates output one token at a time, each token depending on the one before it. At small batch sizes, that resembles a matrix-vector operation. As NVIDIA explains, latency is generally limited by how quickly weights and other data can move from memory to the GPU, making decode memory-bound.

The vLLM paper adds that decode can't be parallelized along the sequence because of that token-to-token dependency. It therefore accounts for most of a single request's latency.

For the full treatment, including how the two phases interact under load, there's our deeper breakdown of the prefill and decode phases.

Why prefill is compute-bound and decode is memory-bandwidth-bound

The shape of the arithmetic decides it. Prefill multiplies a matrix of many tokens against the weights, so each weight load does a lot of work. At batch size one, decode multiplies a single token's vector against the same weights, so the GPU loads the entire model to produce one token and then does it again. In that memory-bound case, more arithmetic throughput offers limited benefit. The number that moves decode is HBM bandwidth: NVIDIA quotes over 2 TB/s on the A100 80GB and 3 TB/s on the H100, and on memory-bound workloads that ratio limits generation speed.

TTFT and TPOT: the two latencies that matter

Time to first token is what a user experiences as responsiveness, and NVIDIA's benchmarking docs count queueing and network time inside it, not just prefill. Time per output token, also called inter-token latency, measures the generation phase; NVIDIA excludes the first token from this definition to isolate decode time.

Combining them into one average makes diagnosis harder. The two have different causes and different fixes: long prompts hurt TTFT, and memory pressure hurts TPOT.

They also fight each other inside the scheduler. A long prefill occupying the GPU stalls every decode step queued behind it, and users read that as a conversation pausing mid-sentence whenever someone else pastes a large document. Chunked prefill is the standard answer: split the prefill into pieces and interleave them with decode steps so both make progress.

vLLM V1 enables chunked prefill by default when the configuration supports it. The scheduler batches pending decode requests before scheduling prefill. You trade a little TTFT for steadier inter-token latency. In an interactive product, that is almost always the trade you want.

Prefill builds prompt state; decode extends it token by token. End-to-end time to first token also includes waiting and setup.

KV cache: where the memory actually goes

To avoid recomputing attention over the whole sequence on every step, the model stores the key and value tensors for tokens it has already processed and reuses them. That store is the KV cache, and it is the largest dynamic consumer of GPU memory in a serving deployment.

It competes directly with the weights. On a 13B model on a 40 GB A100, the vLLM authors measured roughly 65% of memory in static weights and close to 30% in dynamic request state, making KV cache management a major constraint on maximum batch size. If you want the mechanics rather than the memory accounting, our piece on what KV cache actually stores covers it.

Carry one sizing intuition, and you can predict most capacity surprises. KV cache bytes are approximately batch size times sequence length times 2 times layers times KV heads times head dimension times bytes per element. The 2 accounts for the separate key and value tensors, and this is the standard dense full-context approximation. Because batch size and sequence length both multiply into it, they trade directly against each other on a fixed card. No amount of scheduler tuning changes that arithmetic, which is why context-window announcements and serving costs move together.

PagedAttention and block-based memory management

The naive approach pre-allocates a contiguous buffer per request, sized to its maximum possible length. Profiling pre-PagedAttention systems found that only 20.4% to 38.2% of KV cache memory held actual token state; the rest was reserved and idle. That waste shows up as a hardware problem on a dashboard but is really an allocator problem.

PagedAttention, the algorithm behind vLLM, borrows paging from operating systems: the cache is split into fixed-size blocks that don't have to be contiguous, allocated as the sequence grows. In the 2023 evaluation, vLLM as a complete serving system delivered 2 to 4 times higher throughput than FasterTransformer and Orca at comparable latency, and PagedAttention's reduction in KV memory waste was a central contributor to that system result. vLLM, SGLang, and TensorRT-LLM all now list paged KV cache management as a core feature of their runtimes.

Why cache reuse changes the cost equation

Paging solves waste inside one request. The bigger opportunity is across requests, and most teams haven't taken it. A system prompt fronting every conversation, a document queried forty times, an agent replaying its context on every loop: all of that is prefill you already paid for. Prefix caching reuses those KV blocks when a new request shares the same prefix, avoiding repeated computation without changing model outputs.

The catch is that it only works when the reused text sits at the front of the prompt and matches exactly. Dynamically retrieved or rotating content reduces reuse after the first changed block, although any stable leading prefix can still hit the cache. So put the stable material first and the volatile material last, then measure. That's why our own numbers on cache hit rates in production agent workloads make it a metric worth watching rather than assuming.

PagedAttention maps ordered logical KV blocks to physical blocks allocated as needed, reducing the need for large contiguous reservations.

Concept adapted from Kwon et al., SOSP 2023. Original illustration.

Request batching and scheduling

Batching keeps a GPU busy. Loading model weights to serve one request wastes almost all the load, and spreading it across many concurrent sequences is what throughput optimization means in practice.

Static batching vs. continuous batching

Static batching groups requests, runs them together, and returns when the slowest finishes. The Orca paper describes how this delays both completed responses and newly arriving requests until the entire batch finishes. Since generation lengths vary wildly, most of the batch sits idle waiting for the longest one.

Continuous batching rebuilds the batch after every forward pass, so a finished sequence leaves immediately and an arriving one waits a single iteration. The terms in-flight batching, continuous batching, and iteration-level batching refer to the same technique, as NVIDIA's TensorRT-LLM documentation explains. vLLM's V1 scheduler works this way by default, and TensorRT-LLM and SGLang both list the behavior as a core runtime feature.

Static batching can still suit offline workloads. A homogeneous offline job where every request is roughly the same length, and nobody is waiting on a screen, loses very little to it.

Continuous batching can admit waiting requests as slots become available instead of waiting for the longest sequence in a fixed batch.

Model parallelism: scaling beyond one GPU

When the weights plus the KV cache exceed one device, the model has to be split.

Tensor parallelism vs. pipeline parallelism

Tensor parallelism shards individual layers horizontally across GPUs, so every device holds a slice of every layer, and they communicate on each forward pass. It needs a fast interconnect. That is why it's the within-a-node choice.

Pipeline parallelism splits the model vertically into contiguous groups of layers, one per device. Communication is far lighter, but devices idle waiting on the previous stage, the pipeline bubbles. Across nodes, most deployments combine both: tensor parallel within each node, pipeline parallel between them.

Sharding also buys something unrelated to fit. Spreading weights across GPUs cuts memory pressure per device, freeing KV cache space and increasing the batch size you can hold.

Disaggregated prefill and decode

Since prefill and decode want opposite hardware, running them on the same GPUs means neither gets what it wants. Disaggregation separates them, and LMCache ships a KV-transfer connector for exactly this against vLLM, listed as LMCacheConnectorV1 in vLLM's disaggregated prefilling docs. Read those docs before you plan around it.

vLLM marks the feature as experimental and describes it as a way to tune TTFT and inter-token latency separately, rather than improve raw throughput. The tradeoff is that the KV cache produced by prefill now has to reach the decode workers over the network, which turns a memory problem into a transport one. We go further in our pieces on disaggregating prefill and decode onto separate GPU pools and on a peer-to-peer architecture for sharing KV cache across GPUs.

Tensor parallelism shards work within layers; pipeline parallelism assigns groups of layers to successive devices.

Choosing a serving framework: vLLM, SGLang, and TensorRT-LLM

The hardware entries follow SGLang's README, NVIDIA's supported-hardware page, and vLLM's CPU install guide, which covers basic inference and serving. If you run anything other than NVIDIA silicon, that column decides this before any benchmark does.

A useful distinction: RadixAttention is not a PagedAttention replacement. SGLang's paper describes it as compatible with continuous batching, paged attention, and tensor parallelism, and its runtime lists both as separate features. RadixAttention automates prefix sharing across requests; paging manages the blocks underneath. They sit at different layers.

SGLang's month-long production measurement found cache hit rates of 52.4% on LLaVA-Next-34B and 74.1% on Vicuna-33B. The Vicuna-33B result corresponded to an average 1.7x improvement in time to first token; that gain applies to the measured workload.

In practice, the choice is less fraught than the benchmark discourse suggests, because all three implement the same core ideas and differ mainly at the edges.

Take vLLM unless you have a reason not to; most tooling assumes it by default. Reach for SGLang when your workload has heavy structural repetition, agent traces and multi-turn sessions sharing long prefixes, which is what RadixAttention exploits. Reach for TensorRT-LLM when you're committed to NVIDIA hardware and the extra tuning surface is worth the added build step. Benchmarks between them move with every release, so treat any specific multiple as a snapshot. We put a full framework-by-framework comparison of vLLM and SGLang side by side if you're choosing between those two specifically.

Common LLM serving bottlenecks (and how teams fix them)

KV cache exhaustion is the one you hit first, and it announces itself as preemption rather than an error, which is why it gets misdiagnosed as a slow model. When there isn't room for the batch, vLLM preempts requests and recomputes them later, logging exactly that. The documented remedies, in order: raise the memory utilization ceiling, lower the maximum sequence or token counts, then add tensor and pipeline parallelism.

Memory fragmentation is what paging exists to solve, and it comes back at the eviction layer. Blocks are freed only when completely empty, so evicting most of a sequence's tokens can reclaim almost nothing if the survivors are scattered.

Cold starts are a function of size. Loading tens of gigabytes of weights onto a GPU takes time, so spiky traffic pays either in first-request latency or in idle capacity held warm. Choose based on the latency target and traffic pattern.

Noisy neighbors appear as soon as one endpoint serves mixed traffic. In vLLM V1, the default scheduling policy is FCFS. That's fine while requests look alike. It stops being fine the moment a batch job and an interactive user share a queue, because the interactive request waits behind batch work. vLLM documents a priority policy as the alternative, where a lower value means earlier handling. Splitting workload classes onto separate endpoints is the other answer.

GPU utilization alone does not measure useful throughput. NVIDIA's DCGM defines SM activity as the fraction of time at least one warp was active, and it counts warps waiting on memory as active. Even a value of 0.8 or greater does not establish efficient GPU use on its own.

A card can look busy while it stalls on HBM all day. Latency has a direct commercial cost, covered in our pieces on how latency directly affects revenue and on the true cost of running inference at scale, and so does the compute you're wasting.

Deployment models: self-hosted, serverless, and managed serving

Self-hosting on reserved GPUs gives you control over the whole stack and the best economics at steady, high utilization. It also gives you the operational load: capacity planning, upgrades, and the idle hours you pay for anyway.

Serverless inference removes the capacity planning and bills by token, which suits spiky or early-stage traffic and costs you some control over the serving layer. We show what that path looks like in practice in deploying open-weight models serverlessly in minutes, and our piece on the tradeoffs between self-hosting and serverless inference goes deeper on when to switch.

Managed serving sits between them: someone else runs the engine, you keep model choice and skip the operational load, and you give up control over how requests get scheduled.

The deciding variable is almost always utilization rather than list price, because idle capacity can outweigh a lower hourly rate. A reserved GPU is cheaper per token than serverless only if you keep it busy, and "busy" means useful throughput rather than a utilization percentage on a dashboard. Teams that migrate for cost reasons without measuring their real duty cycle first tend to move their bill rather than reduce it. The second variable is how much of your traffic is repeated context, because that determines whether caching or capacity is your actual lever.

Where Tensormesh fits in

Everything above treats the KV cache as something the engine owns. Serving frameworks reuse it inside one replica through prefix caching, and lose it when the block is evicted or the pod restarts. Carrying it further, across sessions and across replicas, needs a layer that owns the cache instead of the engine. Tensormesh provides this layer.

Tensormesh Platform is a self-hosted caching and context-reuse layer, not a serving engine. We built it on the open-source LMCache project our founders created, and it runs the cache beneath the serving engine rather than replacing it. Tensormesh Platform manages a three-tier hierarchy for vLLM: L0 in GPU HBM, L1 in host RAM, and L2 in filesystem-backed storage.

Tensormesh Platform supports vLLM, and our documented compatibility matrix includes model and configuration details. On our separate Serverless Inference service, we bill cached tokens at $0 per million on most of our listed models; output tokens are still billed.

If your prompts share structure but cache hit rate is low, investigate missed reuse before adding capacity. Chat with our team about improving cache reuse in your serving stack. Measure the hit rate before estimating savings.

Key takeaways

• LLM serving requires careful memory management and scheduling alongside compute capacity. Almost every technique in the stack manages GPU memory or keeps the device busy while it waits on memory.
• Prefill and decode have opposite hardware profiles. Prefill saturates compute; decode is bound by memory bandwidth and dominates single-request latency.
• The KV cache is the largest dynamic memory consumer and it competes with model weights for the same pool, which is what caps your batch size.
• Continuous batching, in-flight batching and iteration-level batching are three names for one technique, and every current engine implements it.
• Framework choice follows workload shape and hardware more than benchmark numbers. Take vLLM as the general default, SGLang where prefixes are heavily shared, TensorRT-LLM for peak throughput on NVIDIA hardware you already own.
• Reuse beats optimization. The cheapest token is the one you never recompute, and cache reuse beyond a single replica is where most deployments still leave money on the table.