A model can answer a test prompt quickly and still become slow under production traffic. Longer inputs delay the first token, concurrent requests compete for GPU memory, and a throughput improvement can slow individual responses.

Running LLM inference well starts with identifying which part of the request is constrained. This guide connects prefill, decode, and KV cache memory to deployment sizing, four practical tuning controls, and the logs and metrics that help diagnose latency problems.

What is LLM inference?

LLM inference runs an already-trained language model on new input to generate a response or prediction. No gradients, no weight updates: the weights are frozen, and you pay for the forward pass. Training is a capital expense you pay once, whereas inference is a per-request cost that scales with every user.

How LLM inference actually runs in production

A request goes through two phases, and your GPU behaves like different hardware in each.

Prefill processes the entire prompt at once. Because the whole input is known up front, the work is a matrix-matrix operation that parallelizes well and can make extensive use of GPU compute. Prefill is compute-bound.

Decode generates output tokens one at a time. Each depends on the previous ones, making decode resemble a matrix-vector operation at small batch sizes. Memory transfer speed therefore tends to limit latency more than arithmetic throughput. Decode is often memory-bandwidth-bound, though larger batches can change the bottleneck.

Long prompts increase time to first token, while longer conversations can also slow decode as the cache grows. For decode, the hardware number worth comparing is HBM bandwidth: over 2 TB/s on the A100 80GB against 3 TB/s on H100. Track these latency measures separately to identify which phase needs tuning.

The bridge is the KV cache. Rather than recomputing key and value tensors for every prior token at every decode step, you keep them in GPU memory and append.

The deployment checklist: what to decide before you serve

Five decisions, in this order, because they interact.

Do the arithmetic first. Weights are parameters times bytes per parameter: a 7B at FP16 is roughly 14 GB. KV cache is the moving part, and for Llama 2 7B, one 4,096-token sequence costs about 2 GB in NVIDIA's worked example.

Run it against real hardware. On an 80 GB card at vLLM v0.28.0's default --gpu-memory-utilization of 0.92, you get roughly 73 GB, minus 14 GB of weights, leaving about 59 GB for KV cache: roughly 29 concurrent 4,096-token sequences.

Treat that as an optimistic memory-only upper bound. It ignores runtime workspaces, CUDA graphs, activation peaks, fragmentation, and the scheduler, all out of the same pool. It's model-specific too: Llama 2 7B uses multi-head attention, and a grouped-query model with a quarter the key/value heads carries roughly a quarter of the cache.

The deployable number is what the engine prints at start-up after profiling the executor: a GPU KV cache size line in tokens and a Maximum concurrency estimate. Size on paper, believe the log.

Tuning knobs that actually move the needle

Four areas are worth tuning: KV cache management, batching, quantization and model parallelism. Offloading and reusing cache can also reduce repeated prefill on workloads that share context.

KV caching

On a 13B model on a 40 GB A100, the vLLM authors measured roughly 65% of memory in static weights and close to 30% in dynamic request state, making KV cache management a major constraint on maximum batch size.

PagedAttention borrows OS paging to eliminate the fragmentation and duplication that naive allocators leak memory to, and prefix caching reuses KV blocks across requests sharing a prompt prefix without changing outputs. Both are on by default in vLLM.

Prefix caching pays off on long system prompts, multi-turn chat, and RAG, and it breaks on exactness. Each block is hashed with the prefix before it, so a timestamp at the top of your system prompt costs you every hit you thought you had.

You can also move the cache off the GPU to CPU RAM or local disk. LMCache's docs describe lower TTFT on long-context, multi-turn, and RAG workloads. Tensormesh builds on this layer. Measure the improvement on your own workload.

Batching (static vs. continuous)

Static batching is the obvious approach and, on interactive traffic, inefficient. As the Orca paper explains, completed requests and new arrivals both wait for the entire batch to finish. Iteration-level scheduling rebuilds the batch after every forward pass, so an arriving request waits one iteration.

For a homogeneous offline job, where nobody waits on a screen, static batching is fine. vLLM uses continuous batching by default, with chunked prefill on in V1 to keep compute-bound prefill and memory-bound decode batched together.

In-flight batching, continuous batching, and iteration-level batching are names for the same technique in TensorRT-LLM.

The dial that matters is max_num_batched_tokens, and it pulls in two ways. vLLM's guidance: smaller values around 2048 give better inter-token latency, because fewer prefills interrupt decodes, while higher values give better TTFT. For throughput, it recommends > 8192, especially for smaller models on large GPUs. Test it on your own traffic first.

Quantization

Quantization lowers weight precision and, in W8A8-style schemes, activation precision too, so the model takes less memory and moves fewer bytes over the same bandwidth. On memory-bound decode, the second effect pays off.

FP8 performance depends on hardware. vLLM reports that FP8 halves model memory requirements and improves throughput by up to 1.6x with a small accuracy impact, and that figure is for W8A8 on Hopper and Ada Lovelace; Turing and Ampere get weight-only FP8 through Marlin kernels, a different trade. Below FP8, derive the memory savings from bytes per parameter and measure accuracy on your own evals.

Weights are the easy half, fixed after training. Activations carry outliers that widen the dynamic range, which is why weight-only schemes keep them at higher precision and lean on a kernel-specific conversion path: the speedup tracks hardware and kernel support rather than the format. Check the matrix first. vLLM documents INT8 computation on compute capability above 7.5, Turing through Hopper, and not on Blackwell at 10.0 and above, where it sends you to FP8.

Model parallelism

Tensor parallelism shards individual layers horizontally across GPUs. Pipeline parallelism splits the model vertically into contiguous groups of layers, one per device, and pays with devices idling on the previous stage: "pipeline bubbles".

Within a node, vLLM says to set tensor_parallel_size to your GPU count. Across nodes, combine both: tensor_parallel_size as GPUs per node, pipeline_parallel_size as node count. If the model fits one node but cannot be evenly divided across its GPUs, drop to pipeline parallelism, which supports uneven splits, with pipeline_parallel_size as your GPU count and tensor_parallel_size=1.

Sharding weights also cuts memory pressure per GPU, freeing KV cache space and raising throughput even when you didn't need it for fit.

Common pitfalls in production LLM inference

The one you'll hit first is KV cache exhaustion, and it announces itself as preemption. When there isn't enough space for the batch, vLLM preempts requests and recomputes them later, logging it: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space. The documented remediations, in order: raise gpu_memory_utilization, lower max_num_seqs or max_num_batched_tokens, then raise tensor_parallel_size, then pipeline_parallel_size.

The second is an inconsistent token budget. With chunked prefill disabled, max_num_batched_tokens must exceed max_model_len or vLLM may crash at start-up.

Mixed workloads can also create queueing delays. vLLM v0.28.0 defaults --scheduling-policy to fcfs, first-come, first-served, which is fine while traffic is uniform. Put a nightly batch job and an interactive user on one endpoint, and you get head-of-line delay. The answers are separate endpoints or the priority policy. Non-zero priorities require that policy. Configure X-Vllm-Priority at the client or gateway and check that requests receive the intended priority.

None of the three is a model problem. They're what happens when the same context gets moved, rebuilt, and thrown away repeatedly: the case we make for treating inference as a data problem rather than a compute problem.

How to measure whether your inference setup is working

Latency splits into two numbers, and teams treat them as one.

Time to first token is what the user feels as responsiveness. NVIDIA's benchmarking docs count queueing and network latency inside it, not just prefill, and longer prompts push it up because the full input builds the KV cache first.

Inter-token latency and time per output token are the same metric, and NVIDIA excludes the first token from its definition to isolate decode time. Tools differ on whether TTFT is folded in, so check yours.

Watch both rather than inferring them. vLLM exposes named Prometheus series on /metrics: vllm:time_to_first_token_seconds, vllm:request_prefill_time_seconds, and vllm:request_queue_time_seconds. Separating queueing from prefill and decode turns "it got slower" into a diagnosis.

Also track a metric that affects both latency and cost: prefix cache hit rate, vllm:prefix_cache_hits over vllm:prefix_cache_queries. The external_ variants only count cross-instance sharing through a KV connector and read zero until one is configured, so don't build on those. A falling hit rate is usually the first sign your prompt template changed, and it can increase costs, especially on agent workloads where every loop iteration rewrites the context.

LLM inference vs. LLM serving

Inference is the model's own computation for a single request: the prefill pass, then one forward pass per generated token, with the KV cache carrying state between them. Serving is everything wrapped around that for more than one caller: queueing, scheduling, KV cache allocation, replica routing.

Almost every knob here lives in the serving layer, which is why a slow endpoint is rarely a model problem: it's how requests are scheduled, how memory is budgeted, and how much work you repeat. Routing across replicas, prefill and decode disaggregation, and shared cache topology are covered in our LLM serving guide.

Where to go from here

Two facts carry the weight: prefill and decode are different problems, and memory rather than compute sets your concurrency ceiling. So put your prefix cache hit rate on the same dashboard as your TTFT. If it's low and your prompts share structure, you're paying for prefill you already did, and offloading and reusing that KV cache is usually the bigger win.

Contact our team to discuss caching for your LLM inference workload.

‍