A serving engine that performs well on independent, short prompts may behave differently when requests share long documents or generate structured output. Choosing between vLLM and SGLang requires matching their scheduling and cache-reuse behavior to the traffic you plan to serve.

Both engines support high-throughput LLM serving. This comparison examines their memory management, prefix reuse, deployment features, and benchmark conditions to help you choose what to test on your own workload. Tensormesh Platform integrates with vLLM for persistent cache reuse.

The short version, before the evidence:

What are vLLM and SGLang?

vLLM is an open-source LLM inference and serving engine originally developed in UC Berkeley's Sky Computing Lab. SGLang is a production inference framework designed for low latency and high throughput, from a single GPU to distributed clusters. Both serve open-weight models.

The engines support many of the same optimizations. SGLang's README lists prefix caching, continuous batching, paged attention, four kinds of parallelism, speculative decoding, and chunked prefill. vLLM's docs list most of the same ideas. These are two implementations of one body of serving research with different defaults, and the latency either produces has a direct commercial cost.

vLLM vs SGLang: architecture and core technology

PagedAttention: allocation

PagedAttention addresses KV cache allocation. As the 2023 vLLM paper explains, cache requirements change with each request, and fragmentation or duplicated state can waste enough memory to limit batch size.

So stop allocating one contiguous region per request. The cache splits into fixed-size blocks, and a sequence's blocks need not be adjacent in GPU memory. The paper reports near-zero KV memory waste and supports sharing cached state within and across requests. This describes the original design, and the docs home now names FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton as the optimized kernels. For current implementation details, consult the version you deploy.

RadixAttention: retention and lookup

RadixAttention retains KV state for prompts and generated output after a request finishes, using a radix tree to support prefix lookup, reuse, insertion, and eviction. The SGLang paper describes this as a systematic way to reuse cache state across requests.

A radix tree maps token sequences to their KV tensors, so a later request sharing a prefix finds them instead of recomputing. Eviction takes least-recently-used leaves first, keeping shared ancestors alive until they become leaves. This describes the design in the June 2024 paper.

The correction most comparisons need

RadixAttention does not replace PagedAttention. SGLang's paper describes it as compatible with continuous batching, paged attention, and tensor parallelism, and its README lists both RadixAttention and PagedAttention as runtime features.

They sit at different layers. Paging is allocation: how blocks are handed out and reclaimed inside a request, so fragmentation doesn't cap your batch size. Radix-tree indexing is retention: what survives a request, and how the next one finds it. Both engines do both. vLLM's design docs note that public APIs such as OpenAI and Anthropic, and open-source frameworks including SGLang, use prefix caching.

The differences are in the surrounding machinery. vLLM supports optional per-request salts that restrict cache reuse to requests with the same salt. SGLang pairs its cache with a scheduling policy that orders requests to raise the hit rate. For the mechanics underneath, see prefill and decode.

vLLM vs SGLang: performance and throughput benchmarks

The figures below come from the original vLLM and SGLang papers, published in 2023 and 2024. They use different baselines and do not provide a head-to-head comparison of the versions you deploy.

The 2–4x and 6.4x gains apply to the systems and baselines evaluated in those papers. Neither predicts performance on a newer deployment.

SGLang's authors measured RadixAttention in Chatbot Arena over one month: cache hit rates were 52.4% for LLaVA-Next-34B and 74.1% for Vicuna-33B. Reuse came from shared system messages, repeated example images and multi-turn history, reducing first-token latency by an average of 1.7x for Vicuna-33B. Across the synthetic suite, hit rates ranged from 50% to 99%, measured as cached prompt tokens divided by prompt tokens.

The 1.7x TTFT improvement applies to Vicuna-33B in that production measurement; results will vary with prefix reuse in your traffic.

Why benchmark numbers for vLLM vs SGLang disagree so much

Four factors can change the result.

Version drift. vLLM shipped v0.22.0 through v0.28.0 between late May and late August 2026, roughly a release every ten days. On the SGLang side you often cannot tell which build was measured at all.

Hardware. The SGLang paper ran most experiments on A10G 24GB cards, and 24GB results don't transfer to an H100 fleet.

Prefix overlap, the largest of the four. Hit rate ran from 50% to 99% across the paper's own suite, and it decides how much prefill work either engine skips.

Configuration. Smaller chunked-prefill values in vLLM improve inter-token latency by reducing prefill interference; higher values improve TTFT. Tuning for different metrics can therefore change the ranking on identical hardware. The SGLang paper excludes optimizations that alter computation results, another condition to check when comparing benchmarks.

Benchmark your own deployment. We publish an open CLI for this, and comparing serving stacks with the tmesh benchmark tool points it at any OpenAI-compatible endpoint, including a vLLM or SGLang deployment you already run.

When to choose vLLM vs when to choose SGLang

The deciding variable is workload shape, so this is a checklist, not a feature grid.

Two operational settings need attention before deployment.

The memory controls require tuning in both engines. vLLM lists a default of 0.92 for --gpu-memory-utilization, while SGLang selects --mem-fraction-static using heuristics rather than a single documented default.

# vLLM: fraction of GPU memory for weights plus KV cache, documented default 0.92--gpu-memory-utilization 0.92# SGLang: the same trade, no default published in the docs--mem-fraction-static <value>

Both engines support separating prefill and decode to manage latency. vLLM marks disaggregated prefilling as experimental and describes it as a way to tune TTFT and inter-token latency independently, rather than improve raw throughput. SGLang uses PD disaggregation to address prefill interference and data-parallel attention imbalance. For the architecture, see how prefill and decode split apart.

Where TensorRT-LLM and TGI fit in

TensorRT-LLM: peak throughput on NVIDIA hardware

NVIDIA's TensorRT LLM accelerates inference on NVIDIA GPUs. Its optimizations include in-flight batching, paged KV cache with block reuse, chunked prefill, speculative decoding and beta disaggregated serving. In-flight batching is another name for continuous or iteration-level batching.

The trade-off is hardware. Its supported-hardware page lists Blackwell, Hopper, Ada Lovelace, and Ampere, NVIDIA and nothing else. NVIDIA claims that FP8 on H100 and later can double performance and halve memory use relative to 16-bit floating point, a vendor claim with no model, benchmark, or baseline. vLLM's docs put the throughput half at up to 1.6x. These figures use different conditions and should not be compared directly.

What happened to TGI?

Hugging Face has put TGI into maintenance mode. As of September 5, 2026, its README limits ongoing work to small bug fixes, documentation updates, and lightweight maintenance. It recommends vLLM, SGLang and compatible local engines such as llama.cpp or MLX for new work.

That is the maintainer's recommendation, not a benchmark, and it sets no end date. If you run TGI, your migration targets are the two engines compared here.

The KV cache problem behind all four engines

Sort the techniques above, and only some of them avoid recomputing tokens. Prefix reuse does. Paging, batching, disaggregation, and memory flags mostly change how work is allocated, scheduled, moved, or executed. Separate the two and one shared limit stands out.

The built-in prefix caches live in memory and are subject to eviction. vLLM reclaims cached blocks from its free queue under memory pressure, while SGLang applies LRU eviction to radix-tree leaves. Cached prefixes can also be lost on restart or redeploy, and a request routed to a cold replica cannot use another replica's local cache.

Connectors can extend reuse beyond the built-in cache. vLLM's disaggregated prefilling interface includes LMCacheConnectorV1, which can connect instances to a standalone LMCache server holding shared KV state. External storage can retain that state beyond the engine process.

Engine choice and caching strategy are two decisions, not one. A layer that owns the KV cache can carry it across requests, sessions and replicas, a lever no single-engine benchmark captures. That is what Tensormesh Platform is: our self-hosted software, built on the open-source LMCache project our founders created. Owning the context caching lifecycle covers what persistence changes, and LMCache's peer-to-peer architecture covers sharing it between replicas.

Tensormesh Platform supports vLLM. For engine, model, and feature compatibility, consult the compatibility matrix before deploying.

Our partner program covers integration opportunities. Our separate Serverless Inference service charges nothing for cached tokens, so greater context reuse can reduce the cost per request.

Choosing an engine

Pick vLLM for the broadest documented model coverage and a release you can pin. Pick SGLang when much of your traffic shares long prefixes, or for reinforcement learning rollouts. Pick TensorRT-LLM if you are committed to NVIDIA hardware. For TGI, follow the maintainer's recommendation and plan a move.

Measure your prefix reuse ratio on your own traffic to assess how much repeated computation either engine can avoid.

Contact our team to discuss caching for your inference stack.