How to Serve LLMs Efficiently Across Multiple Workloads
A chatbot needs a fast response while an overnight classification job needs sustained throughput. Put both on the same GPU pool without workload controls, and batch traffic can delay interactive users. Give each its own capacity, and some of those GPUs may sit idle.
Sharing capacity efficiently requires decisions about model placement, routing, scheduling and cache isolation. This guide explains how to make those decisions across workloads and tenants, including when separate serving instances are necessary and how to prevent one workload from consuming another's resources.
What it means to serve LLMs across multiple workloads
Serving LLMs across multiple workloads means running more than one distinct inference job on a shared GPU pool: different models, different fine-tunes of one model, or different classes of traffic against the same model. The engineering work is in isolation and arbitration. Each workload needs its own memory budget, priority, and cache boundary, without a dedicated cluster per workload.
That's a different problem from making one workload fast. Throughput techniques like continuous batching, PagedAttention, and tensor parallelism operate inside a single serving instance and are covered in our complete guide to LLM serving. Splitting prefill and decode onto separate pools is its own architecture, covered in our guide to disaggregated serving. This piece is about what happens above those: many tenants, one pool.
The constraint everything else works around
With the current implementation, two different base models mean two server processes, two sets of weights resident in GPU memory, and two independent schedulers that know nothing about each other.
That has three practical consequences. Weights are duplicated per instance, so a pool that could hold one 70B model in memory can't hold three of them plus their KV caches. Utilization fragments, because each instance batches only its own traffic, and an idle instance can't lend its slots to a busy one. And routing becomes your problem, since nothing in the engine decides which of your servers a request should reach.
A fourth one shows up inside each instance rather than between them. Every instance divides GPU memory between resident weights and KV cache, so each workload you add takes cache headroom from the ones already there. Every instance also mixes prompt processing with token generation in one batch, which can increase TTFT under heavy load as prefill and decode compete for the same GPU. A second workload with a different prompt shape makes that mix harder to schedule, not easier.
So there are really only three answers, and production systems usually run two of them at once.
Multi-LoRA gives you many behaviors, not many models
LoRA adapters can add specialized behavior without loading another copy of the base weights. vLLM supports per-request adapter selection for models that implement SupportsLoRA, with low serving overhead.
You register adapters at launch:
vllm serve meta-llama/Llama-3.2-3B-Instruct \
--enable-lora \
--lora-modules sql-lora=jeeejeee/llama32-3b-text2sql-spiderSelect a LoRA adapter through the model request parameter, just as you would select a base model. vLLM can process adapter requests alongside base-model requests and, when max_loras permits, requests for other adapters in the same deployment.
The knob that decides how much that costs you is rank. vLLM's guidance is to set --max-lora-rank to the maximum rank among the adapters you plan to serve, because setting it higher than necessary wastes memory and may reduce performance. Rank is set per deployment, not per adapter, so one high-rank adapter taxes every other tenant on that server.
Runtime adapter changes require stricter access controls. vLLM supports loading and unloading adapters through POST /v1/load_lora_adapter and POST /v1/unload_lora_adapter, gated behind the VLLM_ALLOW_RUNTIME_LORA_UPDATING environment variable.
Dynamic adapter loading should be restricted to isolated, fully trusted environments. vLLM also warns against enabling remote downloads through the Hugging Face Hub resolver in production. In a multi-tenant deployment, these features can expose a tenant-controlled loading path inside the serving process.
An adapter only multiplies the behaviors of one base model, so within a single vLLM server process, different base models still need separate instances: a Llama tenant and a Qwen tenant get two. A gateway or an orchestration layer can put those instances behind one endpoint, so the split stops being visible to callers. Underneath, it's still there.
Routing workloads across the GPU pool
With multiple serving instances, a routing layer must choose where requests run. NVIDIA Dynamo coordinates SGLang, TensorRT-LLM, or vLLM across nodes rather than replacing those engines. A single model on one GPU will often need only the inference engine.
llm-d provides two relevant routing components. Multi-Model Routing exposes multiple models and LoRA adapters through one gateway using the Inference Payload Processor. Precise Prefix Cache Routing uses cache state published by model servers to route requests toward reusable context, avoiding unnecessary prefill.
A cache-aware router can weigh queue length against the prefill work avoided by sending a request to a backend that already holds its prefix. If you're picking a stack, our serving-stack benchmark compares them under load rather than on feature lists.
Choosing a scheduling policy for mixed workloads
Queueing policy can create latency problems across workloads. vLLM defaults to first-come, first-served (FCFS), handling requests in arrival order. This can work for similar requests, but interactive requests may wait behind overnight batch jobs on a shared endpoint.
With priority scheduling, lower values run earlier and arrival time breaks ties. The X-Vllm-Priority header accepts an integer on the Completions, Chat Completions, and Responses APIs and overrides the JSON body's priority value. Non-zero priorities require the server to use priority scheduling.
That header is what makes priorities operationally useful: the classification can live at your gateway rather than in every client. A workable scheme is coarse: interactive traffic at 0, internal tooling at 10, batch and backfill at 100. Resist per-tenant integers, because a priority space you can't reason about is one you can't debug.
What priority scheduling doesn't do is create capacity. It reorders a queue; it doesn't put a floor under the low-priority end, and vLLM documents no guarantee either way. So if a batch tier has a deadline, protect it with admission control, reserved capacity, or its own instance, and watch the observed queue length rather than trusting the number you assigned. Our LLM serving pillar covers the noisy-neighbor pattern more generally, including the case for splitting workload classes onto separate endpoints.
Tenant isolation is a cache problem before it's a compute problem
Cache isolation also matters. When several tenants share one pool, the KV cache is shared state, and shared state between tenants is a boundary you declare rather than assume.
Our multi-tenancy documentation splits this into two axes: controlling who may share cached prefixes, and controlling how much cache each tenant may consume. The first is what cache_salt provides. This per-request isolation key determines which requests may share cached KV state. The same model and same salt allow cache access, while a different salt blocks it. It's a first-class field on OpenAI-compatible vLLM requests, so it travels in the request body:
{
"model": "<your-model>",
"cache_salt": "tenant-acme",
"messages": [{"role": "user", "content": "..."}]
}Choose a stable salt for requests that should share cache and a different salt for requests that must remain isolated. Keep the number of distinct values manageable so metrics labels do not grow without bound. Tenant or workspace identifiers such as tenant-acme and workspace-42 are suitable examples.
A random UUID on every request prevents cache sharing, effectively making each request a miss. A salt is validated too, rejected with a ValueError if it contains @, /, \, a NUL byte, or exceeds 128 characters.
The second axis is capacity. Per-tenant quotas give each cache_salt a byte budget. IsolatedLRU maintains a separate eviction order for each salt, so one tenant's evictions affect only that tenant's entries. That's the mechanism that stops one heavy tenant from evicting everyone else. A small admin API manages them at runtime: PUT /quota/{cache_salt} sets a limit_gb, and GET /quota lists every registered quota and its usage.
Register a quota for every tenant whose data should remain cached. A salt without a registered quota has an effective budget of zero: writes are accepted initially but evicted on the next cycle, approximately one second later. Monitor for salts that appear in requests but have no quota entry.
Per-tenant hit rate is then measurable rather than inferred. The engine exports lookup and hit counters labeled by model and salt, so hits divided by lookups, grouped by salt, is a real per-tenant hit rate. A reuse-gap metric records the time between a chunk's last write and its next read.
Validate cache_salt isolation, per-tenant quotas, and IsolatedLRU in staging with representative tenant traffic before relying on them in production.
Use deployment-level isolation when tenants must not share GPUs or memory at all. Cache salts and quotas provide logical separation within a shared pool; they do not replace physical isolation requirements.
The half of the problem most teams don't schedule
Everything above arbitrates compute: scheduling, routing, and quotas decide who gets the GPU and for how long. What they don't address is that in a multi-workload pool, much of the work being scheduled is recomputation of context the pool has already processed, sometimes minutes earlier, for a different tenant hitting the same documents.
llm-d groups Advanced KV-Cache Management as a top-level capability, and Dynamo uses cache-aware routing to avoid repeated computation and higher TTFT. Prefix caching inside one replica is a starting point. Reuse that survives across replicas, tenants, and sessions is where most deployments leave money on the table, an argument we make at length here.
It also gets harder in exactly the multi-workload case, because a shared prefix is a fragile thing. Once each tenant prepends its own system prompt or its own retrieved documents in a different order, the reusable content is no longer a clean prefix, and ordinary prefix caching finds nothing.
CacheBlend reuses contiguous cached chunks even when they move to different prompt positions. It selectively recomputes a small subset of tokens to update reused blocks, extending reuse beyond an exact prefix match.
Tensormesh Platform provides non-prefix caching for supported dense models. To use the plugin image from our private registry, request an access token.
Check our compatibility matrix for per-model coverage before deploying non-prefix caching. Models marked verified are tested in Tensormesh CI and listed as production-ready; models marked supported require testing before production use.
Tensormesh Platform supports vLLM, with prefill/decode disaggregation so those phases can run on separate GPU pools, and peer-to-peer KV cache sharing over RDMA where available, falling back to TCP.
In results we published with AMD in July 2026, Dell servers with eight MI355 accelerators supported twice the model density. On that configuration, TTFT also fell from 3.4 seconds to under half a second for Kimi-K2.6 on a 300 GB document set.
Those are our own figures from one named configuration rather than a general result, and the full write-up carries the setup. Retaining reusable cache can reduce repeated prefill work. On the serverless side, our pricing follows the same logic, with no charge for cached tokens.
Putting the workload plan into practice
If you're serving more than one workload on one GPU pool, four questions decide the architecture, and they're answered in this order.
Do your workloads share a base model? If yes, LoRA adapters give you many behaviors on one set of resident weights, and you can skip a second instance. If no, you're running multiple instances behind a gateway, and the router in front of them should be cache-aware rather than load-only.
Do your workloads have different urgency? Then FCFS is actively working against you. Set a coarse priority scheme, remember that lower values are handled earlier, and give any tier with a real deadline its own capacity rather than a better number.
Do your tenants need to be kept apart? Then declare the cache boundary explicitly with a salt, register a quota for every tenant you want cached, and validate isolation in staging with representative tenant traffic.
Measure cache reuse as well as scheduling efficiency. Group your cache hit rate by tenant and see how much of your pool's compute goes into context it has already processed. If the answer is a lot, capacity isn't your constraint; reuse is. Chat with our team about improving cache reuse across your workloads.
