vLLM's OpenAI-Compatible Server: What Transfers and What Doesn't
The online serving page documents a specific set of OpenAI APIs, each qualified by which model types it applies to and, in three cases, by which request parameters behave differently.
vLLM implements the OpenAI API protocol, allowing existing SDKs to connect to its server. Parameter behavior still needs to be checked for each endpoint.
That distinction matters most for code you didn't write. A framework that builds requests for you, with fields you never chose, is where an ignored parameter can change behavior without producing an error.
The OpenAI endpoints vLLM documents as supported
The supported endpoints have the following model-type requirements.
Two things are easy to misread. The scope notes are about the model you loaded: /v1/embeddings applies only to embedding models, so a server running a text generation model will not serve useful embeddings, whatever your client expects. And /v1/realtime appears only inside the Speech-to-Text section, scoped to ASR models. It isn't a general OpenAI Realtime implementation, so don't plan against it as one.
The same port also carries surfaces that aren't OpenAI's at all: an Anthropic messages API at /v1/messages, a Cohere Embed API at /v2/embed, and a rerank API at /rerank, /v1/rerank, and /v2/rerank implementing Jina AI's v1 rerank API with compatibility for Cohere's v1 and v2 rerank APIs. If you're choosing a serving layer to avoid lock-in, that breadth is the real argument.
The three parameter caveats behind "drop-in replacement"
Check these three parameter differences before migrating an application.
The /v1/completions endpoint does not support suffix. If your code sets suffix for fill-in-the-middle completions, that field has no effect here, and the behavior it produced won't be reproducible.
The /v1/chat/completions endpoint ignores user. Plenty of production code sets user as an end-user identifier for abuse tracking and rate accounting. vLLM accepts the request but ignores this field, so reporting that depends on it needs another source of user identification.
On /v1/chat/completions, parallel_tool_calls=false limits a response to zero or one tool call. The default, true, permits multiple calls but does not guarantee them: the model must support that behavior.
This sets an upper limit. The parameter caps how many tool calls can come back. It doesn't make a model produce them. An agent loop written against a provider whose models batch tool calls, then pointed at one that doesn't, still gets valid responses. It just makes more round trips than its author budgeted for, and looks slower with nothing in the logs broken.
These three are the fine print underneath "drop-in replacement." Test for them before cutover, not after.
Starting the server
Check the documented host requirements: Linux, Python 3.10–3.13 and, for NVIDIA GPUs, compute capability 7.5 or higher. Examples include T4, RTX20xx, A100, L4, H100 and B200. Windows is not supported natively. These requirements establish engine compatibility, not whether a particular model will fit.
Install vLLM with:
uv pip install vllm --torch-backend=auto
And this for the container path:
docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=$HF_TOKEN" \
-p 8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model Qwen/Qwen3-0.6BConfigure shared memory with --ipc=host or --shm-size. vLLM uses PyTorch shared memory for communication between processes, particularly during tensor parallel inference; insufficient shared memory can cause multi-GPU failures.
The launch command from the Quickstart is one line:
vllm serve Qwen/Qwen2.5-1.5B-Instruct
Check any additional flags against the documentation for your installed vLLM version.
The server starts at http://localhost:8000 and currently hosts one base model, with model-listing, chat-completion, and completion endpoints.
Serving three different base models therefore requires three server processes and separate memory budgets. A router can expose them through one endpoint, but it does not remove the underlying instances.
Enable API-key checking with --api-key or the VLLM_API_KEY environment variable. Passing multiple keys after --api-key lets the server accept any of them, supporting key rotation. Key checking is opt-in; without either setting, the server does not require an API key.
Pointing existing client code at the server
The migration is small for a structural reason. vLLM implements the OpenAI API protocol on the paths OpenAI uses, all under /v1, so the client library doesn't change. What changes is the base URL and the credentials. Here is the before, in the shape OpenAI's own Python library documents:
from openai import OpenAI
client = OpenAI()
completion = client.chat.completions.create(
model="gpt-5.5",
messages=[
{"role": "user", "content": "Say this is a test"},
],
)
print(completion.choices[0].message.content)Set base_url to connect the client to your server. base_url is documented on that same client, with OPENAI_BASE_URL as its environment-variable equivalent, and OpenAI's own example points it at a plain http://host:port/v1. That is the shape a self-hosted server takes. The code reads:
import os
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key=os.environ.get("OPENAI_API_KEY"),
)
completion = client.chat.completions.create(
model="Qwen/Qwen2.5-1.5B-Instruct",
messages=[
{"role": "user", "content": "Say this is a test"},
],
)
print(completion.choices[0].message.content)http://localhost:8000 is the Quickstart's default bind, and /v1 is where vLLM puts the OpenAI-shaped routes. The credential is whichever key you passed to --api-key, and if you never passed one, the server does not check it.
This connects the client to the vLLM endpoint. Behavioral parity is a separate question, decided endpoint by endpoint by the notes above. A green response tells you the protocol matched. It says nothing about suffix, user, or your sampling defaults.
The part worth spending time on isn't the swap. It's the model identifier. The server hosts one base model at a time, and /v1/models reports the name it serves under. A hardcoded provider model name may be rejected. Confirm that the configured model matches the one used in your evaluations.
What changes without telling you
Two behaviors shift on migration without producing an error. Both are documented.
By default, vLLM loads generation_config.json from the model's Hugging Face repository when available. This can replace sampling defaults with values recommended by the model creator. Launch with --generation-config vllm to disable that behavior.
That is the likeliest cause of a migration where the API works, the model is right, and the outputs feel different. Temperature and the rest may not be what your code assumes: the model repository supplied recommendations, and vLLM honored them.
The second is chat templates. /v1/chat/completions requires a text generation model with a chat template, and messages are rendered into a prompt through that template. When a model ships without one, or ships one that doesn't match how your prompts were built, the documented override is:
vllm serve <model> --chat-template ./path-to-chat-template.jinja
Both can change output quality even when requests succeed. Compare outputs on a fixed prompt set during migration.
Endpoints you should not expose
Development endpoints require separate access controls and should not be exposed through a production ingress.
Keep development endpoints out of production. vLLM restricts dynamic LoRA loading and unloading to local development, and VLLM_SERVER_DEV_MODE=1 enables endpoints including /collective_rpc, which can execute arbitrary RPC methods on the engine.
Scale-out endpoints are disabled by default on vllm serve. Set VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 only when the deployment requires those APIs.
Production flags and the endpoints worth monitoring
The operational endpoints are plain: /health for a health check, /version for version information, /load for server load metrics, /v1/models to list available models, and /metrics for Prometheus-compatible metrics. Readiness probes should use /health. Anything that needs to know what's loaded should read /v1/models, not deployment configuration.
These settings control concurrency, logging and server behavior.
The default that surprises people is --gpu-memory-utilization at 0.92. Much of the internet still repeats 0.9, older vLLM docs included. Check the current value, not a remembered one. The setting applies per instance; another vLLM instance on the same GPU does not change its value.
Preemption can signal that the KV memory budget is too small for the active batch. vLLM frees space by preempting requests and recomputes them when enough cache capacity becomes available. That repeated work can increase latency under load.
Reducing repeated prefill with KV cache reuse
Protocol compatibility is a portability win. It's not a cost win. Once the endpoint works, the bill depends on how much of each request the GPU computes from scratch, and the preemption note states the problem: the KV cache is finite, contested, and the engine recomputes when it runs short.
vLLM's prefix cache retains KV blocks from processed requests and reuses them for later requests with an identical prefix. A repeated system prompt can therefore skip prefill without changing model outputs.
Built-in reuse has a limited scope. The same page describes eviction from a free queue, so cached blocks compete for finite space and a busy replica loses them. Reuse is prefix-shaped: it matches from the front of the prompt. And the built-in cache lives inside the engine process, so on its own it gives you nothing when a request lands on another replica or a pod restarts.
Carrying it further, across sessions and replicas, is a different layer's job, and it is the layer we build. Tensormesh Platform is a self-hosted caching and context-reuse layer built on the open-source LMCache project our founders created, running beneath the serving engine rather than replacing it. Tensormesh Platform supports vLLM, and our documented compatibility matrix provides model and configuration details.
On Kubernetes, our operator runs a mutating webhook that wires vLLM pods to the cache engine, with its own prerequisites: Kubernetes 1.28 or newer, a labeled supported GPU node (NVIDIA or AMD), cluster-admin permissions. Install the operator with Helm.
On our separate Serverless Inference rate card, the models that show a cached-input line price that input at $0, so check the rate for your chosen model. Output tokens are billed either way, so a hit is not a free request. What it removes is input you already paid to compute once. Chat with our team about inference costs for your workload.
For a managed alternative, Tensormesh's serverless service exposes an OpenAI-compatible API. Applications can connect by updating the base URL and model configuration.
Checking the migration
vLLM's server speaks the OpenAI protocol well enough that your client library and request shapes carry over. It is compatible endpoint by endpoint, with three documented parameter gaps and a set of model-type scopes, not compatible wholesale. Treat "drop-in replacement" as a claim about the wire, and the per-endpoint notes as the acceptance criteria.
Before you cut over, do three things. Check whether your requests set suffix or user, and account for those going nowhere. Decide whether --generation-config vllm should be on. And run a fixed prompt set through both endpoints, comparing outputs, because the failures that matter return 200.
Contact our team to discuss your OpenAI-compatible inference deployment.
