When an AI feature has unpredictable traffic, provisioning GPUs means choosing how much idle capacity to pay for and how much demand the service can absorb. A serverless inference endpoint lets you start with requests and usage-based billing while the provider manages capacity.

As traffic grows, the decision becomes more specific: can the endpoint meet your latency and concurrency targets, and would dedicated capacity cost less at your actual utilization? This guide works through those tradeoffs, including cold starts and cache reuse, so you can evaluate when to stay serverless and when to move.

What is serverless inference?

Serverless inference is a deployment model where you send requests to a managed API and the provider handles the compute behind it. No instances to size, no clusters to provision, no scaling policies to write. You're billed for what you use, and nothing while the endpoint sits idle.

AWS manages the underlying infrastructure for its serverless inference option. However, the category covers two different types of service.

AWS SageMaker Serverless Inference is a container endpoint with 1 GB to 6 GB of RAM and, per AWS's own feature exclusions, no GPU support at all. Token-billed LLM APIs, Together's and Tensormesh's among them, run open-weight models on GPUs and bill per million tokens. Both are serverless. Only the second will serve you a 70B model, and pointing an agent at one mostly means changing a base URL, a model name, and an API key.

How serverless inference works

The general shape is the same across implementations. Your request hits the endpoint, which routes it to a worker that has the model loaded or starts one if none is available. The model runs, the response returns, and the compute goes back to a shared pool. AWS documents one implementation; the numbers below are SageMaker's, not the category's.

Capacity scales automatically in both directions. AWS scales its serverless endpoint to zero when no requests are arriving. With usage-based billing, you trade control over the hardware for never paying for an idle GPU.

Capacity is governed by concurrency quotas rather than instance counts. A single SageMaker serverless endpoint tops out at a maximum concurrency of 200, with 1,000 concurrent invocations shared across all serverless endpoints per Region in eight named Regions and 500 in thirteen others. They're soft quotas you can ask AWS to raise. Read them as the shape of the constraint anyway: you're budgeting concurrency, not machines.

The cold-start tradeoff

After an idle period, a serverless endpoint may need to start compute resources before processing a request. That delay is a cold start.

AWS does not give a fixed cold-start duration: it depends on model size, download time and container startup time. Cold starts can also occur when a traffic spike exceeds the capacity currently running, even if the endpoint has not been idle.

The standard remedy is provisioned concurrency, which keeps a set number of workers warm so they can respond within milliseconds. It works, and it means paying for idle capacity again. Include that warm capacity in your cost comparison.

Why serverless inference is the right call for most teams

For most teams, this isn't a compromise. It's the correct architecture, and the arguments for it don't expire once you grow.

The strongest is that you cannot plan capacity you cannot forecast. AWS recommends considering pay-per-use for infrequent or unpredictable traffic. Overprovisioning leaves paid capacity idle; underprovisioning can leave demand unmet.

Operational overhead is second, and it's routinely underpriced. Running inference yourself means GPU drivers, model loading, autoscaling, health checks, observability, and a pager. For smaller teams, operating cost can outweigh the token savings, so model the expected staffing and on-call burden rather than assuming it nets out.

Then iteration speed. On a compatible API, switching models is a small endpoint change, though behavior, context limits, quotas, tool calling, and evaluation still need validating.

And the workload shapes suit it: spiky consumer traffic, internal tools on business hours, batch jobs twice a week, prototypes still finding an audience. All of them have long idle stretches, which is exactly what serverless doesn't charge for.

Where the economics change as volume grows

Start with the arithmetic most people skip, because it points the opposite way from the received wisdom.

Break-even is monthly dedicated infrastructure and operating cost divided by effective API cost per token, and it has to be computed for the same model, precision, input and output mix, latency target, redundancy level, utilization, and commitment term. What follows is one illustration at list prices, not a threshold to borrow.

We publish gpt-oss-120b at $0.15 per million input tokens and $0.60 per million output tokens. Compare that with published rates for eight H100 nodes over a 730-hour month at full utilization, and ask what the hardware would have to sustain to be cheaper.

The p5.48xlarge rate is on-demand, read from AWS's pricing page on September 5, 2026; region and OS are stated because both affect it. Committed purchasing sits lower, as the Capacity Blocks rows show on the same instance.

Read the last row as a bound, not a forecast: every token billed at the output rate, a full-month commitment, a node that never idles. Even there you're sustaining tens of thousands of tokens per second around the clock. And no row normalizes for whether eight H100s are the right hardware for that model at that precision and latency target, or for redundancy, or for operating cost. That work is yours, and it turns a list-price crossing into a deployment decision.

So the per-token bill is rarely what should move you. Four other things change with volume.

Cold starts stop being a nuisance and become an SLA problem, and the fix reintroduces the cost you left. Concurrency ceilings start shaping your architecture, and queue-and-retry logic written around someone else's quota is infrastructure work without the benefits. Control starts to matter, because at volume you want to pick the engine, the batching policy and the hardware. Then there's cache state, which is the expensive one.

Agents, RAG pipelines, and long-running assistants send substantially the same prefix on every call: the same system prompt, tool definitions, and retrieved documents. As our guide to persistent caching explains, serverless cache state is often temporary. A session ends, the cache goes, and the next request re-encodes a prefix processed moments earlier.

We estimate that for high-context workloads this repeated processing routinely accounts for 60 to 90 percent of compute, an effect we call the Amnesia Tax. That's our own estimate rather than an independent benchmark, so check it against your own traces. It's the one number here you can measure in an afternoon.

Serverless vs. dedicated GPU inference

The cache row usually decides it, and it isn't strictly a serverless-versus-dedicated split. Some serverless providers, our own Serverless Inference among them, price cached tokens at $0 on most listed models. What the dedicated column really buys you is control over cache retention and eviction.

Signals it's time to reconsider serverless

Run these against your own numbers. If none of them describe you, stay where you are.

1. A large share of your input tokens is identical across requests. Sample your production prompts and measure the overlap. If most of it is the same system prompt, tool schema, and retrieved context, you're paying repeatedly for work already done.
2. Cold starts are hitting a latency SLA, especially if you've already enabled provisioned concurrency to fix it. At that point, you're paying for warm capacity without owning it.
3. Rate limits or concurrency quotas are shaping your architecture rather than occasionally annoying you.
4. Your traffic has become steady and forecastable enough that the flexibility you're paying for has no value left.
5. You're already running or evaluating open-weight models. This is the practical gate: self-hosting needs weights you control, and a team on a closed-weight API has nothing to move.

Chat with our team about serverless inference costs for your workload.

What self-hosting involves

The old objection was decisive: self-hosting meant a long infrastructure build before serving a single request. That is no longer accurate.

Open-source engines cut most of the work of exposing an efficient model server. vLLM, from UC Berkeley's Sky Computing Lab and now maintained by over 2,000 contributors, serves a model with one command: vllm serve Qwen/Qwen2.5-1.5B-Instruct. Its OpenAI-compatible server lets applications reuse existing client libraries. Compatibility is endpoint by endpoint rather than blanket, and the server hosts one model at a time.

What the engine doesn't give you is the operating layer: production routing, admission control, autoscaling, upgrades, and observability all remain your problem. SGLang and NVIDIA's TensorRT-LLM occupy similar ground, though TensorRT-LLM runs only on NVIDIA GPUs.

The second problem is harder, and it's why moving to your own GPUs sometimes disappoints: it doesn't by itself stop you from paying to reprocess the same context. A caching layer such as Tensormesh reduces that repeated work. We were founded by the creators of LMCache and built the platform on that open-source project. LMCache grew out of systems research at the University of Chicago and stays vendor-neutral across engines, so adopting the open-source layer and buying the managed one are separate decisions.

Alongside the self-hosted Platform, we offer a separate Serverless Inference product with cached tokens at $0 on most listed models and reserved GPU capacity, so the decision here isn't necessarily a change of vendor. Tensormesh Platform supports vLLM, and the compatibility matrix provides model and configuration details. Savings depend on your workload and infrastructure.

Self-hosting still costs something real: weights you control, a serving stack you understand, and someone accountable when it breaks at 3 am. Serving engines and caching layers reduce implementation work, but operating them remains your responsibility.

Conclusion

Serverless inference is a useful default. Revisit the decision as traffic, latency requirements, and operating costs change.

There is a break-even, but it belongs to your workload rather than to a volume number you can borrow from an article. The generalizable factors are cache lifecycle, cold-start behavior under an SLA, concurrency ceilings, and control. And the largest cost lever in a context-heavy workload isn't which side of the line you sit on; it's how much of your context you're paying to process twice.

Measure your input and output mix, cache-hit rate, utilization, and SLOs, then compare current per-token pricing with a same-model reserved-GPU deployment.

Contact our team to discuss serverless inference for your workload.