An assistant that answers a question in one call can become much more expensive when it starts retrieving documents, calling tools, and carrying conversation history through several model requests. Even if the price per token falls, the total work needed to complete a task can grow.

Reducing inference cost means finding which work drives the bill: model execution, repeated context, output generation, or unused GPU capacity. This guide breaks down those costs and shows how caching, batching, and deployment choices affect them. The broader market and energy picture is covered separately.

Why inference cost dominates the AI budget

Training is a bounded project. Inference runs as long as the model is live and scales with every request. AWS estimates that inference can account for up to 90% of the infrastructure spend for developing and running ML applications. That upper estimate covers ML workloads broadly, rather than a typical LLM deployment.

Gartner predicts inference costs per agentic workflow will rise more than fivefold through 2028, as more capable systems use more tokens and may need more expensive ones. Analyst Will Sommer told CIO Dive that agents can use five to 30 times as many tokens per query as chatbots.

Both are true at once. Stanford's 2025 AI Index found the cost of GPT-3.5-level performance fell more than 280-fold between November 2022 and October 2024, and Epoch AI measured fixed capability milestones falling 9x to 900x per year. Fixed capability gets cheaper while what teams ship gets bigger.

What actually drives inference cost

Six factors drive inference cost: model size, token volume, context length, utilization, runtime efficiency, and cache hit rate.

Model size sets the floor: parameters times bytes per parameter, so a 7B model at FP16 is about 14 GB before it serves a request. Token volume is the meter, and input and output don't count equally. On OpenAI's Standard flagship tier, output runs 5 to 6 times the input rate at short context and roughly 4 times in the long-context tier, where input itself doubles. Context length multiplies the meter: long documents and conversations get reprocessed every turn unless something stops them.

Utilization is the master variable, and the one teams underestimate. GPU-hours bill whether or not the device is busy, so at a fixed hourly rate, doubling useful throughput (tokens per second, or completed requests per GPU-hour) halves cost per token.

Don't read that off a utilization percentage. NVIDIA's DCGM defines SM Activity as the fraction of time at least one warp was active, notes that warps waiting on memory count as active, and notes that even a value of 0.8 or greater does not establish efficient GPU use on its own. Use throughput and latency alongside utilization to assess useful work.

Runtime efficiency decides how much of the hardware the work reaches, since prefill and decode load a GPU very differently, and most of the practical tuning lives there. Cache hit rate governs how much work you skip entirely.

The prefill and decode split, and why caching is the first-order lever

Every request runs in two phases with different economics. Miss that split and everything downstream is a guess.

Prefill processes many tokens in parallel and can make extensive use of GPU compute. Decode generally performs less arithmetic per byte transferred and is often limited by memory bandwidth, as NVIDIA explains in its inference optimization guide. The artifact prefill produces and decode consumes is the KV cache.

Provider pricing reflects the savings from reusing context. On the Standard tier of OpenAI's published rate card, cached input is exactly one tenth of standard input across all three current flagships at short context: GPT-5.6 Sol $4.00 to $0.40 per million tokens, Terra $2.00 to $0.20, Luna $0.20 to $0.02.

Two conditions come with it. Writing to the cache costs 1.25x the uncached input rate, and the minimum cacheable prefix is 1,024 visible input tokens. Net those and a prefix is ahead the second time it's sent: 1.25x to write plus 0.1x to read back, against 2.0x uncached. The write premium is recovered once the prefix is reused.

Six levers that cut inference cost

Six levers do most of the work in LLM inference optimization, and they stack.

Two carry published throughput numbers, and the conditions matter more than the multiples. The Orca authors measured 36.9x over NVIDIA FasterTransformer at equal latency, on GPT-3 175B in 2022. vLLM's PagedAttention improved throughput 2 to 4 times against the state of the art then, more on long sequences. Neither is a multiplier to expect on your workload.

Why caching specifically moves the needle

Every lever above is real. Caching gets the emphasis because it removes work rather than speeding work up: a hit skips prefill you'd pay for twice.

Be precise about what it doesn't do. Cached KV blocks still occupy the finite cache while they're live, and on a hit vLLM pulls the reused block out of the free queue so it can't be evicted, so reclaiming that space costs the cache entry. Shared prefixes cut duplicated KV storage. Larger batches are a separate measurement we'd benchmark, not assume.

The workloads where this compounds are the ones people build: repeated queries against one long document, multi-round conversations that reprocess the whole history, and agent loops that rewrite their context. We describe that repeated work as the amnesia tax, and it's why open-weight models often miss their promised savings.

NVIDIA measured it directly: reusing a shared system prompt across a burst of concurrent users, rather than recomputing it per user, accelerated inference by up to 5x.

It's the mechanism we built Tensormesh on. LMCache, the open-source layer underneath, moves KV cache out of GPU memory into a tiered hierarchy across CPU memory, local storage, and remote backends, so it's reused across requests and sessions rather than recomputed. On our Serverless Inference rate card, most models carry a cached-token rate of $0.00 per million.

Self-hosting vs. managed inference: where the math actually lands

Utilization decides this, and you can work out roughly where you sit before provisioning anything.

On-demand GPU rates, read September 3, 2026: CoreWeave's North America list shows an 8x H100 node at $49.24 per hour on demand and $19.71 spot, plus $6.16 per GPU for inference platform customers. Lambda lists H100 SXM at $3.99 per GPU-hour. Call it roughly $2.45 to $6.20 per H100-hour, depending on provider, region, and spot access. These move: OpenAI flags GPT-5.6 Sol's price as promotional through at least November 21, 2026.

Self-hosting wins when your load is steady enough to keep those GPUs busy, when you need data residency, or when your workload is odd enough that generic serving leaves performance on the table. The utilization arithmetic above swallows most theoretical savings.

Managed inference wins when traffic is spiky, when you'd rather not staff a serving stack, or when the per-token price already reflects optimizations you'd otherwise build. Compare these costs with the capacity and operational work of running your own stack.

Neither is cheaper in the abstract. Your cache hit rate and throughput matter more than any list price.

Where to start

Pull two numbers before you change anything: your cache hit rate and your throughput per GPU-hour. Both help identify where optimization will make a difference.

If the hit rate is low and your prompts share structure, you're paying full price for prefill you already computed. Chat with our team about reducing inference costs for your workload.