A high inference bill can come from idle GPUs, repeated prompt processing or requests that could have been answered from a cache. Each calls for a different fix. A tool that reveals where spending happens may help you diagnose the problem without reducing the underlying computation.

The nine tools below work at different layers: requests, cost reporting, infrastructure and model execution. Comparing what each changes, the conditions it needs and the work it leaves untouched helps you choose a tool for the source of cost in your deployment.

Why the bill grows while token prices fall

A fixed capability keeps getting cheaper, and most teams' inference bills keep going up, because the work per request grows faster than the price per token falls. Model size sets a floor, token volume is the meter, context length multiplies the meter, and how well you fill the GPU decides what any of it costs. The full anatomy, with supporting measurements, sits in the companion piece on what actually drives LLM inference cost.

Two of those drivers are why a tools page exists at all. Idle GPU capacity bills exactly like busy capacity, and repeated context creates prefill work you already paid for once, which is why latency and cost are the same problem twice.

What the tools are implementing on your behalf

LLM inference optimization is a category name, not a technique. Every product below is a packaged version of something you could implement yourself: continuous batching keeps the device busy by rebuilding the batch every iteration, quantization moves fewer bytes per token, speculative decoding spends spare compute to buy back latency, and prefix caching skips prompt computation you already did. They stack, and they pay differently depending on whether your workload is prefill-heavy or decode-heavy, so settle that question before you shop.

The phase split, and which half of it your bill is made of, is worked through in the companion post on prefill versus decode. Buy a decode optimization for a prefill problem and the bill doesn't move.

Three things called caching, and only one of them skips prefill

Every gateway in this roundup lists caching as a cost feature. So does every frontier API provider, and so does the serving engine underneath both. They aren't the same mechanism; they don't fail the same way, and buying one while expecting another is the most common way teams end up with an unchanged bill.

Gateway response caching stores complete LLM answers. LiteLLM returns a stored response when the same request repeats, avoiding another model API call. Helicone offers a similar edge cache using Cloudflare Workers KV across more than 300 locations, though its documentation scopes it to development and testing.

What a response cache saves is the entire call. What it needs in return is a request that matches one already answered, and the two matching methods fail in opposite directions. Exact matching is narrow and safe: change a token of the request and you miss, and you pay the provider in full. Semantic matching is broader and riskier, because a near miss still counts as a hit.

Semantic caching can replay stale answers in multi-turn or agentic workloads. LiteLLM explains that consecutive turns often contain nearly identical conversation histories, with embedding similarities around 0.99, even though each turn needs a different response.

Portkey's version has similar limits. Semantic caching there is restricted to select Enterprise plans, requires a vector database, works on requests under 8,191 tokens with at most four messages, and ignores the system prompt when matching. Check these limits against your requests before enabling semantic caching.

Provider prompt caching reuses computed KV tensors when requests share a prompt prefix. OpenAI stores those tensors rather than the input tokens themselves, allowing it to skip repeated prompt computation.

Reused tokens bill at a discounted cached-input rate, with discounts of up to 90%. For GPT-5.6 and later, reads cost 0.1x the uncached input rate and writes cost 1.25x. Caching requires at least 1,024 tokens, with state retained for at least 30 minutes after the latest write or reuse. The cache lives on individual machines; traffic above 15 requests per minute can trigger overflow routing and reduce hits.

The same mechanism also works inside a self-hosted serving stack. vLLM retains KV blocks from processed requests and reuses them when a later request has an identical prefix, avoiding repeated prefill without changing model outputs.

Free of charge is not the same as free of engineering. Cached blocks occupy finite cache while they're live, and shared deployments need vLLM's per-request cache_salt so one tenant's blocks aren't reused by another.

The distinction that matters when you're choosing a tool is this. A response cache matches on the request and returns text, so it pays on repeated identical calls, and on a moving conversation it either misses or, in semantic mode, answers the wrong turn. A prefix cache matches on the prefix and returns computation, so it pays exactly where agent loops and long documents hurt. A gateway can only do the first. The second happens where the KV cache lives.

Top tools to reduce ML inference costs

If you're looking for the top tools to reduce ML inference costs without sacrificing performance, the useful sort isn't by rank. It's by which part of the stack a tool touches, because products in different categories aren't substitutes for each other.

Gateways and routers

A gateway sits between your application and the model providers, gives you one API across all of them, and becomes the natural place for budgets, keys, and a response cache.

LiteLLM's proxy provides a shared gateway to multiple LLMs, with cost tracking and budgets per project or user and virtual keys for access control.

For licensing, LiteLLM is MIT except for the enterprise/ directory, which carries a separate license.

Portkey, now branded PRISMA AIRS AI Gateway and generally available, combines model routing with observability, guardrails, governance, and prompt management. The gateway is open source under MIT and runs locally with a single npx command, but the hosted platform and the open-source component are different products, and the caching conditions above belong to the platform. The public gateway repository's last commit is dated May 25, 2026.

Helicone is an open-source LLM observability platform licensed under Apache 2.0, with an OpenAI-compatible gateway alongside its tracing. Its documentation distinguishes response caching from provider-level prompt caching.

Bifrost is an Apache 2.0 gateway that provides access to more than 23 providers through one OpenAI-compatible API. It supports semantic caching and hierarchical budgets using virtual keys, teams, and customers. Its benchmarks measure proxy overhead in microseconds rather than inference-cost reductions.

Spend visibility

Neither tool here reduces your bill, and neither claims to. They earn a place because you can't cut spend you can't attribute, and most teams can't produce cost per feature or per customer on request.

Langfuse provides open-source tools for tracing, debugging, and evaluating agent applications, including visibility into cost and latency. Its licensing separates the core and enterprise features: tracing, evaluations, prompt management, and experiments are MIT with no usage limits, while enterprise modules live in marked /ee directories and need a license key when self-hosted.

LangSmith is the commercial equivalent, covering individual request traces and production-wide performance metrics across cloud, hybrid, and self-hosted deployments. Its published tiers run Developer at $0 per seat with up to 5,000 base traces a month, Plus at $39 per seat per month, and Enterprise custom. Include its subscription cost when budgeting for observability.

Kubernetes, GPUs and cloud commitments

Cast AI automates Kubernetes, and its inference product, Kimchi, runs coding and inference workloads inside your own infrastructure. It aims to reduce costs by improving GPU efficiency and scaling capacity with demand, including to zero when idle. Its hybrid routing assigns reasoning to an expensive model and execution to cheaper self-hosted open models.

The utilization argument is real, but be careful which utilization number you plan against. NVIDIA's DCGM documentation defines SM Activity as the fraction of time at least one warp was active, counts warps waiting on memory as active, and notes that even a value of 0.8 or greater does not establish efficient GPU use on its own. Busy is not the same as productive.

nOps optimizes cloud commitments by analyzing usage and purchasing a mix of reserved instances, Savings Plans, and committed-use discounts. This changes infrastructure purchasing rather than request execution. Its AI module attributes hourly spend to models, accounts, teams, customers, and features, and recommends changes to token efficiency, model selection, and cache tuning.

nOps operates at the purchasing and cost-management layer. Commitment coverage lowers the per-hour price. It does not change how much work happens inside it.

Managed KV cache and context reuse

This category acts on computation rather than on the request or the invoice, and it's where Tensormesh Platform, our self-hosted product, sits. We build it as a self-hosted KV cache and context-reuse layer on LMCache, the Apache 2.0 open-source project our founders created, so the caching approach is inspectable rather than proprietary. LMCache moves KV cache out of GPU memory into a tiered hierarchy so it can be reused across requests, sessions, and engine instances, and there's a longer write-up of how its peer-to-peer architecture works.

The pricing mechanic is worth stating precisely. Our separate Serverless Inference service does not charge for cached tokens, and most models on its rate card carry a cached-token rate of $0.00 per million, though not every listed model shows a cached-token line. Savings depend on context reuse: the more your workload reuses context, the more your cost per request can drop. Our hosted inference offering has two deployment modes on the same rate card: Serverless Inference behind an OpenAI-compatible API and Reserved GPUs for dedicated capacity, both with caching.

Tensormesh Platform supports vLLM and runs on Kubernetes with NVIDIA and AMD GPUs, using Helm for installation.

The Platform provides prefill/decode disaggregation and peer-to-peer KV cache sharing. Its cache hierarchy has three tiers: L0 in GPU HBM, L1 in host RAM, and L2 in filesystem-backed storage. Our compatibility matrix provides model and configuration details.

How to choose, by what you're actually short of

Pick the category first. Which tool you pick inside it matters much less than getting the category right.

The last row is the one most teams underestimate. Two numbers decide it: your cache hit rate, and your throughput per GPU-hour. If the hit rate is low and your prompts share structure, you're paying full price for prefill you already computed, and no gateway will fix that for you, because a gateway never sees the prefill.

How to read a cost claim in this category

OpenAI's billing documentation specifies the read multiplier, write premium, minimum prefix length, and routing behavior. Performance claims need comparable workload and hardware details before they can inform a cost estimate.

Compare cost-reduction results only when the baseline, workload, and hardware are specified. Without those conditions, a published multiplier cannot predict savings for your deployment.

These mechanisms provide a more useful basis for comparison: Cast AI on idle GPUs, LiteLLM where semantic caching breaks, and vLLM when prefix caching doesn't change outputs. Test each mechanism on your own workload before estimating savings.

Choosing the right cost-reduction tool

These nine tools address four layers of inference cost. KV cache layers remove repeated prefill work. An exact response cache can remove a repeated call outright. Infrastructure automation cuts placement and idle-capacity waste, and observability identifies where the waste occurs. Those aren't interchangeable, so choose based on the source of waste in your workload.

If your prompts repeat, start with the prefill line, because that's the work you're paying for twice.

Pull your cache hit rate and your throughput per GPU-hour this week. If the hit rate is low, chat with our team about where caching could help.