A support assistant may answer hundreds of different questions using the same product documentation. Some questions repeat closely enough that it can return a cached answer. Others need a new response, even though the model has already processed much of their context.

Semantic caching handles the first case by reusing answers to similar questions. KV caching can reduce repeated computation in the second by reusing previously computed attention state. Understanding that difference helps you choose where to cache, what savings to expect, and when to use both.

What semantic caching actually matches

Semantic caching stores previously generated answers and matches new prompts by meaning rather than exact wording. The incoming prompt is converted into a vector embedding, and a vector similarity search finds the nearest stored prompt. If the similarity score clears a configured threshold, the cache returns the stored answer and the generation model is never called.

The lookup compares the new query's embedding with stored query embeddings to find a previously answered question with similar meaning.

Embedding similarity introduces a tradeoff between matching the right question, lookup latency and computational cost. A 2025 study by researchers at Redis and Virginia Tech examines that tradeoff when tuning embedding models for semantic caching.

The lookup path, step by step

Five steps, and each one runs in front of the model rather than inside it.

1. Embed the prompt to produce a numeric representation of its meaning.
2. Run a vector similarity search against the stored prompt vectors held in a vector database or index.
3. Compare the best match against the threshold. Search returns a similarity score for each candidate, and serving the highest-scoring result is a common policy. A semantic cache can also return multiple results because its lookup is a query rather than a single-key match.
4. On a cache hit, the cached response returns immediately, and the model is never invoked.
5. On a miss, call the model and write the pair back, so the next semantically similar request can hit.

A semantic cache hit still requires an embedding lookup and vector storage. Redis notes that avoiding output-token charges is the clearest saving; embedding and storage costs can offset input-token savings.

Why exact-match caching stops working on natural language

Traditional caching looks up a request using an exact key, such as a URL, user ID, or query string. Semantic caching instead searches vector representations for similar requests, allowing differently worded questions to match.

Natural language isn't stable. Two users who want the same thing will almost never type the same string, and an exact-match cache treats every rewording as a brand new request.

In AWS's example, three users ask an IT help bot to set up the company VPN using different wording. Exact-match caching treats them as separate requests and calls the LLM three times. Semantic caching can reuse the first answer for the other two, requiring only one model call.

GPTCache identifies variation in natural-language queries as a reason exact matching produces low cache hit rates. Its semantic approach matches requests by meaning rather than requiring identical text.

The tradeoff runs in both directions. Traditional caching is narrow, and its usual failure is a miss that costs money rather than a wrong answer. That is not a guarantee of correctness, since an exact-match cache can still serve stale or wrongly scoped data. Semantic matching adds intent collision on top of the ordinary freshness, invalidation, and authorization risks, because a near miss still counts as a hit and gets served to a user.

Semantic caching runs before generation, not inside it

Semantic caching runs in the request path before generation, often in app middleware or an AI gateway, and its job is to stop the upstream call.

With LangCache, the application checks the cache before calling the model. A miss returns an empty response, after which the application requests a fresh answer from its chosen LLM.

A centralized AI gateway can provide this cache lookup for multiple applications. In either deployment pattern, the semantic cache sits around the model call rather than inside inference. LangCache is currently in preview.

GPTCache is the self-hosted equivalent, and it is an in-app library rather than a gateway. It intercepts the call and answers it locally when it can. Several AI gateways ship the same capability alongside routing, budgets, and keys.

Blocking an LLM call outright is the largest saving available on any request, since it removes prefill, decode and the provider's margin in one move. Provider-side prompt caching is a third thing again, sitting inside the API you call rather than in front of it.

The layer sees request and response text, so it cannot directly reuse the model's internal key and value tensors. A cache miss still incurs lookup costs before generation. It embeds the prompt, searches the index, and stores the result after generation.

Why a KV cache is not a semantic cache

A KV cache operates inside the serving engine, at the inference and GPU layer, and it caches computation rather than text. Autoregressive transformer inference commonly uses one, because inference at any useful speed is impractical without it.

Autoregressive generation repeatedly needs the keys and values for earlier tokens. Without KV caching, each step recomputes that state; with caching, it reuses earlier keys and values and computes only those for the current token.

The tensors can be retained in GPU memory for later decode steps. That storage is the KV cache: it holds attention state for a sequence in progress and avoids repeated computation.

Three mechanisms, not one

The term KV cache covers three mechanisms, and only the first corresponds to the within-request reuse described above. Separating them is the difference between an accurate cost model and a hopeful one.

1. Sequence-local KV cache. Inside a single inference request, the engine keeps the K and V tensors it has already computed, so each decode step does not recompute them. This is the within-request reuse described above. It saves nothing across requests and disappears when the sequence ends.
2. Cross-request prefix caching. A later request can reuse an earlier request's KV blocks when both share the same token prefix. vLLM matches hashed blocks and reuses only complete blocks. A differing token breaks the prefix match for all following blocks.
3. A persistent, shared cache layer. LMCache and Tensormesh extend reuse across requests, sessions and serving instances by offloading and retaining KV state beyond GPU memory. LMCache spans GPU and CPU memory plus disk or remote storage. CacheBlend adds non-prefix reuse for supported configurations.

Almost every claim that a KV cache saves money across requests is really a claim about the second or third. The first one never leaves its own request.

Answer caching and KV caching act on different artifacts. KV mechanisms can coexist and reduce repeated computation while the model still generates a fresh response.

Reuse across requests is the part that changes a bill. When a new request shares context with one already processed, its key-value tensors may already exist. That shared context might be a system prompt, a retrieved document, a tool schema, or six earlier turns.

Supplying reusable prefix state from cache skips the corresponding prefill work, provided the blocks remain available. Prefill processes the prompt in parallel and is typically compute-bound, while decode is generally constrained by memory bandwidth.

The last column shows the difference in avoided work. A semantic cache answers a repeated question without invoking the generation model and does nothing for a novel one. Cross-request prefix caching and persistent caching can cut the cost of a new request that shares earlier context. None of them skips generation, so a question answered a thousand times still pays a full decode. Different input, different unit of reuse, different failure mode, different place in the stack.

The last of those rows is where we work. Tensormesh Platform is a self-hosted KV cache and context-reuse layer built on the open-source LMCache project our founders created. It has no view of prompt text or user intent at all. Those are request-path concerns, and they stay request-path concerns.

Running both, and what each one catches

Push a realistic mix of requests through a stack running both caches and watch where each one lands.

Request one is a customer support question a hundred users have already asked, worded slightly differently. The semantic layer can absorb it. A stored prompt's embedding is close enough, and the cached response goes back without invoking the generation model, though the embedding call and the index search still run. KV reuse could reduce prefill work, but it would still require decoding the answer token by token.

Request two is a new question about a 200-page contract the same user asked about ten minutes ago. The semantic layer misses it because the question is genuinely different. Underneath, the contract's key value tensors may still be cached, and if the contract is genuinely the shared token prefix, the model reads them instead of reprocessing the document.

That is conditional rather than automatic. The blocks have to have survived eviction, the request has to land somewhere that can reach them, and the model, configuration, and isolation namespace all have to match. When they do, prefill can skip the cached prefix and process the remaining input.

Request three is turn seven of a conversation. Every turn resends the thread, so consecutive prompts look almost identical to an embedding model while requiring completely different answers. That's the case a semantic cache should be configured to avoid.

Cross-request prefix caching is the better fit here, because the six earlier turns are a genuine shared prefix. It is subject to the same conditions as request two, though. Nothing is handled natively across separate API calls unless the serialized history really is a reusable prefix and the blocks stay reachable under the same model, configuration, and security namespace.

Request four is one step of an agent loop that rewrites its own prompt between iterations. The semantic layer misses, because no two iterations are the same request. Ordinary prefix caching misses too, since the reusable material has moved position rather than staying at the front. Only the third mechanism can help, and only where non-prefix reuse is supported, reusing cached chunks at shifted positions rather than a clean shared prefix.

Of these four requests, semantic caching can answer the first, prefix reuse can help with the next two, and non-prefix reuse may help with the fourth. The layers are not mutually exclusive either. The same request can be eligible for both, and an upstream semantic hit removes it from the downstream denominator. Report the two hit rates conditionally rather than adding them up.

Reuse depends on the request and retained state. A semantic hit bypasses generation; KV hits reduce repeated processing but still produce a new answer.

What a self-hosted caching layer for repeated context adds down here, below anything semantic caching can see, is capacity and reach. The Platform manages a three-tier hierarchy for vLLM: L0 in GPU HBM, L1 in host RAM, and L2 in filesystem-backed storage.

So a long document that won't fit in L0 can still be reused rather than recomputed, and our control plane reports cache hit rate, throughput, latency, and cost savings.

We do not charge for cached tokens, and the more a workload reuses context, the more its cost per request can drop.

Tensormesh Platform supports vLLM. Our compatibility matrix includes model and configuration details. The caching layer is LMCache, licensed under Apache 2.0, so you can inspect its implementation.

When semantic caching returns the wrong answer

A semantic cache can make two matching errors: returning a stored answer that does not fit the new question, or missing a reusable answer. These are false positives and false negatives, respectively.

A high similarity threshold can reject useful matches and store multiple answers to equivalent questions. A low threshold can return irrelevant answers. Tune the threshold on representative requests; there is no universally correct score.

Then there's context collision, which threshold tuning can't fix at all. Microsoft's example is worth walking through because it's so mundane. A user asks for the largest lake in North America and gets Lake Superior. They follow up with "What is the second largest?", and with the conversation in context, they correctly get Lake Huron. That pair is cached.

Later, a different user in a different session asks about the largest stadium in North America. Then they ask, "What is the second largest?" The cache matches on the follow-up alone and answers Lake Huron. This returns a lake name for a stadium question. The fix is to key on part of the chat history instead of the last prompt. That works, and it also shrinks the pool of entries two users can share. It is not an authorization boundary on its own, either.

Per-user and time-sensitive data are the other class, and Oracle's write-up on scaling agentic systems names both. "What is the weather in Seattle today?" and "What is the weather in Seattle tomorrow?" are near-identical to an embedding model and must not share an answer. "What is my account balance?" and "What is John's account balance?" are structurally similar. Those answers must remain isolated between users.

A time-to-live setting bounds staleness and does nothing about permissions. The key has to carry the scope that actually matters. That means the tenant, user, and permission set, the model and system-prompt version in force, and whatever freshness policy the underlying data demands.

Agent traffic compounds all of it, because agent loops defeat caching that matches on the whole request. Every iteration rewrites part of the prompt, producing a long tail of near-duplicate requests that are individually distinct and collectively expensive. Narrow the threshold, and you get no hits. Widen it and you replay the wrong iteration's answer.

So scope it. On high-repetition, self-contained, cross-user questions, it's excellent, and the AI applications AWS names for it, RAG assistants, copilots and documentation bots, are the right ones. Expecting it to cover the rest of your traffic is the mistake.

Using the two caching layers together

Semantic caching and KV caching answer different questions about reuse.

Semantic caching checks whether a similar question has already been answered and whether that answer is still valid. It compares embeddings before generation and returns stored text on a valid hit, skipping the generation model entirely.

"Have I already computed this?" is answered on the GPU in three stages. The sequence-local KV cache answers it within one request. Cross-request prefix caching answers it for a later request that shares a token prefix. A persistent layer answers it once those blocks have left the GPU, or once the request has landed on a different instance. When any of them says yes, you skip prefill for that context and generate the rest, so the model still runs and still produces a fresh response.

Choosing between them is the wrong exercise. The right one is figuring out which requests each layer catches.

So pull two numbers this week, separately. Semantic cache hit rate and KV cache hit rate, with the second read against the requests the first one did not already absorb. If the upper number looks healthy and the bill hasn't moved, the work is happening below it, where it cannot see. Chat with our team about improving KV cache reuse for your workload.