Why LLM Inference Is a Data Problem, Not Just a Compute Problem
The real challenge in modern AI inference is becoming a data problem. Models produce and consume large amounts of reusable state during inference, especially KV cache. That state is no longer a temporary implementation detail. It is becoming one of the most important artifacts in the AI infrastructure stack.
Big data went through a similar transition about fifteen years ago. Before the modern data stack existed, teams tried to make relational databases handle workloads they were never designed for. Hardware vendors poured money into scaling up single machines, and much of the industry assumed the existing infrastructure stack would stretch far enough. It did not.
The shape of the data was different. The access patterns were different. The economics were different. A new stack eventually emerged with distributed storage, new query engines, and new orchestration layers. The companies that built on that stack moved faster than the ones still trying to bolt big data onto OLTP databases.
LLM inference is in the same place today. Most of the industry is approaching LLM inference optimization by building on top of existing infrastructure. The focus is on faster GPUs, larger on-chip memory, and higher token throughput. Those improvements matter, but from where I sit, they do not solve the core problem.
What language models produce and consume during inference is a new kind of data. It needs a new AI inference infrastructure stack built specifically for it.
I think of this as AI-native data. It is state produced by a model, consumed by a model, and increasingly valuable across the entire inference pipeline. KV cache optimization is the most visible example, but it is not the only one. Hidden states, encoder outputs, and intermediate activations for multimodal inputs all belong in the same category.
LLM Inference as a Distributed Systems Problem
My background is in networking and distributed systems, including congestion control, real-time data infrastructure, and the layers underneath whatever the application thinks it is doing. When I look at LLM inference from that angle, the picture looks different from the traditional machine learning view.
I start asking different questions. What is being recomputed that could be reused? Where is the bandwidth actually going? Which tier of the memory hierarchy is oversubscribed, and which one is sitting idle? What does the access pattern look like over time, across users, across sessions, and across workflows?
These are the same kinds of questions that shaped big data infrastructure. They are also the right questions for AI-native data. They are not the questions teams usually ask when the prevailing mental model is that the model is the product and everything underneath it is just plumbing.
The teams that win on AI inference cost, latency, and output quality over the next few years will be the ones that stop treating the inference layer as plumbing. They will treat it as data infrastructure, with abstraction layers, observability, lifecycle management, and a real cost model.
Why Prompt Caching and Bigger HBM Do Not Solve the Full Problem
If AI-native data is the right framing, then many current industry decisions start to look reasonable on the surface but expensive underneath. Two examples make the point clearly.
The first is prompt caching in LLMs as it exists today. Most caching offered by hosted inference providers is append-only. You can extend a cached prefix, but you cannot remove something from the middle of the context without invalidating the cache.
That may sound like a small limitation until you see how it affects long-running conversations, AI agent workflows, and long-context applications. Developers cannot easily prune noisy or irrelevant context because pruning breaks the cache. Once the cache breaks, the LLM inference cost savings disappear. So teams leave the noise in. Then, somewhere between hundreds of thousands of tokens, the model’s attention starts getting pulled toward irrelevant material, and output quality begins to degrade.
The append-only constraint quietly forces a tradeoff between cost and quality. That tradeoff should not exist. A real data layer would let teams edit cached state, remove what no longer matters, and keep the savings on the context that still does.
The second example is where the hardware industry is putting much of its investment. There is enormous investment going into expanding on-chip GPU memory. Bigger HBM. More of it. Closer to compute. That is useful, but it is not the only layer that matters for the scale this data is heading toward.
KV cache requirements for serious AI agent infrastructure, long-context inference, and retrieval-heavy workloads are already moving toward extremely large memory footprints. No realistic amount of HBM will hold everything the next generation of AI applications wants to reuse.
What matters more is the bandwidth between GPU memory and the next tier down, including DRAM, NVMe, and distributed storage. The bottleneck is not only capacity. It is movement. The industry is spending heavily on resources that cannot scale to the data sizes ahead, while underinvesting in the infrastructure needed to store, move, reuse, and manage model-produced state.
Both examples point to the same underlying issue. LLM inference is being treated as a compute problem when it is increasingly a data problem.
The AI Inference Infrastructure Stack That Will Emerge
Three years from now, I expect there to be a recognizable software stack dedicated to AI-native data. That stack will sit apart from the inference engines that run the models, in the same way storage systems sit apart from the query engines that use them. It will treat model-produced state as data with a lifecycle, observability, quality controls, and a cost model that reflects what it takes to store and move that data.
The capabilities of this stack need to go far beyond cache hit or cache miss. State should be computable offline for content you already know you will serve. It should be reusable across non-prefix boundaries, across sessions, and in some cases across models. It should be inspectable in ways that help humans understand what the model is doing and intervene when quality drops.
It should also be priced and optimized for energy, latency, and cost in the same way serious data infrastructure is. Teams building AI agents, long-context applications, and production LLM systems are going to need this layer whether or not the market hands it to them.
The question is whether they build it themselves, ad hoc, on top of whichever inference engine they are using, or whether a real infrastructure layer emerges that they can build on.
Open Source KV Cache Infrastructure and the Production Layer Above It
What we are building toward at Tensormesh has an open source core and a commercial layer on top. That split is intentional.
The core abstractions for managing AI-native data belong in the open. What gets cached, how state moves across the memory hierarchy, and how it gets reused should be shaped by the broader community. That is where the design gets stress-tested, where the contribution model works, and where the foundation of any future AI infrastructure stack has to live.
That work is what LMCache is for. LMCache is an open source KV cache layer for LLM inference. It sits between the inference engine and the memory hierarchy, enabling KV cache optimization across sessions, non-prefix boundaries, and in some cases across models.
This is the kind of LLM token caching that does not exist in most hosted inference providers today. LMCache is available on GitHub, and it is the foundation for the infrastructure we are building at Tensormesh.
The commercial layer is where research-quality infrastructure becomes production-quality infrastructure. That includes high availability, fault tolerance, expert plugins for specific workloads, guaranteed builds, and the operational support enterprises need when infrastructure becomes critical.
The value an enterprise pays for is not just the algorithm. It is the guarantee that the algorithm keeps running at 3 a.m. on a holiday, with real support behind it.
Building Production-Grade AI Inference Infrastructure That Compounds
There is one final layer that often gets overlooked, even though it is what enables AI infrastructure to compound in performance and efficiency over time. Building research-grade implementations of clever ideas is much easier than building a maintainable, evolvable codebase that other teams can rely on.
The difference shows up in CI/CD discipline, quality assurance, documentation, testing, compatibility, and long-term maintainability. It shows up in whether someone can pick up an algorithm two years from now and extend it rather than rewrite it. The majority of the LLM infrastructure shipping today is closer to research-grade than production-grade. That is understandable given how quickly the field is moving.
If there is going to be a real software stack for AI-native data, one that other companies build serious products on, somebody has to do the unglamorous work. The layers need to be clean. The abstractions need to be stable. The maintenance cost needs to be manageable. It is a deliberate bet that getting the foundation right will create more leverage than shipping the next clever optimization on top of a foundation that will not hold.
What LLM Inference Cost Means for Your AI Stack
LLM inference is not a commodity, and the layer underneath it is not a commodity either. AI-native data is a new category. It is large enough, valuable enough, and structurally different enough from what came before that it needs its own AI infrastructure stack.
The LLM inference cost curve will not flatten only by squeezing more tokens per second out of the same architecture. It will flatten when teams treat model-produced state as data, with a lifecycle, observability, and a cost model that reflects what it takes to store, move, and reuse that state.
The teams that recognize this early will have a meaningful advantage over the ones still trying to force LLM inference into infrastructure built for a different era.
If you are building anything serious on top of an LLM, the question is not whether this layer matters. The question is whether you are going to rebuild it yourself every time, or whether you are going to build on AI inference infrastructure designed for the data your model actually produces.
