As agentic AI strikes from experiments to actual manufacturing workloads, a quiet however critical infrastructure downside is coming into focus: reminiscence. Not compute. Not fashions. Reminiscence.
Underneath the hood, at present’s GPUs merely don’t have sufficient house to carry the Key-Worth (KV) caches that fashionable, long-running AI brokers rely upon to keep up context. The result’s lots of invisible waste — GPUs redoing work they’ve already executed, cloud prices climbing, and efficiency taking a success. It’s an issue that’s already displaying up in manufacturing environments, even when most individuals haven’t named it but.
At a current cease on the VentureBeat AI Affect Sequence, WEKA CTO Shimon Ben-David joined VentureBeat CEO Matt Marshall to unpack the trade’s rising “reminiscence wall,” and why it’s turning into one of many largest blockers to scaling really stateful agentic AI — programs that may bear in mind and construct on context over time. The dialog didn’t simply diagnose the problem; it laid out a brand new manner to consider reminiscence solely, by means of an strategy WEKA calls token warehousing.
The GPU reminiscence downside
“Once we're wanting on the infrastructure of inferencing, it’s not a GPU cycles problem. It's principally a GPU reminiscence downside,” mentioned Ben-David.
The foundation of the problem comes right down to how transformer fashions work. To generate responses, they depend on KV caches that retailer contextual info for each token in a dialog. The longer the context window, the extra reminiscence these caches devour, and it provides up quick. A single 100,000-token sequence can require roughly 40GB of GPU reminiscence, famous Ben-David.
That wouldn’t be an issue if GPUs had limitless reminiscence. However they don’t. Even essentially the most superior GPUs high out at round 288GB of high-bandwidth reminiscence (HBM), and that house additionally has to carry the mannequin itself.
In real-world, multi-tenant inference environments, this turns into painful rapidly. Workloads like code growth or processing tax returns rely closely on KV-cache for context.
“If I'm loading three or 4 100,000-token PDFs right into a mannequin, that's it — I've exhausted the KV cache capability on HBM,” mentioned Ben-David. That is what’s often known as the reminiscence wall. “Out of the blue, what the inference surroundings is pressured to do is drop information," he added.
Which means GPUs are always throwing away context they’ll quickly want once more, stopping brokers from being stateful and sustaining conversations and context over time
The hidden inference tax
“We always see GPUs in inference environments recalculating issues they already did,” Ben-David mentioned. Methods prefill the KV cache, begin decoding, then run out of house and evict earlier information. When that context is required once more, the entire course of repeats — prefill, decode, prefill once more. At scale, that’s an unlimited quantity of wasted work. It additionally means wasted power, added latency, and degraded person expertise — all whereas margins get squeezed.
That GPU recalculation waste exhibits up straight on the steadiness sheet. Organizations can undergo practically 40% overhead simply from redundant prefill cycles That is creating ripple results within the inference market.
“In the event you have a look at the pricing of enormous mannequin suppliers like Anthropic and OpenAI, they’re truly instructing customers to construction their prompts in ways in which enhance the probability of hitting the identical GPU that has their KV cache saved,” mentioned Ben-David. “In the event you hit that GPU, the system can skip the prefill section and begin decoding instantly, which lets them generate extra tokens effectively.”
However this nonetheless doesn't clear up the underlying infrastructure downside of extraordinarily restricted GPU reminiscence capability.
Fixing for stateful AI
“How do you climb over that reminiscence wall? How do you surpass it? That's the important thing for contemporary, cost- efficient inferencing,” Ben-David mentioned. “We see a number of corporations attempting to resolve that in several methods.”
Some organizations are deploying new linear fashions that attempt to create smaller KV caches. Others are centered on tackling cache effectivity.
“To be extra environment friendly, corporations are utilizing environments that calculate the KV cache on one GPU after which attempt to copy it from GPU reminiscence or use an area surroundings for that,” Ben-David defined. “However how do you try this at scale in an economical method that doesn't pressure your reminiscence and doesn't pressure your networking? That's one thing that WEKA helps our clients with.”
Merely throwing extra GPUs on the downside doesn’t clear up the AI reminiscence barrier. “There are some issues that you just can’t throw sufficient cash at to resolve," Ben-David mentioned.
Augmented reminiscence and token warehousing, defined
WEKA’s reply is what it calls augmented reminiscence and token warehousing — a method to rethink the place and the way KV cache information lives. As an alternative of forcing every little thing to suit inside GPU reminiscence, WEKA’s Augmented Reminiscence Grid extends the KV cache into a quick, shared “warehouse” inside its NeuralMesh structure.
In apply, this turns reminiscence from a tough constraint right into a scalable useful resource — with out including inference latency. WEKA says clients see KV cache hit charges soar to 96–99% for agentic workloads, together with effectivity positive factors of as much as 4.2x extra tokens produced per GPU.
Ben-David put it merely: "Think about that you’ve got 100 GPUs producing a certain quantity of tokens. Now think about that these hundred GPUs are working as in the event that they're 420 GPUs."
For big inference suppliers, the outcome isn’t simply higher efficiency — it interprets on to actual financial influence.
“Simply by including that accelerated KV cache layer, we're taking a look at some use instances the place the financial savings quantity can be hundreds of thousands of {dollars} per day,” mentioned Ben-David
This effectivity multiplier additionally opens up new strategic choices for companies. Platform groups can design stateful brokers with out worrying about blowing up reminiscence budgets. Service suppliers can supply pricing tiers primarily based on persistent context, with cached inference delivered at dramatically decrease value.
What comes subsequent
NVIDIA tasks a 100x enhance in inference demand as agentic AI turns into the dominant workload. That strain is already trickling down from hyperscalers to on a regular basis enterprise deployments— this isn’t only a “huge tech” downside anymore.
As enterprises transfer from proofs of idea into actual manufacturing programs, reminiscence persistence is turning into a core infrastructure concern. Organizations that deal with it as an architectural precedence relatively than an afterthought will acquire a transparent benefit in each value and efficiency.
The reminiscence wall is just not one thing organizations can merely outspend to beat. As agentic AI scales, it is likely one of the first AI infrastructure limits that forces a deeper rethink, and as Ben-David’s insights made clear, reminiscence can also be the place the following wave of aggressive differentiation begins.
