Cracking the Memory Wall: Agents, State and the Missing Tier

Sudeep Bhoja founded d-Matrix in 2019, when CNNs set the agenda. He traced the shift in one line.

“CNNs was a compute-bound problem…ever since autoregressive decoding in GPT-2 it became a memory-bound problem.”

Sudeep Bhoja, d-Matrix

Angela Yeung of Cerebras tied the next step to agents. Context per request keeps growing, and agents reuse the same context across many queries.

“Your agent is only useful if it actually has information about you, right? All the information about you is not going to fit into 100K of context.”

Angela Yeung, Cerebras

That context, she said, “has to be stored somewhere easily accessible, whether that’s on the accelerator or in close access to the accelerator.”

Weights stay in the fastest tier because every token touches them. KV cache for a million-token context is still large after years of compression work. When an agent pauses for a tool call, that state sits idle in the most expensive memory in the system.

“If you go for a tool call and spend thirty seconds, it doesn’t make a lot of sense to keep all of that state on the HBM tier, and this is where memory tiering is very helpful.”

Sudeep Bhoja, d-Matrix

Kartik Srinivasan, CEO of Napatech, laid out three ways to handle the KV cache:

• Tier it. Keep hot blocks next to the accelerator and move cold blocks to SSD.
• Share it. Let multiple agent workers reuse one stored KV cache instead of regenerating it.
• Move fractions. Send only the parts the next step needs.

“Why have the entire KV cache move over the network? Because compute is expensive, and traversing on the network is expensive.”

Kartik Srinivasan, Napatech

Angela matched each performance metric to a task:

• Throughput for batch jobs such as evals and synthetic data
• Time to first token for voice agents and classification
• Time between tokens for answers that run to thousands of tokens

“Time between tokens starts to matter, and it’s really the end-to-end response time.”

Angela Yeung, Cerebras

Ryan Wang of Nebius took the economics question. Token pricing lets a provider serve the same tokens on fewer GPUs. Lower unit prices then pull in more usage, as the Jevons paradox predicts.

“Maybe you have two times lower cost, five times more usage.”

Ryan Wang, Nebius

At the end of the panel, Kartik made a final point on how to measure what matters to end users.

“Tokens is not the right measure. Outcome is the right measure. Task completions are the right measure.”

Kartik Srinivasan, Napatech

Rakesh Cheerla, VP of Product at Netpreme, sorted agent state by how fast it must come back. Long tool calls can wait on SSD. Short calls, some in the microsecond range, need state in a warmer tier, and a planning agent can predict when to promote it. He then made the rack the design point.

“The rack is the new unit of compute. Agent state belongs in a shared warm memory tier, managed apart from the model.”

Rakesh Cheerla, Netpreme

That tier is what Netpreme builds. X-Mem, based on Netpreme’s Memory Processing Unit, places standard DDR memory on the scale-up fabric. Every accelerator in the scale-up domain reads one shared copy of KV cache and agent state at scale-up bandwidth. Netpreme designs X-Mem to extend accelerator memory by up to 100x and add hundreds of terabytes of shared memory per rack, at about 10x the bandwidth of PCIe or CXL attach. Kartik’s tiering and reuse both need a destination faster than SSD that every worker can reach. A rack-level warm tier is that destination.

Rakesh closed the speed round with a goal for 2029: inference that no longer has to shard across a whole rack. “We don’t need to shard as much, we don’t need to complicate the deployments as much,” he said. When state fits within reach of every accelerator, fewer GPUs wait on memory and the cost per completed task falls.

Netpreme thanks the panelists and the OCP 2026 attendees who joined us for this event! To learn more about X-Mem, visit netpreme.com.