Accelerating SGLang HiCache with Netpreme X-Mem™ MPU

Prefix caching is designed to reduce expensive prefill compute for workloads where requests repeatedly share long, token-identical prompts that only differ in the short last part. SGLang has already built a strong software foundation for prefix reuse with RadixAttention and HiCache. RadixAttention enables efficient prefix caching in GPU memory, while HiCache extends it into a hierarchy beyond GPU HBM.

However, as GPUs become faster, the speed of loading KV cache into the HBM becomes important. To estimate the bandwidth requirement for KV-cache offloading, we model a long-context coding-agent workload with 64K-token prompts and prefill concurrency of 8. As the prefix hit rate increases, residual computation drops quickly, but the amount of KV data fetched from secondary memory grows. When hit rate exceeds 95%, scaling the bandwidth of KV-cache tiering can significantly reduce TTFT.

But is 95% of prefix cache hit rate really practical? Let us look at KV-cache reuse in coding agents, the largest market for AI applications as of today.

To quantify this, we measure prefix-cache hit ratio using traces collected from Claude Code agent while running multi-turn coding workflows on SWE-bench. We observe that the cache-hit ratio consistently exceeds 95% in subsequent rounds for the same session, with the mean hit ratio being around 98%. In this regime, scaling KV-caching bandwidth beyond what is achievable with Host CPU offloading results in a 3x performance boost.

Netpreme X-Mem™ MPU is a purpose-built solution to expand GPU memory by tens of TB over fast network fabrics. Conceptually, X-Mem™ is a dedicated memory node in the AI rack, peer-to-peer to all other GPUs. A single memory node comes with up to 24 TB of memory accessible by all GPUs in the same rack at 4 TB/s. There is no limit on how many memory nodes can be provisioned in the rack, and the aggregate bandwidth is multiplied if several MPU nodes are used. On the software side, X-Mem™'s address space is exposed as a part of the unified virtual memory, and semantically it is not any different from accessing a local HBM or remote GPU's memory.

When integrating with SGLang HiCache, Netpreme X-Mem™ performs as a dedicated memory tier for KV cache offload. Instead of using Host DRAM or RDMA as the primary L2 KV cache tier, MPU provides a purpose-built, accelerator-oriented memory tier optimized for KV movement.

This makes X-Mem™ especially useful for workloads with:

• strict SLO requirements on TTFT;
• long shared prefixes and high-concurrency sessions;
• repeated prefill-heavy requests.

Netpreme X-Mem™ integrates SGLang via CUDA- and PyTorch-compatible APIs. These APIs allow any ML application to tap into the high-bandwidth memory tier transparently and with minimal modification of the application code.

If you would like to try it out yourself, contact us to get access to our X-Mem™ pilot.

We run two experiments, one is a micro-benchmark that measures TTFT of a single request and the other is an end-to-end LLM inference experiment with agentic workloads. Throughout benchmarking, we compare three configurations:

The result shows that Netpreme's MPU reduces single request TTFT by up to about 6.7x compared to Host DRAM, making it almost flat even under very long context.

How does the above TTFT reduction translate into improvements in end-to-end LLM inference? To answer this question, we run the end-to-end LLM serving benchmark, NVIDIA AIPerf. We use workloads that represent agentic AI inference, consisting of 1K tokens of system prompt and 20K tokens of per-user context. Each user inputs 26 tokens in a single turn, and the average number of turns is 20. Solid points indicate Pareto-optimal points; shaded points indicate non-Pareto-optimal ones.

The result demonstrates that Netpreme's X-Mem™ provides 33% higher TPS/user, which corresponds to interactivity, in a medium-load case and 50% higher interactivity and 30% higher TPS, which corresponds to system capacity, in a high-load case. This is because the GPU is utilized more efficiently, spending less time waiting for data to be copied on prefix cache hits.

As coding agents, long-context assistants, and recommendation systems grow, serving systems will need to preserve and move much larger KV working sets. To this end, Netpreme will facilitate:

• Larger persistent KV stores with MPU and SSDs
• Multi-instance KV sharing
• Near-memory KV compression
• Near-memory computation

The long-term vision for Netpreme is to provide a full-stack memory solution for agentic workloads. This includes not only KV cache but also model weights, activations, embeddings, and other data structures. By providing a high-performance memory tier optimized for ML workloads, Netpreme enables new software architectures and optimizations that are not possible with traditional Host DRAM and GPU HBM.

SGLang already provides a strong software foundation for prefix caching. RadixAttention and HiCache make it possible to reuse KV cache across long-context, multi-turn, and prefix-heavy workloads. But as prefix caching becomes more effective, the bottleneck shifts.

The challenge is no longer only finding reusable prefixes. It is moving cached KV back to GPU memory fast enough so that reuse delivers real latency and throughput improvement.

Host DRAM-based offload increases capacity, but its PCIe-level bandwidth can become the limiting factor for KV-intensive inference, especially when TTFT SLO is tight. Netpreme X-Mem™ addresses this bottleneck by providing SGLang HiCache with a high-bandwidth purpose-built KV memory tier.

Together, SGLang HiCache and Netpreme X-Mem™ make prefix caching faster and more scalable for the next generation of LLM workloads, including coding agents, long-context assistants, and recommendation systems.

We thank the SGLang team for their constructive technical discussions and guidance throughout this integration.

1. SGLang Documentation: HiCache Design
2. SGLang HiCache: Fast Hierarchical KV Caching with Your Favorite Storage Backends
3. Netpreme SGLang X-Mem Integration
4. LinkedIn Engineering Blog: Turbocharging LinkedIn's Recommendation Systems with SGLang
5. Qwen3-235B-A22B-Thinking-2507
6. SWE-bench
7. Netpreme Coding Agents Experiments