XM-Sparse: Pushing the Throughput Frontier of Sparse Attention with Netpreme X-Mem™

Sparse attention accesses only selected KV entries at each decode step, but still retains the full KV cache because the selection varies during token generation. SGLang HiSparse uses a hot KV buffer to reduce external-memory reads and keeps the indexer key cache (K-index) in GPU HBM (Figure 1).

Agentic workloads raise the stakes. Coding agents keep hundreds of sessions open with up to 95% KV reuse per session (see our previous blog). For these workloads, sustained decode throughput matters more than time to first token, since decode throughput determines serving capacity at high concurrency.

Enlarge

Figure 1

Figure 1. Existing KV cache offloading architecture. Full KV caches reside in host DRAM; decode GPUs retain hot KV buffers in HBM and fetch missing selected KV over PCIe.

This architecture imposes two key hardware requirements:

1. A high-bandwidth interconnect to fetch missing selected KV entries quickly and reduce decode latency.
2. Sufficient external-memory capacity to store full KV caches for large batches.

We compare modeled decode throughput across four external-memory configurations: PCIe, CXL, NVLink-C2C, and Scale-up expanded memory. For the PCIe baseline, we assume 64 GB/s of bandwidth per GPU and a total host-memory budget of 1 TB for the four-GPU instance, proportionally scaled from a standard eight-GPU DGX configuration. CXL expands capacity to 24 TB at the same bandwidth. NVLink-C2C increases bandwidth to 225 GB/s per GPU in Grace-Blackwell with the same 1 TB total capacity. In the model, Scale-up expanded memory delivers 900 GB/s per GPU and 24 TB of total capacity via a high-bandwidth fabric.

We measure decode compute time without KV I/O, then add estimated I/O overhead using a representative 20% miss rate based on trace analysis in prior work. The model assumes each GPU sustains its modeled bandwidth, with no compute/I/O overlap:

The maximum batch size is limited by both HBM and external-memory capacity:

Capacities are usable budgets per GPU; memory footprints are per request. With 4-way EP and DP attention, we allocate 60 GiB of HBM per GPU to the hot KV buffer and K-index after accounting for weights and other allocations.

The following plots show modeled GLM-5.3 decode throughput on a four-GPU instance (DP4 for attention + EP4 for experts). We focus on 128K as a representative context length for coding-agent workloads reported in TraceLab, with 16K and 1M as complementary cases. Each request uses a 4,096-token hot KV buffer per layer and selects 2,048 KV entries per layer at each decode step. Throughput (output tokens/s) and batch size are both reported per GPU, with each configuration plotted up to its capacity limit. The dashed Ideal curve uses the measured compute timings, assuming no I/O overhead and sufficient external-memory capacity.

Enlarge

Figure 2

Figure 2. Modeled decode throughput at 128K context length.

At 128K, both external-memory capacity and I/O bandwidth constrain throughput. Larger full KV caches limit the 1 TB configurations to batch 36 per GPU, even with the higher bandwidth of NVLink-C2C. Expanding capacity to 24 TB allows batch 109, but the increased KV traffic leaves PCIe-CXL bandwidth-bound. At that batch, the 900 GB/s scale-up fabric delivers 1.39× the throughput of 64 GB/s PCIe-CXL. High throughput therefore requires both sufficient external-memory capacity and a high-bandwidth interconnect.

Enlarge

Figure 3

Figure 3. Modeled decode throughput at (a) 16K and (b) 1M context. Both memory configurations in the 1M panel use 64 GB/s per GPU.

The shorter 16K and longer 1M cases help distinguish the two constraints seen together at 128K. At 16K, external-memory capacity is already sufficient: batch 241 per GPU uses only 0.85 TB across four GPUs, within the 1 TB budget. I/O bandwidth is the primary bottleneck, and the 900 GB/s scale-up fabric delivers 1.80× the throughput of 64 GB/s PCIe at that batch. At 1M, larger full KV caches make external-memory capacity the main constraint for the 1 TB configuration. Expanding to 24 TB raises the feasible batch from 4 to 20 per GPU and delivers 2.34× the throughput at the same 64 GB/s bandwidth. Throughput remains close to Ideal at each batch, indicating a smaller I/O penalty.

Netpreme X-Mem™ Memory Processing Unit (MPU) is purpose-built to expand GPU memory by tens of terabytes over high-bandwidth fabrics within a scale-up domain. X-Mem™ acts as a dedicated memory node in the AI rack and supports peer-to-peer GPU access. A single node holds four MPUs with up to 24 TB of DDR5 memory and about 4 TB/s of aggregate memory bandwidth, shared by GPUs in the rack. Adding memory nodes expands available capacity and aggregate bandwidth. This allows the memory tier to scale independently of compute.

XM-Sparse integrates X-Mem™ as fast, high-capacity shared memory for SGLang HiSparse’s prefill and decode workers. Prefill GPUs publish KV caches to X-Mem™. Decode GPUs retain hot KV buffers in HBM and fetch missing selected KV entries over the scale-up fabric (Figure 4).

Enlarge

Figure 4

Figure 4. KV cache offloading with Netpreme X-Mem™.

X-Mem™ exposes GPU-addressable memory through CUDA- and PyTorch-compatible APIs. XM-Sparse preserves HiSparse’s sparse selection, cache management, and prefetch mechanism, and supports CUDA Graph capture and replay for decode computation and selected-KV fetching.

Requests with a common prefix can share the corresponding KV cache in X-Mem™, avoiding duplicate storage for that prefix. An owner process creates the shared memory region, which GPU processes attach to using a portable descriptor. The owner releases the physical memory only after all processes have detached their mappings.

KV management follows the request lifecycle:

• Publish KV cache. Prefill GPUs write prompt KV directly to X-Mem™. Once publication completes, decode can access the history without first staging the full prompt KV in decode HBM.
• Fetch selected KV. Decode GPUs check their local hot buffers and fetch only the missing selected KV entries from X-Mem™ into HBM for attention.
• Write back generated KV. After each successful decode step, newly generated KV is appended to the history in X-Mem™, making it available to subsequent steps.
• Release KV storage. When a request finishes, the system waits for outstanding KV accesses to complete before reclaiming its storage for new requests.

We evaluate GLM-5.3 on four NVIDIA B300 decode GPUs with attention DP4 and MoE EP4. We compare SGLang HiSparse backed by host DRAM with XM-Sparse, our X-Mem™ integration, labeled “SGLang HiSparse + X-Mem” in the figures. Four additional B300 GPUs are used as the memory banks, with their HBM standing in for X-Mem™ memory behind the same access path the silicon will expose. X-Mem™ uses a 24 TB capacity budget, with batch size determined by modeled capacity limits rather than the physical memory of the four bank GPUs.

Each request has a 4,096-token hot KV buffer per layer. We use synthetic KV prefixes and fixed miss rates, with prefetch disabled. The top-k miss rate is the fraction of selected KV entries absent from the GPU hot buffer and therefore fetched from external memory. We use the same 20% miss rate as in the analysis and add a 40% case to evaluate performance under higher I/O pressure. We report average TPOT and per-GPU throughput. The first comparison measures TPOT at matched batch sizes; the second compares each configuration at its highest-throughput tested batch.

At 16K, both configurations use the same batch sizes and the same amount of external memory. This comparison therefore evaluates the latency benefit of a high-bandwidth interconnect rather than additional capacity.

Enlarge

Figure 5

Figure 5. Average TPOT (lower is better) at 16K with prefetch disabled: (a) 20% miss rate; (b) 40% miss rate. Batch size is per GPU. Arrows mark the reductions at batch 192.

At batch 192 per GPU, X-Mem™ reduces TPOT from 167.48 to 84.99 ms at 20% misses, a 49.25% reduction, and from 255.69 to 93.13 ms at 40% misses, a 63.58% reduction. Small batches show little difference, and X-Mem™ can be slightly slower; the benefit of faster KV access becomes more pronounced as the batch grows. The larger reduction at 40% misses reflects the increased value of fast external access when more selected KV must be fetched from external memory.

We next compare each configuration at its highest-throughput tested batch, using 20% misses with prefetch disabled. Host DRAM uses a 1 TB external-memory capacity budget, while X-Mem™ uses a 24 TB budget across all context lengths.

Enlarge

Figure 6

Figure 6. Bars show peak decode throughput in tokens/s per GPU at a 20% miss rate; dashed lines show batch size per GPU. Labels above the blue bars show X-Mem™ throughput relative to host DRAM.

X-Mem™ achieves 2.09–3.04× the host-DRAM throughput across different contexts. At 16K, both configurations run batch 241 per GPU, so the 2.09× gain reflects the benefit of a high-bandwidth interconnect at the same concurrency. At longer contexts, larger external-memory capacity enables larger batches, so the throughput gains reflect both faster KV access and increased concurrency.