All articles

From SGLang Summit 2026

Don’t Over-Buy HBM, Tier It!

Our Co-Founder & CTO Nikita Lazarev gave a well-received talk on memory tiering for LLM inference at SGLang Summit 2026. The LMSYS event featured researchers and engineering leaders from OpenAI, NVIDIA, Google and Meta.

Nikita LazarevCo-Founder & CTO, Netpreme
Nikita Lazarev, Netpreme Co-Founder & CTO, presenting the SGLang HiSparse case study on stage at SGLang Summit 2026.
Nikita on stage at SGLang Summit, November 13, 2026.
Fort Mason, San Francisco.
Memory architecture · Sparse attention · LLM inference

Longer contexts and larger batches put pressure on GPU memory, limiting how many requests an LLM serving system can handle. At SGLang Summit, Netpreme’s session explored how an external memory hierarchy can ease that constraint.

Nikita used SGLang HiSparse to show how memory tiering can work for LLM inference. HiSparse reads selected entries from the key-value (KV) cache during decoding but retains the full history. A small hot buffer stays in high-bandwidth memory (HBM), while the full cache can live in an external tier. The talk then examined the bandwidth and software needed to fetch selected entries efficiently.

Highlights

  • HBM + external memory

    Keep a hot KV buffer in HBM and the full history in an external tier.

  • Top-k sparse attention

    Read selected entries at each step. Retain the full cache for later selections.

  • Data movement

    Fetch primitives, attention kernels and memory management make the tier usable.

Why HBM capacity is difficult to scale

Nikita’s opening slides contrast hundreds of gigabytes of memory near compute with an illustrative target of 5 to 10 TB at HBM bandwidth.

Figure 1. Nikita’s opening comparison: 250–450 GB near compute versus an illustrative target of 5–10 TB. Adapted from presentation slide 2.
Figure 1
HBM is Great slide comparing 250–450 GB near compute with an illustrative 5–10 TB target, using diagrams of a GPU surrounded by memory.

Nikita’s opening comparison: 250–450 GB near compute versus an illustrative target of 5–10 TB. Adapted from presentation slide 2.

That increase runs into limits in packaging, interconnect technology and manufacturing. Power, cooling and supply add further constraints. Even if a design is technically possible, its cost and reliability still need to hold up.

A memory hierarchy offers another way to add capacity. The design depends on which data must stay near the processor and how long the workload can wait for data stored farther away.

Models are changing how they use memory

Model architectures differ in how much state they retain and how they access it. Nikita discussed multi-head and grouped-query attention alongside multi-head latent attention, sparse attention and linear attention. He also pointed to Engrams and loop transformers as designs being explored.

These access patterns give hardware designers different options for placing data. Model design and memory design can develop together within physical and cost constraints.

The hierarchy in the talk leaves SRAM and HBM near compute and adds a larger external tier through a low-latency, high-bandwidth link. It calls for XPU-native unified memory and a memory-tiering library. Candidate uses include KV caches, sparse attention, Engrams and expert offloading.

Figure 2. The memory hierarchy proposed in the talk, combining SRAM and HBM with a large external tier and supporting software. The capacities and bandwidths shown describe the illustrated architecture. Adapted from presentation slide 6.
Figure 2
SGLang Summit slide showing SRAM, HBM and a 24 TB external memory tier, linked by a 3.6 TB/s connection, with software support for memory tiering.

The memory hierarchy proposed in the talk, combining SRAM and HBM with a large external tier and supporting software. The capacities and bandwidths shown describe the illustrated architecture. Adapted from presentation slide 6.

Why HiSparse retains the full KV cache

During dense-attention decoding, a new token’s query is compared with keys across the retained context. The resulting attention weights determine how the corresponding values are combined.

In the top-k sparse-attention example, an indexer selects a subset of historical positions before attention uses their keys and values. Each decode step reads much less KV data than dense attention.

The selection can change with the next token, so an entry skipped now may be needed later. This approach still needs the full KV history. If all of it stays in HBM, storage capacity can limit both context length and the number of requests served at once.

Figure 3. Top-k sparse attention reads selected KV entries while retaining the full history. Tiering keeps a hot set in HBM and the full cache in host memory. The 4.7× figure comes from the HiSparse paper; Netpreme’s modeled results are presented separately below. Adapted from presentation slide 10.
Figure 3
HiSparse slide comparing dense attention, top-k sparse attention and memory tiering: selective reads still require the full KV history, while tiering keeps a hot set in HBM and the full cache in host memory.

Top-k sparse attention reads selected KV entries while retaining the full history. Tiering keeps a hot set in HBM and the full cache in host memory. The 4.7× figure comes from the HiSparse paper; Netpreme’s modeled results are presented separately below. Adapted from presentation slide 10.

How SGLang HiSparse uses KV cache offloading

In SGLang HiSparse, KV cache offloading keeps the full KV history in host memory while a small hot KV buffer remains in GPU HBM. Selected entries already in the buffer can be used locally; missing entries must be fetched from the external tier. In the DSA-style path shown here, the indexer key cache (K-index) also remains in HBM.

Nikita cited the HiSparse paper by Xie and colleagues, which reports up to 4.7× the peak generation throughput of its full-KV-in-HBM baseline on the long-context workloads it evaluated. Our study below examines how changing the external tier’s capacity and bandwidth affects performance.

In the Netpreme architecture shown in the talk, X-Mem™ holds the full KV cache. Prefill GPUs publish KV data to the pool, while decode GPUs keep their hot buffers in HBM and fetch missing selected entries over the scale-up fabric.

Figure 4. The study’s architecture uses a 24 TB shared external-memory pool and 900 GB/s of modeled bandwidth per GPU. Prefill publishes KV; decode fetches entries missing from its hot buffer. Adapted from presentation slide 11.
Figure 4
Netpreme prototype system architecture: prefill GPUs publish full KV caches to X-Mem, and decode GPUs fetch missing selected KV through a 900 GB/s per-GPU scale-up fabric.

The study’s architecture uses a 24 TB shared external-memory pool and 900 GB/s of modeled bandwidth per GPU. Prefill publishes KV; decode fetches entries missing from its hot buffer. Adapted from presentation slide 11.

Capacity vs. bandwidth: the 128K-context study

Our GLM-5.3 study at 128K context compares four external-memory configurations. They vary the space available for the full KV cache and the bandwidth available to fetch selected entries during decode.

Figure 5. Modeled GLM-5.3 decode performance at 128K context. The approximately 3× batch-size and 2.4× peak-throughput comparisons use PCIe host memory as the baseline. The slide’s “up to 5×” at 1+ TB/s is a forward-looking projection, separate from the 900 GB/s result plotted here. Adapted from presentation slide 12.
Figure 5
Modeled GLM-5.3 128K decode-throughput chart comparing PCIe host memory, PCIe-CXL, C2C and a 900 GB/s scale-up tier; maximum batch rises from 36 to 109 per GPU.

Modeled GLM-5.3 decode performance at 128K context. The approximately 3× batch-size and 2.4× peak-throughput comparisons use PCIe host memory as the baseline. The slide’s “up to 5×” at 1+ TB/s is a forward-looking projection, separate from the 900 GB/s result plotted here. Adapted from presentation slide 12.

In our study, the modeled 900 GB/s tier supported roughly 3× the maximum batch size (36 → 109 requests per GPU) and 2.4× the peak decode throughput per GPU of the PCIe host-memory baseline. Peak throughput is compared at each configuration’s own peak batch size.

The C2C configuration raises bandwidth but keeps capacity at 1 TB, so the full-cache storage limit remains. PCIe-CXL adds capacity while holding bandwidth at 64 GB/s. It permits larger batches, but fetching their selected KV entries still limits throughput.

A fast memory tier still needs an efficient data path

To keep a high-bandwidth link busy, the system needs enough data transfers in flight while earlier requests complete. The amount of data needed is the bandwidth-delay product: bandwidth multiplied by latency.

Nikita’s example pairs terabytes-per-second bandwidth with microsecond-scale latency and accesses of about 800 bytes. Keeping enough of these small transfers in flight takes GPU resources to issue requests, track completion, stage results and coordinate dependencies.

Sparse attention also has to wait for the indexer’s selections before fetching missing entries. The implementation must handle data hazards and hide latency where dependencies allow.

If that fetch path is inefficient, much of the external tier’s bandwidth goes unused.

Figure 6. Data movement becomes a system-design problem at high bandwidth. The slide uses 1 to 2 μs median latency, 10+ μs tail latency and roughly 800-byte accesses to illustrate the challenge. Adapted from presentation slide 13.
Figure 6
Slide showing GPU streaming multiprocessors, HBM and external X-Mem, with data-movement challenges including bandwidth-delay product, fine-grained accesses and latency hiding.

Data movement becomes a system-design problem at high bandwidth. The slide uses 1 to 2 μs median latency, 10+ μs tail latency and roughly 800-byte accesses to illustrate the challenge. Adapted from presentation slide 13.

The software needed for memory tiering

Nikita outlined three parts of the software stack:

  • Data-movement primitives.

    GPU-initiated DMA, scatter-gather operations and swap-in kernels move selected data. The slide includes DMA, TMA and cp.async as building blocks.

  • Workload-aware kernels.

    Nikita described work on PoolSparse, combining HiSparse with a high-bandwidth external memory tier. He also discussed prefix-cache prefetchers; the appendix outlines GPU-initiated DMA and TMA-based fused, out-of-order attention.

  • Memory management.

    An HBM-tier memory-management library would unite memory across the scale-up domain, including C2C-connected CPU memory, peer HBM and X-Mem. It would incorporate the movement primitives and kernels.

Figure 7. The proposed software stack brings together data-movement primitives, workload-aware kernels and a memory-management library. Adapted from presentation slide 14.
Figure 7
Slide grouping efficient HBM-tiering software into data-movement primitives, kernels and an HBM-tier memory-management library.

The proposed software stack brings together data-movement primitives, workload-aware kernels and a memory-management library. Adapted from presentation slide 14.

Kernel design has to account for both the model’s access pattern and the memory fabric.

Where model and memory design meet

The top-k sparse-attention approach in the talk gives memory-tiering designs a specific workload to support: a large retained KV history with a smaller set of entries used at each step. As HBM capacity remains difficult to scale, that difference creates room to design models and memory systems together.

There is still work to do on efficient HBM tiering. Our study shows how capacity and bandwidth affect the available batch size and peak throughput under the modeled assumptions. The data-movement primitives, kernels and memory-management library described in the talk address how to put that memory to use. Netpreme is exploring these components together as models’ access patterns evolve.

Read the study methodology Continue reading · Memory architectureThe Scale-Up Memory Tier

Essential storage

Always on

Remembers your cookie choice on this browser for 180 days. If browser storage is unavailable, your choice applies to this page session.

You can withdraw consent anytime here. Turning analytics off stops further collection and removes accessible Google Analytics cookies from this website. It does not delete data already collected. Read our privacy policy.