Long-context inference puts pressure on GPU HBM memory. As context grows, KV caches take up more space, competing with model weights and other data for HBM capacity. Netpreme’s scale-up memory tier provides a pooled DRAM layer for this data. The G1–G4 labels follow the terminology used in NVIDIA’s CMX memory hierarchy. Figure 1 shows Netpreme’s adaptation of this hierarchy:
- G1: HBM on the GPU package, where active data lives
- G2: CPU-dedicated system DRAM, reached through a PCIe or C2C interface
- G1.5 and G2.EXP: the scale-up memory tier on the scale-up fabric, with load/store access on compatible configurations
- G3: Flash / NVMe-oF, reached through block I/O over the scale-out network
- G4: Elastic Storage across racks
GPUs reach memory and storage through different access paths. They access HBM and compatible system memory through memory operations; the G3 tier shown here uses block I/O over the scale-out network. From G3 down, every access pays the block I/O tax — microseconds of protocol and queuing latency per transfer. The G2 scale-up memory tier operates with native memory semantics, keeping access latencies orders of magnitude lower.
The scale-up memory tier sits between HBM and storage. It pools DRAM across the rack on the scale-up fabric, with load/store access on compatible accelerator and fabric configurations. Netpreme extends NVIDIA’s G1–G4 terminology with G1.5 and G2.EXP to describe how this pooled tier extends G1 and G2.
Netpreme Memory Processing Units (MPUs) connect the pool to the scale-up fabric. The memory tray’s aggregate bandwidth scales with the fabric ports it serves, while memory capacity grows independently of CPU or GPU count.
Workloads
The tier addresses workloads that outgrow their available HBM and need to fetch data during execution. KV cache reuse and sparse attention place different demands on it. X-Mem also targets embedding-table caching, while Engram and RL rollouts remain applications to evaluate.
Published evaluations
Product target
Active KV cache for sparse attention
The full KV cache can live in the scale-up memory tier, while the GPU keeps a hot buffer and fetches only missing selected entries during decode. See our XM-Sparse evaluation for the emulated setup and results.
Embedding tables for recommendation models
X-Mem targets embedding-table and vector-search caching. A scale-up memory tier can hold large tables for the small, irregular lookups used in recommendation systems.
Engram tables
DeepSeek’s Engram adds lookup-based conditional memory to a model. These large lookup tables are another use case to evaluate for the scale-up memory tier.
RL rollouts for post-training
Generated trajectories could stay in a pooled scale-up memory tier while they wait for scoring and model updates.
As context and model state grow, more of the working set will live in the scale-up memory tier.