TCO Calculator.
Estimate inference efficiency with Netpreme X-Mem™ at the rack scale.
The set of analytical roofline models estimates efficiency gains of running inference with Netpreme X-Mem™ MPU for various use cases, models, and hardware configurations served on an NVL72-class rack.
The following models are currently supported:
- Sparse attention decode. LLM decode with DeepSeek-style sparse attention, where each token reads only the top-k selected KV entries. The hot KV stays in HBM; the rest lives in the X-Mem™ Warm Memory Tier, and misses are fetched over each GPU's memory tier link. Read about XM-Sparse
- Long-Context Prefix caching. Time to first token for running long-context prefill in agentic or multi-turn workloads. Read about accelerating HiCache
Sparse Attention Decode.
Example configuration.Any XPU with a similar profile (compute, HBM capacity and bandwidth) fits into the same category.
The number of GPUs the model is sharded across.
The share of top-k KV entries found in HBM; the rest are fetched from the X-Mem™ Warm Memory Tier.It depends on user input and on caching policy: in effect, on how well each request's working set of top-k tokens stays in HBM.
%
900 GB/s
16 TB
Decode throughput vs. interactivity
Modelled results, represented similar to InferenceX : tokens/s per GPU against tokens/s per user, one point per batch size.Each decode step is timed on a roofline — GPU compute and HBM — plus the top-k KV misses fetched from the X-Mem™ tier over each GPU's memory tier link. Capacity caps how many requests fit at once (the large dot); bandwidth sets what each miss costs.Ideal assumes external memory is never the bottleneck (INF bandwidth and capacity).Netpreme X-Mem™Ideal
Prefix Caching.
Example configuration.Any XPU with a similar profile (compute, HBM capacity and bandwidth) fits into the same category.
The number of GPUs the model is sharded across.
The application class sets the prefix hit rate: the share of each prompt reused from previous turns and prefetched from the X-Mem™ Warm Memory Tier. Only the rest is prefilled.General autonomous agents (85%): each turn adds fresh tool output and observations.Documents Q&A (95%): the same documents stay cached; only new questions are prefilled.Agentic coding (98%): each turn appends a small edit or tool result to the same session.
How many requests are prefilled together, across the GPUs. Their KV must all fit in HBM at once.
req
Time to first token vs. memory tier bandwidth
Time to first token for a batch of prefills whose prompt prefix is cached in the X-Mem™ tier, across the memory tier bandwidth of each GPU's link to it (log scale).Prefetch: the cached prefix's KV loaded into HBM over the link. Prefill compute: the uncached suffix, on a roofline of GPU compute and HBM; it does not depend on the link. The two run one after the other, so TTFT is their sum.Prefill prefetchPrefill compute