Large language model (LLM) inference demands ever more memory capacity for larger models and batch sizes. High-Bandwidth Flash (HBF) replaces the DRAM dies of HBM with NAND flash and provides up to 16x the capacity at comparable read bandwidth. However, HBF can incur a read latency of nearly 20μs, 20x that of HBM, so on-demand HBF accesses slow inference by up to 12.8x. Conventional cache-line prefetchers require 20x more in-flight tracking state to cover this latency. Tensor-streaming designs, such as H3, follow virtual-address order as opposed to kernel-demand order, and therefore need hundreds of megabytes of buffering. We propose SlipStream, a software-hardware co-designed prefetcher that turns prefetching into a demand-paced producer-consumer pipeline. In software, SlipStream profiles the repeatable weight-page demand order of data-independent LLM kernels and inserts a staging warp that replays this order with non-blocking STAGE instructions. In hardware, the staging warp acts as the producer and the work warps as consumers. Our key insight is to retire each STAGE tracking entry only when demand first accesses the page, as opposed to when the page returns from HBF. Demand thus paces staging and bounds how far staging leads consumption. SlipStream tracks 4 KB pages instead of cache lines and allocates buffer slots only when pages return, which decouples the buffer size from outstanding traffic. Across seven dense models spanning 16-800 GB, SlipStream with a 40 MB staging buffer achieves 92.4% of the performance of an ideal HBF with HBM-like latency. H3 achieves only 43.1%, even with a 320 MB buffer. The buffer sizing of SlipStream is independent of HBF latency and workload. SlipStream with 16 MB buffer, 20x smaller than that of H3, performs within 4% of SlipStream-40MB. SlipStream also supports Mixture-of-Experts models and reaches 91.2% of ideal performance (40 MB buffer).
Figures & tables
Fig. 1: (a) Increasing load-to-use latency from 1 μ s with HBM to 20 μ s with HBF increases the required outstanding data by 20 × . (b) Prefetching tensor pages by virtual address does not reproduce the order in which each kernel requests them, resulting in inaccurate and untimely prefetching. (c) Goal: minimize slowdown, tracking SRAM, and buffer SRAM. (d) SlipStream uses software guidance and hardware flow control to bound tracking and buffer SRAM.
Property
HBM
HBF
Capacity per stack
16GB
256GB
Aggregate read bandwidth
2 TB/s
2 TB/s
SM load-to-use latency
∼ 1 μ s
∼ 20 μ s
Read granularity
32B line
4 KB page
Outstanding data †
2MB
40MB
Power consumption
∼ 40W
∼ 160W
TABLE I: Memory device properties.
Fig. 2: (a) Execution time normalized to ideal-HBF of LLM on-demand fetch from HBF for different HBF access latencies. 20 μ s access incurs 10.8x average slowdown, and up to 12.8x (seven LLMs, Section III ). (b) Profiled accesses of two LLM kernels. LLM kernel accesses do not follow tensor’s VA-order (x=y line). (c) Performance of H3 for Llama-8B and Llama-70B, when prefetcher streams in VA-order (high slowdown due to order stalls) and in demand-order (provided by SlipStream in Section IV , capacity stalls dominate). (d) Seven-model performance vs buffer SRAM for H3 . Our goal is ideal-HBF parity within a small buffer.
QKV-Proj
O-Proj
MLP-Gate
MLP-Up
MLP-Down
Total
Model
Time
Wt. pages
CTAs
Time
Wt. pages
CTAs
Time
Wt. pages
CTAs
Time
Wt. pages
CTAs
Time
Wt. pages
CTAs
Time
Wt. pages
CTAs
Llama-8B [ 1 ]
50.3
12K
256
34.7
8K
64
91.5
29K
56
89.8
29K
56
86.7
29K
96
353.0
107K
528
Mistral-24B [ 30 ]
60.2
15K
256
41.0
10K
80
219.5
82K
256
218.9
82K
256
234.2
82K
100
773.8
271K
948
Qwen2.5-32B [ 31 ]
73.1
18K
200
37.2
13K
80
160.5
69K
108
163.2
69K
108
193.0
69K
100
627.0
238K
596
DeepSeek-67B [ 32 ]
121.1
41K
272
117.3
33K
64
242.4
88K
86
245.6
88K
86
238.1
88K
96
964.5
338K
604
Llama-70B [ 1 ]
121.0
41K
272
117.3
33K
64
420.5
115K
224
424.0
115K
224
308.2
115K
96
1391.0
418K
880
TABLE II: Workload characterization (per-layer, batch size 64, FP16). CTA counts include split-K slices (grid z ). Time is in μ s.
Fig. 3: H 3 -like system organization [ 13 ] : HBF stores read-only weights, HBM stores mutable data, and a staging buffer hides HBF latency from SMs.
Parameter
Value
SM count
108
GPU core / memory clock
1.410 / 1.512 GHz
GPU, HBM Cacheline size
128B, 32B
L1 / shared memory per SM
192 KB
L2 capacity
40 MB
Memory channels
40
TABLE III: Configuration of the baseline HBM-HBF system.
Fig. 4: Overview of SlipStream profiler. The stream profiler collects access streams and stores them in a reusable stream cache database. We integrate a staging warp per CTA of the workload to use the access streams at runtime.
Fig. 5: SlipStream bootstrapping. The binder gets kernel parameters from CUDA graph capture. It then uses the Stream Cache and serving system’s weight registry to build the bind map which staging warps use at runtime.
Fig. 6: Real-GPU decode overhead of replaying SlipStream’s access streams.
Fig. 7: Overview of SlipStream’s architecture. Staging warps issue STAGE instructions, the per-SM SSHR tracks each staged page until its first demand access, staging buffer serves demands at near-HBM bandwidth/latency.
Fig. 8: Performance of on-demand fetch, H3 , and SlipStream (40 MB and 16 MB buffers), normalized to an ideal-HBF system with HBM-like latency (left), and geometric-mean performance against buffer SRAM (right). SlipStream performs within 8% (12%) of ideal-HBF at just 40MB (16MB) of buffer SRAM.
Fig. 9: Stall attribution at runtime. Bars per model, left to right: H3 -40MB (VA order), H3 -40MB with the oracle demand order, SlipStream 40 MB, and SlipStream 16 MB. SlipStream eliminates order and capacity stalls.
Fig. 10: Sustained HBF read bandwidth (norm. to 2TB/s) for different designs. SlipStream retains most bandwidth utilization in all evaluated models.
Fig. 11: Per-page lifetime in the 16 MB buffer (log scale): the time SlipStream needs each page versus the time it holds it. The gap is post-consumption retention; measured to be 8–23 μ s (so sufficient margin against jitters).
Fig. 12: Contribution breakdown normalized to ideal-HBF’s performance. Each optimization is necessary (especially for 16MB buffer) and additive.
Fig. 13: Sensitivity to HBF request-to-return latency, normalized to ideal-HBF. SlipStream retains low average overheads at various HBF latencies.
Structure
Storage
Area
Access
Leak.
Dyn.
(mm 2 )
(ns)
(W)
(W)
GPU die (per SM)
SSHR CAM
1.25 KB
0.0036
0.10
0.001
0.001
SSHRMax register
4 B
—
—
—
—
Total (108 SMs)
135 KB
0.39
—
0.153
0.09
HBM base die (per stack)
TABLE IV: Physical cost of SlipStream’s added structures. Top-cell value is for SlipStream-40MB, bottom is 16MB variant.
Fig. 14: Performance of MoE kernels [ 53 ] , normalized to ideal-HBF. SlipStream-40MB reaches 91.2% of ideal-HBF (87.3% at 16MB).
Fig. 15: Serving throughput, energy per token, and EDP, normalized to HBM-only system. HBF with SlipStream increases lowers EDP by 1.27–4.35x.
Fig. 16: Prior prefetchers adapted to HBF, normalized to ideal-HBF. All designs that fit within the GPU-die area of SlipStream but incur 1.5-5.2x slowdown, while Warped PE with 90-page lead requires 8% extra die area.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
LLM inference is increasingly constrained by accelerator memory capacity rather than compute throughput. This constraint is especially acute in single-accelerator and small-node inference systems, where limited on-package memory capacity restricts the size of deployable models. HBF is an emerging 3D-stacked NAND flash technology that provides multi-terabyte near-accelerator capacity, making it a promising capacity tier for storing LLM weights. However, existing HBF-based proposals face three adoption challenges: they (1) rely on coarse-grained static prefetching for LLM weights aiming to hide the microsecond-level read latency of the NAND flash device while maximizing HBF's read throughput, (2) expose NAND flash management tasks (e.g., refresh operations) to the accelerator-visible critical inference path, and (3) miss optimization opportunities to specialize and optimize the flash-management mechanisms to the workload behavior. Our goal is to design an efficient HBF substrate that integrates HBF as a memory-capacity tier alongside HBM while addressing these three challenges. To this end, we propose FLINT, a workload-driven HBF substrate for capacity-scalable LLM inference. FLINT introduces three mechanisms: (1) a hardware burst-buffer controller that dynamically coalesces and pipelines HBF reads aiming to utilize existing NAND flash buffers while sustaining high HBF bandwidth, (2) a phantom-plane refresh mechanism, which removes refresh from the critical inference path by moving refresh-related NAND flash operations outside the read foreground back via low-cost resource duplication, and (3) a read-only FTL, which replaces SSD-class support for arbitrary writes with a compact table that translates logical weight bursts to physical HBF locations.
Geraldo F. Oliveira, Arash Tavakkol, Xiangyu Zhu +10
Huawei Technologies Switzerland AG · Huawei Technologies Co., Ltd. · ETH Zürich +1
Large language model (LLM) serving requires substantial memory to store model weights and KV caches. As models grow larger and contexts become longer, memory capacity and bandwidth increasingly become bottlenecks for serving performance. Agentic workloads compound this pressure through repeated interactions over growing contexts, making it increasingly important to retain KV state for reuse. High-bandwidth flash (HBF) offers a way to expand accelerator memory capacity for large language model (LLM) serving, but its access costs and limited write endurance complicate its use. We evaluate HBF for high-throughput agentic serving across system design and scheduling choices to understand when additional capacity improves serving performance and energy efficiency. We introduce an HBM-HBF-host hierarchical storage system and buffered cache-aware scheduling, and use trace-driven simulations to analyze their effects on performance, energy consumption, and HBF write lifetime. Across the evaluated workloads, the fastest HBF-augmented systems reduce completion time by 36.1-87.7% relative to HBM-only systems. Modeled energy savings reach 59.1%, with benefits depending on the workload and weight placement. Buffered cache-aware scheduling extends estimated HBF write lifetime from 1.21 to 14.82 years in the evaluated configuration. These results demonstrate the importance of coordinating data placement and scheduling to improve serving efficiency while sustaining a practical HBF write lifetime.
Zack Yu, Chloe Wong, Coleman Hooper +7
University of California, Berkeley Berkeley, California, USA · FuriosaAI Seoul, South Korea · University of California, Berkeley; ICSI; LBNL Berkeley, California, USA
GPU memory bandwidth and capacity limit throughput in large language model (LLM) inference. The GPU memory system consists of a primary tier of high-bandwidth memory (HBM) and a secondary tier of host memory connected via CPU-to-GPU interconnect. Current serving systems treat the tiers hierarchically: they serve exclusively from HBM when data fits, and otherwise prefetch data from host memory to HBM before use. In both cases, the host memory bandwidth is never well utilized. Prefetching expands capacity by utilizing host memory, but consumes HBM bandwidth for writes, reducing the bandwidth available for demand loads. We observe that fully utilizing both host and HBM bandwidth requires each wave of GPU threadblocks to access both tiers concurrently and in proportion to their bandwidth ratio. Existing bandwidth-proportional placement strategies fail to provide concurrency because they are not aware of GPU waves, and the large 2MB GPU page size. This paper presents BOOST, the first runtime system that provides concurrent and proportional access to both GPU memory tiers, extracting the combined bandwidth of host memory and HBM for LLM inference without kernel changes. The key insight in BOOST is to use kernel access patterns to make page allocation and runtime data management wave-aware. For static model weights, BOOST applies modulo-based page placement that eliminates access-ratio variance; for dynamically provisioned attention key-value (KV) pairs, it makes the free KV page pool wave-aware. We integrate BOOST into vLLM and evaluate on a Grace Hopper system. At iso-batch size, BOOST improves Time-per-Output-Token (TPOT) by 4.3% over HBM-only serving, whereas prefetching degrades TPOT by 6%. In high-throughput serving, BOOST improves throughput by 31% on average, outperforming prefetching by 15%.
Anish Saxena, Jae Hyung Ju, Hritvik Taneja +4
Georgia Tech Atlanta, USA · Nvidia Research Westford, USA · Nvidia Research & Stanford Santa Clara, USA