Large language model (LLM) inference demands ever more memory capacity for larger models and batch sizes. High-Bandwidth Flash (HBF) replaces the DRAM dies of HBM with NAND flash and provides up to 16x the capacity at comparable read bandwidth. However, HBF can incur a read latency of nearly 20μs, 20x that of HBM, so on-demand HBF accesses slow inference by up to 12.8x. Conventional cache-line prefetchers require 20x more in-flight tracking state to cover this latency. Tensor-streaming designs, such as H3, follow virtual-address order as opposed to kernel-demand order, and therefore need hundreds of megabytes of buffering. We propose SlipStream, a software-hardware co-designed prefetcher that turns prefetching into a demand-paced producer-consumer pipeline. In software, SlipStream profiles the repeatable weight-page demand order of data-independent LLM kernels and inserts a staging warp that replays this order with non-blocking STAGE instructions. In hardware, the staging warp acts as the producer and the work warps as consumers. Our key insight is to retire each STAGE tracking entry only when demand first accesses the page, as opposed to when the page returns from HBF. Demand thus paces staging and bounds how far staging leads consumption. SlipStream tracks 4 KB pages instead of cache lines and allocates buffer slots only when pages return, which decouples the buffer size from outstanding traffic. Across seven dense models spanning 16-800 GB, SlipStream with a 40 MB staging buffer achieves 92.4% of the performance of an ideal HBF with HBM-like latency. H3 achieves only 43.1%, even with a 320 MB buffer. The buffer sizing of SlipStream is independent of HBF latency and workload. SlipStream with 16 MB buffer, 20x smaller than that of H3, performs within 4% of SlipStream-40MB. SlipStream also supports Mixture-of-Experts models and reaches 91.2% of ideal performance (40 MB buffer).
Figures & tables
Fig. 1: (a) Increasing load-to-use latency from 1 μ s with HBM to 20 μ s with HBF increases the required outstanding data by 20 × . (b) Prefetching tensor pages by virtual address does not reproduce the order in which each kernel requests them, resulting in inaccurate and untimely prefetching. (c) Goal: minimize slowdown, tracking SRAM, and buffer SRAM. (d) SlipStream uses software guidance and hardware flow control to bound tracking and buffer SRAM.
Property
HBM
HBF
Capacity per stack
16GB
256GB
Aggregate read bandwidth
2 TB/s
2 TB/s
SM load-to-use latency
∼ 1 μ s
∼ 20 μ s
Read granularity
32B line
4 KB page
Outstanding data †
2MB
40MB
Power consumption
∼ 40W
∼ 160W
TABLE I: Memory device properties.
Fig. 2: (a) Execution time normalized to ideal-HBF of LLM on-demand fetch from HBF for different HBF access latencies. 20 μ s access incurs 10.8x average slowdown, and up to 12.8x (seven LLMs, Section III ). (b) Profiled accesses of two LLM kernels. LLM kernel accesses do not follow tensor’s VA-order (x=y line). (c) Performance of H3 for Llama-8B and Llama-70B, when prefetcher streams in VA-order (high slowdown due to order stalls) and in demand-order (provided by SlipStream in Section IV , capacity stalls dominate). (d) Seven-model performance vs buffer SRAM for H3 . Our goal is ideal-HBF parity within a small buffer.
QKV-Proj
O-Proj
MLP-Gate
MLP-Up
MLP-Down
Total
Model
Time
Wt. pages
CTAs
Time
Wt. pages
CTAs
Time
Wt. pages
CTAs
Time
Wt. pages
CTAs
Time
Wt. pages
CTAs
Time
Wt. pages
CTAs
Llama-8B [ 1 ]
50.3
12K
256
34.7
8K
64
91.5
29K
56
89.8
29K
56
86.7
29K
96
353.0
107K
528
Mistral-24B [ 30 ]
60.2
15K
256
41.0
10K
80
219.5
82K
256
218.9
82K
256
234.2
82K
100
773.8
271K
948
Qwen2.5-32B [ 31 ]
73.1
18K
200
37.2
13K
80
160.5
69K
108
163.2
69K
108
193.0
69K
100
627.0
238K
596
DeepSeek-67B [ 32 ]
121.1
41K
272
117.3
33K
64
242.4
88K
86
245.6
88K
86
238.1
88K
96
964.5
338K
604
Llama-70B [ 1 ]
121.0
41K
272
117.3
33K
64
420.5
115K
224
424.0
115K
224
308.2
115K
96
1391.0
418K
880
TABLE II: Workload characterization (per-layer, batch size 64, FP16). CTA counts include split-K slices (grid z ). Time is in μ s.
Fig. 3: H 3 -like system organization [ 13 ] : HBF stores read-only weights, HBM stores mutable data, and a staging buffer hides HBF latency from SMs.
Parameter
Value
SM count
108
GPU core / memory clock
1.410 / 1.512 GHz
GPU, HBM Cacheline size
128B, 32B
L1 / shared memory per SM
192 KB
L2 capacity
40 MB
Memory channels
40
TABLE III: Configuration of the baseline HBM-HBF system.
Fig. 4: Overview of SlipStream profiler. The stream profiler collects access streams and stores them in a reusable stream cache database. We integrate a staging warp per CTA of the workload to use the access streams at runtime.
Fig. 5: SlipStream bootstrapping. The binder gets kernel parameters from CUDA graph capture. It then uses the Stream Cache and serving system’s weight registry to build the bind map which staging warps use at runtime.
Fig. 6: Real-GPU decode overhead of replaying SlipStream’s access streams.
Fig. 7: Overview of SlipStream’s architecture. Staging warps issue STAGE instructions, the per-SM SSHR tracks each staged page until its first demand access, staging buffer serves demands at near-HBM bandwidth/latency.
Fig. 8: Performance of on-demand fetch, H3 , and SlipStream (40 MB and 16 MB buffers), normalized to an ideal-HBF system with HBM-like latency (left), and geometric-mean performance against buffer SRAM (right). SlipStream performs within 8% (12%) of ideal-HBF at just 40MB (16MB) of buffer SRAM.
Fig. 9: Stall attribution at runtime. Bars per model, left to right: H3 -40MB (VA order), H3 -40MB with the oracle demand order, SlipStream 40 MB, and SlipStream 16 MB. SlipStream eliminates order and capacity stalls.
Fig. 10: Sustained HBF read bandwidth (norm. to 2TB/s) for different designs. SlipStream retains most bandwidth utilization in all evaluated models.
Fig. 11: Per-page lifetime in the 16 MB buffer (log scale): the time SlipStream needs each page versus the time it holds it. The gap is post-consumption retention; measured to be 8–23 μ s (so sufficient margin against jitters).
Fig. 12: Contribution breakdown normalized to ideal-HBF’s performance. Each optimization is necessary (especially for 16MB buffer) and additive.
Fig. 13: Sensitivity to HBF request-to-return latency, normalized to ideal-HBF. SlipStream retains low average overheads at various HBF latencies.
Structure
Storage
Area
Access
Leak.
Dyn.
(mm 2 )
(ns)
(W)
(W)
GPU die (per SM)
SSHR CAM
1.25 KB
0.0036
0.10
0.001
0.001
SSHRMax register
4 B
—
—
—
—
Total (108 SMs)
135 KB
0.39
—
0.153
0.09
HBM base die (per stack)
TABLE IV: Physical cost of SlipStream’s added structures. Top-cell value is for SlipStream-40MB, bottom is 16MB variant.
Fig. 14: Performance of MoE kernels [ 53 ] , normalized to ideal-HBF. SlipStream-40MB reaches 91.2% of ideal-HBF (87.3% at 16MB).
Fig. 15: Serving throughput, energy per token, and EDP, normalized to HBM-only system. HBF with SlipStream increases lowers EDP by 1.27–4.35x.
Fig. 16: Prior prefetchers adapted to HBF, normalized to ideal-HBF. All designs that fit within the GPU-die area of SlipStream but incur 1.5-5.2x slowdown, while Warped PE with 90-page lead requires 8% extra die area.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
University of California, Berkeley Berkeley, California, USA · FuriosaAI Seoul, South Korea · University of California, Berkeley; ICSI; LBNL Berkeley, California, USA