GPU memory bandwidth and capacity limit throughput in large language model (LLM) inference. The GPU memory system consists of a primary tier of high-bandwidth memory (HBM) and a secondary tier of host memory connected via CPU-to-GPU interconnect. Current serving systems treat the tiers hierarchically: they serve exclusively from HBM when data fits, and otherwise prefetch data from host memory to HBM before use. In both cases, the host memory bandwidth is never well utilized. Prefetching expands capacity by utilizing host memory, but consumes HBM bandwidth for writes, reducing the bandwidth available for demand loads. We observe that fully utilizing both host and HBM bandwidth requires each wave of GPU threadblocks to access both tiers concurrently and in proportion to their bandwidth ratio. Existing bandwidth-proportional placement strategies fail to provide concurrency because they are not aware of GPU waves, and the large 2MB GPU page size. This paper presents BOOST, the first runtime system that provides concurrent and proportional access to both GPU memory tiers, extracting the combined bandwidth of host memory and HBM for LLM inference without kernel changes. The key insight in BOOST is to use kernel access patterns to make page allocation and runtime data management wave-aware. For static model weights, BOOST applies modulo-based page placement that eliminates access-ratio variance; for dynamically provisioned attention key-value (KV) pairs, it makes the free KV page pool wave-aware. We integrate BOOST into vLLM and evaluate on a Grace Hopper system. At iso-batch size, BOOST improves Time-per-Output-Token (TPOT) by 4.3% over HBM-only serving, whereas prefetching degrades TPOT by 6%. In high-throughput serving, BOOST improves throughput by 31% on average, outperforming prefetching by 15%.
Figures & tables
Figure 1. (a) LLM serving systems use HBM exclusively, leaving host CPU memory idle. (b) To improve serving capacity, current systems migrate (prefetch) data, degrading HBM demand bandwidth. (c) Our goal is to perform concurrent and proportional (CAP) accesses at each wave to fully utilize system bandwidth. (d) BOOST facilitates CAP accesses by making page placement and runtime data management aware of CTA-level kernel data access patterns.
Figure 2. The GPU memory system connects HBM and host CPU memory (attached via C2G links) to the GPU in parallel.
Figure 3. Left: Demand bandwidth to the SM in HBM-only baseline and prefetching for Llama-3.3 70B on GH200, both at batch size of 10 (details in section 5 ). Middle: Breakdown of system bandwidth utilization. Right: Lifecycle of offloaded data. The prefetch engine migrates data from the C2G tier to HBM before use, reducing demand bandwidth available to the SMs.
Figure 4. Top: Bandwidth usage of random page placement (BAPP-R) over HBM-only accesses. BAPP-R incurs a 3.5% lower usage while the ideal bandwidth improvement is 10.4%. Bottom: Bandwidth utilization of HBM and C2G links over time for BAPP-R. The temporal variance in bandwidth utilization degrades system bandwidth utilization.
Migration
Placement
Goal
Property
Prefetching ( vLLM Team and NVIDIA, 2026 )
cudaMalloc Managed
BAPP-R † ( Agarwal et al., 2015b )
Weighted interleave † ( Kerrisk, )
Use host capacity
✓
✓
✓
✓
✓
Migration-free
✗
✗
✓
✓
✓
Proportional accesses
✗
✗
✓
✓
✓
Per-wave concurrency
✗
✗
✗
✗
✓
Works for dyn. data
✗
✗
✗
✗
✓
Table 1. Summary of comparisons of current and proposed schemes. All schemes add host capacity but fail to provide per-wave concurrency and do not maintain the α ratio for dynamic KV data. † No GPU/LLM implementation exists ( section 5 ).
Figure 5. Key questions in enabling concurrent and proportional memory access to fully utilize system bandwidth.
Figure 6. (a) Variable access ratio with BAPP-R. (b) Deterministic access ratio with CAP accesses. (c) Add-kernel speedup at varying access ratios.
Figure 7. Performance of random and modulo-based page placement for Llama3.3-70B fused gate+up projection matmul kernel (GH200 GPU, details in section 5.1 ) at different batch sizes. Modulo-based page placement consistently outperforms random page placement.
Figure 8. Profiling of Qwen3-Next 80B’s expert compute kernel (GH200 GPU) for ShareGPT and LongWriter datasets. Active threads are 264K, CTA size variation is low, but CTAs per wave vary widely depending on problem size.
Figure 9. Speedup of Llama-70B up-projection on GH200 compared to HBM-only with different access partitioning.
Figure 10. Speedup of FlashAttention3 ( Shah et al., 2024 ) kernel for Llama3.1-8B on GH200 for different mappings. Managing page-to-KV head chunk mapping maximizes TPOT speedup.
Figure 11. Top: MPP ensures CAP with static data (model weights). Bottom: Wave-aware LLM system for CAP with dynamic KV data. LPP is linear (BW-proportional) placement.
Figure 12. Design overview of BOOST.
Figure 13. Left: TPOT speedup for Llama3.3 dense LLM (batch size of 10) and Qwen3-Next MoE (batch size of 88) for low latency serving, normalized to HBM-only baseline. BOOST improves TPOT by 4.3% on average (geometric mean) while prefetching degrades TPOT by 6% for iso-batch serving. Right: Normalized throughput speedup in high throughput serving, where batch size scales to fill the available memory capacity. BOOST provides 31% average throughput speedup, 15% higher than prefetching.
Figure 14. Bandwidth utilization (left) and its breakdown (right) of HBM and C2G for Llama-3.3 (512/4K context). BOOST improves bandwidth utilization (HBM+C2G) by 3% over HBM-only baseline and by 12% over prefetching.
Model/ Config
HBM-only
BOOST (MLP-only)
BOOST
Llama-3.3 70B
1026
1263 (1.23x)
1352 (1.32x)
Table 2. Performance of BOOST for Llama-3.3 70B (512/4K in/out context) with PAM (BOOST-MLP-only) and with both PAM+DSR (BOOST). Our optimizations are additive.
Configuration
Capacity (BS)
Throughput (TPS)
TPOT (ms)
HBM-only (85% HBM mem-util)
81.6GB (BS=6)
222
27.1
HBM-capacity (95% HBM)
91.2GB (BS=10)
344 (1.55x)
29.0
BOOST (85% HBM + CPU-mem)
91.2GB (BS=10)
359 (1.62x)
27.9
Table 3. BOOST speedup decomposition for Llama-3.3 70B (1K/8K). The capacity-matched baseline holds the same capacity as BOOST in HBM, isolating bandwidth-induced gains.
Figure 15. TPOT speedup of BOOST and BAPP-R for Qwen3-Next and Llama-3.3. BAPP-R provides 0.4% and 2.7% TPOT degradation for MoE and dense LLM, respectively.
Llama Norm. perf
Weighted interleave
Proportional interleave
cudaMallocManaged
BOOST
up-proj kernel
1.06
0.60
1.00
1.07
down-proj kernel
0.99
1.00
1.00
1.03
Iso-batch TPOT
0.99
0.75
1.00
1.04
Max throughput
1.30
1.08
0.19
1.35
Table 4. Comparison of capacity expansion techniques using Llama3.3-70B (512/4K). Existing techniques fail to deliver consistent speedup (per-row maximum in bold). All four techniques (columns) use identical batch sizes.
Figure 16. P99 ITL versus request rate for BOOST and BAPP-R on the decode instance (Llama-3.3 70B, LongWriter). BOOST reduces P99 ITL by 7.8% (geometric mean) over BAPP-R across the serviceable load range.
Figure 17. Normalized throughput speedup of Llama3.3 model with longer context (output fixed at 8K). BOOST scales to long context workloads.
Figure 18. Speedup of expert kernel from a MoE model (Qwen3-Next) and matmul kernel from a dense model (Llama3.3) across different batch sizes.
Figure 19. DSR keeps the host ratio of KV cache near ideal despite low number of attention heads (H=4).
Figure 20. Speedup of expert kernel from a MoE model (Qwen3-Next) and matmul kernel from a dense model (Llama3.3) across different hardware platforms. Input activations use batch size of 256 (MoE) and 64 (dense). H100-emulation provides better speedup due to higher α .
Figure 21. TPOT speedup normalized to HBM-only serving for larger models with tensor parallelism (TP=2) on H100-emulation. BOOST improves TPOT by 6.5% on average.
Figure 22. Projected normalized throughput speedup of BOOST and prefetching for GB200 running DeepSeek R1 FP4 at 128K context/user at varying α (high-TPS setting). BOOST outperforms prefetching by up to 60%.
University of California, Berkeley Berkeley, California, USA · FuriosaAI Seoul, South Korea · University of California, Berkeley; ICSI; LBNL Berkeley, California, USA