GPU memory bandwidth and capacity limit throughput in large language model (LLM) inference. The GPU memory system consists of a primary tier of high-bandwidth memory (HBM) and a secondary tier of host memory connected via CPU-to-GPU interconnect. Current serving systems treat the tiers hierarchically: they serve exclusively from HBM when data fits, and otherwise prefetch data from host memory to HBM before use. In both cases, the host memory bandwidth is never well utilized. Prefetching expands capacity by utilizing host memory, but consumes HBM bandwidth for writes, reducing the bandwidth available for demand loads. We observe that fully utilizing both host and HBM bandwidth requires each wave of GPU threadblocks to access both tiers concurrently and in proportion to their bandwidth ratio. Existing bandwidth-proportional placement strategies fail to provide concurrency because they are not aware of GPU waves, and the large 2MB GPU page size. This paper presents BOOST, the first runtime system that provides concurrent and proportional access to both GPU memory tiers, extracting the combined bandwidth of host memory and HBM for LLM inference without kernel changes. The key insight in BOOST is to use kernel access patterns to make page allocation and runtime data management wave-aware. For static model weights, BOOST applies modulo-based page placement that eliminates access-ratio variance; for dynamically provisioned attention key-value (KV) pairs, it makes the free KV page pool wave-aware. We integrate BOOST into vLLM and evaluate on a Grace Hopper system. At iso-batch size, BOOST improves Time-per-Output-Token (TPOT) by 4.3% over HBM-only serving, whereas prefetching degrades TPOT by 6%. In high-throughput serving, BOOST improves throughput by 31% on average, outperforming prefetching by 15%.
Figures & tables
Figure 1. (a) LLM serving systems use HBM exclusively, leaving host CPU memory idle. (b) To improve serving capacity, current systems migrate (prefetch) data, degrading HBM demand bandwidth. (c) Our goal is to perform concurrent and proportional (CAP) accesses at each wave to fully utilize system bandwidth. (d) BOOST facilitates CAP accesses by making page placement and runtime data management aware of CTA-level kernel data access patterns.
Figure 2. The GPU memory system connects HBM and host CPU memory (attached via C2G links) to the GPU in parallel.
Figure 3. Left: Demand bandwidth to the SM in HBM-only baseline and prefetching for Llama-3.3 70B on GH200, both at batch size of 10 (details in section 5 ). Middle: Breakdown of system bandwidth utilization. Right: Lifecycle of offloaded data. The prefetch engine migrates data from the C2G tier to HBM before use, reducing demand bandwidth available to the SMs.
Figure 4. Top: Bandwidth usage of random page placement (BAPP-R) over HBM-only accesses. BAPP-R incurs a 3.5% lower usage while the ideal bandwidth improvement is 10.4%. Bottom: Bandwidth utilization of HBM and C2G links over time for BAPP-R. The temporal variance in bandwidth utilization degrades system bandwidth utilization.
Migration
Placement
Goal
Property
Prefetching ( vLLM Team and NVIDIA, 2026 )
cudaMalloc Managed
BAPP-R † ( Agarwal et al., 2015b )
Weighted interleave † ( Kerrisk, )
Use host capacity
✓
✓
✓
✓
✓
Migration-free
✗
✗
✓
✓
✓
Proportional accesses
✗
✗
✓
✓
✓
Per-wave concurrency
✗
✗
✗
✗
✓
Works for dyn. data
✗
✗
✗
✗
✓
Table 1. Summary of comparisons of current and proposed schemes. All schemes add host capacity but fail to provide per-wave concurrency and do not maintain the α ratio for dynamic KV data. † No GPU/LLM implementation exists ( section 5 ).
Figure 5. Key questions in enabling concurrent and proportional memory access to fully utilize system bandwidth.
Figure 6. (a) Variable access ratio with BAPP-R. (b) Deterministic access ratio with CAP accesses. (c) Add-kernel speedup at varying access ratios.
Figure 7. Performance of random and modulo-based page placement for Llama3.3-70B fused gate+up projection matmul kernel (GH200 GPU, details in section 5.1 ) at different batch sizes. Modulo-based page placement consistently outperforms random page placement.
Figure 8. Profiling of Qwen3-Next 80B’s expert compute kernel (GH200 GPU) for ShareGPT and LongWriter datasets. Active threads are 264K, CTA size variation is low, but CTAs per wave vary widely depending on problem size.
Figure 9. Speedup of Llama-70B up-projection on GH200 compared to HBM-only with different access partitioning.
Figure 10. Speedup of FlashAttention3 ( Shah et al., 2024 ) kernel for Llama3.1-8B on GH200 for different mappings. Managing page-to-KV head chunk mapping maximizes TPOT speedup.
Figure 11. Top: MPP ensures CAP with static data (model weights). Bottom: Wave-aware LLM system for CAP with dynamic KV data. LPP is linear (BW-proportional) placement.
Figure 12. Design overview of BOOST.
Figure 13. Left: TPOT speedup for Llama3.3 dense LLM (batch size of 10) and Qwen3-Next MoE (batch size of 88) for low latency serving, normalized to HBM-only baseline. BOOST improves TPOT by 4.3% on average (geometric mean) while prefetching degrades TPOT by 6% for iso-batch serving. Right: Normalized throughput speedup in high throughput serving, where batch size scales to fill the available memory capacity. BOOST provides 31% average throughput speedup, 15% higher than prefetching.
Figure 14. Bandwidth utilization (left) and its breakdown (right) of HBM and C2G for Llama-3.3 (512/4K context). BOOST improves bandwidth utilization (HBM+C2G) by 3% over HBM-only baseline and by 12% over prefetching.
Model/ Config
HBM-only
BOOST (MLP-only)
BOOST
Llama-3.3 70B
1026
1263 (1.23x)
1352 (1.32x)
Table 2. Performance of BOOST for Llama-3.3 70B (512/4K in/out context) with PAM (BOOST-MLP-only) and with both PAM+DSR (BOOST). Our optimizations are additive.
Configuration
Capacity (BS)
Throughput (TPS)
TPOT (ms)
HBM-only (85% HBM mem-util)
81.6GB (BS=6)
222
27.1
HBM-capacity (95% HBM)
91.2GB (BS=10)
344 (1.55x)
29.0
BOOST (85% HBM + CPU-mem)
91.2GB (BS=10)
359 (1.62x)
27.9
Table 3. BOOST speedup decomposition for Llama-3.3 70B (1K/8K). The capacity-matched baseline holds the same capacity as BOOST in HBM, isolating bandwidth-induced gains.
Figure 15. TPOT speedup of BOOST and BAPP-R for Qwen3-Next and Llama-3.3. BAPP-R provides 0.4% and 2.7% TPOT degradation for MoE and dense LLM, respectively.
Llama Norm. perf
Weighted interleave
Proportional interleave
cudaMallocManaged
BOOST
up-proj kernel
1.06
0.60
1.00
1.07
down-proj kernel
0.99
1.00
1.00
1.03
Iso-batch TPOT
0.99
0.75
1.00
1.04
Max throughput
1.30
1.08
0.19
1.35
Table 4. Comparison of capacity expansion techniques using Llama3.3-70B (512/4K). Existing techniques fail to deliver consistent speedup (per-row maximum in bold). All four techniques (columns) use identical batch sizes.
Figure 16. P99 ITL versus request rate for BOOST and BAPP-R on the decode instance (Llama-3.3 70B, LongWriter). BOOST reduces P99 ITL by 7.8% (geometric mean) over BAPP-R across the serviceable load range.
Figure 17. Normalized throughput speedup of Llama3.3 model with longer context (output fixed at 8K). BOOST scales to long context workloads.
Figure 18. Speedup of expert kernel from a MoE model (Qwen3-Next) and matmul kernel from a dense model (Llama3.3) across different batch sizes.
Figure 19. DSR keeps the host ratio of KV cache near ideal despite low number of attention heads (H=4).
Figure 20. Speedup of expert kernel from a MoE model (Qwen3-Next) and matmul kernel from a dense model (Llama3.3) across different hardware platforms. Input activations use batch size of 256 (MoE) and 64 (dense). H100-emulation provides better speedup due to higher α .
Figure 21. TPOT speedup normalized to HBM-only serving for larger models with tensor parallelism (TP=2) on H100-emulation. BOOST improves TPOT by 6.5% on average.
Figure 22. Projected normalized throughput speedup of BOOST and prefetching for GB200 running DeepSeek R1 FP4 at 128K context/user at varying α (high-TPS setting). BOOST outperforms prefetching by up to 60%.
Mixed batching (MB)--interleaving prefill and decode in a single batch--has become the standard scheduling strategy for large language model (LLM) inference due to its efficiency in maximizing compute and memory utilization. However, through controlled experiments, we find that prefill-decode interference inflates MB's per-step marginal cost above that of pure decode. On the high-bandwidth H200 (4.8 TB/s), this occurs only when decode tokens exceed 80% of the batch; however, on the bandwidth-constrained RTX PRO 6000 (1.792 TB/s), this threshold plummets to just 20%. Consequently, the optimal choice between MB and exclusive batching (EB) fundamentally depends on GPU memory bandwidth, model size, and workload composition. We derive a closed-form condition for this EB-MB performance crossover, along with asymptotically optimal phase-switching thresholds and memory-safe batch sizing for EB. Optimized EB achieves up to 41.9% higher throughput on bandwidth-constrained GPUs, while MB retains its advantage on high-bandwidth hardware with larger models. Our hybrid scheduler EB+ applies this condition online to dynamically switch between EB and MB without manual intervention. Under non-stationary traffic with distribution or concurrency shifts, EB+ attains the highest or near-highest throughput in every setting, outperforming MB by up to 36.4%.
Weifang Zhang, Yuzhou Nie, Bowen Pang +2
The Hong Kong Polytechnic University · UC Santa Barbara · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences +1
Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation (RAG), and compressed contextual memory, to support complex reasoning. We show that these optimizations can be unified into a four-step memory processing pipeline: Prepare Memory, Compute Relevancy, Retrieval, and Apply to Inference. Through systematic profiling, we identify a 22%-97% memory processing overhead in LLM inference and strong heterogeneity in its computational characteristics. Motivated by this insight, we argue that heterogeneous systems are well-suited to accelerate memory processing and thus end-to-end inference. We demonstrate this approach on a GPU-FPGA system by offloading sparse, irregular, and memory-bounded operations to FPGAs while retaining compute-intensive operations on GPUs. Evaluated on an AMD MI210 GPU and an Alveo U55C FPGA, our system is up to 2.2× faster and achieves up to 4.7× less energy across multiple LLM inference optimizations than the GPU baseline (similar results hold on NVIDIA A100). These results establish heterogeneous systems as a practical direction for efficient LLM memory processing and inform future heterogeneous hardware design.
Zifan He, Rui Ma, Yizhou Sun +1
Department of Computer Science, University of California, Los Angeles · Microsoft Research Asia
Large language model (LLM) serving requires substantial memory to store model weights and KV caches. As models grow larger and contexts become longer, memory capacity and bandwidth increasingly become bottlenecks for serving performance. Agentic workloads compound this pressure through repeated interactions over growing contexts, making it increasingly important to retain KV state for reuse. High-bandwidth flash (HBF) offers a way to expand accelerator memory capacity for large language model (LLM) serving, but its access costs and limited write endurance complicate its use. We evaluate HBF for high-throughput agentic serving across system design and scheduling choices to understand when additional capacity improves serving performance and energy efficiency. We introduce an HBM-HBF-host hierarchical storage system and buffered cache-aware scheduling, and use trace-driven simulations to analyze their effects on performance, energy consumption, and HBF write lifetime. Across the evaluated workloads, the fastest HBF-augmented systems reduce completion time by 36.1-87.7% relative to HBM-only systems. Modeled energy savings reach 59.1%, with benefits depending on the workload and weight placement. Buffered cache-aware scheduling extends estimated HBF write lifetime from 1.21 to 14.82 years in the evaluated configuration. These results demonstrate the importance of coordinating data placement and scheduling to improve serving efficiency while sustaining a practical HBF write lifetime.
Zack Yu, Chloe Wong, Coleman Hooper +7
University of California, Berkeley Berkeley, California, USA · FuriosaAI Seoul, South Korea · University of California, Berkeley; ICSI; LBNL Berkeley, California, USA