Fine-tuning large language models (LLMs) often exceeds GPU memory limits, prompting systems to offload model states to CPU memory. However, existing offloaded training frameworks like ZeRO-Offload treat all parameters equally and update the full model on the CPU, causing severe GPU stalls, where fast, expensive GPUs sit idle, waiting for slow CPU updates and limited-bandwidth PCIe transfers. We present ZenFlow, a new offloaded training framework that decouples updates between GPU and CPU by prioritizing important gradients. ZenFlow performs in-place updates of important gradients on GPU while offloading and accumulating less important ones on CPU without blocking training iterations, fully overlapping CPU work with GPU computation. To make this importance-aware design scalable, ZenFlow leverages a novel spatio-temporal locality of important gradients, concentrated in specific positions that remain stable across iterations, to make important gradient identification efficient and lightweight while preserving fidelity. Evaluation shows that ZenFlow achieves up to 5x end-to-end speedup, 2x lower PCIe traffic, 85% less GPU stalls, and up to 72% lower cloud GPU cost, all while preserving accuracy.
Figures & tables
Figure 1. GPU vs. CPU peak performance evolution. Line chart comparing the growth of peak GPU and CPU floating-point performance from 2016 to 2025.
Figure 2. Comparison of GPU utilization over time. GPU utilization timelines for three consecutive training steps under ZeRO-Offload and ZenFlow. ZeRO-Offload repeatedly leaves the GPU idle during CPU-side updates, while ZenFlow keeps GPU utilization high.
Figure 3. Offloading strategy comparison. (a) ZeRO-Offload ( Ren et al., 2021b ) sequentially executes FP and BP on GPU, then offloads gradients and performs UP on CPU, leaving the GPU idle. (b) StrongHold ( Sun et al., 2022 ) overlaps CPU updates with GPU backward computation and gradient offloading by offloading and updating gradients layer by layer. However, CPU-side update is still too slow to hide, causing GPU stalls. (c) Example: iteration with accumulation: Update occurs every two steps—one for accumulation (i.e., gradients are summed without applying updates) ( ACC ), one for normal update ( UP ). (d) ZenFlow prioritizes parameter updates for important gradient ( § 3.3 ) on fast GPU to leverage its high compute bandwidth, while accumulating unimportant gradients in several rounds on slow CPU to reduce unnecessary parameter update overhead ( § 3.1 ). This design decouples fast GPU computing from slow CPU processing, minimizing GPU stalls and reducing data movement ( § 3.2 ). A chart comparing different offloading methods.
Figure 4
Figure 6. Localities of gradients during fine-tuning. Spatial-temporal locality heatmap showing the top-1% gradients.
Figure 7. Zero-stall pipelining with double buffering. ZenFlow overlaps GPU computation with CPU-side gradient accumulation and updates. Across training steps, important gradients are updated immediately on the GPU, while unimportant gradients are offloaded and accumulated on the CPU. The CPU maintains two buffers and alternates between accumulation ( ACC ) and update ( UP ) phases across rounds. While one buffer accumulates gradients from the current iterations, the other buffer applies parameter updates in the background. Layer-wise scheduling and swap-out/in ensure that CPU updates and parameter transfers are fully overlapped with GPU FP / BP , preventing GPU stalls. Zero-bubble pipeline.
Figure 8. Gradient gathering strategies. Left: Full gradient gathering collects and all-gathers full gradient matrices across GPUs, incurring high communication volume and large peak memory footprint. Right: ZenFlow performs light-weight gradient gathering by first reducing gradients along the channel dimension and exchanging only compact per-channel summaries. Gradient gathering strategies.
Figure 9. CDF of top- k elements and gradient norm. The channel index is sorted by the number of top- k elements. CDF of top-k elements and gradient norm.
Figure 10. Training time breakdown and speedup across hardware configurations and models. Bars correspond to the upper X-axis. Lines (with markers) show relative speedup over ZeRO-Offload (ZO), referenced on the bottom X-axis. SH denotes StrongHold, ZF* denotes ZenFlow* (importance-aware updates without zero-stall pipelining), and ZF denotes ZenFlow (full system). Fifteen bar-and-line charts compare training-time components and speedup for five models on A100 GPUs, H100 GPUs, and H100 GPUs with limited CPU resources. ZenFlow consistently removes most offloading-induced GPU stalls and provides the highest speedup.
Figure 10
Figure 13. Accuracy vs. per-iteration speedup on GLUE tasks for various models and baselines. The red dashed line marks the baseline accuracy achieved by ZeRO-Offload. Results close to the red line indicate accuracy comparable to the baseline while achieving significant speedup. Accuracy vs. per-iteration speedup on GLUE tasks.
Figure 14. Cost comparison for fine-tuning Qwen (0.5B–7B) on Alpaca-52K with manually tuned maximum batch sizes. We report the maximum feasible batch size (top), per-iteration time (middle), and total end-to-end training cost (bottom) across different GPU platforms. ZenFlow enables lower training cost by (i) supporting larger batch sizes on cheaper hardware through offloading, and (ii) maintaining competitive iteration speed despite CPU-GPU heterogeneity. As a result, ZenFlow achieves substantial reductions in end-to-end training cost compared to ZeRO-Offload and GPU-only ZeRO, while scaling to larger models without OOM. Batch size, iteration time, and estimated cost for fine-tuning models of different sizes (0.5B-7B) on four hardware configurations (4× RTX2080Ti, 4× RTX4090, 4× A6000, and 4× A100), comparing ZenFlow, ZeRO-Offload, and ZeRO. Costs are calculated using the listed on-demand hourly rates. Red cross marks indicate out-of-memory configurations.
Figure 15. Convergence on GLUE with OPT-350M. ZenFlow ’s loss matches ZeRO-Offload across tasks. Convergence comparison.
Figure 16. Sensitivity analysis of key hyperparameters. Sensitivity of top-k ratio and update interval.