Fine-tuning large language models (LLMs) often exceeds GPU memory limits, prompting systems to offload model states to CPU memory. However, existing offloaded training frameworks like ZeRO-Offload treat all parameters equally and update the full model on the CPU, causing severe GPU stalls, where fast, expensive GPUs sit idle, waiting for slow CPU updates and limited-bandwidth PCIe transfers. We present ZenFlow, a new offloaded training framework that decouples updates between GPU and CPU by prioritizing important gradients. ZenFlow performs in-place updates of important gradients on GPU while offloading and accumulating less important ones on CPU without blocking training iterations, fully overlapping CPU work with GPU computation. To make this importance-aware design scalable, ZenFlow leverages a novel spatio-temporal locality of important gradients, concentrated in specific positions that remain stable across iterations, to make important gradient identification efficient and lightweight while preserving fidelity. Evaluation shows that ZenFlow achieves up to 5x end-to-end speedup, 2x lower PCIe traffic, 85% less GPU stalls, and up to 72% lower cloud GPU cost, all while preserving accuracy.
Figures & tables
Figure 1. GPU vs. CPU peak performance evolution. Line chart comparing the growth of peak GPU and CPU floating-point performance from 2016 to 2025.
Figure 2. Comparison of GPU utilization over time. GPU utilization timelines for three consecutive training steps under ZeRO-Offload and ZenFlow. ZeRO-Offload repeatedly leaves the GPU idle during CPU-side updates, while ZenFlow keeps GPU utilization high.
Figure 3. Offloading strategy comparison. (a) ZeRO-Offload ( Ren et al., 2021b ) sequentially executes FP and BP on GPU, then offloads gradients and performs UP on CPU, leaving the GPU idle. (b) StrongHold ( Sun et al., 2022 ) overlaps CPU updates with GPU backward computation and gradient offloading by offloading and updating gradients layer by layer. However, CPU-side update is still too slow to hide, causing GPU stalls. (c) Example: iteration with accumulation: Update occurs every two steps—one for accumulation (i.e., gradients are summed without applying updates) ( ACC ), one for normal update ( UP ). (d) ZenFlow prioritizes parameter updates for important gradient ( § 3.3 ) on fast GPU to leverage its high compute bandwidth, while accumulating unimportant gradients in several rounds on slow CPU to reduce unnecessary parameter update overhead ( § 3.1 ). This design decouples fast GPU computing from slow CPU processing, minimizing GPU stalls and reducing data movement ( § 3.2 ). A chart comparing different offloading methods.
Figure 4
Figure 6. Localities of gradients during fine-tuning. Spatial-temporal locality heatmap showing the top-1% gradients.
Figure 7. Zero-stall pipelining with double buffering. ZenFlow overlaps GPU computation with CPU-side gradient accumulation and updates. Across training steps, important gradients are updated immediately on the GPU, while unimportant gradients are offloaded and accumulated on the CPU. The CPU maintains two buffers and alternates between accumulation ( ACC ) and update ( UP ) phases across rounds. While one buffer accumulates gradients from the current iterations, the other buffer applies parameter updates in the background. Layer-wise scheduling and swap-out/in ensure that CPU updates and parameter transfers are fully overlapped with GPU FP / BP , preventing GPU stalls. Zero-bubble pipeline.
Figure 8. Gradient gathering strategies. Left: Full gradient gathering collects and all-gathers full gradient matrices across GPUs, incurring high communication volume and large peak memory footprint. Right: ZenFlow performs light-weight gradient gathering by first reducing gradients along the channel dimension and exchanging only compact per-channel summaries. Gradient gathering strategies.
Figure 9. CDF of top- k elements and gradient norm. The channel index is sorted by the number of top- k elements. CDF of top-k elements and gradient norm.
Figure 10. Training time breakdown and speedup across hardware configurations and models. Bars correspond to the upper X-axis. Lines (with markers) show relative speedup over ZeRO-Offload (ZO), referenced on the bottom X-axis. SH denotes StrongHold, ZF* denotes ZenFlow* (importance-aware updates without zero-stall pipelining), and ZF denotes ZenFlow (full system). Fifteen bar-and-line charts compare training-time components and speedup for five models on A100 GPUs, H100 GPUs, and H100 GPUs with limited CPU resources. ZenFlow consistently removes most offloading-induced GPU stalls and provides the highest speedup.
Figure 10
Figure 13. Accuracy vs. per-iteration speedup on GLUE tasks for various models and baselines. The red dashed line marks the baseline accuracy achieved by ZeRO-Offload. Results close to the red line indicate accuracy comparable to the baseline while achieving significant speedup. Accuracy vs. per-iteration speedup on GLUE tasks.
Figure 14. Cost comparison for fine-tuning Qwen (0.5B–7B) on Alpaca-52K with manually tuned maximum batch sizes. We report the maximum feasible batch size (top), per-iteration time (middle), and total end-to-end training cost (bottom) across different GPU platforms. ZenFlow enables lower training cost by (i) supporting larger batch sizes on cheaper hardware through offloading, and (ii) maintaining competitive iteration speed despite CPU-GPU heterogeneity. As a result, ZenFlow achieves substantial reductions in end-to-end training cost compared to ZeRO-Offload and GPU-only ZeRO, while scaling to larger models without OOM. Batch size, iteration time, and estimated cost for fine-tuning models of different sizes (0.5B-7B) on four hardware configurations (4× RTX2080Ti, 4× RTX4090, 4× A6000, and 4× A100), comparing ZenFlow, ZeRO-Offload, and ZeRO. Costs are calculated using the listed on-demand hourly rates. Red cross marks indicate out-of-memory configurations.
Figure 15. Convergence on GLUE with OPT-350M. ZenFlow ’s loss matches ZeRO-Offload across tasks. Convergence comparison.
Figure 16. Sensitivity analysis of key hyperparameters. Sensitivity of top-k ratio and update interval.
We present MegaTrain, a memory-centric system that efficiently trains 100B+ parameter large language models at full precision on a single GPU. Unlike traditional GPU-centric systems, MegaTrain stores parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state. To battle the CPU-GPU bandwidth bottleneck, we adopt two key optimizations. 1) We introduce a pipelined double-buffered execution engine that overlaps parameter prefetching, computation, and gradient offloading across multiple CUDA streams, enabling continuous GPU execution. 2) We replace persistent autograd graphs with stateless layer templates, binding weights dynamically as they stream in, eliminating persistent graph metadata while providing flexibility in scheduling. On a single H200 GPU with 1.5TB host memory, MegaTrain reliably trains models up to 120B parameters. It also achieves 1.84× the training throughput of DeepSpeed ZeRO-3 with CPU offloading when training 14B models. MegaTrain also enables 7B model training with 512k token context on a single GH200.
Training large language models on limited hardware is increasingly a scheduling problem across GPU compute, host memory, PCIe transfer, and storage bandwidth. Existing offloading systems reduce GPU residency, and MegaTrain shows that a CPU-master layer-streaming executor can train large models on a single GPU, but fixed checkpointing and placement heuristics still leave communication exposed on the critical path. We propose LazyTrain, an optimization layer over a layer-streaming executor. LazyTrain formulates checkpoint selection, activation placement, recomputation, and CPU-GPU-NVMe communication overlap as a mixed-integer scheduling problem, then executes the solved policy during training. It further couples 8-bit optimizer states with fast gradient clipping as a single Hybrid 8-bit operator: state compression reduces optimizer-state memory, while fast clipping counteracts the additional CPU-side update overhead. Across H800 experiments from Qwen2.5-3B to Qwen3.6-27B, LazyTrain improves sustained TFLOPS over matched baselines runs by approximately 1.24×; RTX 3090 experiments likewise increase the maximum feasible batch size by one at each model scale. In the primary Qwen3.6-27B H800 MetaMathQA run, LazyTrain reaches 219.95 TFLOPS and 1361 tokens/s at batch size 72, peaks at 68.84,GB of GPU memory, and obtains 95.42% exact-match accuracy on the full evaluation split. The source code is available at https://github.com/DataArcTech/LazyTrain.
Xiaojun Wu, Cehao Yang, Honghao Liu +5
1IDEA Research · 2The Hong Kong University of Science and Technology (Guangzhou) · 3DataArcTech Ltd.
Fine-tuning Large Language Models (LLMs) has become essential for domain adaptation, but its memory-intensive property exceeds the capabilities of most GPUs. To address this challenge and democratize LLM fine-tuning, we present SlideFormer, a novel system designed for single-GPU environments. Our innovations are: (1) A lightweight asynchronous engine that treats the GPU as a sliding window and overlaps GPU computation with CPU updates and multi-tier I/O. (2) A highly efficient heterogeneous memory management scheme significantly reduces peak memory usage. (3) Optimized Triton kernels to solve key bottlenecks and integrated advanced I/O. This collaborative design enables fine-tuning of the latest 123B+ models on a single RTX 4090, supporting up to 8x larger batch sizes and 6x larger models. In evaluations, SlideFormer achieves 1.40x to 6.27x higher throughput while roughly halving CPU/GPU memory usage compared to baselines, sustaining >95% peak performance on both NVIDIA and AMD GPUs.The code is available at https://github.com/RegiaYoung/SlideFormer.
Ruijia Yang, Zeyi Wen
HKUST(GZ) Guangzhou, China · HKUST Guangzhou, China