Accelerating LLM Pretraining with Zeroth-Order Curvature on Idle Serving Compute
Authors: Mingchuan Ma, Dongqi Cai, Daqian Yang, Zhenyan Lu, Chen Chen, Teresa Lv, Yan Gao, Shangguang Wang, Mengwei Xu, Nicholas D. Lane, Yu Huang
Organizations: School of Software & Microelectronics, Peking University · Nanjing University · MingTi Technology · Beijing University of Posts and Telecommunications · Sony Europe Ltd. · University of Cambridge
LLM pretraining is increasingly compute-intensive, while substantial inference serving compute remains idle during off-peak periods. However, such serving hardware is difficult to use for conventional first-order training due to its memory, networking and other system requirements. Zeroth-order (ZO) optimization offers a natural interface to this forward-only compute resource, but its gradient estimates break down in pretraining. Unlike fine-tuning, which can confine updates to a small parameter subspace, pretraining must update every parameter, and the alignment of a ZO gradient estimate with the true gradient shrinks as the number of updated parameters grows. We make a different observation: noisy ZO estimates can still provide useful curvature information, even when they are too inaccurate to serve as gradients. Based on this insight, we propose ZO-Curv, which leverages idle forward-only devices to estimate curvature and uses it to adapt and refine the update used during first-order training, thereby accelerating the process. We provide convergence guarantees that characterize the effect of finite-query curvature estimation. Experiments on LLMs from 125M to 1.5B parameters show that ZO-Curv consistently accelerates pretraining. At 770M parameters, ZO-Curv reaches AdamW's 20K-step validation loss in 3.9 hours instead of 7.5 hours, while achieving a lower final loss of 2.70 versus 2.84 under the same step budget. These results show that idle model serving compute can accelerate LLM pretraining when zeroth-order queries are used to estimate curvature rather than gradients directly.
Figures & tables
Figure 1: Landscape of idle serving devices and the pathway of harvesting them. (a) Diurnal inference demand leaves peak-provisioned capacity idle. (b) Production clusters exhibit low average GPU utilization. (c) Harvesting idle devices doubles the pretraining runs a cluster completes per day. (d) Comparison of ZO curvature and ZO gradients on pretraining.
Figure 2: ZO gradients fail at pretraining. (a) Pure ZO stalls early while AdamW converges under the same budget. (b) Cosine similarity between the ZO gradient and the exact gradient.
Figure 3: Serving devices cannot join training, and curvature estimation burdens the training node. (a) Forward-only execution fits a 24 GB serving card at every size; AdamW and SophiaG training exceed it from 355M onward. (b) A SophiaG curvature-refresh step costs about twice an AdamW step, while a forward-only computation is much cheaper to obtain.
Figure 4: ZO-Curv overview. The strong node runs ordinary first-order pretraining; idle serving devices concurrently evaluate q central-difference queries and return their averaged directional estimate, from which the strong node assembles the diagonal curvature that preconditions its update.
Figure 5
Figure 5: Main result (RQ1): forward-only curvature turns step efficiency into wall-clock speedup (OpenWebText; AdamW receives a doubled step budget). ZO-Curv reaches AdamW’s final loss in roughly half the time at every size, whereas SophiaG spends its step advantage on curvature computed on the training node.
Figure 7
Figure 7: Effect of the query budget q across model sizes. (a)(b) Final validation loss for different query budgets; (c)(d) corresponding training trajectories.
Figure 9
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Model
dmodel
Heads
Depth
AdamW lr
SophiaG / ZO-Curv lr
γ
GPT-2 Small (125M)
768
12
12
6×10−4
6×10−4
0.05
GPT-2 Medium (355M)
1024
16
24
3×10−4
7×10−4
0.08
GPT-2 Large (770M)
1280
20
36
2×10−4
3×10−4
0.05
GPT-NeoX (1.5B)
1536
24
48
1.5×10−4
1.2×10−4
0.05
Appendix
Table 2: Model configurations and peak learning rates. ZO-Curv uses the SophiaG hyperparameters unchanged; γ is the curvature scale of equation 3 .
Figure 10: ZO-Curv matches SophiaG’s step efficiency at every scale (first 20K steps of the runs of Figure 5 ). The two curves are nearly indistinguishable on a step axis: the forward-only estimator matches the backpropagated one in per-step quality, and the advantage of ZO-Curv lies in where the estimate is computed.
Figure 11: SophiaG with a ten-step refresh interval. Wall-clock time to complete 20K steps for AdamW, SophiaG refreshing its curvature every ten steps, and ZO-Curv refreshing at every step. Even with its curvature cost amortized, SophiaG is slower than ZO-Curv at every size.
Figure 12: Loss of SophiaG with a ten-step refresh interval. Validation loss after 20K steps for the runs of Figure 11 . With its curvature refreshed ten times less often, SophiaG ends above ZO-Curv at every size.
Figure 13: Training trajectories across query budgets, all four sizes (companion to Figure 7 ). At 125M the budgets collapse onto each other early, whereas at 770M and 1.5B they remain cleanly ordered, matching the scale-dependent saturation of Figure 7 (a,b).
Figure 14: The one-step pipeline offset is free (GPT-2 355M, q=512 ). Training with curvature estimated at θt−1 (pipelined, as deployed) is indistinguishable from using curvature estimated at the current θt (synchronous).
HellaSwag
ARC-E
ARC-C
LAMBADA
Average
Method
Base
FT
Base
FT
Base
FT
Base
FT
Base
FT
AdamW
31.4
33.0
40.2
50.9
22.1
26.1
30.8
–
31.1
35.2
SophiaG
35.6
37.1
47.5
54.7
25.8
28.1
39.1
–
37.0
39.8
ZO-Curv (ours)
35.2
36.3
47.4
54.2
26.8
28.8
39.0
–
37.1
39.6
Appendix
Table 3: Full downstream evaluation of the GPT-2 770M checkpoints at 20K steps (companion to Figure 9 ). Accuracy in %. Base: few-shot before finetuning (10 examples for HellaSwag, 25 for ARC-E and ARC-C, none for LAMBADA). FT: zero-shot after per-task LoRA finetuning. LAMBADA has no public training split and is not finetuned; it enters the FT average with its untuned score.
Fine-tuning large language models (LLMs) with backpropagation achieves high performance but incurs substantial memory overhead, limiting scalability on resource-constrained hardware. Zeroth-order (ZO) optimization provides a memory-efficient alternative by relying solely on forward passes, yet it typically suffers from slow or unstable convergence due to high-variance gradient estimates. Sparse ZO updates partially address this issue by perturbing only a subset of parameters, but their effectiveness hinges on selecting informative parameters, which is challenging in ZO optimization because each query yields only scalar feedback. We propose Adaptive Curvature-Guided Sparse Zeroth-Order Optimization (CurvZO), which tracks curvature signals online from scalar ZO feedback and leverages these signals to construct a parameter-wise sampling distribution for selecting coordinates at each update, reducing the variance of the sparse ZO gradient estimator. Moreover, CurvZO dynamically adapts the perturbation budget to the evolving curvature signal distribution, yielding sparse ZO updates that remain both focused and sufficiently exploratory. Extensive experiments on OPT and Llama across diverse NLP tasks show that CurvZO consistently improves fine-tuning performance and reduces training time over ZO baselines. It improves accuracy by up to 4.4 points and achieves up to a 2× speedup, while preserving memory efficiency.
Shuo Wang, Ziyu Chen, Ming Tang
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China. · Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.
Zeroth-order (ZO) methods train large language models using only forward passes, yet common ZO execution paths perform substantial work beyond what the algorithm itself requires. To remove this extra execution overhead, we present Infer-ZO, which separates the evaluations required by the algorithm from how they are executed, allowing them to reuse existing inference-engine optimizations. After removing inherited execution work, a complete Infer-ZO step on Qwen3-14B adds only 1.16% wall-clock time over matched inference, bringing ZO execution close to its fundamental inference workload. With Infer-ZO, ZO evaluations run at near-inference cost with frozen base weights and can share GPU batches with ordinary inference requests. In co-serving experiments, foreground and background throughput remain within 1-3% of the corresponding background-request baseline. Across 15 Qwen3, Llama, and OPT models, Infer-ZO achieves 2.10x-9.94x end-to-end step speedups over released LoZO. Code is available at https://github.com/playeriv65/zo-vllm.
Zeroth-order (ZO) optimization offers a memory-efficient alternative for LLM fine-tuning by estimating updates only from forward evaluations of perturbed parameters, without backpropagation or activation storage. However, in billion-parameter LLMs, isotropic perturbations often waste many forward evaluations on weakly informative directions. To make these evaluations more informative, existing ZO methods restrict perturbations to low-dimensional subspaces. Yet the quality of these subspaces is critical: overly compressed or poorly maintained spaces can miss useful update directions. To obtain a high-quality subspace for ZO updates, this paper proposes AIM-ZO, a ZO fine-tuning method based on Activation-Informed Subspace Maintenance. AIM-ZO uses forward activations as local directional information and continuously integrates them into a broad, evolving subspace over training. To access broader gradient-relevant structure while keeping individual perturbations low-dimensional, AIM-ZO activates only a smaller set of shared and sampled directions, decoupling the maintained width from the active width. We evaluate AIM-ZO across 5 LLMs and 11 downstream tasks under matched forward-evaluation budgets; its six-task average exceeds the strongest fully evaluated ZO baseline by 1.26 percentage points on OPT-2.7B and MeZO by 2.85 percentage points on OPT-30B. Our code is available at https://github.com/EkkoXy/AIM-ZO
Yue Xie, Zhi Zheng, Yunpeng Ba +5
Southern University of Science and Technology · National University of Singapore · Huawei Technologies Ltd. +1