Accelerating LLM Pretraining with Zeroth-Order Curvature on Idle Serving Compute

Mingchuan Ma; Dongqi Cai; Daqian Yang; Zhenyan Lu; Chen Chen; Teresa Lv; Yan Gao; Shangguang Wang

cs.LGNov 2, 2026

Accelerating LLM Pretraining with Zeroth-Order Curvature on Idle Serving Compute

Authors: Mingchuan Ma, Dongqi Cai, Daqian Yang, Zhenyan Lu, Chen Chen, Teresa Lv, Yan Gao, Shangguang Wang,

Organizations: School of Software & Microelectronics, Peking University · Nanjing University · MingTi Technology · Beijing University of Posts and Telecommunications ·

Abstract

LLM pretraining is increasingly compute-intensive, while substantial inference serving compute remains idle during off-peak periods. However, such serving hardware is difficult to use for conventional first-order training due to its memory, networking and other system requirements. Zeroth-order (ZO) optimization offers a natural interface to this forward-only compute resource, but its gradient estimates break down in pretraining. Unlike fine-tuning, which can confine updates to a small parameter subspace, pretraining must update every parameter, and the alignment of a ZO gradient estimate with the true gradient shrinks as the number of updated parameters grows. We make a different observation: noisy ZO estimates can still provide useful curvature information, even when they are too inaccurate to serve as gradients. Based on this insight, we propose ZO-Curv, which leverages idle forward-only devices to estimate curvature and uses it to adapt and refine the update used during first-order training, thereby accelerating the process. We provide convergence guarantees that characterize the effect of finite-query curvature estimation. Experiments on LLMs from 125M to 1.5B parameters show that ZO-Curv consistently accelerates pretraining. At 770M parameters, ZO-Curv reaches AdamW's 20K-step validation loss in 3.9 hours instead of 7.5 hours, while achieving a lower final loss of 2.70 versus 2.84 under the same step budget. These results show that idle model serving compute can accelerate LLM pretraining when zeroth-order queries are used to estimate curvature rather than gradients directly.

Figures & tables

Figure 1: Landscape of idle serving devices and the pathway of harvesting them. (a) Diurnal inference demand leaves peak-provisioned capacity idle. (b) Production clusters exhibit low average GPU utilization. (c) Harvesting idle devices doubles the pretraining runs a cluster completes per day. (d) Comparison of ZO curvature and ZO gradients on pretraining.

Figure 2: ZO gradients fail at pretraining. (a) Pure ZO stalls early while AdamW converges under the same budget. (b) Cosine similarity between the ZO gradient and the exact gradient.

Figure 3: Serving devices cannot join training, and curvature estimation burdens the training node. (a) Forward-only execution fits a

24

GB serving card at every size; AdamW and SophiaG training exceed it from 355M onward. (b) A SophiaG curvature-refresh step costs about twice an AdamW step, while a forward-only computation is much cheaper to obtain.

Figure 4: ZO-Curv overview. The strong node runs ordinary first-order pretraining; idle serving devices concurrently evaluate $q$ central-difference queries and return their averaged directional estimate, from which the strong node assembles the diagonal curvature that preconditions its update.

Figure 5

Figure 5: Main result (RQ1): forward-only curvature turns step efficiency into wall-clock speedup (OpenWebText; AdamW receives a doubled step budget). ZO-Curv reaches AdamW’s final loss in roughly half the time at every size, whereas SophiaG spends its step advantage on curvature computed on the training node.

Figure 7: Effect of the query budget

q

across model sizes. (a)(b) Final validation loss for different query budgets; (c)(d) corresponding training trajectories.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$d_{\mathrm{model}}$	Heads	Depth	AdamW lr	SophiaG / ZO-Curv lr	$\gamma$
GPT-2 Small (125M)	768	12	12	$6\times 10^{-4}$	$6\times 10^{-4}$	0.05
GPT-2 Medium (355M)	1024	16	24	$3\times 10^{-4}$	$7\times 10^{-4}$	0.08
GPT-2 Large (770M)	1280	20	36	$2\times 10^{-4}$	$3\times 10^{-4}$	0.05
GPT-NeoX (1.5B)	1536	24	48	$1.5\times 10^{-4}$	$1.2\times 10^{-4}$	0.05

Appendix

Table 2: Model configurations and peak learning rates. ZO-Curv uses the SophiaG hyperparameters unchanged;

\gamma

is the curvature scale of equation 3 .

Figure 10: ZO-Curv matches SophiaG’s step efficiency at every scale (first 20K steps of the runs of Figure 5 ). The two curves are nearly indistinguishable on a step axis: the forward-only estimator matches the backpropagated one in per-step quality, and the advantage of ZO-Curv lies in where the estimate is computed.

Figure 11: SophiaG with a ten-step refresh interval. Wall-clock time to complete 20K steps for AdamW, SophiaG refreshing its curvature every ten steps, and ZO-Curv refreshing at every step. Even with its curvature cost amortized, SophiaG is slower than ZO-Curv at every size.

Figure 12: Loss of SophiaG with a ten-step refresh interval. Validation loss after 20K steps for the runs of Figure 11 . With its curvature refreshed ten times less often, SophiaG ends above ZO-Curv at every size.

Figure 13: Training trajectories across query budgets, all four sizes (companion to Figure 7 ). At 125M the budgets collapse onto each other early, whereas at 770M and 1.5B they remain cleanly ordered, matching the scale-dependent saturation of Figure 7 (a,b).

Figure 14: The one-step pipeline offset is free (GPT-2 355M,

q=512

). Training with curvature estimated at

\theta_{t-1}

(pipelined, as deployed) is indistinguishable from using curvature estimated at the current

\theta_{t}

(synchronous).

	HellaSwag		ARC-E		ARC-C		LAMBADA		Average
Method	Base	FT	Base	FT	Base	FT	Base	FT	Base	FT
AdamW	31.4	33.0	40.2	50.9	22.1	26.1	30.8	–	31.1	35.2
SophiaG	35.6	37.1	47.5	54.7	25.8	28.1	39.1	–	37.0	39.8
ZO-Curv (ours)	35.2	36.3	47.4	54.2	26.8	28.8	39.0	–	37.1	39.6

Appendix

Table 3: Full downstream evaluation of the GPT-2 770M checkpoints at 20K steps (companion to Figure 9 ). Accuracy in %. Base: few-shot before finetuning (10 examples for HellaSwag, 25 for ARC-E and ARC-C, none for LAMBADA). FT: zero-shot after per-task LoRA finetuning. LAMBADA has no public training split and is not finetuned; it enters the FT average with its untuned score.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 461 papers

CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
Mar 23, 2026Shuo Wang, Ziyu Chen, Ming TangZeroth-Order Optimization Memory-Efficient Fine-Tuning
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China. · Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.
Inference-Native Zeroth-Order Optimization for LLMs
May 27, 2026Zelin Li, Caiwen DingLLM Serving Zeroth-Order Optimization
University of Minnesota
AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuning
Sep 28, 2026Yue Xie, Zhi Zheng, Yunpeng Ba +5Subspace Optimization Memory-Efficient Fine-Tuning
Southern University of Science and Technology · National University of Singapore · Huawei Technologies Ltd. +1

Mar 23, 2026J/K move · Enter open · S save

CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning

Shuo Wang, Ziyu Chen, Ming Tang

Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China. · Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.

Zeroth-Order Optimization Memory-Efficient Fine-Tuning LLM Fine-Tuning Sparse Fine-Tuning

Open paper AskPDF arXiv

Fine-tuning large language models (LLMs) with backpropagation achieves high performance but incurs substantial memory overhead, limiting scalability on resource-constrained hardware. Zeroth-order (ZO) optimization provides a memory-efficient alternative by relying solely on forward passes, yet it typically suffers from slow or unstable convergence due to high-variance gradient estimates. Sparse ZO updates partially address this issue by perturbing only a subset of parameters, but their effectiveness hinges on selecting informative parameters, which is challenging in ZO optimization because each query yields only scalar feedback. We propose Adaptive Curvature-Guided Sparse Zeroth-Order Optimization (CurvZO), which tracks curvature signals online from scalar ZO feedback and leverages these signals to construct a parameter-wise sampling distribution for selecting coordinates at each update, reducing the variance of the sparse ZO gradient estimator. Moreover, CurvZO dynamically adapts the perturbation budget to the evolving curvature signal distribution, yielding sparse ZO updates that remain both focused and sufficiently exploratory. Extensive experiments on OPT and Llama across diverse NLP tasks show that CurvZO consistently improves fine-tuning performance and reduces training time over ZO baselines. It improves accuracy by up to 4.4 points and achieves up to a

2\times

speedup, while preserving memory efficiency.

Abstract

Language Model Pretraining

Explore similar work

CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning

Inference-Native Zeroth-Order Optimization for LLMs

AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuning