Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning

Cristian McGee; El Houcine Bergou; Aritra Dutta

cs.LGOct 1, 2026

Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning

Authors: Cristian McGee, El Houcine Bergou, Aritra Dutta

Organizations: University of Central Florida Orlando, FL, USA · Mohammed VI Polytechnic University Ben Guerir, Morocco

Abstract

Step-size selection remains a central challenge in large-scale neural network optimization; conservative steps slow convergence, while aggressive steps can destabilize it. We combine \textbf{Z}ero-and-\textbf{F}irst-\textbf{O}rder optimization~(ZFO) and propose a lightweight framework that decouples direction selection from step-size. ZFO uses a trusted first-order optimizer to determine the direction and performs zeroth-order evaluations only along this one-dimensional subspace to choose how far to move. Using the current {gradient information} and two additional objective function evaluations, ZFO instances construct a local model of the objective function along the proposed direction and select a curvature-aware step within a bounded search interval. This yields an adaptive step-selection mechanism that costs less than a full line search. We provide theoretical guarantees to show that shared-sample evaluations produce reliable finite-difference curvature estimates, that the induced local model selects a near-optimal step along the search interval, and that ZFO converges to a neighborhood of a stationary point. Across the evaluated settings, language models and datasets, ZFO frequently improves optimization and final performance relative to fixed-step first-order baselines, with the magnitude and preferred local model depending on the objective. Our code is publicly available at: https://github.com/nizswan/Zeroth-First-Order-Framework.

Figures & tables

Figure 1 : The Zeroth First -Order ( Z F O) Framework. Left: A first-order optimizer proposes the direction

z_{t}

\theta_{t}

, which restricts the objective to the line

\theta_{t}+sz_{t}

. Right: Along this line,

\theta_{t}+sz_{t}

, the forward pass gives

\phi_{t}(0)

, the backward pass gives

\phi_{t}^{\prime}(0)

, and two symmetric probes at

\pm\epsilon

supply curvature estimates. ZFO fits a local model

M(s)

and takes the optimized step

s_{M}

over the interval

[0,R_{t}]

; see Algorithm 1 .

Model	Dataset	Baseline		ZFO Methods (This Paper)
Model	Dataset	AdamW (FO)	MeZO (ZO)	Taylor 2	Taylor 3	Padé 2	Padé 3
Qwen-2.5-Math-1.5B	GSM8K	78.24 $\pm$ 0.59	0.00	81.00 $\pm$ 0.11	75.56 $\pm$ 0.24	80.95 $\pm$ 0.49	80.97 $\pm$ 0.73
	MATH	31.77 $\pm$ 1.63	22.01 $\pm$ 3.93	40.10 $\pm$ 1.19	20.18 $\pm$ 3.03	37.50 $\pm$ 1.79	39.58 $\pm$ 0.45
	SVAMP	88.11 $\pm$ 0.69	68.67 $\pm$ 10.33	90.78 $\pm$ 0.96	89.11 $\pm$ 0.51	90.89 $\pm$ 1.02	90.33 $\pm$ 2.19
	AsDiv	91.45 $\pm$ 0.36	64.72 $\pm$ 3.51	94.09 $\pm$ 0.29	88.27 $\pm$ 0.35	93.04 $\pm$ 0.22	93.46 $\pm$ 0.18
	OpenBookQA	26.07 $\pm$ 2.53	8.00 $\pm$ 10.05	37.80 $\pm$ 17.85	39.73 $\pm$ 20.99	60.67 $\pm$ 0.95	64.07 $\pm$ 1.22

Table 1: Evaluation performance for baseline first-order (FO) fine-tuning (using AdamW) and zeroth-order (ZO) fine-tuning (using MeZO) against ZFO methods across models and reasoning benchmarks. All runs report the final evaluation and mean results over 3 seeds. Green is the best and red is the second best result.

Method	ARC-Challenge	StrategyQA	FOLIO	CODAH
AdamW (FO)	49.29 $\pm$ 16.08	68.56 $\pm$ 0.91	34.48 $\pm$ 1.13	57.91 $\pm$ 3.24
Taylor 2	58.10 $\pm$ 1.33	69.63 $\pm$ 1.82	36.6 $\pm$ 3.26	62.83 $\pm$ 3.14
Taylor 3	54.24 $\pm$ 0.94	61.67 $\pm$ 1.46	34.48 $\pm$ 1.13	55.28 $\pm$ 1.66
Padé 2	53.58 $\pm$ 2.77	65.55 $\pm$ 4.43	36.27 $\pm$ 2.73	53.96 $\pm$ 17.72
Padé 3	57.05 $\pm$ 1.24	70.06 $\pm$ 0.61	36.44 $\pm$ 4.53	41.85 $\pm$ 20.58

Table 2: Qwen-2.5-Math-1.5B study on ZFO methods in our settings via additional reasoning benchmarks. Results report evaluation performance averaged over 3 seeds for FO and ZFO methods.

Figure 2 : Results on classic optimization problems. We plot the median

\log_{10}

objective over three perturbed initializations. GeN has a similar computational budget as ZFO, as both methods perform two additional function evaluations per iteration. We study the extended classic optimization problems in Table 8 and Figure 5 .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Study	Model	Model Parameters	Dataset	Dataset Size (Train / Eval)
Main Study	Qwen-2.5-Math-1.5B	1.54B	GSM8K	7,473 / 1,319
			MATH	512 / 256
			SVAMP	700 / 300
			AsDiv	1,844 / 461
			OpenBookQA	4,957 / 500
	Phi-2	2.70B	SVAMP	700 / 300

Appendix

Table 3 : Summary of the models and datasets used in the large-model experiments. Parameter counts denote the total number of model parameters. Dataset sizes are reported as the number of training/evaluation examples used in our experiments. MATH uses the 512/256 train/evaluation subset described in the main text.

Table 7

Model	Dataset	Configuration
Model	Dataset	FO LR	ZO LR	BT2	BT3	BP2	BP3	Config Type
Qwen-2.5-Math-1.5B	GSM8K	5e-6	ALL	10	5	10	10	a
	MATH	5e-6	1e-6	5	5	5	5	c
	SVAMP	1e-5	5e-7	3	5	3	3	a
	AsDiv	5e-6	5e-7	5	5	5	5	c
	OpenBookQA	1e-5	1e-7	10	5	5	5	a

Appendix

Table 6: Hyperparameter Configuration. Learning rates, bounds, and configuration types used for the main empirical study. FO LR and ZO LR denote the learning rates used for first-order and zeroth-order fine-tuning, respectively. BT2, BT3, BP2, and BP3 represent the selected bounds for Taylor2, Taylor3, Padé2, and Padé3, respectively. Learning rate search is over

\{1\mathrm{e}-6,5\mathrm{e}{-6},1\mathrm{e}{-5}\}

and bound search is over

\{3,5,10\}

. ALL indicates that every lr for MeZO produced the same result.

Model	Dataset	Configuration
Model	Dataset	Learning Rate	BT2	BT3	BP2	BP3	Config Type
Qwen-2.5-Math-1.5B	ARC-Challenge	1e-5	3	5	3	3	c
	StrategyQA	1e-5	5	5	3	5	c
	FOLIO	1e-5	3	3	3	3	c
	CODAH	1e-5	5	5	5	5	c

Appendix

Table 7: Learning rate, bound, and configuration type used for the Qwen sub-study on additional reasoning benchmarks. Learning rate search is over

\{

5e-6, 1e-5

\}

and bound search is over

\{3,5\}

Problem	Objective	Initial position
Linear least squares	$\min_{\theta}\dfrac{1}{2n}\lVert A\theta-y\rVert^{2}$	$\theta_{0}=\mathbf{0}_{20}$
Nonlinear least squares	$\min_{a,b,c}\dfrac{1}{2N}\sum_{i}\big(a\,e^{bx_{i}}+c-y_{i}\big)^{2}$	$(a_{0},b_{0},c_{0})=(0.4,\,0.8,\,1.0)$
Logistic regression	$\min_{\theta}\dfrac{1}{n}\sum_{i}\big[\log(1+e^{x_{i}^{\top}\theta})-y_{i}x_{i}^{\top}\theta\big]+\dfrac{\lambda}{2}\lVert\theta\rVert^{2}$	$\theta_{0}=\mathbf{0}_{16}$
Low-rank matrix factorization	$\min_{U,V}\dfrac{1}{2mn}\lVert UV^{\top}-M\rVert_{F}^{2}+\dfrac{\lambda}{2\|\theta\|}\lVert\theta\rVert^{2}$	$(U_{0},V_{0})\sim\mathcal{N}(0,0.15^{2})$
Rosenbrock	$\min_{x_{1},x_{2}}100(x_{2}-x_{1}^{2})^{2}+(1-x_{1})^{2}$	$(x_{1},x_{2})_{0}=(-1.2,\,1.0)$
Beale	$\min_{x_{1},x_{2}}\sum_{k=1}^{3}\big(c_{k}-x_{1}+x_{1}x_{2}^{k}\big)^{2}$	$(x_{1},x_{2})_{0}=(-1.0,\,1.0)$

Appendix

Table 8 : Compact summary of the Classical Optimization Problems used in Figure 5 . Before optimization, each listed initial position

\theta_{0}

is perturbed independently per seed as

\theta_{0,s}=\theta_{0}+\xi_{s}

, where

\xi_{s}\sim\mathcal{N}(0,0.04I)

applies independent Gaussian noise (standard deviation

0.2

) to every coordinate, drawn from a dedicated, task-specific random stream and shared across all FO, ZFO, and GeN methods evaluated on that task–seed pair.

Figure 5 : Median

\log_{10}

objective over three seeds for six optimization problems. Solid curves use AdamW directions, and dashed curves use SGD directions. Each method uses its best learning rate from the same 18-point search. Lower values indicate better optimization.

Figure 6 : ZFO approximations on a variety of locally analytic functions. Each panel plots Taylor 2 , Taylor 3 , Padé 2 , and Padé 3 around an $x_{0}.$ Top row consists of panels where all four interpolants are visually indistinguishable from the actual function. The bottom row consists of panels that expose the qualitative differences between the Taylor and Padé approximants.

Figure 7 : Toy RLVR validation on the Grouped Sequence Bandit and Grouped Path-Matching environments. ZFO methods improve over FO and ZO baselines under the same search scale $\beta=1$ , showing that local curvature models can select more effective steps along update directions.

Figure 8 : Stability of ZFO. Empirical evaluation curves demonstrating ZFO’s capacity to stabilize and improve over fine-tuning with AdamW (FO FT). Demonstrations involve AdamW FT (FO FT) curves of multiple learning rates.

Figure 9 : Ablation study inspecting the relationship between the learning rate $\alpha$ and search hyperparameter $\beta$ on the reward rate. We report the performance over a training set of 512 elements and an evaluation set of 256 elements on the Qwen-2.5-MATH-1.5B model and datasets, GSM8K and SVAMP, on the top and bottom row, respectively. From left to right, the panels represent all four ZFO methods ( Taylor 2 , Taylor 3 , Padé 2 , Padé 3 ) presented in this paper.

Method	Main Results Wins	Qwen Sub-Study Wins	Win Ratio
Taylor 2	10/12	4/4	14/16
Taylor 3	6/12	1/4	7/16
Padé 2	11/12	2/4	13/16
Padé 3	11/12	3/4	14/16

Appendix

Table 9: Win ratios for each ZFO method across the main results and Qwen sub-study. A win denotes a setting in which the method achieves a higher mean accuracy than the corresponding FO baseline.

Model	Dataset	AdamW + Cosine	Prodigy	Worst ZFO	Best ZFO
Qwen-2.5-Math-1.5B	GSM8K	72.88 $\pm$ 0.54	64.32 $\pm$ 0.90	75.56 $\pm$ 0.24 (Taylor 3 )	81.00 $\pm$ 0.11 (Taylor 2 )
	SVAMP	86.00 $\pm$ 0.33	83.22 $\pm$ 0.19	89.11 $\pm$ 0.51 (Taylor 3 )	90.89 $\pm$ 1.02 (Padé 2 )
	AsDiv	81.98 $\pm$ 0.52	80.82 $\pm$ 0.30	88.27 $\pm$ 0.35 (Taylor 3 )	94.09 $\pm$ 0.29 (Taylor 2 )
	OpenBookQA	59.60 $\pm$ 0.87	38.73 $\pm$ 0.31	37.80 $\pm$ 17.85 (Taylor 2 )	64.07 $\pm$ 1.22 (Padé 3 )
	ARC-Challenge	55.15 $\pm$ 1.11	30.80 $\pm$ 0.15	53.58 $\pm$ 2.77 (Padé 2 )	58.10 $\pm$ 1.33 (Taylor 2 )
	StrategyQA	66.28 $\pm$ 0.51	60.80 $\pm$ 0.95	61.67 $\pm$ 1.46 (Taylor 3 )	70.06 $\pm$ 0.61 (Padé 3 )

Appendix

Table 10: Comparison with additional first-order baselines. AdamW with linear warmup and cosine decay and Prodigy are compared with the worst- and best-performing ZFO variants for each Qwen-2.5-Math-1.5B dataset. The ZFO extrema are selected by mean final evaluation performance across Taylor 2 , Taylor 3 , Padé 2 , and Padé 3 ; the selected variant is shown in parentheses. The best result in each row is bold.

Dataset	ZFO-Seq	Armijo	BB	Polyak	PLS
CODAH	55.47	32.81	52.73	30.47	31.25
ARC-Challenge	55.86	57.81	54.69	35.16	35.16
SVAMP	83.59	85.55	85.55	82.81	76.17

Appendix

Table 11 : Comparison with line-search algorithms. Extended line-search results comparing the basic ZFO-Seq method against Armijo, Barzilai-Borwein, Polyak, and a simplified probabilistic line search (PLS) on the Qwen Model. The best result is in bold , and the second best result is underlined .

Dataset	Base LR	AdamW (FO)	Taylor 2	Taylor 3	Padé 2	Padé 3
ARC-Challenge	$1\times 10^{-5}$	31.71 $\pm$ 8.11	56.77 $\pm$ 0.61	56.51 $\pm$ 0.36	55.49 $\pm$ 0.98	49.86 $\pm$ 0.27
StrategyQA	$1\times 10^{-5}$	59.24 $\pm$ 10.14	70.69 $\pm$ 5.10	74.87 $\pm$ 0.37	67.25 $\pm$ 13.64	48.18 $\pm$ 0.44
ASDiv	$5\times 10^{-6}$	82.08 $\pm$ 0.08	81.82 $\pm$ 0.26	78.05 $\pm$ 1.41	81.87 $\pm$ 0.26	81.92 $\pm$ 0.46

Appendix

Table 12: Qwen-2.5-Math-1.5B sub-study ablation with

{\beta=1}

. Results are mean

\pm

standard deviation, over three seeds. All datasets are trained with 3 epochs (ARC-Challenge, StrategyQA; config a’), (ASDiv; config c’). Under this 3-epoch configuration, reducing the ARC FO learning rate from

10^{-5}

5\times 10^{-6}

improves accuracy from

31.71\pm 8.11

55.12\pm 0.60

. The ASDiv FO result uses the third-epoch checkpoint of the matched FO runs.

$\boldsymbol{\beta}$	Method	Selected-step frequency (%)				Median $\boldsymbol{s_{t}/R}$	Fallback frequency (%)
$\boldsymbol{\beta}$	Method	$s_{t}=0$	$0<s_{t}<R$	$0.01<s_{t}/R<0.99$	$s_{t}=R$	Median $\boldsymbol{s_{t}/R}$	Fallback frequency (%)
$1$	Taylor 2	1.37	40.97	3.76	57.66	1.00000	0.61
	Taylor 3	0.86	65.83	23.88	33.31	0.00156	0.67
	Padé 2	43.41	0.00	0.00	56.59	1.00000	0.76
	Padé 3	76.27	3.45	2.55	20.27	0.00000	12.49
$3$	Taylor 2	5.04	30.37	15.87	64.59	1.00000	$<0.01$

Appendix

Table 13: Distribution of ZFO-selected steps over the search interval

[0,R]

. Frequencies are macro-averaged across runs. Exact and 1%-trimmed interior selections are reported separately, and median steps are normalized by

R

. For Padé 3 , fallback frequency denotes activation of the quadratic safeguard. The largest interior frequency for each value of

\beta

is bold.

Explore similar work

Aug 16, 2026cs.LG

SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces

Zeroth-order (ZO) optimization with SGD in random subspaces enables memory-efficient fine-tuning of large language models without backpropagation. However, high gradient estimation noise fundamentally undermines adaptive optimizers like Adam. We propose SubZero+, which achieves practical adaptive ZO optimization through a carefully designed dual low-dimensionality strategy: (i) multi-query forward-difference gradient estimation in periodically refreshed random subspaces to mitigate noise amplification in moment buffers, and (ii) Adam updates with periodic restarts performed directly in low-dimensional space rather than full-parameter space. In experiments, this dual design retains memory overhead comparable to momentum-free ZO methods while achieving stronger optimization performance than the evaluated ZO baselines. Theoretically, in the exact-directional limit,

K

-query averaging preserves conditional unbiasedness, while the coefficient estimator's covariance and mean-squared error, as well as query-induced second-moment inflation, scale exactly as

1/K

. Extensive experiments across SuperGLUE with models from 1.3B to 32B parameters under both full fine-tuning and LoRA schemes demonstrate consistent improvements over competing ZO methods. SubZero+ significantly narrows the performance gap with first-order optimization while preserving ZO's inference-time memory efficiency.

Ziming Yu, Shuyao Xiao, Xingyu Zhao +6

Beijing Normal University · Singapore Management University · Xiaomi Inc. +2

Zeroth-Order Large Language Model Fine-Tuning Subspace Bounded-Memory

Ask PDF

May 5, 2026cs.LG

On Adaptivity in Zeroth-Order Optimization

We investigate the effectiveness of adaptive zeroth-order (ZO) optimization for memory-constrained fine-tuning of large language models (LLMs). Contrary to prior claims, we show that adaptive ZO methods such as ZO-Adam offer no convergence advantage over well-tuned ZO-SGD, while incurring significant memory overhead. Our analysis reveals that in high dimensions, ZO gradients lack coordinate-wise heterogeneity, rendering adaptive mechanisms memory inefficient. Leveraging this insight, we propose MEAZO, a memory-efficient adaptive ZO optimizer that tracks only a single scalar for global step size adaptation. We support our method with theoretical convergence guarantees under standard assumptions. Experiments across multiple LLM families and tasks demonstrate that MEAZO matches ZO-Adam's performance with the memory footprint of ZO-SGD. Additional experiments on synthetic quadratic problems and LLM fine-tuning further demonstrate MEAZO's enhanced robustness to step size choices, particularly in grouped or block-structured optimization settings.

Hassan Dbouk, Nidham Gazagnadou, Matthias Reisser +1

Qualcomm AI Research

Zeroth-Order Optimization Stochastic Gradient Descent

Ask PDF

Apr 20, 2026cs.LG

Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling

Zeroth-Order optimization presents a promising memory-efficient paradigm for fine-tuning Large Language Models by relying solely on forward passes. However, its practical adoption is severely constrained by slow wall-clock convergence and high estimation variance. In this work, we dissect the runtime characteristics of ZO algorithms and identify a critical system bottleneck where the generation of perturbations and parameter updates accounts for over 40% of the training latency. We argue that the standard uniform exploration strategy is fundamentally flawed as it fails to account for the heterogeneous sensitivity of layers in deep networks, resulting in computationally wasteful blind searches. To address this structural mismatch, we propose AdaLeZO, an Adaptive Layer-wise ZO optimization framework. By formulating the layer selection process as a non-stationary Multi-Armed Bandit problem, AdaLeZO dynamically allocates the limited perturbation budget to the most sensitive parameters. We further introduce an Inverse Probability Weighting mechanism based on sampling with replacement, which guarantees unbiased gradient estimation while effectively acting as a temporal denoiser to reduce variance. Extensive experiments on LLaMA and OPT models ranging from 6.7B to 30B parameters demonstrate that AdaLeZO achieves 1.7x to 3.0x wall-clock acceleration compared to state-of-the-art methods. Crucially, AdaLeZO functions as a universal plug-and-play module that seamlessly enhances the efficiency of existing ZO optimizers without incurring additional memory overhead.

Fei Wang, Li Shen, Liang Ding +3

South China University of Technology · Shenzhen Campus of Sun Yat-sen University · University of Sydney +2

Zeroth-Order Optimization Zeroth-Order

Ask PDF