Fine-tuning large language models at scale is increasingly limited by the memory cost of backpropagation. Zeroth-Order (ZO) optimizers reduce this burden by relying only on forward function evaluations, but they still exhibit a substantial performance gap to first-order (FO) optimizers such as Adam. We argue that this gap does not arise solely from the absence of explicit gradients: FO fine-tuning also benefits from informative low-rank optimization geometry, large-scale signal aggregation, and well-conditioned update directions. Motivated by this, we introduce StructZO, a structured Zeroth-Order optimization framework for memory-efficient LLM fine-tuning. StructZO performs large-query ZO estimation within an activation-induced informative subspace and applies spectral control to the resulting reduced-space update before mapping it back to the original parameters. This lets ZO aggregate useful directional signals in a compact, geometry-aware space while avoiding the activation and optimizer-state memory costs of backpropagation. We show theoretically that, when the induced subspace captures concentrated optimization-relevant directions, StructZO reduces the effective ZO estimation dimension from mn to mr, with the remaining projection error governed by the spectral tail of the activation Gram matrix. Empirically, across OPT-series and Llama3.1-8B on diverse downstreamtasks, StructZO improves over the strongest existing ZO baselines by up to 8.7 relative on average and up to 26.9 relative on individual tasks, while remaining within 1.5 of FO-Adam on average. These results suggest that structured and spectrally controlled ZO updates can substantially narrow the gap between ZO and FO optimization.
Figures & tables
Figure 1: Empirical geometry diagnostics of StructZO on SST-2, OPT1.3B. (a) Subspace alignment r1∥P⊤Vr∥F2 between the constructed basis P and the top- r right singular subspace Vr of the true gradient G . (b) Projected gradient energy ∥GPP⊤∥F2/∥G∥F2 . (c) Cosine similarity between the pre-spectral estimator and the raw gradient G . (d) Cosine similarity between the final spectrally shaped update and msign(G) . The -R suffix indicates that a random subspace is used to replace the original informative one, and the -S suffix indicates that the Muon-style approach is not used.
Model
Method
SST-2
RTE
CB
BoolQ
WiC
MultiRC
COPA
ReCoRD
DROP
AVG
Task type
classification
multiple choice
generation
OPT-1.3B
MeZO
88.8
66.4
67.8
63.4
57.9
56.2
73.0
70.7
22.4
63.0
LOZO
89.4
61.3
67.8
60.8
55.1
58.6
73.0
71.2
21.9
62.1
HiZOO
89.6
61.7
67.8
63.1
60.2
58.9
73.0
71.8
22.5
63.2
StructZO
92.4
69.0
71.4
68.5
62.6
63.9
74.0
71.8
26.7
66.7
FO-Adam
93.0
70.3
71.4
69.4
63.0
64.2
74.0
71.7
27.7
67.2
Table 1: Evaluation on OPT-1.3B and Llama3.1-8B using 1,024 training and 512 validation samples per task (for CB and COPA we reserve 100 validation samples and use the remainder for training). With the exception of the DROP task, where F1 is used, all other figures refer to accuracy (%). Each model block lists zeroth-order (ZO) methods first, followed by the first-order (FO) baseline separated by a rule. Bold entries indicate the best score among ZO methods; FO rows are provided for reference.
Model
Method
SST-2
RTE
BoolQ
WiC
AVG
OPT-13B
MeZO
91.3
67.5
68.0
59.5
71.6
LOZO
91.7
70.4
71.9
60.8
73.7
HiZOO-L
92.1
68.2
66.6
59.4
71.6
StructZO
93.7
72.3
78.8
64.0
77.2
OPT-30B
MeZO
90.7
67.1
67.4
56.3
70.4
LOZO
92.0
65.3
72.3
57.2
71.7
Table 2: Full-parameter ZO fine-tuning results on OPT-13B and OPT-30B.
Figure 2: Training loss versus forward passes for (a) BoolQ on OPT1.3B and (b) CB on Llama3.1-8B, comparing MeZO, LOZO ,HiZOO(-L) and StructZO.
Model
Method
SST-2
RTE
CB
BoolQ
WiC
MultiRC
COPA
ReCoRD
DROP
AVG
Task type
classification
multiple choice
generation
OPT-1.3B
StructZO-R
91.7
59.5
71.4
64.3
56.1
59.0
74.0
72.6
22.8
63.5
StructZO-S
91.8
62.0
67.8
63.2
57.5
60.2
72.0
72.0
25.3
63.5
StructZO
92.4
69.0
71.4
68.5
62.6
63.9
74.0
71.8
26.7
66.7
Llama3.1-8B
StructZO-R
88.3
66.7
73.2
77.7
62.5
77.1
90.0
86.5
62.4
76.0
StructZO-S
92.8
76.1
75.0
84.0
61.5
73.0
90.0
86.4
57.3
77.3
Table 3: Ablation study of StructZO on OPT-1.3B and Llama3.1-8B across classification, multiple-choice, and generation tasks. StructZO-R replaces the activation-induced subspace with a random subspace, while StructZO-S replaces Muon-style spectral shaping with SGD-style updates.
Figure 3: Training loss versus forward passes for (a) BoolQ on OPT1.3B and (b) CB on Llama3.1-8B, comparing different query number q .
Figure 7
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Experiment
Hyperparameters
Values
MeZO
Batch size
16
Learning rate
{1e−6,1e−7}
μ
1e−3
HiZOO
Batch size
16
Learning rate
{1e−6,1e−7}
μ
1e−3
Appendix
Table 5: The hyperparameter grids for OPT-1.3B and Llama3.1-8B; the weight decay is set to zero.
Model
MeZO
LOZO
HiZOO-L
StructZO
FO-Adam
OPT1.3B
4.13
4.08
4.61
4.29
27.14
Llama3.1-8B
20.55
19.59
23.35
20.8
85.29
OPT13B
27.27
26.91
29.49
28.33
316
OPT30B
60.98
60.08
64.64
63.67
633
Appendix
Table 6: Peak GPU memory usage (GiB) on MultiRC with batch size 1 and an average sequence length of approximately 400 tokens. HiZOO-L denotes the low-rank version of HiZOO. FO-Adam results for OPT-13B and OPT-30B are taken from MeZO [ 9 ] .
Model
Task
Method
Time / step (s)
Training steps
Total time (h)
OPT1.3B
SST-2
MeZO
0.146
20K
0.811
LOZO
0.145
20K
0.804
HiZOO
0.290
20K
1.613
StructZO
6.890
400
0.766
RTE
MeZO
0.358
20K
1.99
LOZO
0.360
20K
2.00
Appendix
Table 7: Detailed wall-clock comparison on OPT-1.3B and Llama3.1-8B. For MeZO, LOZO, and HiZOO/HiZOO-L, we report the average time per optimization step measured over the first 100 steps and estimate the total training time using 20K steps. For StructZO, we report the average time per optimization step measured over the first 10 steps and estimate the total training time using 400 steps, matching the main experimental setting with q=100 .
Model
Task
Time / Subspace construction P (%)
Time / ZO forward (%)
Time / ZO update (%)
OPT1.3B
SST-2
5.43
83.91
10.66
RTE
6.37
89.55
4.08
BoolQ
6.25
91.15
2.60
Llama3.1-8B
SST-2
4.95
92.77
2.28
RTE
6.03
93.26
0.71
BoolQ
6.20
93.32
0.48
Appendix
Table 8: Component-wise wall-clock time breakdown of StructZO. The ZO forward evaluations dominate the per-step cost, while constructing the activation-induced subspace P and applying the final ZO update introduce only modest overhead.
Model
Task
r=16
r=32
r=64
r=128
r=256
OPT1.3B
RTE
63.2
67.8
69.0
66.1
68.9
Llama3.1-8B
SST-2
94.0
93.8
93.7
93.8
92.8
Appendix
Table 9: Performance of the reduced-space rank r in StructZO.
Model
Task
q=4
q=10
q=20
q=50
q=100
q=200
OPT1.3B
BoolQ
63.5
62.4
65.3
66.9
68.5
66.1
Llama3.1-8B
CB
64.3
66.1
69.7
80.4
92.9
78.6
Appendix
Table 10: Performance of StructZO with different query numbers q on OPT-1.3B BoolQ and Llama3.1-8B CB. All results use the same StructZO pipeline while varying the number of ZO queries aggregated at each optimization step.
Zeroth-order optimization uses function values or comparisons, rather than gradients, to optimize objectives when differentiation is unavailable or costly.
Zeroth-order (ZO) optimization with SGD in random subspaces enables memory-efficient fine-tuning of large language models without backpropagation. However, high gradient estimation noise fundamentally undermines adaptive optimizers like Adam. We propose SubZero+, which achieves practical adaptive ZO optimization through a carefully designed dual low-dimensionality strategy: (i) multi-query forward-difference gradient estimation in periodically refreshed random subspaces to mitigate noise amplification in moment buffers, and (ii) Adam updates with periodic restarts performed directly in low-dimensional space rather than full-parameter space. In experiments, this dual design retains memory overhead comparable to momentum-free ZO methods while achieving stronger optimization performance than the evaluated ZO baselines. Theoretically, in the exact-directional limit, K-query averaging preserves conditional unbiasedness, while the coefficient estimator's covariance and mean-squared error, as well as query-induced second-moment inflation, scale exactly as 1/K. Extensive experiments across SuperGLUE with models from 1.3B to 32B parameters under both full fine-tuning and LoRA schemes demonstrate consistent improvements over competing ZO methods. SubZero+ significantly narrows the performance gap with first-order optimization while preserving ZO's inference-time memory efficiency.
Ziming Yu, Shuyao Xiao, Xingyu Zhao +6
Beijing Normal University · Singapore Management University · Xiaomi Inc. +2
Zeroth-order (ZO) optimization offers a memory-efficient alternative for LLM fine-tuning by estimating updates only from forward evaluations of perturbed parameters, without backpropagation or activation storage. However, in billion-parameter LLMs, isotropic perturbations often waste many forward evaluations on weakly informative directions. To make these evaluations more informative, existing ZO methods restrict perturbations to low-dimensional subspaces. Yet the quality of these subspaces is critical: overly compressed or poorly maintained spaces can miss useful update directions. To obtain a high-quality subspace for ZO updates, this paper proposes AIM-ZO, a ZO fine-tuning method based on Activation-Informed Subspace Maintenance. AIM-ZO uses forward activations as local directional information and continuously integrates them into a broad, evolving subspace over training. To access broader gradient-relevant structure while keeping individual perturbations low-dimensional, AIM-ZO activates only a smaller set of shared and sampled directions, decoupling the maintained width from the active width. We evaluate AIM-ZO across 5 LLMs and 11 downstream tasks under matched forward-evaluation budgets; its six-task average exceeds the strongest fully evaluated ZO baseline by 1.26 percentage points on OPT-2.7B and MeZO by 2.85 percentage points on OPT-30B. Our code is available at https://github.com/EkkoXy/AIM-ZO
Yue Xie, Zhi Zheng, Yunpeng Ba +5
Southern University of Science and Technology · National University of Singapore · Huawei Technologies Ltd. +1
Fine-tuning large language models (LLMs) with backpropagation achieves high performance but incurs substantial memory overhead, limiting scalability on resource-constrained hardware. Zeroth-order (ZO) optimization provides a memory-efficient alternative by relying solely on forward passes, yet it typically suffers from slow or unstable convergence due to high-variance gradient estimates. Sparse ZO updates partially address this issue by perturbing only a subset of parameters, but their effectiveness hinges on selecting informative parameters, which is challenging in ZO optimization because each query yields only scalar feedback. We propose Adaptive Curvature-Guided Sparse Zeroth-Order Optimization (CurvZO), which tracks curvature signals online from scalar ZO feedback and leverages these signals to construct a parameter-wise sampling distribution for selecting coordinates at each update, reducing the variance of the sparse ZO gradient estimator. Moreover, CurvZO dynamically adapts the perturbation budget to the evolving curvature signal distribution, yielding sparse ZO updates that remain both focused and sufficiently exploratory. Extensive experiments on OPT and Llama across diverse NLP tasks show that CurvZO consistently improves fine-tuning performance and reduces training time over ZO baselines. It improves accuracy by up to 4.4 points and achieves up to a 2× speedup, while preserving memory efficiency.
Shuo Wang, Ziyu Chen, Ming Tang
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China. · Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.