Organizations: Michigan State University · University of Minnesota · University of California, Santa Barbara · DAMO Academy, Alibaba Group US · IBM Research, USA
Zeroth-order (ZO) optimization provides a gradient-free alternative to first-order (FO) methods by estimating gradients via finite differences of function evaluations, and has recently emerged as a memory-efficient paradigm for fine-tuning large-scale models by avoiding backpropagation. However, ZO optimization has a fundamental tension between accuracy and query efficiency. In this work, we show that ZO optimization can be substantially improved by unifying two complementary principles: (i) a projection-based subspace view that reduces gradient estimation variance by exploiting the intrinsic low-rank structure of model updates, and (ii) Muon-style spectral optimization that applies gradient orthogonalization to extract informative spectral structure from noisy ZO gradients. These findings form a unified framework of subspace gradient orthogonalization, which we instantiate in a new method, ZO-Muon, admitting a natural interpretation as a low-rank Muon optimizer in the ZO setting. Extensive experiments on large language models (LLMs) and vision transformers (ViTs) demonstrate that ZO-Muon significantly accelerates convergence and achieves a win-win improvement in accuracy and query/runtime efficiency. Notably, compared to the popular MeZO baseline, ZO-Muon requires only 24.7% of the queries to reach the same SST-2 performance for LLM fine-tuning, and improves accuracy by 25.1% on ViT-B fine-tuning on CIFAR-100.
Figures & tables
Figure 1: ZO-Muon achieves faster convergence and better fine-tuning accuracy than the SOTA ZO baselines MeZO ( Malladi et al., 2023 ) and LOZO ( Chen et al., 2025a ) against runtime. (a) OPT-1.3B fine-tuned on the RTE task: training loss ( left ) and test accuracy ( right ) versus runtime, with the cumulative query count indicated by the color bar in the right subplot. (b) ViT-B fine-tuned on CIFAR-10 ( left ) and CIFAR-100 ( right ), shown in the same format as in (a, right).
Figure 2: ZO-Muon yields lower runtime and fewer function queries than all baselines (under comparable GPU memory usage), when reaching the target fine-tuning accuracies: (a) 0.90 on SST-2 and (b) 0.93 on CIFAR-10. Baselines include MeZO ( Malladi et al., 2023 ) , SparseMeZO (S-MeZO) ( Liu et al., 2025b ) , HiZOO ( Zhao et al., 2025b ) , and LOZO ( Chen et al., 2025a ) .
Figure 3: The necessity and advantages of Subspace RGE, illustrated by fine-tuning OPT-1.3B on SST-2: (a) The low-rank structure of model components during FO training with Adam across steps; (b) Performance comparison of Subspace RGE–based MeZO ( i.e. , Subspace-MeZO) with the SOTA baselines SparseMeZO ( i.e. , S-MeZO) and MeZO; And (c) accuracy vs. runtime.
Figure 4: Training (a) and testing (b) performance comparison of the proposed Subspace RGE–based MeZO ( i.e. , Subspace-MeZO) with existing low-rank perturbation–based ZO baselines, LOZO ( Chen et al., 2025a ) and SubZero ( Yu et al., 2025 ) , for fine-tuning OPT-13B on SQuAD whose performance is measured by F1 (%).
Figure 5: Demonstration of the ineffectiveness of ZO-Muon-V0 vs. MeZO and the effectiveness of ZO-Muon through fine-tuning OPT-13B on SST-2.
Figure 6: Comparison of ZO-Muon and Subspace-MeZO for fine-tuning OPT-1.3B on SST-2 and SQuAD with r∈{16,64,256} . (Left) Fine-tuning accuracy (shown by cell color and numeric value) for different r . ZO-Muon consistently outperforms Subspace-MeZO, with r=64 achieving the best performance. (Right) Training loss curves of ZO-Muon under different projection ranks for fine-tuning OPT-1.3B on SQuAD.
Method
LLM Fine-tuning
ViT Fine-tuning
SST-2
RTE
CB
BoolQ
WiC
SQuAD
CIFAR-10
CIFAR-100
LlaMA3-8B
ViT-B
{\color[rgb]{1,0.5,0}\bullet} Adam
96.0
92.0
92.0
86.6
72.6
90.4
98.9
91.2
{\color[rgb]{1,0.5,0}\bullet} LoRA
95.0
80.9
73.2
86.4
70.7
89.4
98.3
86.4
MeZO
92.7
74.4
69.6
76.7
57.8
86.7
93.5
64.5
S-MeZO
92.1
69.7
69.6
80.5
56.9
87.5
93.7
64.3
Table 1: ZO-Muon outperforms the ZO baselines by a non-negligible margin, demonstrated by fine-tuning LLMs (LlaMA3-8B and OPT-13B) on SuperGLUE benchmark and fine-tuning ViTs (ViT-B and ViT-L) on CIFAR datasets. FO methods (marked by {\color[rgb]{1,0.5,0}\bullet} ) include Adam and LoRA. The best ZO fine-tuning result for each task is highlighted in bold .
Method
Steps ↓
Queries ↓
Runtime ↓
Memory ↓
GPU s ↓
Accuracy ↑
{\color[rgb]{1,0.5,0}\bullet} Adam
625
625
10 min
4×63.7 GB
4×A100
95.3
{\color[rgb]{1,0.5,0}\bullet} LoRA
625
625
6 min
2×52.7 GB
2×A100
94.8
MeZO
20k
40k
3 h 37 min
28.2 GB
1×A100
91.4
S-MeZO
20k
40k
6 h 46 min
30.1 GB
1×A100
90.4
HiZOO
13.3k
40k
4 h 46 min
55.4 GB
1×A100
92.1
LOZO
20k
40k
3 h 19 min
27.6 GB
1×A100
91.7
Table 2: Efficiency comparison in terms of the number of optimization steps, number of queries, runtime, memory cost, GPU usage, and accuracy for fine-tuning OPT-13B on SST-2. Note that the total query budget is fixed to 40k for all ZO methods.
Figure 7: Runtime and accuracy comparison of ZO-Muon using NS and SVD for fine-tuning OPT-1.3B/13B on SST-2 under the same query budget (5,000).
Figure 8: Accuracy and GPU memory comparison under varying batch sizes. ZO-Muon vs. baselines on ViT-B fine-tuning for CIFAR-100 with a fixed query budget.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Fine-tune accuracy and runtime of full space gradient orthogonalization methods JAGUAR-Muon and ZO-Muon-V0 compared to MeZO and ZO-Muon, demonstrated by fine-tuning OPT-1.3B on SST-2.
Figure A2: Fine-tuning accuracy vs. query cost for (a) ZO-Muon and (b) Subspace-MeZO when fine-tuning ViT-B on CIFAR-100, with marker color indicating the used iteration number. The per-iteration query budget is set to Nq∈{1,4,16} . (a) ZO-Muon benefits from increasing Nq , with a notable performance gain when moving from Nq=1 to Nq=4 . (b) Under the same setting, Subspace-MeZO exhibits a multi-query paradox, where performance degrades when Nq>1 .
Methods
Hyperparameters
Values
LLM
ViT
MeZO
Learning rate
{1e−7,1e−6} or {1e−7,5e−7,1e−6} for SQuAD
{1e−5,1e−4}
Train steps
20k
20k
S-MeZO
Learning rate
{1e−7,1e−6} or {1e−7,5e−7,1e−6} for SQuAD
{1e−5,1e−4}
Sparsity
0.8
0.8
Train steps
20k
20k
Appendix
Table A1: Hyperparameter settings of ZO and FO methods used for fine-tuning LLMs and ViTs.
Figure A3: ZO-Muon achieves faster convergence rate compared to MeZO and LOZO, demonstrated through fine-tuning OPT-13B on SST-2, SQuAD, RTE and BoolQ.
Method
Gemma2-2B Fine-tuning
SST-2
RTE
CB
BoolQ
WiC
SQuAD
{\color[rgb]{1,0.5,0}\bullet} Adam
94.0
84.0
84.0
90.0
72.0
89.4
{\color[rgb]{1,0.5,0}\bullet} LoRA
91.6
57.8
55.4
75.2
56.1
86.9
MeZO
91.7
53.0
60.7
66.1
57.1
74.1
S-MeZO
92.7
53.8
53.6
63.9
55.8
76.1
HiZOO
92.1
53.1
60.7
66.6
55.2
73.4
Appendix
Table A2: Performance of ZO-Muon and baselines on Gemma2-2B under the same settings as Tab. 1 .
Figure A4: Performance of ZO-Muon with different projection matrix resampling interval v , demonstrated by (a) fine-tuning OPT-1.3B on SST-2 and (b) fine-tuning ViT-B on CIFAR-100.
Method
Sampling Strategy
LlaMA3-8B Fine-tuning
ViT-B Fine-tuning
SST-2
RTE
SQuAD
CIFAR-10
CIFAR-100
Acc. (%) ↑
Mem (MiB) ↓
Acc. (%) ↑
Mem (MiB) ↓
F1 (%) ↑
Mem (MiB) ↓
Acc. (%) ↑
Mem (MiB) ↓
Acc. (%) ↑
Mem (MiB) ↓
Subspace-MeZO
Random
92.3
19058
68.6
28242
84.5
32102
94.2
1634
58.2
4086
Sketching
92.3
34814
67.5
38874
87.2
44734
81.2
1898
29.1
4088
ZO-Muon
Random
94.3
19176
81.2
27224
88.2
30718
94.9
1686
72.6
4410
Sketching
93.8
35268
78.3
44304
88.3
45248
95.3
1950
59.8
4412
Appendix
Table A3: Performance (evaluated by accuracy and F1) and GPU memory (short for “Mem”) of Subspace-MeZO and ZO-Muon with different projection matrix sampling strategies.
Figure A5: Fine-tuning accuracy and runtime of ZO-Muon using SVD vs. NS for gradient orthogonalization under the same query budget. Runtime is measured in seconds (s).
Figure A6: Fine-tuning accuracy and GPU memory comparison under varying batch sizes. ZO-Muon versus baselines on LlaMA3-8B fine-tuning for RTE with the same number of iterations.
Zeroth-order optimization uses function values or comparisons, rather than gradients, to optimize objectives when differentiation is unavailable or costly.
Zeroth-order (ZO) optimization has become increasingly popular and important in fine-tuning large language models (LLMs), especially on edge devices due to its ability to adjust the model to local data without the need for memory-intensive back-propagation. Recent works try to reduce ZO variance through low-dimensional subspace search, but subspace restriction alone leaves key optimization geometry under-exploited, motivating additional acceleration. In this work, we focus on the hidden layer training problem in which spectral optimizers like Muon outperform AdamW due to its ability to exploit weak spectral directions by orthogonalization. However, we have discovered that unlike in the first-order setting, full orthogonalization works poorly in the ZO setting since the gradient estimates are highly noisy and unreliable. To address this issue, we propose applying partial spectral orthogonalization to accelerate ZO optimization. To do so, we replace the iconic Newton-Schulz procedure in Muon with the faster, more concentrated power-iteration method so that it only amplifies dominant spectral directions. Furthermore, to improve the efficiency and generalization of the algorithm, we adopted a streaming variant of power-iteration that requires low variance in gradients, which was achieved through constraining our search inside a subspace obtained through the projection of momentum, echoing recent advances. Experiments on LLM fine-tuning show that our method can achieve from 1.5x to 4x the convergence speed of ZO-Muon, the current SOTA algorithm, across SuperGlue datasets in the OPT-13B model. Across different models, we also reach competitive final accuracies with less time in most cases compared with strong ZO baselines such as MeZO, LOZO and ZO-Muon. Code is available at https://github.com/MOFA-LAB/ZO-MOPI.git.
Jiahe Chen, Ziye Ma
Department of Computer Science City University of Hong Kong
Zeroth-order (ZO) optimization with SGD in random subspaces enables memory-efficient fine-tuning of large language models without backpropagation. However, high gradient estimation noise fundamentally undermines adaptive optimizers like Adam. We propose SubZero+, which achieves practical adaptive ZO optimization through a carefully designed dual low-dimensionality strategy: (i) multi-query forward-difference gradient estimation in periodically refreshed random subspaces to mitigate noise amplification in moment buffers, and (ii) Adam updates with periodic restarts performed directly in low-dimensional space rather than full-parameter space. In experiments, this dual design retains memory overhead comparable to momentum-free ZO methods while achieving stronger optimization performance than the evaluated ZO baselines. Theoretically, in the exact-directional limit, K-query averaging preserves conditional unbiasedness, while the coefficient estimator's covariance and mean-squared error, as well as query-induced second-moment inflation, scale exactly as 1/K. Extensive experiments across SuperGLUE with models from 1.3B to 32B parameters under both full fine-tuning and LoRA schemes demonstrate consistent improvements over competing ZO methods. SubZero+ significantly narrows the performance gap with first-order optimization while preserving ZO's inference-time memory efficiency.
Ziming Yu, Shuyao Xiao, Xingyu Zhao +6
Beijing Normal University · Singapore Management University · Xiaomi Inc. +2
Zeroth-order (ZO) optimization is a memory-efficient alternative to backpropagation for fine-tuning large language models, but its deployment is limited by the high variance of gradient estimation. We propose GRZO, a Group-Relative Zeroth-Order optimizer that draws one pseudo-independent perturbation per mini-batch example and aggregates the per-example losses through group-relative normalization, raising the effective gradient-direction count from one to the batch size at no additional forward cost while preserving inference-level memory. We prove that GRZO is directionally unbiased with variance shrinking proportionally to the batch size, yielding a tighter nonconvex convergence bound than MeZO. Across RoBERTa-large, Llama3-8B, and OPT-13B over multiple tasks, GRZO improves average accuracy on Llama3-8B by +3.0 over MeZO at 23% lower peak GPU memory; as a drop-in replacement for the MeZO core, it lifts sparse, low-rank, and quantized ZO variants by +6.0 on average.