Zeroth-Order Riemannian Optimization on Fixed-Rank Manifolds for LLM Fine-Tuning
Authors: Xinjue Wang, Xiuheng Wang, Yejun Zhang, Sergiy A. Vorobyov, Esa Ollila
Organizations: Department of Information and Communications Engineering, Aalto University · School of Future Technology, South China University of Technology · Department of Computer Science, Aalto University
Zeroth-order (ZO) optimization estimates descent directions from function values alone, which removes the activation storage required by backpropagation and makes it attractive for memory-efficient fine-tuning of large language models (LLMs). Existing low-rank ZO methods either parameterize the additive update through Euclidean factors with a non-unique representation, or restrict each finite-difference direction to be low-rank. In this work, we propose to constrain each additive update block to a fixed-rank matrix manifold while keeping the pretrained weights frozen, so the learned update remains rank-constrained throughout fine-tuning. On this geometry, we build a Riemannian ZO gradient estimator that projects a random matrix onto the tangent space, normalizes it to correct the scale mismatch induced by projection, and further propose Zeroth-order Riemannian Gradient Descent (ZO-RGD) and its smoothed variant, ZO-RGD-S. We show that the ambient finite difference matches its retracted counterpart, the normalized estimator recovers the Riemannian gradient in expectation, and ZO-RGD converges under trajectory-regularity conditions. Experiments on LLM fine-tuning tasks show that the proposed algorithms are competitive with strong forward-only baselines across model families and tasks, with gains in several settings.
Figures & tables
Figure 1: Two loss evaluations determine a fixed-rank ZO-RGD update. The orthogonal projection onto TΔWtMr gives Zt=PTΔWtMr(Zt) , and normalization gives Zt=mnZt/∥Zt∥F . The two points ΔWt±ϵZt along these directions jointly perturb all selected blocks to obtain F± . Their difference gives the directional derivative s^t and thus the zeroth-order Riemannian gradient Gt . Truncated SVD retracts ΔWt−ηGt back to the rank- r manifold.
Method
SST-2
RTE
CB
BoolQ
WSC
WiC
MultiRC
COPA
ReCoRD
SQuAD
DROP
Zero-shot
58.8
59.6
46.4
59.0
38.5
55.0
46.9
80.0
81.0
46.2
14.6
ICL
87.0
62.1
57.1
66.9
39.4
50.5
53.1
87.0
82.3
75.9
29.5
MeZO
91.3
68.2
66.1
68.1
61.5
59.4
59.4
88.0
81.3
81.8
31.3
MeZO-LoRA
89.6
67.9
67.8
73.5
63.5
60.2
61.3
84.0
81.5
82.1
31.3
LOZO
91.7
70.4
69.6
71.9
63.5
60.8
63.0
89.0
81.3
84.9
30.7
ZO-RGD
92.43
72.20
66.07
71.40
59.62
62.54
57.80
87.00
Fail
87.19
30.02
Table 1: OPT-13B main comparison on the LOZO eleven-task suite. The results of Zero-shot, ICL, MeZO, MeZO-LoRA, and LOZO are from Chen et al. (2025) ; ZO-RGD and ZO-RGD-S are evaluated under the setup described above, with each cell selected by the development metric within that variant’s 20,000 -step sweep cells. Best and second-best results in each column are shown in bold and underlined , respectively. Fail denotes that the ZO-RGD’s ReCoRD run diverged and produced no valid evaluation (see Appendix D.18 for details).
Method
SST-2
RTE
BoolQ
WSC
WiC
SQuAD
Zero-shot
56.7
52.0
39.1
38.5
50.2
46.5
ICL
81.9
66.8
66.2
56.7
51.3
78.0
MeZO
90.7
64.3
68.2
63.5
56.3
86.1
LOZO
92.8
65.3
72.3
64.4
57.2
85.6
ZO-RGD
92.89
68.59
70.50
63.46
57.68
84.89
ZO-RGD-S
94.04
71.12
72.30
61.54
58.15
85.11
Table 2: OPT-30B scaling comparison on the LOZO task subset. Zero-shot, ICL, MeZO, and LOZO are cited from Chen et al. (2025) ; ZO-RGD and ZO-RGD-S are evaluated with the same training budget, with checkpoint-selection details reported in Appendix D.1 . Best and second-best results in each column are shown in bold and underlined , respectively. SQuAD uses F1, and the other columns use the OPT task metrics.
OPT-13B
OPT-30B
Method
Passes/update
RTE
SQuAD
BoolQ
MeZO
2F
60.4±1.5
82.8±1.0
66.80
HiZOO †
4F
62.1±2.6
NR
NR
LOZO
2F
62.1±0.6
82.2±0.7
69.40
ZO-RGD
2F
61.9±2.9
83.3±1.0
72.70
ZO-RGD-S
2F
69.4±3.3
84.9±1.3
72.50
Table 3: Controlled OPT comparison with fixed report-seed hyperparameters and symmetric development-loss checkpoint selection. OPT-13B: mean ± sd over three seeds (RTE: accuracy; SQuAD: F1). OPT-30B BoolQ: single-seed accuracy at learning rate 10−7 . ZO-RGD uses r=4 , ZO-RGD-S uses r=2 . Passes/update counts forward (F) and backward (B) evaluations; bold marks the best forward-only result. NR denotes a matched result not reported in this comparison; OOM denotes an out-of-memory failure.
Llama-3.1-8B
Qwen2.5-7B
Method
RTE
BoolQ
SQuAD
GSM8K
RTE
BoolQ
SQuAD
GSM8K
LOZO
56.9±7.1
61.6±2.7
86.8±2.2
31.2±2.4
86.6±1.9
80.7±0.6
87.0±0.8
74.2±2.8
ZO-RGD
62.5±3.2
66.9±1.5
87.1±0.8
37.7±5.8
88.3±0.8
82.4±1.2
87.9±0.3
73.2±2.8
ZO-RGD-S
62.9±8.4
85.8±1.1
85.5±1.0
34.0±1.0
88.3±1.0
82.3±0.5
87.6±0.2
70.3±2.8
Table 4: Modern-model results (mean ± sd, three seeds, %; RTE/BoolQ: accuracy; SQuAD: F1; GSM8K: exact match on n=200 ).
Finite-difference rule
SST-2
Tangent-projected, normalized
92.4±0.2
Retracted evaluation points
89.6±0.9
Ambient rank- r outer product
54.3±1.1
Outer product + LR compensation
64.7±3.9
Table 5: OPT-1.3B geometry controls (%; mean ± sd, three paired seeds). Left: finite-difference geometry on SST-2. Right: manifold versus BA-factor ZO with matched budgets (accuracy; SQuAD: F1). Learning rates are selected separately for each panel; compare within a panel. Right-panel rates use a shared 2,000 -step development pilot: 10−6/10−7/10−6 for the manifold arm on SST-2/RTE/SQuAD and 10−4 for BA-factor.
Method
Passes/update
Time/step (s)
Peak reserved (GB)
MeZO
2F
2.03
39.6
LOZO
2F
1.93
39.4
ZO-RGD
2F
3.51
39.4
ZO-RGD-S
2F
3.54
39.4
LoRA+AdamW
1F+1B
OOM
126.2 (H200, RTE)
Table 6: Measured cost at OPT-13B SQuAD, averaged over the two A100-80G hardware repeats in Table 23 . LoRA+AdamW exhausts the A100 on SQuAD (batch 16, no activation checkpointing); its 126.2 GB entry is a separate three-run RTE mean on H200.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Panel
Arms
Rank
LR selection
Steps × batch
Report seeds
OPT-13B controlled
MeZO, LOZO, ZO-RGD, ZO-RGD-S, LoRA+AdamW
−,1,4,2,8
shared grid, dev loss, seed 0
20,000×16
{1,2,3}
OPT-1.3B breadth / mechanism
6 forward-only methods / controls
published
shared grid, dev loss, seed 0
20,000×16
{1,2,3} (breadth: single fresh seed)
Modern panel
ZO-RGD, ZO-RGD-S, LOZO
4,2,1
per-method dev-only pilot, seed 7
20,000×16
{0,1,2}
Appendix
Table 7: Unified-protocol configuration card. All methods use two forward passes per step; ranks are each method’s published setting; the shared grid is the MeZO grid, verbatim; selection is development-loss only (grid/pilot seed disjoint from report seeds).
Method
Estimator setting
Tuned over
Deviations from official
MeZO
full-parameter ZO, q=1
shared MeZO grid
none
LOZO
official r=1 , ν=100
shared MeZO grid
none (an earlier internal matrix used ν=50 )
MeZO-LoRA / LOZO+LoRA
LoRA r=8,α=16
official LoRA grid (LR {10−4,5×10−5} , +10−5 for QA; ϵ=10−2 )
none
LoRA+AdamW (first-order ref.)
LoRA r=8,α=16 , AdamW
LR {10−5,5×10−5,10−4} , dev loss
not a forward-only method; reference row only
Appendix
Table 8: Baseline implementation audit for the unified-protocol tables. All baselines run through the same data, prompt, and evaluation path as our methods, from the local baseline implementations, with the deviations column exhaustive.
Method
SST-2
RTE
CB
BoolQ
WSC
WiC
MultiRC
COPA
ReCoRD
SQuAD
DROP
MeZO
87.61
53.43
69.64
62.40
64.42
57.21
56.70
73.00
70.60
73.25
22.82
MeZO-LoRA
90.94
53.07
67.86
62.80
61.54
58.31
55.50
74.00
72.20
76.59
22.48
LOZO
88.88
58.12
66.07
62.20
65.38
56.43
56.10
74.00
70.20
77.00
22.23
LOZO+LoRA
89.45
54.87
69.64
64.00
61.54
58.46
56.10
74.00
71.70
74.76
22.39
ZO-RGD
90.83
59.57
71.43
64.70
61.54
58.31
57.20
75.00
71.80
76.61
24.51
ZO-RGD-S
90.48
60.65
73.21
64.60
61.54
58.15
58.00
78.00
71.50
77.65
24.64
Appendix
Table 9: OPT-1.3B complete local comparison on the eleven-task suite. Best and second-best results in each column are shown in bold and underlined , respectively.
Method
SST-2
RTE
CB
BoolQ
WSC
WiC
MultiRC
COPA
ReCoRD
SQuAD
DROP
MeZO
89.33
65.34
69.64
64.40
60.58
58.31
60.90
76.00
70.30
75.04
24.85
LOZO
87.61
65.34
67.86
64.70
61.54
60.03
57.20
73.00
70.20
72.71
23.27
MeZO-LoRA
90.02
63.54
69.64
65.00
61.54
58.62
60.30
74.00
70.50
75.84
24.82
LOZO+LoRA
90.71
63.54
71.43
65.20
58.65
58.93
60.10
74.00
71.90
75.84
25.55
ZO-RGD
92.43
65.34
71.43
66.90
55.77
57.84
59.40
79.00
68.60
77.66
26.30
ZO-RGD-S
92.55
63.90
71.43
65.30
59.62
61.60
59.60
70.00
67.70
77.96
25.97
Appendix
Table 10: OPT-1.3B breadth, unified protocol, single fresh seed per cell (%). No bolding: single-seed cells are not ranked. SQuAD/DROP report F1.
Task
neval
Zero-shot
Best trained
Δ best
Δ /sd
SQuAD
1000
27.67
77.65
+49.98
31.6
SST-2
872
53.56
90.94
+37.38
22.1
BoolQ
1000
47.03
64.70
+17.67
11.2
DROP
1000
11.23
24.64
+13.41
8.5
MultiRC
1000
46.00
58.00
+12.00
7.6
CB
56
37.50
73.21
+35.71
5.3
Appendix
Table 11: Trainability floor at OPT-1.3B (%). Δ /sd normalizes the best-case trained gain by evaluation sampling noise; seed-to-seed training variance is not included, so every ratio is optimistic.
RTE (acc)
SQuAD (F1)
Method
s1
s2
s3
s1
s2
s3
MeZO
58.84
61.73
60.65
82.67
81.88
83.77
LOZO
62.45
61.37
62.45
82.98
81.61
82.00
ZO-RGD
62.09
64.62
58.84
84.23
82.21
83.46
ZO-RGD-S
68.95
72.92
66.43
86.08
83.50
85.08
LoRA+AdamW
83.39
80.51
86.28
NR
NR
NR
Appendix
Table 12: Per-seed values (%) for the non-dagger rows of the OPT-13B controlled comparison (Table 3 ), report seeds {1,2,3} . NR: no formal LoRA+AdamW SQuAD seed results; the A100 grid attempt ran out of memory.
ZO-RGD
ZO-RGD-S
LOZO
Model
Task
s0
s1
s2
s0
s1
s2
s0
s1
s2
Llama-3.1-8B
RTE
61.37
66.06
59.93
70.40
53.79
64.62
58.84
49.10
62.82
Llama-3.1-8B
BoolQ
65.70
66.50
68.60
85.50
84.90
87.10
63.00
63.30
58.50
Llama-3.1-8B
SQuAD
87.34
86.19
87.62
84.42
86.44
85.52
88.77
84.39
87.13
Llama-3.1-8B
GSM8K
32.50
44.00
36.50
33.00
35.00
34.00
33.00
28.50
32.00
Qwen2.5-7B
RTE
87.36
88.81
88.81
87.73
89.53
87.73
84.48
88.09
87.36
Appendix
Table 13: Per-seed values (%) for the modern-model panel (Table 4 ), report seeds {0,1,2} .
Task
Method
Learning rate
Score (mean ± sd)
RTE (acc.)
ZO-RGD
10−6
64.50±3.26
ZO-RGD-S
10−6
69.43±3.28
SQuAD (F1)
ZO-RGD
10−7
85.38±0.34
ZO-RGD-S
10−7
84.89±1.30
Appendix
Table 14: Scalar smoothing at fixed rank and learning rate on OPT-13B. Both methods use r=2 and paired seeds {1,2,3} . The rates were selected on the ZO-RGD-S development grid and fixed before the ZO-RGD runs. Scores are mean ± sd (%); all runs completed.
Task
Seed
ZO-RGD
ZO-RGD-S
Difference
RTE
1
67.87
68.95
+1.08
2
61.37
72.92
+11.55
3
64.26
66.43
+2.17
SQuAD
1
85.67
86.08
+0.42
2
85.00
83.50
-1.50
3
85.48
85.08
-0.40
Appendix
Table 15: Matched-rank OPT-13B scores (%). All twelve runs completed. Differences are ZO-RGD-S minus ZO-RGD, calculated before rounding.
Figure 2: OPT-13B RTE development loss against cumulative forward passes at the shared learning rate 10−7 , truncated at the common 40 K-forward budget. Lines are means and bands are ranges over completed runs. HiZOO uses four forward passes per step; the other methods use two.
ZO-RGD SQuAD +3.3 , 3/3 wins, p=0.125 , strict separation; RTE: both variants p>0.18 , no separation.
Appendix
Table 16: Shared-learning-rate core (%; mean ± sd over completed seeds; completed/attempted in parentheses). Paired differences vs. LOZO use only seeds where both methods completed; the sign test is exact and one-sided. “Strict separation” means the worst seed of ours beats the best seed of LOZO on the paired subset.
Method
Rank
LR 10−6
LR 10−7
LOZO
1
63.0
80.9
ZO-RGD
4
65.7
85.5
ZO-RGD-S
2
67.3
85.5
Appendix
Table 17: Llama-3.1-8B BoolQ learning-rate diagnostic (accuracy, %). All entries use seed 0 and 20,000 updates; each method retains its panel rank.
Baselines at ZO-RGD’s working point
SST-2
RTE
SQuAD
MeZO @ ZO-RGD working point
89.3±0.6
62.9±4.2
72.9±0.8
LOZO @ ZO-RGD working point
89.6±1.3
63.8±1.4
73.1±0.9
ZO-RGD (same point; single seed)
92.4
65.3
77.7
Appendix
Table 18: Baselines at ZO-RGD’s working point on OPT-1.3B (%). MeZO and LOZO report mean ± sd over three seeds; ZO-RGD is the single fresh-seed breadth result in Appendix D.4 . This diagnostic uses a different working point from the parameterization comparison in Table 5 .
Task
Setting
Unnormalized
Normalized ref.
SST-2
raw, ϵ=10−3
60.89
90.83
RTE
raw, ϵ=10−3
55.23
59.57
SQuAD (F1) †
raw, ϵ=10−3
0.00
76.61
SST-2
raw, ϵ=1.6×10−2
56.88
90.83
Appendix
Table 19: Normalization ablation on OPT-1.3B. The normalized reference reproduces the historical ZO-RGD r=4 row (Table 9 ). SQuAD reports F1; the other tasks report accuracy. † The SQuAD unnormalized run diverged and was cancelled before completion; intermediate checkpoints at steps 4 K, 8 K, and 12 K all gave F1 0 , and the entry is a truncated run reported at its collapsed value, not a completed evaluation. A rerun cannot repair it: with normalization off, the SQuAD ablation diverges deterministically (first evaluation NaN on all seeds; Appendix D.18 ), and the divergence itself is the ablation conclusion.
Task
Variant
r=2
r=4
Δ
SST-2
ZO-RGD
91.86
90.83
−1.03
SST-2
ZO-RGD-S
90.48
91.86
+1.38
COPA
ZO-RGD
76.00
77.00
+1.00
COPA
ZO-RGD-S
75.00
75.00
−0.00
SQuAD
ZO-RGD-S
77.94
78.47
+0.53
DROP
ZO-RGD-S
24.64
24.35
−0.29
Appendix
Table 20: OPT-1.3B cross-rank auxiliary runs for the two ranks used in the main table. These runs use 20K steps, η=10−7 , and final-step evaluation; they are separate from the main-table checkpoint-selection convention. SQuAD and DROP report F1; the remaining tasks report accuracy. Bold marks the better rank for each available pair.
Axis
Arm
SST-2
—
base ( r=4 , ϵ=10−3 , σinit=10−6 , FP16)
92.55±0.65
rank
r=1 / r=2 / r=8
91.40±0.49 / 92.60±0.24 / 92.37±0.41
ϵ
10−4 / 10−2
91.28±0.00 / 69.84±2.11
init
10−4 / 10−8
92.60±0.08 / 92.20±0.65
dtype
bf16
77.64±0.65
Appendix
Table 21: Stability atlas (SST-2 accuracy, %; mean ± sd over two paired seeds; one factor varied per row).
Model / task
Method
σmin , step 0
σmin , end
max cond.
13B RTE
ZO-RGD
1.013×10−6
0.52 – 0.60
≤1.07
13B RTE
ZO-RGD-S
1.013×10−6
0.63 – 0.72
≤1.07
13B SQuAD
ZO-RGD / -S
1.013×10−6
0.87 – 1.12
≤1.07
30B BoolQ
ZO-RGD / -S
1.013×10−6
1.00 – 1.15
≤1.07
Appendix
Table 22: Manifold-wide σmin and conditioning across the instrumented runs (ranges over completed seeds). All runs start at σmin=1.013×10−6 by construction.
Time / step (s)
Peak reserved memory (GB)
Method
Repeat 1
Repeat 2
Repeat 1
Repeat 2
MeZO
2.025
2.030
39.62
39.62
LOZO
1.929
1.931
39.43
39.43
ZO-RGD
3.516
3.508
39.40
39.40
ZO-RGD-S
3.551
3.525
39.38
39.38
Appendix
Table 23: Repeated OPT-13B SQuAD cost profile. Each repeat runs all four methods back to back for 1,000 steps each in one allocation, on a different A100-80G card, with intermediate evaluation disabled. These are hardware repeats, not training seeds.
Zeroth-order optimization uses function values or comparisons, rather than gradients, to optimize objectives when differentiation is unavailable or costly.
Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs). However, ZO optimization suffers from high variance in high-dimensional parameter spaces. While recent methods restrict perturbations to random low-rank subspaces, these subspaces often misalign with task-specific descent directions. We propose AdaSub-ZO, a novel forward-only fine-tuning framework in learnable low-rank subspaces. Instead of relying on fixed random matrices, AdaSub-ZO iteratively learns layer-wise projection matrices directly from finite-difference feedback to track task-aware descent directions. Furthermore, AdaSub-ZO incorporates a subspace momentum acceleration mechanism to smooth noisy gradient estimates across evolving subspace bases. Extensive experiments on natural language understanding tasks demonstrate that AdaSub-ZO outperforms existing ZO baselines, substantially reducing gradient estimation variance and accelerating convergence.
Chuang Tong, Ling Chen
State Key Laboratory of Blockchain and Data Security, Zhejiang University · College of Computer Science and Technology, Zhejiang University
Zeroth-order (ZO) optimization enables fine-tuning large language models when backpropagation is unavailable or memory-prohibitive, but existing methods often perturb full model weights or randomly constructed low-dimensional subspaces, yielding high-variance estimates and limited performance. We propose ZO-Act, an activation-informed ZO fine-tuning method that restricts perturbations to a fixed low-rank subspace derived from input activations. For each linear layer, ZO-Act computes a small activation basis once at initialization and optimizes only lightweight coefficient matrices using forward-only loss evaluations. This reduces the effective perturbation dimension, exposes explicit trainable variables compatible with momentum-based optimizers such as Adam, and naturally supports quantized LLM fine-tuning by keeping low-bit weights frozen. We analyze ZO-Act as zeroth-order optimization over a restricted coefficient space and show that perturbing the low-dimensional coefficients reduces both the variance-dependent convergence term and the finite-difference error of the ZO estimator, at the cost of a controlled subspace approximation bias that is mitigated by the low-rank structure of LLM activations and gradients. Experiments on Llama-3-8B, OPT-13B, and INT4 Llama-3-8B show consistent gains over strong ZO fine-tuning baselines across language understanding, question answering, and commonsense reasoning.
Xun Dong, Yibo Xu, Naigang Wang +3
University at Albany, SUNY · IBM T. J. Watson Research Center
Fine-tuning large language models at scale is increasingly limited by the memory cost of backpropagation. Zeroth-Order (ZO) optimizers reduce this burden by relying only on forward function evaluations, but they still exhibit a substantial performance gap to first-order (FO) optimizers such as Adam. We argue that this gap does not arise solely from the absence of explicit gradients: FO fine-tuning also benefits from informative low-rank optimization geometry, large-scale signal aggregation, and well-conditioned update directions. Motivated by this, we introduce StructZO, a structured Zeroth-Order optimization framework for memory-efficient LLM fine-tuning. StructZO performs large-query ZO estimation within an activation-induced informative subspace and applies spectral control to the resulting reduced-space update before mapping it back to the original parameters. This lets ZO aggregate useful directional signals in a compact, geometry-aware space while avoiding the activation and optimizer-state memory costs of backpropagation. We show theoretically that, when the induced subspace captures concentrated optimization-relevant directions, StructZO reduces the effective ZO estimation dimension from mn to mr, with the remaining projection error governed by the spectral tail of the activation Gram matrix. Empirically, across OPT-series and Llama3.1-8B on diverse downstreamtasks, StructZO improves over the strongest existing ZO baselines by up to 8.7 relative on average and up to 26.9 relative on individual tasks, while remaining within 1.5 of FO-Adam on average. These results suggest that structured and spectrally controlled ZO updates can substantially narrow the gap between ZO and FO optimization.