StructZO: Structured Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning

Xuanlin Yang; Jia Li; Zhouchen Lin

cs.LGOct 12, 2026

StructZO: Structured Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning

Authors: Xuanlin Yang, Jia Li, Zhouchen Lin

Organizations: Peking University · Beijing Normal University

Abstract

Fine-tuning large language models at scale is increasingly limited by the memory cost of backpropagation. Zeroth-Order (ZO) optimizers reduce this burden by relying only on forward function evaluations, but they still exhibit a substantial performance gap to first-order (FO) optimizers such as Adam. We argue that this gap does not arise solely from the absence of explicit gradients: FO fine-tuning also benefits from informative low-rank optimization geometry, large-scale signal aggregation, and well-conditioned update directions. Motivated by this, we introduce StructZO, a structured Zeroth-Order optimization framework for memory-efficient LLM fine-tuning. StructZO performs large-query ZO estimation within an activation-induced informative subspace and applies spectral control to the resulting reduced-space update before mapping it back to the original parameters. This lets ZO aggregate useful directional signals in a compact, geometry-aware space while avoiding the activation and optimizer-state memory costs of backpropagation. We show theoretically that, when the induced subspace captures concentrated optimization-relevant directions, StructZO reduces the effective ZO estimation dimension from $mn$ to $mr$ , with the remaining projection error governed by the spectral tail of the activation Gram matrix. Empirically, across OPT-series and Llama3.1-8B on diverse downstreamtasks, StructZO improves over the strongest existing ZO baselines by up to $8.7%$ relative on average and up to $26.9%$ relative on individual tasks, while remaining within $1.5%$ of FO-Adam on average. These results suggest that structured and spectrally controlled ZO updates can substantially narrow the gap between ZO and FO optimization.

Figures & tables

Panel of Figure 1 — Figure 1: Empirical geometry diagnostics of StructZO on SST-2, OPT1.3B. (a) Subspace alignment $\frac{1}{r}\|P^{\top}V_{r}\|_{F}^{2}$ between the constructed basis $P$ and the top- $r$ right singular subspace $V_{r}$ of the true gradient $G$ . (b) Projected gradient energy $\|GPP^{\top}\|_{F}^{2}/\|G\|_{F}^{2}$ . (c) Cosine similarity between the pre-spectral estimator and the raw gradient $G$ . (d) Cosine similarity between the final spectrally shaped update and $\operatorname{msign}(G)$ . The -R suffix indicates that a random subspace is used to replace the original informative one, and the -S suffix indicates that the Muon-style approach is not used.

Model	Method	SST-2	RTE	CB	BoolQ	WiC	MultiRC	COPA	ReCoRD	DROP	AVG
Task type		classification						multiple choice		generation
OPT-1.3B	MeZO	88.8	66.4	67.8	63.4	57.9	56.2	73.0	70.7	22.4	63.0
	LOZO	89.4	61.3	67.8	60.8	55.1	58.6	73.0	71.2	21.9	62.1
	HiZOO	89.6	61.7	67.8	63.1	60.2	58.9	73.0	71.8	22.5	63.2
	StructZO	92.4	69.0	71.4	68.5	62.6	63.9	74.0	71.8	26.7	66.7
	FO-Adam	93.0	70.3	71.4	69.4	63.0	64.2	74.0	71.7	27.7	67.2

Table 1: Evaluation on OPT-1.3B and Llama3.1-8B using 1,024 training and 512 validation samples per task (for CB and COPA we reserve 100 validation samples and use the remainder for training). With the exception of the DROP task, where F1 is used, all other figures refer to accuracy (%). Each model block lists zeroth-order (ZO) methods first, followed by the first-order (FO) baseline separated by a rule. Bold entries indicate the best score among ZO methods; FO rows are provided for reference.

Model	Method	SST-2	RTE	BoolQ	WiC	AVG
OPT-13B	MeZO	91.3	67.5	68.0	59.5	71.6
	LOZO	91.7	70.4	71.9	60.8	73.7
	HiZOO-L	92.1	68.2	66.6	59.4	71.6
	StructZO	93.7	72.3	78.8	64.0	77.2
OPT-30B	MeZO	90.7	67.1	67.4	56.3	70.4
	LOZO	92.0	65.3	72.3	57.2	71.7

Table 2: Full-parameter ZO fine-tuning results on OPT-13B and OPT-30B.

Panel of Figure 2 — Figure 2: Training loss versus forward passes for (a) BoolQ on OPT1.3B and (b) CB on Llama3.1-8B, comparing MeZO, LOZO ,HiZOO(-L) and StructZO.

Model	Method	SST-2	RTE	CB	BoolQ	WiC	MultiRC	COPA	ReCoRD	DROP	AVG
Task type		classification						multiple choice		generation
OPT-1.3B	StructZO-R	91.7	59.5	71.4	64.3	56.1	59.0	74.0	72.6	22.8	63.5
	StructZO-S	91.8	62.0	67.8	63.2	57.5	60.2	72.0	72.0	25.3	63.5
	StructZO	92.4	69.0	71.4	68.5	62.6	63.9	74.0	71.8	26.7	66.7
Llama3.1-8B	StructZO-R	88.3	66.7	73.2	77.7	62.5	77.1	90.0	86.5	62.4	76.0
	StructZO-S	92.8	76.1	75.0	84.0	61.5	73.0	90.0	86.4	57.3	77.3

Table 3: Ablation study of StructZO on OPT-1.3B and Llama3.1-8B across classification, multiple-choice, and generation tasks. StructZO-R replaces the activation-induced subspace with a random subspace, while StructZO-S replaces Muon-style spectral shaping with SGD-style updates.

Panel of Figure 3 — Figure 3: Training loss versus forward passes for (a) BoolQ on OPT1.3B and (b) CB on Llama3.1-8B, comparing different query number $q$ .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Hyperparameters	Values
MeZO	Batch size	16
	Learning rate	$\left\{1e-6,1e-7\right\}$
	$\mu$	$1e-3$
HiZOO	Batch size	16
	Learning rate	$\left\{1e-6,1e-7\right\}$
	$\mu$	$1e-3$

Appendix

Table 5: The hyperparameter grids for OPT-1.3B and Llama3.1-8B; the weight decay is set to zero.

Model	MeZO	LOZO	HiZOO-L	StructZO	FO-Adam
OPT1.3B	4.13	4.08	4.61	4.29	27.14
Llama3.1-8B	20.55	19.59	23.35	20.8	85.29
OPT13B	27.27	26.91	29.49	28.33	316
OPT30B	60.98	60.08	64.64	63.67	633

Appendix

Table 6: Peak GPU memory usage (GiB) on MultiRC with batch size 1 and an average sequence length of approximately 400 tokens. HiZOO-L denotes the low-rank version of HiZOO. FO-Adam results for OPT-13B and OPT-30B are taken from MeZO [ 9 ] .

Model	Task	Method	Time / step (s)	Training steps	Total time (h)
OPT1.3B	SST-2	MeZO	0.146	20K	0.811
		LOZO	0.145	20K	0.804
		HiZOO	0.290	20K	1.613
		StructZO	6.890	400	0.766
	RTE	MeZO	0.358	20K	1.99
		LOZO	0.360	20K	2.00

Appendix

Table 7: Detailed wall-clock comparison on OPT-1.3B and Llama3.1-8B. For MeZO, LOZO, and HiZOO/HiZOO-L, we report the average time per optimization step measured over the first 100 steps and estimate the total training time using 20K steps. For StructZO, we report the average time per optimization step measured over the first 10 steps and estimate the total training time using 400 steps, matching the main experimental setting with

q=100

Model	Task	Time / Subspace construction $P$ (%)	Time / ZO forward (%)	Time / ZO update (%)
OPT1.3B	SST-2	5.43	83.91	10.66
	RTE	6.37	89.55	4.08
	BoolQ	6.25	91.15	2.60
Llama3.1-8B	SST-2	4.95	92.77	2.28
	RTE	6.03	93.26	0.71
	BoolQ	6.20	93.32	0.48

Appendix

Table 8: Component-wise wall-clock time breakdown of StructZO. The ZO forward evaluations dominate the per-step cost, while constructing the activation-induced subspace

P

and applying the final ZO update introduce only modest overhead.

Model	Task	$r=16$	$r=32$	$r=64$	$r=128$	$r=256$
OPT1.3B	RTE	63.2	67.8	69.0	66.1	68.9
Llama3.1-8B	SST-2	94.0	93.8	93.7	93.8	92.8

Appendix

Table 9: Performance of the reduced-space rank

r

in StructZO.

Model	Task	$q=4$	$q=10$	$q=20$	$q=50$	$q=100$	$q=200$
OPT1.3B	BoolQ	63.5	62.4	65.3	66.9	68.5	66.1
Llama3.1-8B	CB	64.3	66.1	69.7	80.4	92.9	78.6

Appendix

Table 10: Performance of StructZO with different query numbers

q

on OPT-1.3B BoolQ and Llama3.1-8B CB. All results use the same StructZO pipeline while varying the number of ZO queries aggregated at each optimization step.

Zeroth-Order Optimization

Zeroth-order optimization uses function values or comparisons, rather than gradients, to optimize objectives when differentiation is unavailable or costly.

See all 86 papers

SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces
Aug 16, 2026Ziming Yu, Shuyao Xiao, Xingyu Zhao +6Memory-Efficient Fine-Tuning Zeroth-Order Optimization
Beijing Normal University · Singapore Management University · Xiaomi Inc. +2
AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuning
Sep 28, 2026Yue Xie, Zhi Zheng, Yunpeng Ba +5Subspace Optimization Memory-Efficient Fine-Tuning
Southern University of Science and Technology · National University of Singapore · Huawei Technologies Ltd. +1
CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
Mar 23, 2026Shuo Wang, Ziyu Chen, Ming TangZeroth-Order Optimization Memory-Efficient Fine-Tuning
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China. · Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.

Aug 16, 2026J/K move · Enter open · S save

SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces

Ziming Yu, Shuyao Xiao, Xingyu Zhao, Sike Wang+5

Beijing Normal University · Singapore Management University · Xiaomi Inc.+2

Memory-Efficient Fine-Tuning Zeroth-Order Optimization LLM Fine-Tuning Memory-Efficient Optimization

Open paper AskPDF arXiv

Zeroth-order (ZO) optimization with SGD in random subspaces enables memory-efficient fine-tuning of large language models without backpropagation. However, high gradient estimation noise fundamentally undermines adaptive optimizers like Adam. We propose SubZero+, which achieves practical adaptive ZO optimization through a carefully designed dual low-dimensionality strategy: (i) multi-query forward-difference gradient estimation in periodically refreshed random subspaces to mitigate noise amplification in moment buffers, and (ii) Adam updates with periodic restarts performed directly in low-dimensional space rather than full-parameter space. In experiments, this dual design retains memory overhead comparable to momentum-free ZO methods while achieving stronger optimization performance than the evaluated ZO baselines. Theoretically, in the exact-directional limit,

K

-query averaging preserves conditional unbiasedness, while the coefficient estimator's covariance and mean-squared error, as well as query-induced second-moment inflation, scale exactly as

1/K

. Extensive experiments across SuperGLUE with models from 1.3B to 32B parameters under both full fine-tuning and LoRA schemes demonstrate consistent improvements over competing ZO methods. SubZero+ significantly narrows the performance gap with first-order optimization while preserving ZO's inference-time memory efficiency.

Abstract

Zeroth-Order Optimization

Explore similar work

SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces

AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuning

CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning