Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces

Chuang Tong; Ling Chen

cs.LGOct 10, 2026

Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces

Authors: Chuang Tong, Ling Chen

Organizations: State Key Laboratory of Blockchain and Data Security, Zhejiang University · College of Computer Science and Technology, Zhejiang University

Abstract

Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs). However, ZO optimization suffers from high variance in high-dimensional parameter spaces. While recent methods restrict perturbations to random low-rank subspaces, these subspaces often misalign with task-specific descent directions. We propose AdaSub-ZO, a novel forward-only fine-tuning framework in learnable low-rank subspaces. Instead of relying on fixed random matrices, AdaSub-ZO iteratively learns layer-wise projection matrices directly from finite-difference feedback to track task-aware descent directions. Furthermore, AdaSub-ZO incorporates a subspace momentum acceleration mechanism to smooth noisy gradient estimates across evolving subspace bases. Extensive experiments on natural language understanding tasks demonstrate that AdaSub-ZO outperforms existing ZO baselines, substantially reducing gradient estimation variance and accelerating convergence.

Figures & tables

Method	SST-2	RTE	CB	BoolQ	WSC	WIC	COPA	AVG.
SGD	92.1	64.0	76.2	67.4	63.7	62.5	77.8	6.50%
MeZO (NeurIPS 2023)	89.4	59.2	69.3	64.1	60.0	59.5	72.0	0.00%
ZO-AdaMU (AAAI 2024)	89.4	59.4	67.9	63.2	60.0	59.8	72.7	-0.23%
S-MeZO (NeurIPS 2026)	89.5	59.3	69.5	64.2	60.1	59.6	72.0	0.15%
LOZO (ICLR 2025)	90.9	56.4	66.8	63.0	63.5	60.2	72.2	-0.16%
SubZero (ICCV 2025)	90.3	56.6	68.5	64.6	64.0	60.8	73.0	0.93%

Table 1: Performance comparison of fine-tuning methods on OPT-1.3B. SGD is included as a first-order performance reference. Bold denotes the best result among zeroth-order methods, and underline denotes the second-best result among zeroth-order methods.

Figure 1: Relative variance of ZO gradient estimates.

Figure 2: Training loss curves on CB.

Figure 3: Effect of low-rank dimension

r

on SST-2.

Figure 4: Effect of subspace momentum coefficient

\beta_{m}

on SST-2.

Method	SST-2	RTE	CB	BoolQ	WSC	WIC	COPA	AVG.
AdaSub-ZO w/o Learnable Module	91.7	61.2	71.1	64.5	59.2	56.6	76.7	0.00%
AdaSub-ZO	92.0	61.6	71.4	64.8	59.6	57.0	77.0	+0.52%

Table 2: Ablation study of learnable module on OPT-1.3B. AVG denotes the average relative improvement over AdaSub-ZO w/o Learnable Module.

Method	SST-2	RTE	CB	BoolQ	WSC	WIC	COPA	AVG.
AdaSub-ZO w/o Subspace Momentum	91.1	61.4	70.0	64.9	63.0	57.0	74.0	0.00%
AdaSub-ZO	92.0	61.6	71.4	64.8	59.6	57.0	77.0	+0.26%

Table 3: Ablation study of subspace momentum on OPT-1.3B. AVG denotes the average relative improvement over AdaSub-ZO w/o Subspace Momentum.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Search Space
Model Learning Rate $\eta_{\theta}$	$\{1\text{e-}7,1\text{e-}6,1\text{e-}5\}$
Model Perturbation Scale $\epsilon$	$\{1\text{e-}4,1\text{e-}3,1\text{e-}2\}$
Low-Rank Dimension $r$	$\{24,36,48\}$
Projection Learning Cycle $K$	$\{50,100,200,500\}$
Projection Learning Steps $M$	$\{1,5,10,20\}$
Projection Learning Rate $\eta_{\phi}$	$\{1\text{e-}3,1\text{e-}2,1\text{e-}1\}$

Appendix

Table 4: Hyperparameter search space for AdaSub-ZO.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 754 papers

SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces
Aug 16, 2026Ziming Yu, Shuyao Xiao, Xingyu Zhao +6Memory-Efficient Fine-Tuning Zeroth-Order Optimization
Beijing Normal University · Singapore Management University · Xiaomi Inc. +2
ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces
Jul 1, 2026Xun Dong, Yibo Xu, Naigang Wang +3Zeroth-Order Optimization LLM Fine-Tuning
University at Albany, SUNY · IBM T. J. Watson Research Center
CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
Mar 23, 2026Shuo Wang, Ziyu Chen, Ming TangZeroth-Order Optimization Memory-Efficient Fine-Tuning
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China. · Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.

Aug 16, 2026J/K move · Enter open · S save

SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces

Ziming Yu, Shuyao Xiao, Xingyu Zhao, Sike Wang+5

Beijing Normal University · Singapore Management University · Xiaomi Inc.+2

Memory-Efficient Fine-Tuning Zeroth-Order Optimization LLM Fine-Tuning Memory-Efficient Optimization

Open paper AskPDF arXiv

Zeroth-order (ZO) optimization with SGD in random subspaces enables memory-efficient fine-tuning of large language models without backpropagation. However, high gradient estimation noise fundamentally undermines adaptive optimizers like Adam. We propose SubZero+, which achieves practical adaptive ZO optimization through a carefully designed dual low-dimensionality strategy: (i) multi-query forward-difference gradient estimation in periodically refreshed random subspaces to mitigate noise amplification in moment buffers, and (ii) Adam updates with periodic restarts performed directly in low-dimensional space rather than full-parameter space. In experiments, this dual design retains memory overhead comparable to momentum-free ZO methods while achieving stronger optimization performance than the evaluated ZO baselines. Theoretically, in the exact-directional limit,

K

-query averaging preserves conditional unbiasedness, while the coefficient estimator's covariance and mean-squared error, as well as query-induced second-moment inflation, scale exactly as

1/K

. Extensive experiments across SuperGLUE with models from 1.3B to 32B parameters under both full fine-tuning and LoRA schemes demonstrate consistent improvements over competing ZO methods. SubZero+ significantly narrows the performance gap with first-order optimization while preserving ZO's inference-time memory efficiency.

Abstract

LLM Fine-Tuning

Explore similar work

SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces

ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces

CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning