cs.LGAug 16, 2026

SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces

Authors: Ziming Yu, Shuyao Xiao, Xingyu Zhao, Sike Wang, Pan Zhou, Peiyu Zang, Xiangda Yan, Yongjie Yang, +1 more

Organizations: Beijing Normal University · Singapore Management University · Xiaomi Inc. · Beijing Key Laboratory of Artificial Intelligence for Education · Engineering Research Center of Intelligent Technology and Educational Application (MOE)

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Zeroth-order (ZO) optimization with SGD in random subspaces enables memory-efficient fine-tuning of large language models without backpropagation. However, high gradient estimation noise fundamentally undermines adaptive optimizers like Adam. We propose SubZero+, which achieves practical adaptive ZO optimization through a carefully designed dual low-dimensionality strategy: (i) multi-query forward-difference gradient estimation in periodically refreshed random subspaces to mitigate noise amplification in moment buffers, and (ii) Adam updates with periodic restarts performed directly in low-dimensional space rather than full-parameter space. In experiments, this dual design retains memory overhead comparable to momentum-free ZO methods while achieving stronger optimization performance than the evaluated ZO baselines. Theoretically, in the exact-directional limit, KK-query averaging preserves conditional unbiasedness, while the coefficient estimator's covariance and mean-squared error, as well as query-induced second-moment inflation, scale exactly as 1/K1/K. Extensive experiments across SuperGLUE with models from 1.3B to 32B parameters under both full fine-tuning and LoRA schemes demonstrate consistent improvements over competing ZO methods. SubZero+ significantly narrows the performance gap with first-order optimization while preserving ZO's inference-time memory efficiency.

Figures & tables

Figure 1: Training dynamics and memory cost of SubZero+. (a) Under full-parameter tuning of OPT-1.3B on SST-2, SubZero+ reaches a lower loss than MeZO and SubZero within the same forward-pass budget. (b) Under LoRA tuning of LLaMA3.1-8B on WiC, SubZero+ gives a smoother trajectory and reaches a lower loss. (c) For the setting of (a), SubZero+ introduces only ∼\sim 0.6GB additional GPU memory overhead compared with MeZO and SubZero.
Figure 2
ModelMethodSST-2RTEBoolQWICMultiRCCOPAReCoRDSQuADDROPAvg.
 Task type—— Classification ——— Multi-choice —— Generation — 
OPT-1.3BZero-shot53.553.445.557.545.475.070.527.211.148.8
FO (Adam)92.078.075.866.070.977.071.181.731.171.5
MeZO90.264.365.058.359.774.071.076.422.564.6
SubZero90.967.565.458.159.575.070.976.425.465.5
LOZO91.167.266.758.961.475.072.174.525.565.8
Table 1: Main results on autoregressive LLMs under the FT scheme. SubZero+ uses K=99K=99 perturbation queries plus one baseline evaluation per update. All methods use a matched 40K total forward-evaluation budget.
Figure 2: Selected loss trajectories corresponding to Table 1 .
ModelMethodSST-2RTEBoolQWICMultiRCCOPAReCoRDSQuADDROPAvg.
 Task type—— Classification ——— Multi-choice —— Generation — 
OPT-13BMeZO93.662.169.156.957.886.081.483.028.768.7
SubZero92.262.566.756.758.387.081.883.728.368.6
LOZO92.861.464.755.560.985.082.183.729.568.4
TeZO92.165.366.059.459.585.081.080.729.068.7
ZO-Muon92.965.365.954.960.084.081.379.427.667.9
Table 2: Main results on autoregressive LLMs under the LoRA scheme. The best result in each model/task column, including Avg., is bolded.
Figure 3: Selected loss trajectories corresponding to Table 2 .
MethodSST-2RTEBoolQWICSQuADAvg.
Zero-shot61.587.084.361.885.576.0
MeZO87.590.386.765.787.983.6
MeZO-LoRA62.087.484.864.784.776.7
SubZero89.190.387.768.889.685.1
LOZO89.491.087.069.390.585.4
ZO-Muon90.591.388.671.090.586.4
Table 3: Performance comparison on Qwen2.5-32B. Averages are computed over five tasks.
 FTLoRA
TaskMeZOSubZeroSubZero+Δ\Delta %MeZOSubZeroSubZero+Δ\Delta %
RTE30.430.530.7+1.030.430.430.4+0.0
BoolQ34.134.234.5+1.234.134.134.10.0
SQuAD30.830.931.1+1.030.830.830.8+0.0
DROP50.450.550.8+0.850.450.450.40.0
Table 4: Peak GPU memory on OPT-13B (GB, batch size 16, fp16).
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Multi-query estimation reduces gradient noise. (a) A 2D loss surface with steep walls and a flat valley. (b) SubZero with K=1K=1 . The estimated gradient-norm field is noisy. (c) SubZero+ with K=100K=100 . Averaging produces a visibly smoother gradient-norm field. All panels share the same domain and starting point.
Figure 5: Peak GPU memory for full-parameter tuning of OPT-13B on SST-2 (batch size 16, fp16). The blue bars apply single- or multi-query SGD/Adam in full parameter space on top of MeZO; the red bar is SubZero+ with multi-query coefficient-space Adam. The 97.24 GB bar represents straightforward full-space query accumulation rather than a direct reproduction of MUZO’s layer-wise implementation.
Dataset15102030507080100150200Δ\Delta
SST-250.952.656.769.888.390.991.591.692.289.989.7+41.3
RTE47.752.749.857.062.863.958.160.357.061.461.0+9.3
MultiRC55.555.552.359.857.660.057.659.359.658.761.1+4.1
Avg.51.453.652.962.269.671.669.370.469.670.070.6+18.2
Appendix
Table 5: Ablation study on perturbation-query count KK across tasks under FT scheme. Results on OPT-1.3B with SubZero+ (sign-corrected QR subspaces, r=24r=24 ) across SST-2, RTE, and MultiRC.
Figure 6: Learning-rate robustness of SubZero+ and MeZO for LoRA of Qwen3-0.6B ( Yang et al., 2025 ) on SST-2. (a) Training accuracy versus learning rate (LR). SubZero+ maintains high accuracy over a wide LR range, whereas MeZO degrades quickly outside its optimal value. (b) Normalized results, with each LR scaled by its optimum. SubZero+ shows a symmetric plateau, while MeZO collapses when LR is larger than the optimum.
MethodConfiguration
MeZOlearning rate {10−7,5×10−7,10−6}\{10^{-7},5\times 10^{-7},10^{-6}\} ; 20,000 steps (2 forward passes/step)
SubZerolearning rate {10−7,5×10−7,10−6}\{10^{-7},5\times 10^{-7},10^{-6}\} ; rank r=16r=16 (OPT-1.3B) or 128128 (others); update frequency F∈{500,1000,2000}F\in\{500,1000,2000\} ; 20,000 steps (2 forward passes/step)
LOZOlearning rate {10−7,5×10−7,10−6}\{10^{-7},5\times 10^{-7},10^{-6}\} ; rank r=8r=8 ; reuse interval ν=100\nu=100 ; 20,000 steps (2 forward passes/step)
TeZOlearning rate {10−7,5×10−7,10−6}\{10^{-7},5\times 10^{-7},10^{-6}\} ; fixed rank r=64r=64 ; 20,000 steps (2 forward passes/step)
ZO-Muonlearning rate {3×10−3,10−2,3×10−2}\{3\times 10^{-3},10^{-2},3\times 10^{-2}\} ; projection rank r∈{64,128}r\in\{64,128\} ; Nq=4N_{q}=4 ; projection interval 100; 5 Newton–Schulz iterations; 8,000 steps (4 perturbation queries + 1 baseline forward pass/step)
SubZero+learning rate {10−3,5×10−3,10−2}\{10^{-3},5\times 10^{-3},10^{-2}\} ; rank r=16r=16 (OPT-1.3B) or 128128 (others); update frequency F∈{50,100}F\in\{50,100\} ; K=99K=99 ; 400 steps (99 perturbation queries + 1 baseline forward pass/step); Adam (β1,β2)=(0.9,0.95)(\beta_{1},\beta_{2})=(0.9,0.95) and ϵadam=10−8\epsilon_{\mathrm{adam}}=10^{-8}
Appendix
Table 6: Hyperparameter configurations and search grids for the FT scheme. Common settings are batch size 16, perturbation scale 10−310^{-3} , constant learning rate, and zero weight decay.
MethodConfiguration
MeZOlearning rate {1.5×10−5,3×10−5,5×10−5}\{1.5\times 10^{-5},3\times 10^{-5},5\times 10^{-5}\} ; 20,000 steps (2 forward passes/step)
SubZerolearning rate {1.5×10−5,3×10−5,5×10−5}\{1.5\times 10^{-5},3\times 10^{-5},5\times 10^{-5}\} ; rank r∈{4,8,16}r\in\{4,8,16\} ; update frequency F∈{500,1000,2000}F\in\{500,1000,2000\} ; 20,000 steps (2 forward passes/step)
LOZOlearning rate {1.5×10−5,3×10−5,5×10−5}\{1.5\times 10^{-5},3\times 10^{-5},5\times 10^{-5}\} ; rank r=8r=8 ; reuse interval ν=100\nu=100 ; 20,000 steps (2 forward passes/step)
TeZOlearning rate {1.5×10−5,3×10−5,5×10−5}\{1.5\times 10^{-5},3\times 10^{-5},5\times 10^{-5}\} ; fixed rank r=64r=64 ; 20,000 steps (2 forward passes/step)
ZO-Muonlearning rate {10−3,3×10−3,10−2}\{10^{-3},3\times 10^{-3},10^{-2}\} ; projection rank r∈{8,16}r\in\{8,16\} (selected: 16); Nq=4N_{q}=4 ; projection interval 100; 5 Newton–Schulz iterations; 8,000 steps (4 perturbation queries + 1 baseline forward pass/step)
SubZero+learning rate {5×10−3,10−2,5×10−2}\{5\times 10^{-3},10^{-2},5\times 10^{-2}\} ; rank r∈{4,8,16}r\in\{4,8,16\} ; update frequency F∈{50,100}F\in\{50,100\} ; K=99K=99 ; 400 steps (99 perturbation queries + 1 baseline forward pass/step); Adam (β1,β2)=(0.9,0.95)(\beta_{1},\beta_{2})=(0.9,0.95) and ϵadam=10−8\epsilon_{\mathrm{adam}}=10^{-8}
Appendix
Table 7: Hyperparameter configurations and search grids for the LoRA scheme. LoRA rank is 8 and α=16\alpha=16 for all methods; common ZO settings are as in Table 6 .

Explore similar work

CardsList

  1. On Adaptivity in Zeroth-Order Optimization

    May 5, 2026Hassan Dbouk, Nidham Gazagnadou, Matthias Reisser +1Zeroth-Order OptimizationStochastic Gradient Descent

    Qualcomm AI Research

  2. ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces

    Jul 1, 2026Xun Dong, Yibo Xu, Naigang Wang +3Zeroth-Order OptimizationZeroth-Order

    University at Albany, SUNY · IBM T. J. Watson Research Center

  3. Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling

    Apr 20, 2026Fei Wang, Li Shen, Liang Ding +3Zeroth-Order OptimizationZeroth-Order

    South China University of Technology · Shenzhen Campus of Sun Yat-sen University · University of Sydney +2