Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models through forward-only evaluations, but high-dimensional random perturbations can limit optimization efficiency. Sparse perturbations reduce the search dimension, making the selection of informative coordinates critical. We propose Wanda-Importance Sparsity-MeZO (WIS-MeZO), which uses Wanda weight--activation saliency to construct task-conditioned sparse perturbation masks while keeping the pretrained model dense. We analyze support quality through gradient energy capture and relate it to gradient estimation and convergence. Across 16 benchmarks with LLaMA and Qwen models ranging from 3B to 32B parameters, WIS-MeZO achieves higher aggregate scores than MeZO and reaches comparable validation losses in fewer optimization steps. Comparisons with random and magnitude-based sparse perturbations further highlight the importance of informative support selection.
Figures & tables
Figure 1: Validation loss versus training steps for fine-tuning LLaMA3.2-3B, averaged over five random seeds. WIS-MeZO reaches comparable validation-loss levels in fewer optimization steps.
Figure 2: Gradient energy capture ratio ρ on LLaMA-3.2-3B at a density ratio of r=0.05 . We compare uniform random masking, the small-magnitude mask used to represent the selection principle of Sparse-MeZO (W-small), the corresponding large-magnitude mask (W-large), and the Wanda-based WIS mask.
Models
Methods
MNLI
RTE
QQP
MRPC
SST-2
SQuAD
Avg.
LLaMA3.2-3B
Zero-shot
38.3
49.4
53.7
78.3
58.8
55.0
55.6
FT
82.6 ±0.6
87.3 ±0.8
79.2 ±1.0
86.9 ±1.0
95.1 ±0.5
88.2 ±0.4
86.5
LoRA
83.4 ±0.7
88.8 ±0.7
79.2 ±1.0
85.8 ±1.3
94.8 ±0.1
88.0 ±0.0
86.6
MeZO
49.4 ±3.0
70.3 ±2.1
65.0 ±2.1
81.4 ±0.6
93.9 ±0.8
86.7 ±0.6
74.4
Sparse-MeZO
56.7 ±2.6
77.6 ±3.8
70.6 ±0.6
81.8 ±0.9
93.6 ±0.3
86.4 ±0.7
77.7
Random-MeZO
52.1 ±1.8
74.3 ±0.7
67.4 ±1.3
82.6 ±1.1
93.9 ±0.4
88.1 ±0.6
76.4
Table 1: Results on five GLUE tasks and SQuAD (five random seeds). We report F1 scores for QQP, MRPC, and SQuAD, and accuracy (%) for all other tasks ( ↑ ). FT and LoRA denote full parameter fine-tuning and Low-Rank Adaptation, respectively, both utilizing the AdamW optimizer.
Models
Methods
CB
MultiRC
WSC
ReCoRD
BoolQ
Avg.
LLaMA3.2-3B
Zero-shot
44.6
47.0
36.5
79.7
61.6
53.9
FT
91.0
83.6
57.6
81.6
83.1
79.4
LoRA
92.8
85.2
62.5
81.4
84.3
81.2
MeZO
71.4
72.4
61.5
81.2
77.6
72.8
Sparse-MeZO
67.8
79.7
62.5
81.4
81.3
74.5
Random-MeZO
75.0
74.2
61.5
81.0
78.9
74.1
Table 2: Experimental results on SuperGLUE. WIS-based methods achieve the strongest aggregate ZO performance on both model families.
Models
Methods
ARC-E
ARC-C
HellaSwag
WinoGrande
OpenBookQA
Avg.
Qwen2.5-3B
Zero-shot
70.0
48.4
68.0
66.3
40.8
58.7
MeZO
82.9
57.8
69.9
66.3
49.0
65.1
WIS-MeZO
85.4
60.8
74.4
69.7
54.0
68.8
Qwen2.5-7B
Zero-shot
74.9
52.5
73.8
68.9
43.8
62.7
MeZO
83.0
59.2
74.9
66.5
51.2
66.9
WIS-MeZO
85.0
61.2
79.4
70.7
53.8
70.0
Table 3: Experimental results for Qwen2.5-3B to 32B. WIS-MeZO consistently improves average performance over MeZO across all model scales.
Methods
SST-2
ReCoRD
CB
Mem.
Iter/s
Mem.
Iter/s
Mem.
Iter/s
FT
35.5
3.37
87.4
0.92
94.5
0.75
LoRA
22.1
4.73
73.5
1.33
85.0
0.94
MeZO
9.70
3.50
21.5
1.39
45.1
0.99
Sparse-MeZO
16.2
2.98
27.3
1.30
51.0
0.95
WIS-MeZO
11.6
3.16
25.1
1.33
47.0
0.97
Table 4: Peak GPU memory (GB, batch size=16) and iteration speed (iteration/s) for fine-tuning Qwen2.5-3B.
Methods
SST-2
ReCoRD
CB
MeZO
1.59
4.00
5.61
Sparse-MeZO
1.86
4.27
5.85
WIS-MeZO
1.76
4.18
5.73
Table 5: End-to-end wall-clock time (hours) for Qwen2.5-3B under the common ZO training budget.
Methods
ARC-C
HellaSwag
Mem.
Iter/s
Mem.
Iter/s
FT
600.5
0.126
664.8
0.074
LoRA
256.0
0.161
282.2
0.105
MeZO
97.29
0.174
104.8
0.104
WIS-MeZO
122.0
0.169
132.3
0.102
WIS-MeZO (1-bit)
98.84
0.160
107.7
0.099
Table 6: Peak GPU memory (GB, batch size=16) and iteration speed (iteration/s) for fine-tuning Qwen2.5-32B.
Ratio- r
ARC-E
ARC-C
HellaSwag
WinoGrande
OpenBookQA
0.05
85.0
57.8
71.8
68.6
54.0
0.1
85.4
60.8
74.4
69.7
52.8
0.2
83.8
59.5
72.0
69.5
53.6
0.4
84.2
59.8
71.4
67.1
52.8
MeZO
82.9
57.8
69.9
66.3
49.0
Table 7: The effects of density ratio- r for Fine-tuning Qwen2.5-3B with WIS-MeZO.
Calibration size
ARC-E
ARC-C
HellaSwag
WinoGrande
OpenBookQA
32
85.0
60.5
74.0
67.4
54.0
64
85.2
60.5
74.3
67.8
53.0
128 (default)
85.4
60.8
74.4
69.7
54.0
256
85.4
60.5
74.6
69.7
54.0
512
85.9
60.8
74.5
69.7
53.9
Table 8: Effects of the calibration size on WIS-MeZO fine-tuning of Qwen2.5-3B.
Calibration size
ARC-E
ARC-C
HellaSwag
WinoGrande
OpenBookQA
32
0.32
0.33
0.43
0.26
0.27
64
0.49
0.49
0.72
0.34
0.38
128 (default)
0.80
0.89
1.26
0.53
0.60
256
1.49
1.55
2.36
0.91
1.00
512
2.70
2.98
4.59
1.66
1.84
Table 9: Total calibration time (seconds) for Qwen2.5-3B.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Description
W,M,Z
Weights, binary support mask, and Gaussian perturbation.
G,G^ϵ,G^0
Stochastic gradient, finite-difference estimate, and its directional limit.
x,δ
Per-observation input and backpropagated signal.
X∈Rn×B
Matrix of calibration activation observations for one layer.
vj,v^j
Population and empirical second moments of feature j .
Si,j
Wanda score of weight Wi,j .
Appendix
Table 10: Summary of notation.
Methods
Hyperparameters
Values
FT
Batch size
8
Learning rate
{1.0e−6,5.0e−6,1.0e−5,5.0e−5}
(β1,β2)
(0.9, 0.999)
LoRA
Batch size
8
Learning rate
{1.0e−5,5.0e−5,1.0e−4,5.0e−4}
Rank
8
Appendix
Table 11: The hyperparameter grids used for the experiments in the main text.
Models
Methods
MNLI
RTE
QQP
MRPC
SST-2
SQuAD
LLaMA3.2-3B
FT
1.0e-5
1.0e-5
1.0e-5
1.0e-5
1.0e-5
1.0e-5
LoRA
5.0e-4
5.0e-4
5.0e-4
5.0e-4
1.0e-4
5.0e-5
MeZO
5.0e-7
2.5e-7
5.0e-7
2.5e-7
5.0e-7
2.5e-7
LOZO
1.0e-7
2.5e-7
1.0e-7
2.5e-7
2.5e-7
2.5e-7
Sparse-MeZO
2.5e-6, 0.2
1.0e-6, 0.2
1.0e-6, 0.4
1.0e-6, 0.4
1.0e-6, 0.4
1.0e-6, 0.2
Random-MeZO
5.0e-7, 0.2
5.0e-7, 0.2
5.0e-7, 0.4
5.0e-7, 0.2
5.0e-7, 0.2
5.0e-7, 0.2
Appendix
Table 12: Selected hyperparameters for the GLUE and SQuAD experiments.
Models
Methods
CB
MultiRC
WSC
ReCoRD
BoolQ
LLaMA3.2-3B
FT
1.0e-5
1.0e-5
1.0e-5
1.0e-6
1.0e-5
LoRA
5.0e-4
1.0e-4
5.0e-4
1.0e-5
5.0e-4
MeZO
2.5e-7
5.0e-7
2.5e-7
2.5e-7
1.0e-7
LOZO
1.0e-7
2.5e-7
1.0e-7
2.5e-7
1.0e-7
Sparse-MeZO
1.0e-6, 0.4
1.0e-6, 0.2
5.0e-6, 0.4
1.0e-6, 0.2
2.5e-6, 0.2
Random-MeZO
5.0e-7, 0.2
5.0e-7, 0.2
2.5e-7, 0.1
5.0e-7, 0.4
5.0e-7, 0.2
Appendix
Table 13: Selected hyperparameters for the SuperGLUE experiments.
Figure 3: Gradient energy capture ratio ρ on LLaMA-3.2-3B at a density ratio of r=0.1 . Random denotes uniform random masking. W-small and W-large select coordinates with the smallest and largest weight magnitudes, respectively, while WIS selects coordinates using Wanda weight–activation scores.
Figure 4: Gradient energy capture ratio ρ on LLaMA-3.2-3B at a density ratio of r=0.2 . All mask construction strategies use the same row-wise perturbation density, model parameters, gradient snapshots, and evaluation data.
Figure 5: Gradient energy capture ratio ρ on LLaMA-3.2-3B at a density ratio of r=0.4 . Random denotes uniform random masking. W-small and W-large select coordinates with the smallest and largest weight magnitudes, respectively, while WIS uses Wanda weight–activation scores.
Zeroth-order optimization uses function values or comparisons, rather than gradients, to optimize objectives when differentiation is unavailable or costly.
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China. · Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.