CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
Authors: Shuo Wang, Ziyu Chen, Ming Tang
Organizations: Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China. · Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.
Fine-tuning large language models (LLMs) with backpropagation achieves high performance but incurs substantial memory overhead, limiting scalability on resource-constrained hardware. Zeroth-order (ZO) optimization provides a memory-efficient alternative by relying solely on forward passes, yet it typically suffers from slow or unstable convergence due to high-variance gradient estimates. Sparse ZO updates partially address this issue by perturbing only a subset of parameters, but their effectiveness hinges on selecting informative parameters, which is challenging in ZO optimization because each query yields only scalar feedback. We propose Adaptive Curvature-Guided Sparse Zeroth-Order Optimization (CurvZO), which tracks curvature signals online from scalar ZO feedback and leverages these signals to construct a parameter-wise sampling distribution for selecting coordinates at each update, reducing the variance of the sparse ZO gradient estimator. Moreover, CurvZO dynamically adapts the perturbation budget to the evolving curvature signal distribution, yielding sparse ZO updates that remain both focused and sufficiently exploratory. Extensive experiments on OPT and Llama across diverse NLP tasks show that CurvZO consistently improves fine-tuning performance and reduces training time over ZO baselines. It improves accuracy by up to 4.4 points and achieves up to a 2× speedup, while preserving memory efficiency.
Figures & tables
Figure 1: Visualization of anisotropic local curvature in the attention output weights of OPT-6.7B. The x - and y -axes index the columns and rows of the weight matrix, while the z -axis shows curvature magnitude approximated via the diagonal Fisher information (a standard local-curvature surrogate in neural networks).
Task
SST-2
RTE
CB
BoolQ
WSC
WIC
SQuAD
DROP
Average
—————-————-classification————————–
—-generation—-
Zero-shot
56.3
54.1
50.0
47.6
36.5
52.6
29.8
9.5
42.1
FT
93.8
80.1
82.1
72.6
63.4
69.1
82.9
33.2
72.2
LoRA
94.7
81.6
82.7
74.1
59.6
62.1
84.5
35.1
71.8
MeZO
92.1
63.3
67.8
66.7
58.6
59.8
79.0
26.5
64.2
DiZO
94.4
65.6
66.0
66.8
56.7
57.1
62.1
22.5
61.4
Table 1: Accuracy (%) on OPT-2.7B fine-tuning with 1,000 training samples. FT and LoRA are first-order (backpropagation-based) oracle baselines. Better results among MeZO, DiZO, and CurvZO are highlighted in bold .
Task
SST-2
RTE
WIC
WSC
SQuAD
-————-classification———–
-generation-
MeZO
94.1
70.3
62.5
60.5
81.6
DiZO
91.6
66.4
61.6
59.6
67.8
CurvZO
94.7
72.2
61.7
61.5
83.7
MeZO + LoRA
93.0
64.9
61.4
60.5
80.1
DiZO + LoRA
91.4
58.8
57.2
58.6
71.5
Table 2: Accuracy (%) on OPT-6.7B fine-tuning with 1,000 training samples.
Figure 2: Accuracy (%) on fine-tuning Llama2-7B (top) and Llama2-13B (bottom) with 1,000 training samples.
Figure 3: Convergence curves of fine-tuning OPT-2.7B with MeZO and CurvZO on (a) RTE, (b) BoolQ tasks.
Model
Task
SST-2
RTE
WIC
WSC
SQuAD
———-classification———
-generation-
OPT-2.7B
FT
42.88
45.28
43.48
43.21
45.28
OPT-2.7B
MeZO
5.91
5.91
5.91
5.91
5.91
OPT-2.7B
CurvZO
5.91
5.91
5.91
5.91
5.91
OPT-6.7B
FT
> 80.00
> 80.00
> 80.00
> 80.00
> 80.00
OPT-6.7B
MeZO
13.94
13.94
13.94
13.94
13.94
Table 3: Memory usage (batch size = 1) of fine-tuning OPT-2.7B and OPT-6.7B (1,000 examples). For OPT-6.7B, FT exceeds the 80 GB memory limit of a single GPU, while MeZO and CurvZO remain memory-efficient.
Figure 4: GPU hours of MeZO and CurvZO across tasks, together with convergence iterations and forward pass time.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Task
SST-2
RTE
CB
BoolQ
WSC
WIC
SQuAD
DROP
Average
—————-————-classification————————–
—-generation—-
US + AB
90.8
59.9
66.7
64.2
58.6
56.5
77.8
25.1
62.5
CurvZO ( B=0.1d )
91.3
64.9
60.1
69.1
59.6
59.7
78.6
24.6
63.5
CurvZO ( B=0.4d )
94.4
67.5
67.8
65.1
56.7
60.1
80.2
24.1
64.5
CurvZO ( B=0.7d )
93.0
63.8
63.2
67.3
57.6
61.1
80.0
23.1
63.6
CurvZO
94.1
67.7
69.8
68.9
62.9
61.9
81.5
27.6
66.8
Appendix
Table 4: Fine-tuning results on OPT-2.7B across tasks. CurvZO combines curvature-guided sampling with adaptive budget selection. US+AB ( Uniform Sampling + Adaptive Budget ) uses the same adaptive budget selection as CurvZO, but samples parameters uniformly at random. CurvZO ( B=ρd ) uses curvature-guided sampling with a fixed perturbation budget B=ρd (e.g., ρ∈{0.1,0.4,0.7} ). The best results in each block are highlighted.
Experiment
Hyperparameters
Values
FT
Batch size
8
Learning rate
{1e−5,5e−5}
LR schedule
Constant
MeZO
Batch size
{64,16}
Learning rate η
{2e−7,5e−7,1e−6}
ϵ
1e−3
Appendix
Table 5: Hyperparameters used in our experiments. CurvZO and CurvZO+LoRA use the same settings as MeZO and MeZO+LoRA, respectively.
Zeroth-order optimization uses function values or comparisons, rather than gradients, to optimize objectives when differentiation is unavailable or costly.