Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Training
Organizations: Department of Computer Science University of Maryland - College Park · Department of Computer Science and Engineering University of Texas at Arlington
Abstract
Fine-tuning pre-trained Large Language Models (LLMs) for downstream tasks using First-Order (FO) optimizers presents significant computational challenges. Parameter-Efficient Fine-Tuning (PEFT) methods address these by freezing most model parameters and training only a small subset. However, PEFT often underperforms compared to full fine-tuning when high task-specific accuracy is required. Zeroth-Order (ZO) methods fine-tune the entire pre-trained model without back-propagation, estimating gradients through forward passes only. While memory-efficient, ZO methods suffer from slow convergence and high sensitivity to prompt selection. We bridge these two worlds with Bilevel-ZOFO, a bilevel optimization method that couples fast, local FO-PEFT adaptation at the inner level with stable, memory-efficient ZO updates of the full backbone at the outer level. The FO-PEFT inner loop performs fast, low-memory local adaptation that reduces the variance of ZO estimates and stabilizes the search, guiding the outer ZO updates of the full backbone and reducing prompt sensitivity. In the mean time, the outer ZO provides better generalization ability for PEFT. We provide theoretical convergence guarantees and empirically demonstrate that Bilevel-ZOFO significantly outperforms existing ZO and FO-PEFT methods, achieving 2-4 times faster training while maintaining similar memory efficiency. Additionally, we show by updating the backbone with ZO and adapting only a tiny FO-PEFT block per task, Bilevel-ZOFO combines full-model capacity with few-shot efficiency, making it a very efficient meta-learning algorithm that quickly adapts to new tasks.
Figures & tables
| (8) |
| Method | Task | w/ prompt (%) | w/o prompt (%) | Diff. |
|---|---|---|---|---|
| MeZO | SST-2 | 89.6 | 51.9 | -38.6 |
| COPA | 70.0 | 54.8 | -15.2 | |
| Ours | SST-2 | 93.3 | 92.9 | -0.4 |
| COPA | 76.7 | 73.6 | -3.1 |
| Trainer | Mode | BoolQ | CB | Copa | ReCoRD | RTE | SST2 | WIC | WinoGrande | WSC | Average |
|---|---|---|---|---|---|---|---|---|---|---|---|
| MeZO | ft | 0.6927 | 0.7767 | 0.7000 | 0.6980 | 0.6587 | 0.8214 | 0.5543 | 0.5480 | 0.5054 | 0.6617 |
| lora | 0.6860 | 0.7607 | 0.7200 | 0.7083 | 0.6755 | 0.8501 | 0.5549 | 0.5607 | 0.5570 | 0.6748 | |
| prefix | 0.6573 | 0.7945 | 0.7033 | 0.7047 | 0.6972 | 0.8218 | 0.5622 | 0.5370 | 0.5105 | 0.6654 | |
| prompt | 0.6260 | 0.5821 | 0.7067 | 0.7070 | 0.5415 | 0.7463 | 0.5574 | 0.5556 | 0.4654 | 0.6098 | |
| average | 0.6655 | 0.7285 | 0.7075 | 0.7045 | 0.6432 | 0.8099 | 0.5572 | 0.5503 | 0.5096 | 0.6529 | |
| FO | lora | 0.7456 | 0.8512 | 0.7500 | 0.7206 | 0.7292 | 0.9258 | 0.6463 | 0.5806 | 0.6474 | 0.7330 |
| Trainer | Mode | BoolQ | ReCoRD | SQuAD | SST2 | Average |
|---|---|---|---|---|---|---|
| MeZO | ft | 0.7915 | 0.7890 | 0.7737 | 0.8646 | 0.8047 |
| lora | 0.8020 | 0.7970 | 0.7412 | 0.8529 | 0.7983 | |
| prefix | 0.7830 | 0.7905 | 0.7093 | 0.8364 | 0.7798 | |
| prompt | 0.7787 | 0.7935 | 0.7014 | 0.8246 | 0.7746 | |
| average | 0.7888 | 0.7925 | 0.7489 | 0.8397 | 0.7825 | |
| FO | lora | 0.8420 | 0.7920 | 0.8197 | 0.9557 | 0.8524 |
| Method | BoolQ | CB | COPA | SST2 |
|---|---|---|---|---|
| MeZO | 0.6927 | 0.7767 | 0.7000 | 0.8214 |
| FO PEFT | 0.7150 | 0.7142 | 0.7466 | 0.8016 |
| Bilevel-ZOZO | 0.6280 | 0.6092 | 0.7146 | 0.7633 |
| Two-Stage Pipeline | 0.7060 | 0.6786 | 0.7433 | 0.8016 |
| Jointly Optimized | 0.7209 | 0.7500 | 0.7466 | 0.8148 |
| Bilevel-ZOFO | 0.7367 | 0.7679 | 0.7633 | 0.8335 |
| Method | Training | Inference | class | non_class | qa | non_qa | ||
|---|---|---|---|---|---|---|---|---|
| FLOPS | Peak Mem(bs=1) | Step Duration | Tokens / Sample | class | class | qa | qa | |
| Zero-shot | 0 | - | - | X | 34.2 | 34.2 | 40.2 | 40.2 |
| Few-shot | 0 | - | - | X | 34.9 (1.4) | 34.9 (1.4) | 40.5 (0.3) | 40.5 (0.4) |
| MetaICL | 32GB | 0.48s | X | 46.4 (1.1) | 37.7 (1.7) | 45.5 (0.3) | 40.2 (0.6) | |
| Ours (Zero-shot) | 12GB | 0.18s | X | 34.5 | 34.3 | 41.8 | 40.4 | |
| Ours(Tuned) | 12GB | 0.19s | X | 47.1 | 42.4 | 43.5 (1.3) | 41.9 | |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Trainer | Mode | BoolQ | CB | Copa | ReCoRD | RTE | SST2 | WIC | WinoGrande | WSC | Average |
|---|---|---|---|---|---|---|---|---|---|---|---|
| MeZO | ft | ||||||||||
| lora | |||||||||||
| prefix | |||||||||||
| prompt | |||||||||||
| average | |||||||||||
| FO | lora |
| Trainer | Mode | BoolQ | ReCoRD | SQuAD | SST2 | Average |
| MeZO | ft | 0.7915 ± 0.0516 | 0.7890 ± 0.0001 | 0.7737 ± 0.1634 | 0.8646 ± 0.0216 | 0.8047 |
| lora | 0.8020 ± 0.0014 | 0.7970 ± 0.0001 | 0.7412 ± 0.0013 | 0.8529 ± 0.0117 | 0.7983 | |
| prefix | 0.7830 ± 0.0131 | 0.7905 ± 0.0007 | 0.7093 ± 0.0207 | 0.8364 ± 0.0010 | 0.7798 | |
| prompt | 0.7787 ± 0.0049 | 0.7935 ± 0.0007 | 0.7014 ± 0.0451 | 0.8246 ± 0.0216 | 0.7746 | |
| FO | lora | 0.8420 ± 0.0104 | 0.7920 ± 0.0053 | 0.8197 ± 0.0043 | 0.9557 ± 0.0007 | 0.8524 |
| prefix | 0.7783 ± 0.0021 | 0.8013 ± 0.0012 | 0.7946 ± 0.0419 | 0.9243 ± 0.0053 | 0.8246 |
| Method | Experiment (COPA) | Acc (%) |
|---|---|---|
| Two-Stage | After Stage 1 | 74.33 |
| After Stage 2 (lr ) | 51.66 | |
| After Stage 2 (lr ) | 70.33 | |
| After Stage 2 (lr ) | 72.66 | |
| After Stage 2 (lr ) | 74.33 | |
| Bilevel-ZOFO | - | 76.33 |
| Meta-train Setting | # tasks | # examples | Target Setting | # tasks |
|---|---|---|---|---|
| Classification | 43 | 384,022 | Classification | 20 |
| Non-Classification | 37 | 368,768 | ||
| QA | 37 | 486,143 | QA | 22 |
| Non-QA | 33 | 521,342 |
| Method | BoolQ | CB | COPA | ReCoRD | RTE | SST-2 | WiC | WinoGrande | WSC |
|---|---|---|---|---|---|---|---|---|---|
| FO-FT | 0.660 | 0.821 | 0.730 | 0.719 | 0.690 | 0.937 | 0.586 | 0.526 | 0.635 |
| Method | BoolQ | ReCoRD | SQuAD | SST-2 |
|---|---|---|---|---|
| FO-FT | 0.863 | 0.814 | 0.801 | 0.952 |
| Method | With prompt (“Question:{Q}\nAnswer:{A}”) | Raw format |
|---|---|---|
| MeZO | 0.329 | 0.122 |
| Bilevel-ZOFO | 0.762 | 0.744 |
| Method | GSM8K (acc) | Math500 (acc) | HumanEval (pass@1) |
|---|---|---|---|
| Before tuning | 0.420 | 0.18 | 0.476 |
| LoRA | 0.727 | 0.28 | 0.518 |
| MeZO | 0.329 | 0.05 | 0.110 |
| Bilevel-ZOFO (ours) | 0.762 | 0.31 | 0.543 |
| Method | BoolQ | SST-2 |
|---|---|---|
| DePT | 0.813 | 0.932 |
| MeZO | 0.792 | 0.865 |
| Bilevel-ZOFO + DePT | 0.852 | 0.946 |
LLM Fine-Tuning
LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.
See all 787 papers