ENGINE: Endogenous Variational MultiScale Optimization for Zeroth-Order LLM Fine-Tuning
Organizations: Department of Artificial Intelligence, School of Engineering, Westlake University · Department of Electronic and Electrical Engineering, Southern University of Science and Technology · Zhejiang University
Abstract
Zeroth-order (ZO) optimization enables memory-efficient large language model (LLM) fine-tuning, and recent methods incorporate curvature-aware strategies. However, in the high-dimensional parameter space of LLMs, these methods inherently suffer from exploding gradient variance and prohibitive costs. Conversely, confining updates to a low-dimensional subspace reduces variance but discards critical high-dimensional information, degrading performance. This creates a fundamental dilemma: full-space optimization is informative but has high variance, while subspace optimization has low variance but loses information. To break this trade-off, we propose ENGINE (Endogenous Variational MultiScale). Inspired by computational mechanics, we adapt static Variational MultiScale (VMS) theory to dynamic optimization, treating low- and high-dimensional spaces as resolved coarse and unresolved fine scales. Through a multigrid-style V-cycle, ENGINE alternates between spaces, dynamically probing fine-grained information in the high-dimensional space and injecting it to correct the efficient low-dimensional sampling process. Under appropriate assumptions, statistical analysis characterizes variance reduction and correction of curvature-induced bias, and convergence analysis establishes a stationarity bound for the corrected coarse updates. Evaluations on standard LLM benchmarks show that ENGINE achieves higher accuracy and faster convergence than existing ZO methods, achieving up to a speed-up in forward passes with zero additional peak memory overhead. Code is available at https://github.com/AI4Science-WestlakeU/ENGINE.
Figures & tables
| Space | Effic. | Var. | Perf. | Curv. |
|---|---|---|---|---|
| Full space | ||||
| Subspace | ||||
| Engine (Ours) |
| GLUE (FP32) | GPT-2-XL (1.5B) | OPT-2.7B | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | MNLI | QNLI | SST-2 | CoLA | Avg. | MNLI | QNLI | SST-2 | CoLA | Avg. |
| MeZO | 38.4 | 59.2 | 74.3 | 65.8 | 59.4 | 50.4 | 75.2 | 88.7 | 68.4 | 70.7 |
| MeZO-Adam | 50.8 | 71.9 | 91.2 | 71.4 | 71.3 | 45.9 | 72.3 | 86.9 | 68.5 | 68.4 |
| MeZO-SVRG | 49.8 | 65.5 | 84.5 | 65.6 | 66.4 | 41.5 | 59.8 | 62.9 | 66.5 | 57.7 |
| LOZO | 42.9 | 60.6 | 80.0 | 66.2 | 62.4 | 43.2 | 59.5 | 66.2 | 66.3 | 58.8 |
| HiZOO | 48.4 | 66.8 | 88.9 | 71.2 | 68.8 | 50.9 | 74.7 | 88.2 | 66.3 | 70.0 |
| GLUE (FP32) | GPT-2-XL (1.5B) | OPT-2.7B | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | MNLI | QNLI | SST-2 | CoLA | Avg. | MNLI | QNLI | SST-2 | CoLA | Avg. |
| LOREN | 50.6 | 74.1 | 89.0 | 72.0 | 71.4 | 54.1 | 72.6 | 88.2 | 68.1 | 70.8 |
| Engine w/o VMS-EO | 48.8 | 72.7 | 90.2 | 71.5 | 70.8 | 54.7 | 73.8 | 89.5 | 69.1 | 71.8 |
| Engine (Ours) | 52.0 | 77.0 | 91.8 | 72.7 | 73.4 | 55.5 | 75.4 | 91.0 | 69.5 | 72.9 |
| vs LOREN | +1.4 | +2.9 | +2.8 | +0.7 | +2.0 | +1.4 | +2.8 | +2.8 | +1.4 | +2.1 |
| SuperGLUE (BF16) | LLaMA-3-8B | OPT-13B | ||||||||
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Estimator | Bias | Trace variance | MSE |
|---|---|---|---|
| Isotropic LoRA RGE | |||
| LoRA+LOREN | |||
| Restricted full-space LOREN | |||
| Ideal Engine ( ) | |||
| Ideal Engine ( ) |
| GLUE (FP32) | GPT-2-XL (1.5B) | OPT-2.7B | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | MNLI | QNLI | SST-2 | CoLA | Avg. | MNLI | QNLI | SST-2 | CoLA | Avg. |
| MeZO | 38.4 1.0 | 59.2 0.2 | 74.3 1.0 | 65.8 0.3 | 59.4 | 50.4 3.3 | 75.2 1.0 | 88.7 0.2 | 68.4 2.3 | 70.7 |
| MeZO-Adam | 50.8 1.4 | 71.9 4.4 | 91.2 0.5 | 71.4 0.8 | 71.3 | 45.9 1.2 | 72.3 3.2 | 86.9 2.8 | 68.5 1.0 | 68.4 |
| MeZO-SVRG | 49.8 1.1 | 65.5 1.0 | 84.5 1.5 | 65.6 0.4 | 66.4 | 41.5 0.2 | 59.8 1.0 | 62.9 1.9 | 66.5 0.2 | 57.7 |
| LOZO | 42.9 0.9 | 60.6 1.1 | 80.0 1.0 | 66.2 0.5 | 62.4 | 43.2 1.1 | 59.5 0.8 | 66.2 3.0 | 66.3 0.3 | 58.8 |
| HiZOO | 48.4 0.2 | 66.8 3.4 | 88.9 0.2 | 71.2 0.9 | 68.8 | 50.9 1.0 | 74.7 1.0 | 88.2 0.6 | 66.3 0.2 | 70.0 |
| GLUE (FP32) | GPT-2-XL (1.5B) | OPT-2.7B | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | MNLI | QNLI | SST-2 | CoLA | Avg. | MNLI | QNLI | SST-2 | CoLA | Avg. |
| LOREN | 50.6 0.5 | 74.1 1.2 | 89.0 0.9 | 72.0 0.8 | 71.4 | 54.1 0.7 | 72.6 1.8 | 88.2 0.8 | 68.1 0.2 | 70.8 |
| Engine w/o VMS-EO | 48.8 0.6 | 72.7 1.1 | 90.2 0.5 | 71.5 0.7 | 70.8 | 54.7 0.8 | 73.8 1.3 | 89.5 0.4 | 69.1 1.0 | 71.8 |
| Engine (Ours) | 52.0 0.5 | 77.0 1.0 | 91.8 0.5 | 72.7 0.6 | 73.4 | 55.5 0.5 | 75.4 1.2 | 91.0 0.4 | 69.5 0.2 | 72.9 |
| vs LOREN | +1.4 | +2.9 | +2.8 | +0.7 | +2.0 | +1.4 | +2.8 | +2.8 | +1.4 | +2.1 |
| SuperGLUE (BF16) | LLaMA-3-8B | OPT-13B | ||||||||
| Method | GPT-2-XL | OPT-2.7B | LLaMA-3-8B, OPT-13B |
|---|---|---|---|
| MeZO | {1e-4, 5e-5, 1e-5, 5e-6 , 1e-6} | {1e-4, 5e-5, 1e-5, 5e-6 , 1e-6} | {1e-5, 5e-6, 1e-6 , 5e-7, 1e-7} |
| MeZO-Adam | {1e-3, 5e-4, 1e-4 , 5e-5, 1e-5} | {1e-3, 5e-4, 1e-4, 5e-5 , 1e-5} | {1e-3, 5e-4, 1e-4, 5e-5 , 1e-5} |
| MeZO-SVRG full | {1e-4, 5e-5 , 1e-5, 5e-6, 1e-6} | {1e-4, 5e-5 , 1e-5, 5e-6, 1e-6} | {1e-5, 5e-6, 1e-6 , 5e-7, 1e-7} |
| MeZO-SVRG mini | {1e-5, 5e-6, 1e-6 , 5e-7, 1e-7} | {1e-5, 5e-6, 1e-6 , 5e-7, 1e-7} | {1e-6, 5e-7, 1e-7 , 5e-8, 1e-8} |
| LOZO | {1e-4, 5e-5, 1e-5, 5e-6 , 1e-6} | {1e-4, 5e-5, 1e-5, 5e-6 , 1e-6} | {1e-5, 5e-6, 1e-6, 5e-7 , 1e-7} |
| HiZOO | {1e-3, 5e-4, 1e-4, 5e-5 , 1e-5} | {1e-4, 5e-5, 1e-5, 5e-6 , 1e-6} | {1e-5, 5e-6, 1e-6 , 5e-7, 1e-7} |
| GPT-2-XL | OPT-2.7B | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| MNLI | QNLI | CoLA | SST-2 | MNLI | QNLI | CoLA | SST-2 | ||
| 5 | 1.00 | 1.00 | 0.94 | 1.00 | 5 | 1.00 | 1.00 | 1.00 | 1.00 |
| 10 | 0.95 | 1.00 | 1.00 | 0.94 | 10 | 1.00 | 0.92 | 0.94 | 0.91 |
| 20 | 0.92 | 1.00 | 1.00 | 1.00 | 20 | 1.00 | 1.00 | 1.00 | 0.91 |
| 100 | 0.34 | 0.27 | 0.35 | 0.38 | 100 | 0.31 | 0.36 | 0.45 | 0.22 |
| 1000 | 0.00 | 0.00 | 0.00 | 0.00 | 1000 | 0.00 | 0.00 | 0.00 | 0.00 |
| 1.00 | 0.65 | 0.44 | |
| 0.91 | 0.76 | 0.34 | |
| 0.82 | 0.71 | 0.00 |
| Method | GPT-2-XL | OPT-2.7B | LLaMA-3-8B | OPT-13B | ||||
|---|---|---|---|---|---|---|---|---|
| SST-2 (70%) | QNLI (70%) | CB (65%) | CB (65%) | |||||
| Queries | Time | Queries | Time | Queries | Time | Queries | Time | |
| MeZO | 1.0 | 1.0 | 1.0 | 1.0 | 1.2 | 1.2 | 2.3 | 2.5 |
| MeZO-Adam | 10.3 | 9.8 | 9.7 | 9.2 | 1.0 | 1.0 | 1.0 | 1.0 |
| MeZO-SVRG | 3.9 | 2.1 | 4.1 | 4.5 | 2.1 | 1.8 | 3.2 | 2.9 |
| LOZO | 1.6 | 4.7 | 2.2 | 6.2 | 2.2 | 6.4 | 3.5 | 7.3 |
Zeroth-Order Optimization
Zeroth-order optimization uses function values or comparisons, rather than gradients, to optimize objectives when differentiation is unavailable or costly.
See all 99 papers