The Cost of Mismatch: Noise Amplification in Zeroth-Order Reinforcement Learning
Organizations: School of Computer Science and Technology, Huazhong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou) · The University of Hong Kong · The Hong Kong University of Science and Technology
Abstract
Zeroth-order (ZO) optimization has emerged as a promising approach for memory-efficient large language model (LLM) post-training, yet its reliability in reinforcement learning with verifiable rewards (RLVR) remains largely unexplored. Motivated by the distinct data regimes of on-policy RL and fixed-data optimization, we develop a generic variance-theoretic framework to investigate what limits the reliability of ZO gradient estimation. Our analysis reveals that estimation reliability is structurally determined by whether the two finite-difference evaluations are performed on matched data. On-policy reinforcement learning (RL) under the mismatched-data regime produces a oracle-noise amplification that sets an irreducible noise floor as the perturbation scale decreases, whereas Boltzmann Targeted SFT (BOLT) under the matched-data regime eliminates this amplification via noise cancellation. We further derive convergence guarantees with explicit constants that cleanly separate the two regimes. These results motivate the ZO-BOLT algorithm, which builds on BOLT's matched-data advantage for reliable ZO optimization of an RLVR-aligned objective. Controlled perturbation-scale sweeps and matched-versus-mismatched end-to-end comparisons validate the predicted effect of data matching. Experiments across three models and two mathematical reasoning benchmarks further show that ZO-BOLT consistently improves over the pretrained baseline while retaining a near-inference memory footprint. The code is publicly available at https://github.com/lianmin-chen/the_cost_of_mismatch.
Figures & tables
| Model | Task | Baseline | BOLT (FO) | ZO-BOLT (ZO) | Rec. (%) |
|---|---|---|---|---|---|
| Qwen2.5-1.5B | GSM8K | 52.24 | 63.99 | 63.78 | 98.2 |
| Qwen2.5-1.5B | MATH | 29.82 | 33.60 | 32.84 | 79.9 |
| Qwen2.5-7B | GSM8K | 75.93 | 86.56 | 85.73 | 92.2 |
| Qwen2.5-7B | MATH | 50.30 | 61.42 | 58.00 | 69.2 |
| DeepSeek-R1-Distill-Qwen-1.5B | GSM8K | 60.97 | 70.77 | 66.89 | 60.4 |
| DeepSeek-R1-Distill-Qwen-1.5B | MATH | 52.28 | 54.66 | 52.78 | 21.0 |
| Method | Qwen2.5-1.5B | Qwen2.5-7B | ||
|---|---|---|---|---|
| Peak (MB) | over Inference | Peak (MB) | over Inference | |
| Inference only | 3,281 | – | 14,900 | – |
| BOLT (FO) | 10,363 | 28,436 | ||
| GRPO (FO) | 12,012 | 29,379 | ||
| ZO-BOLT | 4,201 | 16,014 | ||
| ZO-GRPO (ZO) | 5,245 | 16,598 | ||
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Step | Matched | Mismatched | Step | Matched | Mismatched |
|---|---|---|---|---|---|
| 0 | 52.24 | 52.24 | 450 | 60.82 | 42.60 |
| 50 | 52.85 | 49.35 | 500 | 61.43 | 45.56 |
| 100 | 53.61 | 48.52 | 550 | 63.33 | 45.25 |
| 150 | 55.88 | 46.85 | 600 | 63.48 | 44.95 |
| 200 | 54.37 | 48.75 | 650 | 63.78 | 43.58 |
| 250 | 56.72 | 45.25 | 700 | 63.55 | 40.24 |
| Step | Matched | Mismatched | Step | Matched | Mismatched |
|---|---|---|---|---|---|
| 0 | 52.24 | 52.24 | 500 | 54.82 | 47.68 |
| 50 | 53.68 | 53.08 | 550 | 55.66 | 49.35 |
| 100 | 52.47 | 51.40 | 600 | 55.05 | 46.39 |
| 150 | 52.62 | 49.58 | 650 | 56.72 | 42.60 |
| 200 | 53.76 | 49.81 | 700 | 57.33 | 42.22 |
| 250 | 55.50 | 51.10 | 750 | 59.00 | 42.52 |
Zeroth-Order Optimization
Zeroth-order optimization uses function values or comparisons, rather than gradients, to optimize objectives when differentiation is unavailable or costly.
See all 99 papers