cs.LGNov 2, 2026

The Cost of Mismatch: Noise Amplification in Zeroth-Order Reinforcement Learning

Authors: Lianmin Chen, Junbin Qiu, Chenxing Wei, Yao Shu, Kun He

Organizations: School of Computer Science and Technology, Huazhong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou) · The University of Hong Kong · The Hong Kong University of Science and Technology

Abstract

Zeroth-order (ZO) optimization has emerged as a promising approach for memory-efficient large language model (LLM) post-training, yet its reliability in reinforcement learning with verifiable rewards (RLVR) remains largely unexplored. Motivated by the distinct data regimes of on-policy RL and fixed-data optimization, we develop a generic variance-theoretic framework to investigate what limits the reliability of ZO gradient estimation. Our analysis reveals that estimation reliability is structurally determined by whether the two finite-difference evaluations are performed on matched data. On-policy reinforcement learning (RL) under the mismatched-data regime produces a 1/μ21/μ^2 oracle-noise amplification that sets an irreducible noise floor as the perturbation scale μμ decreases, whereas Boltzmann Targeted SFT (BOLT) under the matched-data regime eliminates this amplification via noise cancellation. We further derive convergence guarantees with explicit constants that cleanly separate the two regimes. These results motivate the ZO-BOLT algorithm, which builds on BOLT's matched-data advantage for reliable ZO optimization of an RLVR-aligned objective. Controlled perturbation-scale sweeps and matched-versus-mismatched end-to-end comparisons validate the predicted effect of data matching. Experiments across three models and two mathematical reasoning benchmarks further show that ZO-BOLT consistently improves over the pretrained baseline while retaining a near-inference memory footprint. The code is publicly available at https://github.com/lianmin-chen/the_cost_of_mismatch.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning

    Jun 1, 2026Liyan Tan, Yequan Zhao, Yifan Yang +3Zeroth-Order OptimizationMemory-Efficient Training

  2. Zeroth-Order Riemannian Optimization on Fixed-Rank Manifolds for LLM Fine-Tuning

    Nov 7, 2026Xinjue Wang, Xiuheng Wang, Yejun Zhang +2Zeroth-Order OptimizationRiemannian Optimization