cs.LGSep 28, 2026

ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards

Authors: Hsiao-Ru Pan, Florent Draye, Bernhard Schölkopf

Organizations: MPI-IS · ELLIS Institute Tübingen

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recent progress in reinforcement learning with verifiable rewards (RLVR) has highlighted the effectiveness of simple critic-free policy-gradient methods such as Group Relative Policy Optimization (GRPO). In contrast, actor-critic methods rely on learned value functions whose approximation error can introduce bias through commonly used advantage estimators such as temporal-difference error. Motivated by this observation, we revisit trajectory-level control variates through an advantage-value formulation, which we call Advantage-Based Control Variates (ABC). This formulation reveals that the covariance structure is closely related to the return decomposition used in Direct Advantage Estimation (DAE). Finally, we combine ABC with DAE into a single actor-critic algorithm and evaluate it in an offline-to-online RLVR setting, where the critic is first trained on previously collected trajectories and adapted during online learning. On mathematical reasoning tasks, ABC achieves performance competitive with GRPO using substantially fewer online optimization steps.

Figures & tables

Figure 1: Comparison of unbiased policy-gradient estimators. REINFORCE uses the sampled return directly, while a value baseline removes state-dependent variation. ABC additionally uses estimated advantages as a trajectory-level control variate and adds an analytic correction term to preserve unbiasedness.
Figure 2: Overview of the offline-to-online training pipeline. We first collect trajectories with verifiable rewards and use DAE to pretrain the critic. During online training, trajectories sampled from the current actor are used both for ABC policy-gradient updates and continued DAE critic training.
 g^=1N∑i=1N∑t=0∣τi∣−1g^ABC(Aϕ,Vϕ)(st,at)\hat{g}=\frac{1}{N}\sum_{i=1}^{N}\sum_{t=0}^{|\tau_{i}|-1}\hat{g}_{\mathrm{ABC}}(A_{\phi},V_{\phi})(s_{t},a_{t}) 
Algorithm 1 Online Actor-Critic Training with ABC and DAE
Figure 4
Figure 5: ABC training curves.
Model sizeMethodMATH500AMC23AIME25AIME26
0.6BDr. GRPO47.4 /80.425.9 /67.51.4 /13.30.8 /10.0
REINFORCE29.5 /72.215.3 /65.00.5 /6.70.1 /3.3
ABC53.2 /83.629.3 /75.00.6 /16.70.7 /10.0
1.7BDr. GRPO67.1 /90.039.5 /85.03.5 /20.03.5 /16.7
REINFORCE62.3 /89.636.7 /82.52.4 /13.32.4 /16.7
ABC68.8 /92.240.7 /82.54.8 /23.33.1 /20.0
Table 1: Final-model evaluation scores in percent, reported as (mean/pass)@32. Mean is the average reward over 32 sampled completions per problem; pass is the percentage of problems with at least one correct completion.
Figure 6: Comparison with biased gradients.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Dr. GRPO training curves.
Figure 8: Residual ( ww ) squared distribution.
Figure 9: REINFORCE training curves.
Figure 11
HyperparameterValue
Optimization and objective
ObjectiveDr. GRPO [ 17 ]
OptimizerAdamW
Learning rate1×10−61\times 10^{-6} (constant)
Adam (β1,β2,ϵ)(\beta_{1},\beta_{2},\epsilon)(0.9,0.95,10−8)(0.9,0.95,10^{-8})
Weight decay0
Appendix
Table 2: Hyperparameters for the GRPO baselines.
HyperparameterValue
Optimization and objective
OptimizerAdamW
Batch size256 trajectories
Learning rate (advantage head)2.5×10−62.5\times 10^{-6}
Learning rate (others)2.5×10−52.5\times 10^{-5}
Learning rate scheduleLinear warmup (5%) followed by linear annealing
Appendix
Table 3: Hyperparameters for the offline critic training.
HyperparameterValue
Shared Hyperparameters
OptimizerAdamW
Learning rate scheduleconstant
Weight decay0
Rollout Batch size81928192 trajectories
Actor Hyperparameters
Appendix
Table 4: Hyperparameters for the online actor-critic training.
More in this topic

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,668 papers

Explore similar work

CardsList

  1. Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards

    Oct 11, 2026Yibo Wang, Yuanxin Liu, Zhaoran WangReinforcement Learning with Verifiable RewardsPolicy Gradient Methods

  2. Discounted Beta-Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewards

    Mar 19, 2026Haechan Kim, Soohyun Ryu, Gyouk Chu +2Reinforcement LearningRL for Language Model Reasoning

    Kim Jaechul Graduate School of AI, KAIST, Daejeon, South Korea · KRAFTON, Seoul, South Korea · AITRICS, Seoul, South Korea

  3. BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards

    Jun 27, 2026Yupeng Chang, Yuan Wu, Yi ChangReinforcement LearningRL for Language Model Reasoning

    School of Artificial Intelligence, Jilin University · 2Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · 3International Center of Future Science, Jilin University