cs.LGAug 24, 2026

Best Practice Critic Optimization

Authors: Penghui Qi, Xiangxin Zhou, Wee Sun Lee

Organizations: National University of Singapore · Tencent Hunyuan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable. We study this instability and develop Best Practice Critic Optimization (BPCO), a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of each design choice. Across mathematical reasoning tasks with models ranging from 1.5B parameters to 30B-A3B mixtures of experts, BPCO improves a strong critic-based baseline consistently, and matches or exceeds a group-based baseline while sampling one response per prompt. The same recipe also improves learning with rubric-based rewards. These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation. Code is available at https://github.com/QPHutu/golden_critic.

Figures & tables

Figure 1: PPO and DPPO in the sanity test. With λ=1\lambda=1 , PPO’s training reward collapses after an initial increase, whereas DPPO remains stable. Using λ=0.99\lambda=0.99 makes DPPO unstable again.
Figure 2: Effect of bounding critic predictions to the reward range. The unbounded linear head predicts values outside the binary-reward range [0,1][0,1] (right), leading to unstable training reward (left) and AIME 2025 avg@32 (middle). The bounded value keeps predictions within the reward range and yields stable training.
Figure 3: Effect of the unbiased value target. Regressing to the final reward gives more stable training reward (left) and AIME 2025 avg@32 (middle) than a target bootstrapped with λ=0.99\lambda=0.99 . Explained variance against the bootstrapped target rapidly approaches one (right) because that target is biased; this unexpectedly high value does not imply accurate prediction of the observed return.
Figure 4: Effect of removing batch-wise advantage normalization. Removing normalization achieves comparable training reward (left) while mitigating the overfitting risk on AIME 2025 (middle). Batch-wise advantage normalization forces small residual advantages to a large, growing range when the policy approaches to optimal (right).
Figure 5: Effect of giving the reference answer only to the critic. Privileged input accelerates training-reward improvement (left) and increases explained variance (right). Its AIME 2025 avg@32 rises faster but peaks earlier (middle), revealing a greater risk of overfitting in this small-data setting.
Figure 6: Effect of length-adaptive GAE. A fixed λπ=0.99\lambda_{\pi}=0.99 attains rapid reward improvement (left), but its AIME 2025 avg@32 declines after the initial peak (middle). LA-GAE with α=0.4\alpha=0.4 retains better training efficiency than λ=1\lambda=1 while mitigating the overfitting risk.
Figure 7: Results on a larger DeepScaleR dataset with DeepSeek-R1-Distill-Qwen-1.5B.
Figure 8: Ablation of bounded value prediction and batch-wise advantage normalization.
Figure 9: Ablation of privileged information for critic training.
Figure 10: Results with larger MoE models on DAPO-Math-17k dataset.
Figure 11: Results under rubric-based rewards.

Explore similar work

CardsList

  1. EasyPPO: Stabilizing the Critic Is Key

    Sep 29, 2026Xuanyi Zhou, Qiuyang Mang, Huanzhi Mao +7RL for Language ModelsProximal Policy Optimization

    University of California, Berkeley · Princeton University

  2. SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation

    Aug 10, 2026Jefferson Hernandez, Jaywon Koo, Zilin Xiao +2RL for Language Model ReasoningGroup Relative Policy Optimization

    Rice University

  3. AGPO: Adaptive Group Policy Optimization with Dual Statistical Feedback

    May 20, 2026Miaobo Hu, Shuhao Hu, Bokun Wang +5RL for Language Model ReasoningGroup Relative Policy Optimization

    Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100092, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China