A key strength of Proximal Policy Optimization (PPO) is its learned critic, which uses historical trajectories collected during reinforcement learning to estimate expected returns and reduce policy-gradient variance. However, we find that the critic is also a major source of instability in reinforcement learning for large language models (LLMs). We identify two critic failure modes that destabilize PPO. First, filtering truncated rollouts from both actor and critic shifts the policy objective to reward conditioned on completion, allowing truncation to increase even as conditional reward improves. Second, heterogeneous return noise can cause high-variance prompts to dominate critic updates in finite batches. We introduce EasyPPO to address these failures. Actor-only overlong filtering trains the critic on returns from both completed and truncated rollouts. Noise-normalized critic regression weights each prompt's critic loss by the inverse standard deviation of its sampled returns, balancing noise contributions across prompts. Moderately smaller critic mini-batches confine outlier influence to fewer rollouts during gradient clipping. Across continuous-reward coding on FrontierCS, binary-reward mathematical reasoning on AIME24, and multi-turn search on Search-R1, EasyPPO remains stable throughout the full training horizon and consistently outperforms vanilla PPO, VAPO, and HL-Gauss PPO. Its best validation scores show relative gains of 14.89%, 2.28%, and 9.47% over PPO, respectively.
Figures & tables
Figure 1: EasyPPO addresses two sources of critic instability with three simple modifications. Left: Filtering truncated rollouts from both actor and critic updates can increase truncation and lower overall reward; EasyPPO retains these rollouts for critic updates only. Middle: Return variability differs across prompts, causing unequal critic gradient scales; EasyPPO weights prompts inversely by their standard deviation. Right: Smaller critic mini-batches limit outlier influence through gradient clipping.
Figure 2: Actor-only overlong filtering improves stability but does not eliminate instability. Rollout statistics during PPO training of Qwen3.5-9B in the FrontierCS setting, using 200 problems generated by FrontierSmith, rollout batches of 512 , a 30 -step critic warm-up (gray), and a 32,768 -token response limit. Left: fraction of truncated rollouts. Middle: mean reward among non-truncated rollouts. Right: mean reward over all rollouts. Actor + critic filtering improves reward among completed rollouts while the truncation ratio approaches 1 , leaving overall reward low. No filtering exhibits repeated reward collapses. Actor-only filtering achieves higher reward and lower truncation overall, but the spike and reward drop near step 160 reveal residual instability.
Figure 3: Noise normalization balances critic gradient contributions across prompts. Offline Qwen3.5-9B PPO diagnostics in the FrontierCS setting, using the data and sampling settings of \Cref fig:frontiercs-overlong-filtering. Columns: steps 30 , 60 , 90 , and their pooled responses. Top: without noise normalization. Bottom: with noise normalization. Boxes summarize per-response gradient norms before clipping within each return-standard-deviation interval. Consistent with \Cref eq:prompt-bias, prompts with more variable returns contribute larger gradients; noise normalization markedly reduces this dependence.
Figure 4: EasyPPO sustains learning across tasks. Left: FrontierCS. Middle: AIME. Right: Search-R1. Top: training scores. Bottom: validation scores. Gray marks critic warmup.
Figure 5: EasyPPO stabilizes critic learning. Critic diagnostics on FrontierCS. Left: critic gradient norm before clipping. Right: explained variance of critic value predictions. Gray marks critic warmup.
Figure 6: Noise normalization stabilizes training with various critic mini-batch configurations. FrontierCS ablation. Left: training score. Right: validation score.
Figure 7: Explained variance across critic mini-batch sizes. FrontierCS. Left: smoothed explained variance; gray marks critic warmup. Right: first training step with EV≥0 versus critic mini-batch count, measured from unsmoothed values.
Figure 8: Critic diagnostics on AIME and Search-R1. First two panels: AIME. Last two: Search-R1. Each pair shows critic gradient norm before clipping and explained variance of value predictions; the latter uses symmetric-log axes. Gray marks critic warmup.
Figure 9: EasyPPO maintains stable learning across critic initializations. FrontierCS. Left: training score. Middle: validation mean and min–max. Right: explained variance. Gray marks critic warmup.
Figure 10: Noise normalization reduces overall gradient imbalance in the AIME setting. Offline diagnostics with 512 responses per checkpoint ( 32 prompts, 16 responses each). Columns: steps 30 , 60 , 90 , and their pooled responses. Top: without noise normalization. Bottom: with noise normalization. Boxes summarize per-response critic-gradient norms before clipping, grouped by prompt return standard deviation.
Figure 11: The FrontierCS gradient-scale trend weakens after noise normalization at each checkpoint. Columns: steps 30 , 60 , and 90 . Top/bottom: the same gradients before/after noise normalization. Each panel shows 512 responses; lines are descriptive linear fits.
Figure 12: Response-level gradient variability remains in the AIME training setting. Columns: steps 30 , 60 , and 90 . Top/bottom: the same gradients before/after noise normalization. Each panel shows 512 responses; lines are descriptive linear fits.
Figure 13: Joint filtering exhibits increasing truncation and declining performance on AIME. Left: truncation ratio. Middle: overall training score. Right: AIME24 validation score. Gray marks critic warmup. Training statistics use a five-point moving average with faint raw values; validation is unsmoothed.
Figure 14: EasyPPO sustains performance across all seven Search-R1 validation sets. Each panel shows one dataset from the aggregate comparison in \Cref fig:main-results. Gray marks the 30 -step critic warmup.
Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instability and memory overhead. Even where a critic is trained, it is discarded once training ends, although it has learned to predict outcomes. We revisit this trend and show that a pretrained critic's ability to predict future outcomes can make it a valuable asset for efficient long-horizon reasoning. First, we find that instability in critic-based RL for long chain-of-thought reasoning is largely an optimization artifact: keeping policy updates small and low in variance restores stable convergence. Second, a well-pretrained critic estimates the posterior probability of eventual success from later trajectory states and unfinished prefixes. Its predictions provide outcome-derived, dense, per-prefix learning signals that, during policy optimization, require neither completed rollouts, step-level annotations, nor external reward labels. Building on this insight, we introduce Reward-Free Policy Optimization (RFPO), which repurposes a single calibrated, frozen critic as a rollout-level reward, a value baseline for generalized advantage estimation, and a success forecaster for unfinished prefixes. We further show that binarizing the debiased score stops the policy from exploiting the critic's length bias. Binarized, RFPO matches supervised PPO without a single label in the training loop, while cutting compute and memory overhead. This makes RFPO well suited to long-horizon reasoning tasks, where outcomes arrive late and generation dominates cost: because rollouts can be rewarded before they finish, training no longer has to pay for waiting on every trajectory to complete. Our findings challenge the prevailing critic-free paradigm and establish critic-based, reward-free optimization as a scalable and computationally efficient path for LLM post-training.
Hongyang Li, Xiao Li, Caesar Wu +3
University of Luxembourg · Seafill Open-Source Community · Université Paris-Saclay
Reinforcement learning (RL) has become a cornerstone for improving the reasoning capabilities of large language models (LLMs), but the need for on-policy data substantially limits training efficiency. Reusing off-policy data through importance sampling (IS) can improve efficiency but introduce considerable instability. Hence, algorithms such as PPO and GRPO widely adopt IS-ratio clipping to stabilize training. However, training stability and gradient estimate are mainly determined by the product of IS ratio and advantage. To further stabilize training, we propose ACPO, which clips the product of the IS ratio and the advantage, leading to more stable gradient estimates. We also establish a connection between ACPO and gradient clipping in policy mirror descent (PMD), which is a standard technique to stabilize optimization process, and prove the convergence of clipped-PMD under the standard RL setting. Experiments on widely used mathematical reasoning benchmarks show that ACPO consistently outperforms PPO and GRPO in both accuracy and training efficiency, delivering 4-6 percentage points gains on standard math benchmarks, with Qwen3-8B+PPO. Hence, ACPO is a practical and effective alternative to conventional IS-ratio clipping for RL post-training of LLMs.
Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable. We study this instability and develop Best Practice Critic Optimization (BPCO), a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of each design choice. Across mathematical reasoning tasks with models ranging from 1.5B parameters to 30B-A3B mixtures of experts, BPCO improves a strong critic-based baseline consistently, and matches or exceeds a group-based baseline while sampling one response per prompt. The same recipe also improves learning with rubric-based rewards. These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation. Code is available at https://github.com/QPHutu/golden_critic.
Penghui Qi, Xiangxin Zhou, Wee Sun Lee
National University of Singapore · Tencent Hunyuan