Towards Better Training Signal: Advantage Clipped Policy Optimization

Ruichuan Huang; Jinghan Liu; Congliang Chen

cs.LGSep 29, 2026

Towards Better Training Signal: Advantage Clipped Policy Optimization

Authors: Ruichuan Huang, Jinghan Liu, Congliang Chen

Organizations: Shenzhen Loop Area Institute

Abstract

Reinforcement learning (RL) has become a cornerstone for improving the reasoning capabilities of large language models (LLMs), but the need for on-policy data substantially limits training efficiency. Reusing off-policy data through importance sampling (IS) can improve efficiency but introduce considerable instability. Hence, algorithms such as PPO and GRPO widely adopt IS-ratio clipping to stabilize training. However, training stability and gradient estimate are mainly determined by the product of IS ratio and advantage. To further stabilize training, we propose ACPO, which clips the product of the IS ratio and the advantage, leading to more stable gradient estimates. We also establish a connection between ACPO and gradient clipping in policy mirror descent (PMD), which is a standard technique to stabilize optimization process, and prove the convergence of clipped-PMD under the standard RL setting. Experiments on widely used mathematical reasoning benchmarks show that ACPO consistently outperforms PPO and GRPO in both accuracy and training efficiency, delivering 4-6 percentage points gains on standard math benchmarks, with Qwen3-8B+PPO. Hence, ACPO is a practical and effective alternative to conventional IS-ratio clipping for RL post-training of LLMs.

Figures & tables

Panel of Figure 1 — Figure 1: $r_{t}(\theta)\widehat{A}_{t}$ across prompts

\pi_{k+1}(\cdot\mid s)=\underset{p(\cdot\mid s)\in\Delta(\mathcal{A})}{\operatorname{arg\,min}}\left\{\eta_{k}\frac{\alpha}{\max\{\alpha,\|A^{\pi_{k}}(s,\cdot)\|\}}\left[\left\langle A^{\pi_{k}}(s,\cdot),p(\cdot\mid s)\right\rangle+h^{p}(s)\right]+D_{\pi_{k}}^{p}(s)\right\},\forall s\in\mathcal{S}

Algorithm 1 The policy mirror descent (PMD) method with gradient clipping

Qwen2.5-7B-Math (avg@32)
Method	Weighted Acc.	MATH500	MINERVA	OLYMPIAD	AIME-like
PPO	48.4	79.0	33.0	41.2	21.4
PPO-AC	50.0	80.5	35.3	42.8	21.9
GRPO	51.2	82.0	37.2	43.2	24.1
GRPO-AC	52.6	82.8	37.8	45.6	24.9
Qwen3-1.7B (avg@16)

Table 1: Performance of methods on math benchmarks, where we choose the best weighted accuracy result during training. Bold marks the better result within each matched PPO or GRPO pair.

Figure 2: Accuracy during Training Process

Panel of Figure 3 — Figure 3: Policy entropy over the first 100 training steps across four models.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Data and rollout
	PPO-type		GRPO-type
Hyperparameter	Qwen2.5-Math-7B a	Qwen3	Qwen2.5-Math-7B a	Qwen3
Training dataset	DAPO	DAPO	DAPO	DAPO
Prompt batch size	512	512	512	256
Rollouts per prompt	1	1	8	4
Maximum prompt length	1024	1024	1024	1024

Appendix

Table 2: RL training hyperparameters. Qwen3 includes the 1.7B, 4B, and 8B models. Ratio clipping thresholds apply to the baselines, whereas

\alpha

applies to ACPO variants. All sampling parameters refer to training rollouts.

Figure 4: Qwen2.5-7B-Math on benchmarks

Figure 5: Qwen3-1.7B on benchmarks

Figure 6: Qwen3-4B on benchmarks

Figure 7: Qwen3-8B on benchmarks

Figure 8: Qwen3-1.7B with PPO-Type lr=1e-6 on benchmarks

Figure 9: Weighted Accuracy of Qwen3-1.7B with PPO-Type lr=1e-6

Explore similar work

Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective
May 8, 2026Yuheng Zhang, Chenlu Ye, Shuowei Jin +4RL for Language Model Reasoning Importance Sampling
UIUC · University of Michigan · Amazon
What are Key Factors for Updates in RL for LLM Reasoning?
Jun 21, 2026Peidong Wang, Demi Wang, Xufang Luo +5Reinforcement Learning RL for Language Model Reasoning
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · *Work done during internship at Microsoft Research Asia. · 2Microsoft Research +1
The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning
Jun 6, 2026Zhanke Zhou, Xiangyu Lu, Chentao Cao +4RL for Language Model Reasoning Policy Optimization
1TMLR Group, Department of Computer Science, Hong Kong Baptist University · 2Stanford University · 3Sydney AI Centre, The University of Sydney

May 8, 2026J/K move · Enter open · S save

Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective

Yuheng Zhang, Chenlu Ye, Shuowei Jin, Changlong Yu+3

UIUC · University of Michigan · Amazon

RL for Language Model Reasoning Importance Sampling Policy Gradient Methods RL Post-Training Off-Policy RL

Open paper AskPDF arXiv

Reinforcement learning, including reinforcement learning with verifiable rewards (RLVR), has emerged as a powerful approach for LLM post-training. Central to these approaches is the design of the importance sampling (IS) ratio used in off-policy policy-gradient estimation. Existing methods face a fundamental bias-variance dilemma: token-level IS ratios, as adopted by PPO (Schulman et al., 2017) and GRPO (Shao et al., 2024), introduce bias by ignoring prefix state distribution mismatch; full sequence ratios provide exact trajectory-level correction but suffer from high variance due to the multiplicative accumulation of per-token ratios, while GSPO (Zheng et al., 2025) improves numerical stability via length normalization at the cost of deviating from the exact full-sequence IS correction. In this work, we identify the cumulative token IS ratio, the product of per-token ratios up to position

t

, as a theoretically principled solution to this dilemma. We prove that, under the token-level policy-gradient formulation, this ratio provides an unbiased prefix correction for each token-level gradient term and has strictly lower variance than the full sequence ratio. Building on this insight, we propose CTPO (Cumulative Token Policy Optimization), which combines the cumulative token IS ratio with position-adaptive clipping that scales log-space clip bounds according to the natural

\sqrt{t}

growth of the cumulative log-ratio. This yields more consistent regularization across token positions. We implement and evaluate CTPO in the tool-integrated reasoning setting on several challenging mathematical reasoning benchmarks, achieving the best average performance across both model scales compared with strong GRPO and GSPO baselines. Code will be available at https://github.com/horizon-llm/CTPO.

Abstract

Explore similar work

Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective

What are Key Factors for Updates in RL for LLM Reasoning?

The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning