Organizations: Institute for AI Industry Research (AIR), Tsinghua University · Department of Computer Science and Technology, Tsinghua University · SIA-Lab of Tsinghua AIR and ByteDance Seed
Reinforcement learning (RL) based on PPO-style clipping has become an effective approach to improving the reasoning capabilities of large language models (LLMs). However, standard token-level clipping measures policy change only through the conditional probability of the current token, while the prefix conditioning context is itself generated by the policy. This mismatch becomes especially consequential in long-response and off-policy training. We reformulate autoregressive generation as a finite-horizon Markov decision process and derive its performance-difference identity under behavior-policy sampling. The resulting change of measure identifies the cumulative likelihood ratio of the generated prefix as the exact importance weight for each stepwise policy-improvement term. Starting from the monotonic-improvement principle of trust-region policy optimization, we then construct a prefix-level trust region using the exact reverse-KL coordinate r−1−logr, whose behavior-policy expectation equals the corresponding prefix-distribution KL divergence. This yields EPIC, a PPO-style sequence-clipping method that directly regulates prefix-level policy drift without relying on a local quadratic approximation or expert-routing replay. Experiments on off-policy reinforcement learning with a sparse mixture-of-experts language model show that EPIC consistently outperforms standard and sequence-level clipping baselines across mathematical-reasoning benchmarks, while remaining competitive with routing-replay-based alternatives. Code is available at https://github.com/LiChengxi666/EPIC.
Figures & tables
Figure 1: Over view of EPIC. (a) How GRPO, GSPO, and EPIC compute importance ratios along an autoregressive response. (b) Positive-advantage updates with illustrative clipping bounds. GRPO clips token ratios independently, whereas GSPO uses a shared response-level ratio and can clip all positions together. In these examples, EPIC detects cumulative drift missed by GRPO while preserving early-prefix signals suppressed by GSPO. Hatched boxes indicate clipped positions. (c) Driver-process resident CPU memory during training. EPIC uses substantially less memory than GRPO Clip-Higher combined with R2 or R3, without requiring expert-routing replay. The vertical axis is logarithmic.
Accuracy (%)
Method
AIME 24
AIME 25
AMC 23
HMMT 25
Average
GRPO
23.61
16.81
69.48
4.72
28.65
DAPO
31.67
24.03
76.25
9.86
35.45
GSPO
30.00
20.56
73.44
7.36
32.84
CTPO
30.14
20.97
76.04
8.75
33.98
RIPO
26.94
21.81
74.69
8.47
32.98
Table 1: Main results on Qwen3-30B-A3B-Base. Results are averaged over three independent runs.
Figure 2: Validation and reward trajectories under the aligned off-policy MoE setting. Curves average three seeds at matching training steps.
Figure 3: Position-wise behavior of the calibrated trust region. (a) Normalized late-stage clip fractions across relative response positions. (b) Position-wise clip-fraction variation during training. (c) Clip fraction and finite-lower-bound coverage under raw and geometric-mean behavior-probability calibration.
Figure 4: Comparison with routing replay. (a) Mean@8 accuracy (%) at the checkpoint maximizing the equal-weight average across the four benchmarks. (b) Process-level resident CPU memory during training.
Parameter
GRPO
DAPO
GSPO
CTPO
RIPO
EPIC
Sampling and sequence limits
Temperature (train / val)
1.0 / 1.0
1.0 / 1.0
1.0 / 1.0
1.0 / 1.0
1.0 / 1.0
1.0 / 1.0
Top- p (train / val)
1.0 / 0.7
1.0 / 0.7
1.0 / 0.7
1.0 / 0.7
1.0 / 0.7
1.0 / 0.7
Responses per prompt (train / val)
8 / 8
8 / 8
8 / 8
8 / 8
8 / 8
8 / 8
Oversampling ratio
0.1
0.1
0.1
0.1
0.1
0.1
Maximum prompt length (tokens)
2,048
2,048
2,048
2,048
2,048
2,048
Table 2: Configurations of the main Qwen3-30B-A3B-Base experiments. Batch and minibatch sizes are measured in prompts.
Figure 5: Validation mean@8 on the four individual benchmarks throughout training.
Figure 6: Additional training dynamics under the aligned off-policy MoE setting. Curves average three seeds at matching training steps. Clip fraction, policy entropy, gradient norm, and train–rollout gap are shown on logarithmic vertical scales.
Figure 7: Local-approximation diagnostics for the main EPIC run. (a) Logged mean absolute cumulative prefix log-ratio at each training step; the dashed line marks its median across steps. (b) Relative error of the quadratic coordinate s2/2 , evaluated at positive and negative representative magnitudes from panel (a), with exp(s)−1−s as the reference denominator. These are analytical evaluations at representative magnitudes, not measured averages of token-level approximation errors.
Configuration
Budget calibration
Optimization ratio
Raw-prob
μ(oi,1:t∣x)tδ±
[ri,tpre(θ)]1/t
Cumulative-ratio
[μ(oi,1:t∣x)]1/ttδ±
ri,tpre(θ)
EPIC (default)
[μ(oi,1:t∣x)]1/ttδ±
[ri,tpre(θ)]1/t
Table 3: Configurations used in the prefix-probability ablation.
Figure 8: Ablation of prefix-probability calibration. The three panels show AIME24 validation performance, gradient-norm trajectories on a logarithmic scale, and late-stage position-wise clipping profiles. In the right panel, clipping fractions are normalized by the mean across position bins at each training step; shaded regions indicate the standard deviation over steps 201–300.
Single-copy storage (GiB)
Routing state
Index dtype
Bytes/index
T=9,000
T=18,432
R2 CPU indices
uint8
1
1.65
3.375
R3 rollout indices
int32
4
6.59
13.50
Table 4: Estimated storage for one dense routing-index tensor.
Reinforcement learning (RL) has become a cornerstone for improving the reasoning capabilities of large language models (LLMs), but the need for on-policy data substantially limits training efficiency. Reusing off-policy data through importance sampling (IS) can improve efficiency but introduce considerable instability. Hence, algorithms such as PPO and GRPO widely adopt IS-ratio clipping to stabilize training. However, training stability and gradient estimate are mainly determined by the product of IS ratio and advantage. To further stabilize training, we propose ACPO, which clips the product of the IS ratio and the advantage, leading to more stable gradient estimates. We also establish a connection between ACPO and gradient clipping in policy mirror descent (PMD), which is a standard technique to stabilize optimization process, and prove the convergence of clipped-PMD under the standard RL setting. Experiments on widely used mathematical reasoning benchmarks show that ACPO consistently outperforms PPO and GRPO in both accuracy and training efficiency, delivering 4-6 percentage points gains on standard math benchmarks, with Qwen3-8B+PPO. Hence, ACPO is a practical and effective alternative to conventional IS-ratio clipping for RL post-training of LLMs.
Reinforcement learning with verifiable rewards (RLVR) has become standard for improving LLM reasoning. However, existing PPO-style trust-region mechanisms remain position-agnostic by enforcing uniform thresholds across all tokens independently. This pointwise treatment conflicts with autoregressive generation in two critical ways. First, uniform thresholds ignore autoregressive asymmetry. Early-stage deviations produce compounding sequence-level drift, causing static thresholds to under-regulate early divergence and excessively constrain late-stage exploration. Second, evaluating token-level divergence in isolation overlooks cumulative prefix drift, granting the same divergence allowance regardless of how far the conditioning history has already deviated from the rollout policy. To address this limitation, we propose CPPO (Cumulative Prefix-divergence Policy Optimization), a token-level masking rule that aligns updates with a finite-horizon policy-improvement bound via two coupled mechanisms. First, a position-weighted threshold imposes stricter limits at early positions whose effects persist longer, relaxing constraints for late-stage tokens. Second, a cumulative prefix budget tracks historical deviations, dynamically restricting further token-level deviation to prevent compounding errors along the prefix. Empirically, CPPO enhances training stability and significantly improves reasoning accuracy across various model scales.
Proximal constraints are fundamental to the stability of the Large Language Model reinforcement learning. While the canonical clipping mechanism in PPO serves as an efficient surrogate for trust regions, we identify a critical bottleneck: fixed bounds strictly constrain the upward update margin of low-probability actions, disproportionately suppressing high-advantage tail strategies and inducing rapid entropy collapse. To address this, we introduce Band-constrained Policy Optimization (BandPO). BandPO replaces canonical clipping with Band, a unified theoretical operator that projects trust regions defined by f-divergences into dynamic, probability-aware clipping intervals. Theoretical analysis confirms that Band effectively resolves this exploration bottleneck. We formulate this mapping as a convex optimization problem, guaranteeing a globally optimal numerical solution while deriving closed-form solutions for specific divergences. Extensive experiments across diverse models and datasets demonstrate that BandPO consistently outperforms canonical clipping and Clip-Higher, while robustly mitigating entropy collapse.