Reinforcement learning (RL) post-training has increasingly demonstrated strong ability to elicit reasoning behaviors in large language models (LLMs). For training efficiency, rollouts are typically generated in an off-policy manner using an older sampling policy and then used to update the current target policy. To correct the resulting discrepancy between the sampling and target policies, most existing RL objectives rely on a token-level importance sampling ratio. However, we observe unstable training dynamics in existing importance-weighted objectives under large policy lag. In this paper, we revisit LLM policy optimization under off-policy conditions and show that the exact correction for the prefix-marginal policy gradient is the cumulative prefix importance ratio. Token-level weighting does not explicitly correct for this prefix distribution shift. To stabilize LLM optimization under large off-policy drift, we propose a simple yet effective objective, Minimum Prefix Ratio (MinPRO). MinPRO replaces the unstable cumulative prefix ratio with a biased, non-cumulative surrogate based on the minimum token-level ratio observed in the preceding prefix. Extensive experiments on both dense and mixture-of-experts LLMs, across multiple mathematical reasoning benchmarks, demonstrate that MinPRO substantially improves training stability and peak performance in off-policy regimes.
Figures & tables
Figure 1: (a) An overview of MinPRO. ∇θJ denotes the policy gradient and ρt is the token-level importance sampling (IS) ratio. We take a step back to derive the full IS ratio ρ1:t , referred to as the prefix importance ratio. MinPRO is then developed by relaxing ρ1:t=ρ1:t−1⋅ρt to a non-cumulative proxy ρtρt . (b) AIME24 scores as functions of training steps for Qwen3-30B-A3B-Base under off-policy training.
Figure 2: Plots of (a) reward and (b) entropy as functions of training steps for Qwen3-8B-Base under on-policy and off-policy regimes. In the off-policy setting, each batch of sampled rollouts is stored in a buffer and used for training after a delay of 2 global steps, whereas the on-policy setting applies no delay between sampling and optimization.
Figure 3: Training reward curves as functions of training steps for Qwen3-8B-Base and Qwen3-14B-Base under off-policy optimization.
Method
AMC23
AIME24
AIME25
MATH500
Olympiad
Minerva
GSM8K
Avg
Base
21.6
3.5
3.5
44.5
20.2
20.6
70.3
26.3
GRPO
63.3
23.1
18.5
70.2
40.4
29.0
80.4
46.4
GSPO
64.4
25.9
18.3
78.4
44.4
31.4
87.3
50.0
CISPO
78.9
35.9
25.1
84.5
54.3
32.2
86.7
56.8
M2PO
82.2
33.9
25.8
85.8
56.5
37.7
88.7
58.7
8B-Base
MinPRO
82.5
36.1
30.3
86.5
56.5
33.6
90.2
59.4
Table 1: pass@1 scores under large off-policyness ( n=2 ).
Method
AMC23
AIME24
AIME25
MATH500
Olympiad
Minerva
GSM8K
Avg
n=4
GRPO
60.5
19.1
15.2
74.4
43.6
34.2
82.2
47.0
GSPO
60.8
20.7
15.8
73.6
44.2
34.2
88.2
48.2
CISPO
78.6
34.3
25.4
84.3
53.0
33.0
87.3
56.6
M2PO
77.9
34.8
24.5
83.8
54.0
36.0
87.6
56.9
MinPRO
79.3
38.2
28.3
84.8
54.2
35.3
88.3
58.3
n=0
GRPO
80.7
33.1
26.2
84.8
53.8
36.0
90.4
57.8
Table 2: pass@1 on Qwen3-8B-Base under more off-policy ( n=4 ) and near-on-policy ( n=0 ).
Method
Pass@k
Avg
1
2
4
8
16
32
64
128
GRPO
35.0
43.2
51.1
57.8
63.4
67.7
71.3
73.9
58.0
GSPO
36.2
44.0
51.4
58.0
63.5
68.0
71.2
73.5
58.2
CISPO
46.6
54.3
60.5
65.2
69.2
72.6
75.7
78.3
65.3
M2PO
47.3
54.4
60.7
65.7
69.5
72.8
75.6
77.5
65.4
8B-Base
MinPRO
49.6
56.7
62.5
67.0
70.5
73.1
75.3
77.2
66.5
Table 3: Mean-dataset pass@k scores averaged over AMC23, AIME24, and AIME25 under large off-policyness ( n=2 ). Boldface indicates the highest score within each model.
Figure 4: (a) Training reward curves and (b) average pass@1 scores as functions of training steps for Qwen3-30B-A3B-Base under off-policy optimization.
Method / Statistic
AMC23
AIME24
AIME25
MATH500
Olympiad
Minerva
GSM8K
Avg
MinPRO (minimum)
79.3
38.2
28.3
84.8
54.2
35.3
88.3
58.3
(a) Alternative prefix statistics
Arithmetic mean
72.4
29.8
21.9
83.0
51.6
33.5
86.1
54.0
Geometric mean
78.9
36.5
27.1
85.4
51.8
34.9
86.4
57.3
Cumulative prod
68.6
27.7
21.6
82.6
50.0
32.4
85.7
52.7
(b) Recent baselines
Table 4: Additional pass@1 comparisons (%) on Qwen3-8B-Base. Avg is the mean over seven benchmarks. The shared MinPRO reference and groups (a) – (b) use n=4 ; group (c) varies staleness. The shared reference is reported once.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Category
GRPO
GSPO
CISPO
M2PO
MinPRO
Sampling and Validation
Temperature
1.0
1.0
1.0
1.0
1.0
Top- p / Val Top- p
1.0 / 0.7
1.0 / 0.7
1.0 / 0.7
1.0 / 0.7
1.0 / 0.7
Clipping
Clip ratio (low / high)
0.2 / 0.28
2e−3 / 2e−3
1 / 4
—
1 / 4
Sequence Limits
Appendix
Table 5: Key hyperparameters in RL algorithms. “—” denotes not used.
MinPRO policy-loss implementation in VeRL.
Figure 6: Per-dataset pass@1 scores with Qwen3-30B-A3B-Base under off-policy training.
Setting
Method
AMC23
AIME24
AIME25
MATH500
Olympiad
Minerva
GSM8K
Avg
n=2
GRPO
63.3
23.1
18.5
70.2
40.4
29.0
80.4
46.4
AsymRE
58.8
22.0
14.2
79.2
43.8
34.9
85.8
48.4
MinPRO
82.5
36.1
30.3
86.5
56.5
33.6
90.2
59.4
Appendix
Table 6: pass@1 score comparison with AsymRE at n=2 . The n=4 results are in Table (b).
Token position
Mean
Median
m>0.5
m>0.2
m>0.1
m>0.05
All tokens
0.301
0.258
22.00%
58.55%
76.05%
86.64%
First 10%
0.357
0.292
29.11%
62.70%
78.39%
87.64%
Last 10%
0.281
0.239
19.23%
56.05%
74.39%
85.68%
Appendix
Table 7: Position-conditioned distribution of the minimum-prefix factor for Qwen3-8B-Base at n=4 . Threshold columns give the percentage of tokens whose factor exceeds the stated value.
Relative token position
MinPRO
CISPO
0–10%
0.2502
0.9960
10–20%
0.2232
1.0007
20–30%
0.2185
1.0005
30–40%
0.2155
1.0007
40–50%
0.2140
1.0006
50–60%
0.2129
1.0008
Appendix
Table 8: Average effective token weights, grouped by relative position in the response, for Qwen3-8B-Base at n=4 .
Off-policy reinforcement learning studies how agents learn from data generated by other or earlier policies, improving sample efficiency while managing distribution shift.
Reinforcement learning (RL) has become central to post-training large language models (LLMs). However, popular RL methods like GRPO incur non-negligible overhead by computing both old-policy and current-policy likelihoods to form importance sampling ratios. In this work, we propose Likelihood-Gated Policy Optimization (LGPO), which enforces a soft trust region constraint via likelihood-based gating, eliminating the need to compute old-policy likelihoods. Empirically, we show that removing the importance sampling correction term does not harm training stability, whereas removing the trust region mechanism can lead to collapse. Moreover, ratio-based clipping can fail in fully on-policy training: the importance ratio stays at 1, so the ratio-based trust region constraint never activates. Under standard training settings where GRPO is stable, LGPO achieves comparable training stability and peak performance while reducing training time by ~18% on average relative to a standard implementation that recomputes old-policy likelihoods. In fully on-policy training, where GRPO fails, LGPO remains stable, enabling more efficient and robust LLM RL post-training across training regimes.
Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse. One vital cause is training-inference mismatch: LLM adopts separate inference and training engines for generation efficiency and training precision, which in practice exhibits inconsistent probabilities for the same trajectories on training and inference sides, even with synchronized model parameters. This naturally induces a special type of off-policyness ever existing and poisoning the training. Prior works have made various efforts in addressing the off-policyness to stabilize the training policies under the mismatch. In this paper, we point out the objective misalignment neglected by existing works that an effective update to the policy in the training engine not necessarily ensures the improvement of the inference policy, i.e., the one used in deployment. To this end, we propose a new policy optimization objective for LLM RL, named Monotonic Inference Policy Improvement (MIPI). Following this principle, we introduce Monotonic Inference Policy Update (MIPU), a two-step LLM RL framework that constructs sampler-referenced candidate updates and selectively accepts synchronized candidates using an inference-side gap proxy. Experiments conducted on two model scales under high mismatch show that MIPU improves average reasoning performance and training stability.
Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded. A natural idea is "transferring" these off-policy token into on-policy token, so that the importance scores for correction are unnecessary. Following this idea, we propose Selective Importance Sampling (SIS), which is inspired by rejection sampling. Concretely, SIS implements by viewing off-policy model as proposal distribution, and implement a token-level rejection test: accepted tokens are viewed as on-policy, so that receive unit importance score, while rejected tokens retain the standard IS correction. Our proposed SIS is theoretically proved reducing the gap between token-level and sequence-level off-policy gradient estimators. The SIS acts as a plug-in that only modifies the importance ratio in the policy loss, adding negligible wall-clock overhead, and can be combine with a vast vary of RL post-training algorithms. Experiments on dense and MoE LLMs across math and agent benchmarks show that SIS consistently improves all objectives, while providing substantially stronger robustness under off-policy data.