Reinforcement learning (RL) post-training has increasingly demonstrated strong ability to elicit reasoning behaviors in large language models (LLMs). For training efficiency, rollouts are typically generated in an off-policy manner using an older sampling policy and then used to update the current target policy. To correct the resulting discrepancy between the sampling and target policies, most existing RL objectives rely on a token-level importance sampling ratio. However, we observe unstable training dynamics in existing importance-weighted objectives under large policy lag. In this paper, we revisit LLM policy optimization under off-policy conditions and show that the exact correction for the prefix-marginal policy gradient is the cumulative prefix importance ratio. Token-level weighting does not explicitly correct for this prefix distribution shift. To stabilize LLM optimization under large off-policy drift, we propose a simple yet effective objective, Minimum Prefix Ratio (MinPRO). MinPRO replaces the unstable cumulative prefix ratio with a biased, non-cumulative surrogate based on the minimum token-level ratio observed in the preceding prefix. Extensive experiments on both dense and mixture-of-experts LLMs, across multiple mathematical reasoning benchmarks, demonstrate that MinPRO substantially improves training stability and peak performance in off-policy regimes.
Figures & tables
Figure 1: (a) An overview of MinPRO. ∇θJ denotes the policy gradient and ρt is the token-level importance sampling (IS) ratio. We take a step back to derive the full IS ratio ρ1:t , referred to as the prefix importance ratio. MinPRO is then developed by relaxing ρ1:t=ρ1:t−1⋅ρt to a non-cumulative proxy ρtρt . (b) AIME24 scores as functions of training steps for Qwen3-30B-A3B-Base under off-policy training.
Figure 2: Plots of (a) reward and (b) entropy as functions of training steps for Qwen3-8B-Base under on-policy and off-policy regimes. In the off-policy setting, each batch of sampled rollouts is stored in a buffer and used for training after a delay of 2 global steps, whereas the on-policy setting applies no delay between sampling and optimization.
Figure 3: Training reward curves as functions of training steps for Qwen3-8B-Base and Qwen3-14B-Base under off-policy optimization.
Method
AMC23
AIME24
AIME25
MATH500
Olympiad
Minerva
GSM8K
Avg
Base
21.6
3.5
3.5
44.5
20.2
20.6
70.3
26.3
GRPO
63.3
23.1
18.5
70.2
40.4
29.0
80.4
46.4
GSPO
64.4
25.9
18.3
78.4
44.4
31.4
87.3
50.0
CISPO
78.9
35.9
25.1
84.5
54.3
32.2
86.7
56.8
M2PO
82.2
33.9
25.8
85.8
56.5
37.7
88.7
58.7
8B-Base
MinPRO
82.5
36.1
30.3
86.5
56.5
33.6
90.2
59.4
Table 1: pass@1 scores under large off-policyness ( n=2 ).
Method
AMC23
AIME24
AIME25
MATH500
Olympiad
Minerva
GSM8K
Avg
n=4
GRPO
60.5
19.1
15.2
74.4
43.6
34.2
82.2
47.0
GSPO
60.8
20.7
15.8
73.6
44.2
34.2
88.2
48.2
CISPO
78.6
34.3
25.4
84.3
53.0
33.0
87.3
56.6
M2PO
77.9
34.8
24.5
83.8
54.0
36.0
87.6
56.9
MinPRO
79.3
38.2
28.3
84.8
54.2
35.3
88.3
58.3
n=0
GRPO
80.7
33.1
26.2
84.8
53.8
36.0
90.4
57.8
Table 2: pass@1 on Qwen3-8B-Base under more off-policy ( n=4 ) and near-on-policy ( n=0 ).
Method
Pass@k
Avg
1
2
4
8
16
32
64
128
GRPO
35.0
43.2
51.1
57.8
63.4
67.7
71.3
73.9
58.0
GSPO
36.2
44.0
51.4
58.0
63.5
68.0
71.2
73.5
58.2
CISPO
46.6
54.3
60.5
65.2
69.2
72.6
75.7
78.3
65.3
M2PO
47.3
54.4
60.7
65.7
69.5
72.8
75.6
77.5
65.4
8B-Base
MinPRO
49.6
56.7
62.5
67.0
70.5
73.1
75.3
77.2
66.5
Table 3: Mean-dataset pass@k scores averaged over AMC23, AIME24, and AIME25 under large off-policyness ( n=2 ). Boldface indicates the highest score within each model.
Figure 4: (a) Training reward curves and (b) average pass@1 scores as functions of training steps for Qwen3-30B-A3B-Base under off-policy optimization.
Method / Statistic
AMC23
AIME24
AIME25
MATH500
Olympiad
Minerva
GSM8K
Avg
MinPRO (minimum)
79.3
38.2
28.3
84.8
54.2
35.3
88.3
58.3
(a) Alternative prefix statistics
Arithmetic mean
72.4
29.8
21.9
83.0
51.6
33.5
86.1
54.0
Geometric mean
78.9
36.5
27.1
85.4
51.8
34.9
86.4
57.3
Cumulative prod
68.6
27.7
21.6
82.6
50.0
32.4
85.7
52.7
(b) Recent baselines
Table 4: Additional pass@1 comparisons (%) on Qwen3-8B-Base. Avg is the mean over seven benchmarks. The shared MinPRO reference and groups (a) – (b) use n=4 ; group (c) varies staleness. The shared reference is reported once.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Category
GRPO
GSPO
CISPO
M2PO
MinPRO
Sampling and Validation
Temperature
1.0
1.0
1.0
1.0
1.0
Top- p / Val Top- p
1.0 / 0.7
1.0 / 0.7
1.0 / 0.7
1.0 / 0.7
1.0 / 0.7
Clipping
Clip ratio (low / high)
0.2 / 0.28
2e−3 / 2e−3
1 / 4
—
1 / 4
Sequence Limits
Appendix
Table 5: Key hyperparameters in RL algorithms. “—” denotes not used.
MinPRO policy-loss implementation in VeRL.
Figure 6: Per-dataset pass@1 scores with Qwen3-30B-A3B-Base under off-policy training.
Setting
Method
AMC23
AIME24
AIME25
MATH500
Olympiad
Minerva
GSM8K
Avg
n=2
GRPO
63.3
23.1
18.5
70.2
40.4
29.0
80.4
46.4
AsymRE
58.8
22.0
14.2
79.2
43.8
34.9
85.8
48.4
MinPRO
82.5
36.1
30.3
86.5
56.5
33.6
90.2
59.4
Appendix
Table 6: pass@1 score comparison with AsymRE at n=2 . The n=4 results are in Table (b).
Token position
Mean
Median
m>0.5
m>0.2
m>0.1
m>0.05
All tokens
0.301
0.258
22.00%
58.55%
76.05%
86.64%
First 10%
0.357
0.292
29.11%
62.70%
78.39%
87.64%
Last 10%
0.281
0.239
19.23%
56.05%
74.39%
85.68%
Appendix
Table 7: Position-conditioned distribution of the minimum-prefix factor for Qwen3-8B-Base at n=4 . Threshold columns give the percentage of tokens whose factor exceeds the stated value.
Relative token position
MinPRO
CISPO
0–10%
0.2502
0.9960
10–20%
0.2232
1.0007
20–30%
0.2185
1.0005
30–40%
0.2155
1.0007
40–50%
0.2140
1.0006
50–60%
0.2129
1.0008
Appendix
Table 8: Average effective token weights, grouped by relative position in the response, for Qwen3-8B-Base at n=4 .
Off-policy reinforcement learning studies how agents learn from data generated by other or earlier policies, improving sample efficiency while managing distribution shift.