RLVR improves language-model reasoning, but commonly relies on independent stochastic rollouts for exploration, which can repeatedly sample similar failed strategies. Prior work has seen success with approaches that guide exploration by introducing dependence into the sampling process by including incorrect solutions in the context for subsequent attempts. Still, these methods introduce the additional question of how to assign credit to each attempt. We propose to view the credit assignment through the lens of Stochastic Shortest Paths, a framework in classical reinforcement learning that seeks to minimize the expected cumulative cost to reach a terminal state (in our case, a verified successful response). We show that, under suitable simplifying assumptions, this perspective naturally leads to a simple modification to GRPO, which we call Retry-Aware Policy Optimization (ReAPO). ReAPO allocates additional generation to failed trajectories, allowing successful retries to be reinforced without rewarding preceding failures, and assigns credit in a theoretically principled way to each attempt. Across Qwen3 and OLMo-2 models from 0.6B to 4B and eight mathematical and scientific reasoning benchmarks, ReAPO improves first-attempt performance over GRPO as well as natural recent baseline algorithms, while additional ablations demonstrate the utility of both serial sampling and ReAPO's credit assignment. On Qwen3-0.6B, ReAPO also outperforms GRPO with twice the group size or twice the training steps, reaching their peak accuracy with 39% and 31% fewer generated tokens, respectively. Taken together, our results suggest that ReAPO is a simple yet effective approach for improving LM reasoning performance through improved exploration and retry-aware credit assignment.
Figures & tables
Figure 1: Overview of ReAPO. Each rollout begins with an independent response; failed responses receive a verifier-conditioned retry, while successful rollouts terminate. Advantages are computed for each attempt separately, ensuring that credit is assigned appropriately to both initial and retry attempts, before applying a policy update with the GRPO loss.
Figure 2: Training efficiency on GSM8K-Platinum with Qwen3-0.6B-Base. The first three panels plot accuracy (mean@32) against optimization steps, cumulative generated response tokens, and wall-clock time, respectively. The final panel plots Pass@32 against optimization steps. The GRPO-2 × Steps control extends standard GRPO to 1,160 steps in the compute-normalized panels. Curves show means across eight seeds and shaded regions show standard errors. SCoRe-Stage2 is evaluated on its second response, which is generated regardless of the first response’s correctness and without verifier feedback at inference. Every other method is evaluated on its first response.
Figure 3: Training efficiency on GSM8K-Platinum with OLMo-2-0425-1B-SFT. The first three panels plot first-attempt Mean@32 against optimization steps, cumulative generated response tokens, and wall-clock time; the final panel plots Pass@32 against optimization steps. Results are from a single training seed.
Model / Method
Beyond AIME
AIME 2024
AIME 2025
HMMT 2025
MATH 500
Minerva
GPQA Diamond
Avg.
Qwen3-1.7B
1.2
4.2
3.1
0.2
57.2
21.9
10.8
14.1
GRPO
1.3±0.1
11.5±0.4
6.2±0.3
1.3±0.1
72.0±0.5
30.9±0.2
24.1±0.7
21.1±0.3
ReAPO (ours)
1.6±0.1
13.7±0.3
7.4±0.3
1.3±0.2
72.9±0.3
31.7±0.2
25.6±0.3
22.0±0.1
Qwen3-4B
4.1
9.4
6.4
0.9
67.9
27.4
17.2
19.0
GRPO
7.6±0.3
19.5±1.1
16.5±1.7
5.5±0.8
83.0±0.6
40.8±0.5
28.6±0.6
28.8±0.7
ReAPO (ours)
8.8±0.4
22.4±0.7
19.0±0.9
9.6±0.5
85.3±0.3
41.3±0.3
30.4±0.7
31.0±0.1
Table 1: First-attempt accuracy across model scales and benchmarks. Qwen3-1.7B-Base results are mean ± standard error across seven seeds at the end of training; Qwen3-4B-Base results use three seeds and are evaluated at step 500.
Figure 4: First-attempt Pass@ k for Qwen3-4B-Base. Pass@ k accuracy for each of BeyondAIME, AIME 2025, MATH-500, and Minerva for ReAPO, GRPO, and the base model. GRPO and ReAPO curves are means across three training seeds, with shaded regions denoting standard errors.
Figure 5: Ablations on Qwen3-0.6B-Base and GSM8K-Platinum. We compare ReAPO with K∈{2,3} , ReAPO with an alternative retry prompt, an unconditional-retry variant that generates a second attempt regardless of whether the first attempt is correct, a multi-turn variant that assigns one advantage to the complete trajectory, and MaxRL. The first three panels report first-attempt Mean@32 against training steps, cumulative generated response tokens, and wall-clock time, respectively; the final panel reports first-attempt Pass@32 against training steps.
Figure 6: Answer diversity, retry behavior, and solution coverage. ( Left ) Percentage of incorrect first-attempt answers that are duplicates within the same question group on AIME 2025 for Qwen3-4B-Base. ( Middle ) The percentage of second-attempt answers that exactly match the failed first-attempt answer. ( Right ) Pass@ k on GSM8K-Platinum for Qwen3-0.6B-Base. The SCoRe-Stage2 curve uses second responses generated regardless of first-attempt correctness; all other curves use first responses.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8
Hyperparameter
Value
Hyperparameter
Value
Training dataset
GSM8K
Training hardware
4× NVIDIA B200
Group size
16
Maximum attempts
2
Maximum prompt length
1,024
Maximum response length
2,048
Training batch size
64
Actor learning rate
1×10−6
Learning-rate warmup
0 steps
Gradient clipping
0.3
KL-loss coefficient
0
Entropy coefficient
0
Appendix
Table 2: Qwen3-0.6B-Base training hyperparameters. Results are aggregated across eight training seeds. The maximum-attempt setting K applies to ReAPO; GRPO uses one attempt. The KL is 0 for every experiment besides SCoRe. These default values are used for all 0.6B experiments unless stated otherwise.
Figure 10
Hyperparameter
Value
Hyperparameter
Value
Training dataset
Polaris-53K
Training hardware
4× NVIDIA B200
Group size
16
Maximum attempts
2
Maximum prompt length
1,024
Maximum response length
4,096
Training batch size
256
Actor learning rate
1×10−6
Learning-rate warmup
0 steps
Gradient clipping
0.3
KL-loss coefficient
0
Entropy coefficient
0
Appendix
Table 3: Qwen3-1.7B-Base and Qwen3-4B-Base training hyperparameters. Results are aggregated across seven seeds for Qwen3-1.7B-Base and three seeds for Qwen3-4B-Base. The maximum-attempt setting applies to ReAPO; GRPO uses one attempt.
Figure 7: Per-benchmark training curves for Qwen3-1.7B-Base. The top row reports first-attempt Pass@1 and the bottom row reports Pass@32. Curves are averages across seven training seeds and shaded regions denote standard errors.
Figure 8: Per-benchmark training curves for Qwen3-4B-Base. The top row reports first-attempt Pass@1 and the bottom row reports Pass@32. Curves are averages across three training seeds and shaded regions denote standard errors.
Figure 9: First-attempt Pass@ k for Qwen3-1.7B-Base at the last checkpoint. GRPO and ReAPO curves are means across seven training seeds, with shaded regions denoting standard errors.
Figure 10: Attempt budget, group size, and response length on Qwen3-1.7B-Base (MATH-500) using one training seed. First-attempt accuracy against optimization steps (left two panels) and cumulative generated response tokens (right two panels) for maximum response lengths of 4,096 and 8,192 tokens per trajectory.
Model / Method
Beyond AIME
AIME 2024
AIME 2025
HMMT 2025
MATH 500
Minerva
GPQA Diamond
Avg.
Qwen3-1.7B
1.2
4.2
3.1
0.2
57.2
21.9
10.8
14.1
Max response length 4,096
GRPO
0.8
10.4
6.2
1.0
71.2
30.5
21.2
20.2
GRPO ( 2× group size)
1.8
12.6
7.7
1.8
73.1
32.5
25.0
22.1
ReAPO ( K=2 )
1.4
13.9
7.3
1.1
72.6
32.0
26.2
22.1
ReAPO ( K=3 )
1.9
13.5
8.0
1.4
74.7
32.0
22.2
22.0
Appendix
Table 4: Qwen3-1.7B-Base, seed 42, with different attempt budgets, group sizes, and maximum response lengths. First-attempt accuracy (%) from the high-sample evaluation at the selected checkpoint (as in Table 1 ). Single seed; the best value in each response-length group is bold.
Metric
Qwen3-0.6B Base
GRPO
GRPO (2 × Rollouts)
MaxRL
SCoRe (Stage 2)
ReAPO ( K=2 )
ReAPO ( K=3 )
Mean@32
44.3±0.1
79.3±0.2
80.5±0.3
77.8±0.1
78.4±0.1
81.5±0.2
81.2±0.3
Pass@32
92.6±0.1
94.1±0.2
94.5±0.3
95.3±0.1
96.1±0.1
95.7±0.2
95.3±0.2
Appendix
Table 5: Qwen3-0.6B-Base results on GSM8K-Platinum at step 580. All columns except SCoRe-Stage2 report first-response accuracy; SCoRe-Stage2 reports second-response accuracy without verifier feedback at inference. Values are mean ± standard error across eight seeds.
Figure 11: Percentage of second answers exactly matching the failed first answer on AIME 2025, for one seed. Exact repetition decreases less for the smaller model than for the larger models. Larger models seem to generate less repetitively than smaller ones.
Model / Method
Beyond AIME
AIME 2024
AIME 2025
HMMT 2025
MATH 500
Minerva
GPQA Diamond
Qwen3-1.7B-Base
12.5
26.9
28.1
5.5
90.6
52.9
70.6
GRPO
13.4±0.6
30.8±0.3
28.7±0.3
12.2±1.0
91.6±0.3
55.1±0.3
70.4±2.6
ReAPO (ours)
15.9±0.4
31.1±0.6
29.9±0.9
14.5±0.9
92.5±0.2
56.3±0.2
78.5±1.0
Qwen3-4B-Base
18.2
38.5
37.9
17.0
94.2
58.5
73.5
GRPO
25.1±1.8
42.8±0.9
39.3±1.5
20.5±0.9
94.8±0.1
57.1±0.5
59.5±2.3
ReAPO (ours)
28.1±1.3
48.0±1.7
42.9±2.5
24.5±0.9
95.6±0.4
58.5±0.9
69.5±1.5
Appendix
Table 6: Pass@32 results at the selected checkpoints. Values are percentages. GRPO and ReAPO are reported as mean ± standard error across training seeds, while base-model results use a single evaluation. A result is bold when its uncertainty interval lies strictly above those of the competing methods.
Figure 12: Token-normalized first-attempt accuracy. Performance is plotted against cumulative generated response tokens for Qwen3-1.7B-Base (top) and Qwen3-4B-Base (bottom). Curves show means across seven and three seeds, respectively, and shaded regions denote standard errors. Within the range of overlapping token counts, ReAPO is competitive with or outperforms GRPO on most benchmarks.
Figure 13: Training dynamics for Qwen3-0.6B-Base, Qwen3-1.7B-Base, and Qwen3-4B-Base. From top to bottom, the rows report results for Qwen3-0.6B-Base, Qwen3-1.7B-Base, and Qwen3-4B-Base, averaged across eight, seven, and three seeds, respectively.
First-attempt phrase
Responses
Second-attempt phrase
Responses
“We need to find …”
76.3%
“Let’s approach this problem differently”
75.8%
“Step by step”
58.4%
“We need to find …”
74.9%
“We are given …”
43.8%
“Use the fact that …”
37.9%
“The final answer is …”
34.2%
“Let’s consider …”
16.8%
“Solve the problem”
34.1%
“We need to check …”
15.5%
“Understand the problem”
32.7%
“Use a simpler approach”
13.8%
Appendix
Table 7: Common phrases in first and second attempts. Response-level phrase frequencies for Qwen3-1.7B-Base on MATH-500 at training step 1,040. Percentages are computed over nonempty first attempts and nonempty verifier-conditioned retries extracted from one seed’s per-step evaluation outputs.
First-attempt phrase
Responses
Second-attempt phrase
Responses
“Step by step”
84.0%
“We need to find …”
63.6%
“Final answer”
77.5%
“Let’s rethink this”
37.1%
“To find the …”
68.0%
“Let’s try a different approach”
33.5%
“We need to find …”
43.7%
“Let’s think differently”
28.8%
“To determine the …”
15.8%
“I didn’t arrive at the correct final answer”
25.7%
“We can use …”
10.5%
“Let’s consider …”
22.3%
Appendix
Table 8: Common phrases in first and second attempts. Response-level phrase frequencies for Qwen3-4B-Base on MATH-500 at training step 500. Percentages are computed over nonempty first attempts and nonempty verifier-conditioned retries extracted from one seed’s per-step evaluation outputs.
Figure 14: A verifier-conditioned retry from ReAPO on MATH-500. The retry feedback causes the model to revisit its initial interpretation and recover the correct answer.
Figure 15: Sequential versus parallel sampling on AIME 2025 with Qwen3-1.7B-Base. At total sampling budget k , parallel sampling draws k independent first attempts, while sequential sampling draws k/2 trajectories with up to two verifier-conditioned attempts each. The dotted lines indicate the sampling budgets required to reach the same success rate, and the annotation reports their ratio.
Reinforcement learning with verifiable rewards (RLVR) enhances reasoning in large language models (LLMs) through outcome-level feedback, yet recent approaches to finer-grained credit assignment often require auxiliary models, additional sampling, or privileged information. Although policy entropy provides a readily available signal, prioritizing uncertain positions under both reinforcement and penalization concentrates penalties where failed responses still retain alternatives for recovery, which can suppress opportunities for exploration. To address this, we introduce Entropic Advantage Policy Optimization (EAPO), an entropy-guided credit assignment method that treats success and failure asymmetrically. Specifically, motivated by the observation that success under uncertainty is less repeatable while confident failures tend to recur, EAPO couples normalized policy entropy with the sign of the response advantage to reinforce surprising success and correct repeated failure. It assigns stronger reinforcement to high-entropy decisions in successful responses and stronger penalties to low-entropy decisions in failed responses, while attenuating penalties at uncertain positions to preserve opportunities for recovery. By redistributing the response advantage across tokens, EAPO derives token-level credit directly from existing rollout signals without additional supervision. We validate EAPO on a range of reasoning tasks across both base and reasoning backbones, demonstrating that it achieves the best overall performance. We further show that EAPO promotes more effective exploration, broadening problem coverage and generating more diverse candidate answers.
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing RLVR methods typically rely on on-policy optimization from scratch, resulting in high sampling costs and inefficient utilization of accumulated experience. As model capabilities and policy behaviors evolve during training, recent attempts to reuse experience via fixed reasoning trajectories further suffer from policy mismatch. Motivated by these limitations, we argue that experience in RLVR should not be reused as fixed reasoning trajectories, but instead expressed in a policy-adaptive manner. In this work, we propose Experience-Augmented Policy Optimization (EAPO), which leverages a prior RL-optimized policy as an action-level experience prior and selectively injects experience at critical decision points during rollout. To ensure stable and unbiased learning from experience-augmented rollouts, EAPO further incorporates an adapted importance sampling scheme. Experiments on using Qwen-2.5-math 7b and Qwen-3-8B on five different benchmarks demonstrate that EAPO consistently improves reasoning performance over state-of-the-art RLVR methods.
Jinda Lu, Kexin Huang, Junkang Wu +7
University of Science and Technology of China · 2Peking University · 3Dartmouth College +1
Reinforcement Learning with Verifiable Rewards (RLVR) for language-model reasoning can fail at both extremes of task difficulty: easy prompts often produce all-correct, low-diversity rollout groups with little gradient signal, while hard prompts can produce all-incorrect groups with no positive reward. We introduce ExTra (Exploratory Trajectory Optimization), a GRPO-compatible framework that extracts exploration signals from the model's own rollouts. ExTra combines two mechanisms: (i) a novelty reward that adds embedding-based diversity bonuses after GRPO normalization, rewarding diverse correct solutions; and (ii) entropy-guided prefix regeneration, which scores partial trajectories using entropy signals and continues exploration from promising intermediate steps. Across six mathematical reasoning benchmarks, ExTra improves Qwen3-1.7B over GRPO by about +5 points on pass@1 and +7 points on pass@16, showing that trajectory-level exploration signals can improve both single-sample accuracy and inference-time coverage.