cs.LGOct 20, 2026
SaveReAPO: Retry-Aware Policy Optimization for Efficient Exploration in LLM Reasoning
Organizations: Columbia University · Yale University · UT Austin
Abstract
RLVR improves language-model reasoning, but commonly relies on independent stochastic rollouts for exploration, which can repeatedly sample similar failed strategies. Prior work has seen success with approaches that guide exploration by introducing dependence into the sampling process by including incorrect solutions in the context for subsequent attempts. Still, these methods introduce the additional question of how to assign credit to each attempt. We propose to view the credit assignment through the lens of Stochastic Shortest Paths, a framework in classical reinforcement learning that seeks to minimize the expected cumulative cost to reach a terminal state (in our case, a verified successful response). We show that, under suitable simplifying assumptions, this perspective naturally leads to a simple modification to GRPO, which we call Retry-Aware Policy Optimization (ReAPO). ReAPO allocates additional generation to failed trajectories, allowing successful retries to be reinforced without rewarding preceding failures, and assigns credit in a theoretically principled way to each attempt. Across Qwen3 and OLMo-2 models from 0.6B to 4B and eight mathematical and scientific reasoning benchmarks, ReAPO improves first-attempt performance over GRPO as well as natural recent baseline algorithms, while additional ablations demonstrate the utility of both serial sampling and ReAPO's credit assignment. On Qwen3-0.6B, ReAPO also outperforms GRPO with twice the group size or twice the training steps, reaching their peak accuracy with 39% and 31% fewer generated tokens, respectively. Taken together, our results suggest that ReAPO is a simple yet effective approach for improving LM reasoning performance through improved exploration and retry-aware credit assignment.
Figures & tables
Figure 1: Overview of ReAPO. Each rollout begins with an independent response; failed responses receive a verifier-conditioned retry, while successful rollouts terminate. Advantages are computed for each attempt separately, ensuring that credit is assigned appropriately to both initial and retry attempts, before applying a policy update with the GRPO loss.
Figure 2: Training efficiency on GSM8K-Platinum with Qwen3-0.6B-Base. The first three panels plot accuracy (mean@32) against optimization steps, cumulative generated response tokens, and wall-clock time, respectively. The final panel plots Pass@32 against optimization steps. The GRPO-2 Steps control extends standard GRPO to 1,160 steps in the compute-normalized panels. Curves show means across eight seeds and shaded regions show standard errors. SCoRe-Stage2 is evaluated on its second response, which is generated regardless of the first response’s correctness and without verifier feedback at inference. Every other method is evaluated on its first response.
Figure 3: Training efficiency on GSM8K-Platinum with OLMo-2-0425-1B-SFT. The first three panels plot first-attempt Mean@32 against optimization steps, cumulative generated response tokens, and wall-clock time; the final panel plots Pass@32 against optimization steps. Results are from a single training seed.
| Model / Method | Beyond AIME | AIME 2024 | AIME 2025 | HMMT 2025 | MATH 500 | Minerva | GPQA Diamond | Avg. |
|---|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | ||||||||
| GRPO | ||||||||
| ReAPO (ours) | ||||||||
| Qwen3-4B | ||||||||
| GRPO | ||||||||
| ReAPO (ours) |
Table 1: First-attempt accuracy across model scales and benchmarks. Qwen3-1.7B-Base results are mean standard error across seven seeds at the end of training; Qwen3-4B-Base results use three seeds and are evaluated at step 500.
Figure 4: First-attempt Pass@ for Qwen3-4B-Base. Pass@ accuracy for each of BeyondAIME, AIME 2025, MATH-500, and Minerva for ReAPO, GRPO, and the base model. GRPO and ReAPO curves are means across three training seeds, with shaded regions denoting standard errors.
Figure 5: Ablations on Qwen3-0.6B-Base and GSM8K-Platinum. We compare ReAPO with , ReAPO with an alternative retry prompt, an unconditional-retry variant that generates a second attempt regardless of whether the first attempt is correct, a multi-turn variant that assigns one advantage to the complete trajectory, and MaxRL. The first three panels report first-attempt Mean@32 against training steps, cumulative generated response tokens, and wall-clock time, respectively; the final panel reports first-attempt Pass@32 against training steps.
Figure 6: Answer diversity, retry behavior, and solution coverage. ( Left ) Percentage of incorrect first-attempt answers that are duplicates within the same question group on AIME 2025 for Qwen3-4B-Base. ( Middle ) The percentage of second-attempt answers that exactly match the failed first-attempt answer. ( Right ) Pass@ on GSM8K-Platinum for Qwen3-0.6B-Base. The SCoRe-Stage2 curve uses second responses generated regardless of first-attempt correctness; all other curves use first responses.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value | Hyperparameter | Value |
|---|---|---|---|
| Training dataset | GSM8K | Training hardware | NVIDIA B200 |
| Group size | Maximum attempts | ||
| Maximum prompt length | Maximum response length | ||
| Training batch size | Actor learning rate | ||
| Learning-rate warmup | steps | Gradient clipping | |
| KL-loss coefficient | Entropy coefficient |
Table 2: Qwen3-0.6B-Base training hyperparameters. Results are aggregated across eight training seeds. The maximum-attempt setting applies to ReAPO; GRPO uses one attempt. The KL is 0 for every experiment besides SCoRe. These default values are used for all 0.6B experiments unless stated otherwise.
| Hyperparameter | Value | Hyperparameter | Value |
|---|---|---|---|
| Training dataset | Polaris-53K | Training hardware | NVIDIA B200 |
| Group size | Maximum attempts | ||
| Maximum prompt length | Maximum response length | ||
| Training batch size | Actor learning rate | ||
| Learning-rate warmup | steps | Gradient clipping | |
| KL-loss coefficient | Entropy coefficient |
Table 3: Qwen3-1.7B-Base and Qwen3-4B-Base training hyperparameters. Results are aggregated across seven seeds for Qwen3-1.7B-Base and three seeds for Qwen3-4B-Base. The maximum-attempt setting applies to ReAPO; GRPO uses one attempt.
Figure 7: Per-benchmark training curves for Qwen3-1.7B-Base. The top row reports first-attempt Pass@1 and the bottom row reports Pass@32. Curves are averages across seven training seeds and shaded regions denote standard errors.
Figure 8: Per-benchmark training curves for Qwen3-4B-Base. The top row reports first-attempt Pass@1 and the bottom row reports Pass@32. Curves are averages across three training seeds and shaded regions denote standard errors.
Figure 9: First-attempt Pass@ for Qwen3-1.7B-Base at the last checkpoint. GRPO and ReAPO curves are means across seven training seeds, with shaded regions denoting standard errors.
Figure 10: Attempt budget, group size, and response length on Qwen3-1.7B-Base (MATH-500) using one training seed. First-attempt accuracy against optimization steps (left two panels) and cumulative generated response tokens (right two panels) for maximum response lengths of 4,096 and 8,192 tokens per trajectory.
| Model / Method | Beyond AIME | AIME 2024 | AIME 2025 | HMMT 2025 | MATH 500 | Minerva | GPQA Diamond | Avg. |
|---|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | ||||||||
| Max response length 4,096 | ||||||||
| GRPO | ||||||||
| GRPO ( group size) | ||||||||
| ReAPO ( ) | ||||||||
| ReAPO ( ) | ||||||||
Table 4: Qwen3-1.7B-Base, seed 42, with different attempt budgets, group sizes, and maximum response lengths. First-attempt accuracy (%) from the high-sample evaluation at the selected checkpoint (as in Table 1 ). Single seed; the best value in each response-length group is bold.
| Metric | Qwen3-0.6B Base | GRPO | GRPO (2 Rollouts) | MaxRL | SCoRe (Stage 2) | ReAPO ( ) | ReAPO ( ) |
|---|---|---|---|---|---|---|---|
| Mean@32 | |||||||
| Pass@32 |
Table 5: Qwen3-0.6B-Base results on GSM8K-Platinum at step 580. All columns except SCoRe-Stage2 report first-response accuracy; SCoRe-Stage2 reports second-response accuracy without verifier feedback at inference. Values are mean standard error across eight seeds.
Figure 11: Percentage of second answers exactly matching the failed first answer on AIME 2025, for one seed. Exact repetition decreases less for the smaller model than for the larger models. Larger models seem to generate less repetitively than smaller ones.
| Model / Method | Beyond AIME | AIME 2024 | AIME 2025 | HMMT 2025 | MATH 500 | Minerva | GPQA Diamond |
|---|---|---|---|---|---|---|---|
| Qwen3-1.7B-Base | |||||||
| GRPO | |||||||
| ReAPO (ours) | |||||||
| Qwen3-4B-Base | |||||||
| GRPO | |||||||
| ReAPO (ours) |
Table 6: Pass@32 results at the selected checkpoints. Values are percentages. GRPO and ReAPO are reported as mean standard error across training seeds, while base-model results use a single evaluation. A result is bold when its uncertainty interval lies strictly above those of the competing methods.
Figure 12: Token-normalized first-attempt accuracy. Performance is plotted against cumulative generated response tokens for Qwen3-1.7B-Base (top) and Qwen3-4B-Base (bottom). Curves show means across seven and three seeds, respectively, and shaded regions denote standard errors. Within the range of overlapping token counts, ReAPO is competitive with or outperforms GRPO on most benchmarks.
Figure 13: Training dynamics for Qwen3-0.6B-Base, Qwen3-1.7B-Base, and Qwen3-4B-Base. From top to bottom, the rows report results for Qwen3-0.6B-Base, Qwen3-1.7B-Base, and Qwen3-4B-Base, averaged across eight, seven, and three seeds, respectively.
| First-attempt phrase | Responses | Second-attempt phrase | Responses |
|---|---|---|---|
| “We need to find …” | 76.3% | “Let’s approach this problem differently” | 75.8% |
| “Step by step” | 58.4% | “We need to find …” | 74.9% |
| “We are given …” | 43.8% | “Use the fact that …” | 37.9% |
| “The final answer is …” | 34.2% | “Let’s consider …” | 16.8% |
| “Solve the problem” | 34.1% | “We need to check …” | 15.5% |
| “Understand the problem” | 32.7% | “Use a simpler approach” | 13.8% |
Table 7: Common phrases in first and second attempts. Response-level phrase frequencies for Qwen3-1.7B-Base on MATH-500 at training step 1,040. Percentages are computed over nonempty first attempts and nonempty verifier-conditioned retries extracted from one seed’s per-step evaluation outputs.
| First-attempt phrase | Responses | Second-attempt phrase | Responses |
|---|---|---|---|
| “Step by step” | 84.0% | “We need to find …” | 63.6% |
| “Final answer” | 77.5% | “Let’s rethink this” | 37.1% |
| “To find the …” | 68.0% | “Let’s try a different approach” | 33.5% |
| “We need to find …” | 43.7% | “Let’s think differently” | 28.8% |
| “To determine the …” | 15.8% | “I didn’t arrive at the correct final answer” | 25.7% |
| “We can use …” | 10.5% | “Let’s consider …” | 22.3% |
Table 8: Common phrases in first and second attempts. Response-level phrase frequencies for Qwen3-4B-Base on MATH-500 at training step 500. Percentages are computed over nonempty first attempts and nonempty verifier-conditioned retries extracted from one seed’s per-step evaluation outputs.
Figure 14: A verifier-conditioned retry from ReAPO on MATH-500. The retry feedback causes the model to revisit its initial interpretation and recover the correct answer.
Figure 15: Sequential versus parallel sampling on AIME 2025 with Qwen3-1.7B-Base. At total sampling budget , parallel sampling draws independent first attempts, while sequential sampling draws trajectories with up to two verifier-conditioned attempts each. The dotted lines indicate the sampling budgets required to reach the same success rate, and the annotation reports their ratio.
More in this topic
RL for Language Model Reasoning
Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.
See all 1,103 papers