When every rollout for a problem fails, GRPO receives neither a successful exemplar nor a nonzero group-relative advantage, so it learns nothing from exactly the problems it most needs to learn. The failed reasoning can nevertheless suggest what to try next. We present Replay-Selected Hindsight Distillation (RSHD), which turns failed attempts into an additional, tested source of supervision during GRPO training. The policy writes a corrective note about each failed attempt without seeing a reference answer or solution, then re-solves the problem with the note in context. A note is admitted only if its replays succeed more often than the original group did, after screening for notes that state the answer, and an admitted replay is distilled into the plain-prompt policy. Because admission is decided by fresh successes rather than by the failed rollout itself, the procedure can recover supervision from groups in which every attempt failed, and it applies equally to failures in mixed groups. Only the policy is trained, the only judge is the outcome verifier, and deployment requires no notes or extra inference. Across model families and sizes and two training domains, RSHD consistently improves over GRPO on in-domain benchmarks, and the gains carry over to out-of-domain evaluation sets.
Figures & tables
Figure 1: Groups that GRPO have no learning signal. On the hardest level, half the groups carry no gradient at all.
Figure 2: RSHD overview. From a scored GRPO group, each failed attempt yields a corrective note (left). The note is screened for answer leaks and replayed; it is admitted only if the replays beat the group’s own success rate, with one revision round (middle). An admitted replay is scored by the same weights with and without the note, and the plain-prompt policy imitates it with detached lift weights, added to the GRPO loss (right).
Figure 3
Figure 3: Learning with failure-derived notes in a controlled two-strategy task. Plain-prompt success during training (log scale, mean and 95% interval over ten seeds) at three reasoning budgets H , with initial viable-strategy probability 0.01 and note strength 1.6 . Every arm is allotted two extra attempts per rollout. Unguided retry imitates accepted retries without a note, repair-biased restart proposes a strategy switch with probability 0.1 and a local repair otherwise, and the strategy-switch note always proposes the other strategy. Success is measured without a note.
Backbone
Method
MATH-500
MMLU-Pro
GPQA-Diamond
TheoremQA
Qwen3-4B
Base
65.2
30.2
1.8 / 3.1
30.6
GRPO
76.9
48.7
3.0 / 9.6
40.1
OPSD
73.6
34.5
3.1 / 6.2
34.8
SEED
75.2
41.7
2.3 / 7.5
38.7
RSHD (ours)
84.5
61.2
9.1 / 27.0
56.5
Qwen3-1.7B
Base
64.2
27.3
0.3 / 2.8
25.3
Table 1: RSHD improves over GRPO on every benchmark and every backbone. Accuracy (%, mean over 8 samples per problem); GPQA-Diamond shows exact / matched. The MATH-500 column reports models trained on MATH, and the other three report models trained on the MMLU-Pro pool, so GPQA-Diamond and TheoremQA are out of domain. All trained rows use the same GRPO rollouts and 150-step budget. Best in bold.
Figure 4: Held-out accuracy during training. Rows are the training domain, named on the y -axis (top: MATH, held out on 100 MATH-500 problems; bottom: the MMLU-Pro numeric pool, held out on 100 of its 300 reserved questions), columns are the backbone; every 25 steps, 8 samples per problem at the training temperature. Each panel’s y -range is fitted to its own curves, so compare gaps within a panel rather than across panels. RSHD separates from GRPO by step 50 on all three backbones, and the gap is widest on the science domain, where GRPO plateaus early.
Figure 5: Accuracy by MATH’s own difficulty label (Qwen3-4B); the y -axis starts at 40 . The darker top segment of each RSHD bar is its gain over GRPO, also printed above the pair.
Base model
n
accuracy (%) ↑
Δ
solves it
GRPO
RSHD
never ( 0/8 )
66
6.4
15.0
+8.5
sometimes ( 1 – 7 )
172
84.8
84.4
−0.4
avg
45.6
49.7
+4.1
Table 2: MATH-500 split by what the released Gemma-4-E4B solves in 8 samples. The gain sits entirely on the problems it never solves. Δ is computed before rounding.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol/config
Meaning
Value
GRPO backbone
n
rollouts per prompt
8
problems per optimizer step
14 ( 4 rows/GPU ×4 micro-steps ×7 GPUs)
optimizer steps
150 (about one pass over 2,100 problems)
ϵclip
ratio clip
0.2
loss aggregation
token-level (DAPO)
Appendix
Table 3: Registered training settings, shared by every arm and backbone. All runs are full-parameter (bf16, DeepSpeed ZeRO-2, gradient checkpointing) on 7 training GPUs with a vLLM server on the eighth. GRPO-only baselines differ from RSHD only in the rows marked ∗ . Nothing is fit besides the policy.
The filter — one note per failed rollout
notes written
4,680
returned no usable note
310 (6.6%)
stated the reference answer
517 (11.0%)
replays did not beat the group
2,705 (57.8%)
kept
1,148 ( 24.5% )
What survives
Appendix
Table 4: The note search in numbers (Qwen3-4B on MATH, all 150 steps). Three notes in four are discarded, and the two screens cost no replays at all. A step holds 14 problems and 112 rollouts.
Symbol
Meaning
Rollouts and rewards
x,g
prompt and its reference answer
y=(y1,…,yT) , y<t
a completion and its prefix before token t
{yi}i=1n
the GRPO group of n rollouts for a prompt
R(x,y)∈{0,1} , Ri
verifier reward (final answer matches g ); reward of rollout i
F={i:Ri=0} , p^
the failed set; the group’s no-note success rate n1∑iRi
GRPO-style RLVR trains reasoning models from multiple on-policy attempts per prompt, but typically uses these attempts only through terminal rewards. We show that a mixed group contains a richer process signal: a correct completion is a self-generated witness of how the current policy can solve the problem, while a wrong completion provides on-policy prefixes where the policy needs correction. We introduce Self-Supervised On-Policy Distillation (SSOPD), which distills a teacher distribution conditioned on the shortest correct completion into prefixes of the longest wrong completion. This converts intra-group correct--wrong contrast into dense process supervision without external solution traces. A stopping-time view motivates the shortest-correct / longest-wrong rule as a finite-group approximation to editing persistent failures toward fast-success actions, and a prompt-level frontier weight concentrates the auxiliary loss where correct and wrong branches coexist. Across AIME 2024, AIME 2025, and HMMT 2025, SSOPD improves over GRPO in all nine model-benchmark settings. On Qwen3-8B, it reaches a macro Avg@12 of 65.6, outperforming GRPO by 1.6 points and the solution-conditioned OPSD baseline by 0.8 points. Code will be released at https://github.com/tzq1999/SSOPD.
Group Relative Policy Optimization (GRPO) is effective for training language models on complex reasoning. However, since the objective is defined relative to a group of sampled trajectories, extended deliberation can create more chances to realize relative gains, leading to inefficient reasoning and overthinking, and complicating the trade-off between correctness and rollout efficiency. Controlling this behavior is difficult in practice, considering (i) Length penalties are hard to calibrate because longer rollouts may reflect harder problems that require longer reasoning, penalizing tokens risks truncating useful reasoning along with redundant continuation; and (ii) supervision that directly indicates when to continue or stop is typically unavailable beyond final answer correctness. We propose Weakly Supervised GRPO (WS-GRPO), which improves rollout efficiency by converting terminal rewards into correctness-aware guidance over partial trajectories. Unlike global length penalties that are hard to calibrate, WS-GRPO trains a preference model from outcome-only correctness to produce prefix-level signals that indicate when additional continuation is beneficial. Thus, WS-GRPO supplies outcome-derived continue/stop guidance, reducing redundant deliberation while maintaining accuracy. We provide theoretical results and empirically show on reasoning benchmarks that WS-GRPO substantially reduces rollout length while remaining competitive with GRPO baselines.
Gagan Mundada, Zihan Huang, Rohan Surana +8
University of California, San Diego · University of Toronto · 3Adobe Research +2
Reinforcement Learning with Verifiable Rewards (RLVR) is commonly based on group sampling to estimate advantages and stabilize policy updates. In practice, computational limits often rule out very large groups, so training proceeds with finite rollout sets that can reinforce only the correct behavior they expose. At practical group sizes, updates can miss rare-correct trajectories while still containing mixed rewards, concentrating probability on more common sampled solutions. We derive the probability of such prompt-local tail-miss events as a function of group size, showing non-monotonic behavior, and in the categorical abstraction characterize how unsampled-correct mass can shrink even as total correct mass grows. Motivated by this analysis, we propose a difficulty-aware scaling coefficient, inspired by Focal loss, that down-weights updates on high-success sampled groups. Empirically, categorical simulation illustrates the same effect in the categorical setting, Maze provides a single-solution test, and LLM experiments include a representative GRPO group-size sweep together with fixed-N transfer across GRPO, DAPO, and CISPO. On Qwen2.5-7B at N=8, our method improves average math pass@256 from 64.1 → 70.3 (GRPO), 69.3 → 72.5 (DAPO), and 73.2 → 76.8 (CISPO); OOD pass@256 also improves in all three cases, without increasing group size or computational cost.
Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov +4
T-Tech · Saint Petersburg Electrotechnical University “LETI”