When every rollout for a problem fails, GRPO receives neither a successful exemplar nor a nonzero group-relative advantage, so it learns nothing from exactly the problems it most needs to learn. The failed reasoning can nevertheless suggest what to try next. We present Replay-Selected Hindsight Distillation (RSHD), which turns failed attempts into an additional, tested source of supervision during GRPO training. The policy writes a corrective note about each failed attempt without seeing a reference answer or solution, then re-solves the problem with the note in context. A note is admitted only if its replays succeed more often than the original group did, after screening for notes that state the answer, and an admitted replay is distilled into the plain-prompt policy. Because admission is decided by fresh successes rather than by the failed rollout itself, the procedure can recover supervision from groups in which every attempt failed, and it applies equally to failures in mixed groups. Only the policy is trained, the only judge is the outcome verifier, and deployment requires no notes or extra inference. Across model families and sizes and two training domains, RSHD consistently improves over GRPO on in-domain benchmarks, and the gains carry over to out-of-domain evaluation sets.
Figures & tables
Figure 1: Groups that GRPO have no learning signal. On the hardest level, half the groups carry no gradient at all.
Figure 2: RSHD overview. From a scored GRPO group, each failed attempt yields a corrective note (left). The note is screened for answer leaks and replayed; it is admitted only if the replays beat the group’s own success rate, with one revision round (middle). An admitted replay is scored by the same weights with and without the note, and the plain-prompt policy imitates it with detached lift weights, added to the GRPO loss (right).
Figure 3
Figure 3: Learning with failure-derived notes in a controlled two-strategy task. Plain-prompt success during training (log scale, mean and 95% interval over ten seeds) at three reasoning budgets H , with initial viable-strategy probability 0.01 and note strength 1.6 . Every arm is allotted two extra attempts per rollout. Unguided retry imitates accepted retries without a note, repair-biased restart proposes a strategy switch with probability 0.1 and a local repair otherwise, and the strategy-switch note always proposes the other strategy. Success is measured without a note.
Backbone
Method
MATH-500
MMLU-Pro
GPQA-Diamond
TheoremQA
Qwen3-4B
Base
65.2
30.2
1.8 / 3.1
30.6
GRPO
76.9
48.7
3.0 / 9.6
40.1
OPSD
73.6
34.5
3.1 / 6.2
34.8
SEED
75.2
41.7
2.3 / 7.5
38.7
RSHD (ours)
84.5
61.2
9.1 / 27.0
56.5
Qwen3-1.7B
Base
64.2
27.3
0.3 / 2.8
25.3
Table 1: RSHD improves over GRPO on every benchmark and every backbone. Accuracy (%, mean over 8 samples per problem); GPQA-Diamond shows exact / matched. The MATH-500 column reports models trained on MATH, and the other three report models trained on the MMLU-Pro pool, so GPQA-Diamond and TheoremQA are out of domain. All trained rows use the same GRPO rollouts and 150-step budget. Best in bold.
Figure 4: Held-out accuracy during training. Rows are the training domain, named on the y -axis (top: MATH, held out on 100 MATH-500 problems; bottom: the MMLU-Pro numeric pool, held out on 100 of its 300 reserved questions), columns are the backbone; every 25 steps, 8 samples per problem at the training temperature. Each panel’s y -range is fitted to its own curves, so compare gaps within a panel rather than across panels. RSHD separates from GRPO by step 50 on all three backbones, and the gap is widest on the science domain, where GRPO plateaus early.
Figure 5: Accuracy by MATH’s own difficulty label (Qwen3-4B); the y -axis starts at 40 . The darker top segment of each RSHD bar is its gain over GRPO, also printed above the pair.
Base model
n
accuracy (%) ↑
Δ
solves it
GRPO
RSHD
never ( 0/8 )
66
6.4
15.0
+8.5
sometimes ( 1 – 7 )
172
84.8
84.4
−0.4
avg
45.6
49.7
+4.1
Table 2: MATH-500 split by what the released Gemma-4-E4B solves in 8 samples. The gain sits entirely on the problems it never solves. Δ is computed before rounding.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol/config
Meaning
Value
GRPO backbone
n
rollouts per prompt
8
problems per optimizer step
14 ( 4 rows/GPU ×4 micro-steps ×7 GPUs)
optimizer steps
150 (about one pass over 2,100 problems)
ϵclip
ratio clip
0.2
loss aggregation
token-level (DAPO)
Appendix
Table 3: Registered training settings, shared by every arm and backbone. All runs are full-parameter (bf16, DeepSpeed ZeRO-2, gradient checkpointing) on 7 training GPUs with a vLLM server on the eighth. GRPO-only baselines differ from RSHD only in the rows marked ∗ . Nothing is fit besides the policy.
The filter — one note per failed rollout
notes written
4,680
returned no usable note
310 (6.6%)
stated the reference answer
517 (11.0%)
replays did not beat the group
2,705 (57.8%)
kept
1,148 ( 24.5% )
What survives
Appendix
Table 4: The note search in numbers (Qwen3-4B on MATH, all 150 steps). Three notes in four are discarded, and the two screens cost no replays at all. A step holds 14 problems and 112 rollouts.
Symbol
Meaning
Rollouts and rewards
x,g
prompt and its reference answer
y=(y1,…,yT) , y<t
a completion and its prefix before token t
{yi}i=1n
the GRPO group of n rollouts for a prompt
R(x,y)∈{0,1} , Ri
verifier reward (final answer matches g ); reward of rollout i
F={i:Ri=0} , p^
the failed set; the group’s no-note success rate n1∑iRi