Simplify, Agree, Distill: Label-Free Self-Distillation with Stepping Stones
Organizations: Amazon · Worcester Polytechnic Institute
Abstract
Can a language model teach itself to reason without labels? On-policy self-distillation (OPSD) uses the model as its own teacher, but strengthens that teacher with a ground-truth solution. The natural label-free substitute, voting over the model's attempts, fails precisely on hard problems: attempts disagree, and their most common answer is often wrong. We propose STONE, a label-free alternative that moves agreement to easier related problems. The model writes these stepping stones, keeps those it solves consistently, and shows the teacher their worked solutions, with final answers removed; the student sees only the original problem. No step reads a label, and inference needs no extra context or model calls. Across five math benchmarks, the proposed method achieves the highest average accuracy for Qwen3-4B and Qwen3-8B in thinking mode. Without thinking mode, it gains 13.2 and 22.7 points, 2.5 and 2.7 times the gains of OPSD with ground-truth solutions.
Figures & tables
| Thinking mode | Without thinking mode | ||||||||||||||
| Method | GT | A24 | A25 | H25 | M500 | AMC | Avg. | A24 | A25 | H25 | M500 | AMC | Avg. | ||
| Qwen3-4B | |||||||||||||||
| Base | 74.17 | 64.72 | 45.56 | 94.80 | 95.00 | 74.85 | 25.83 | 17.78 | 10.83 | 84.10 | 66.25 | 40.96 | |||
| SFT | ✓ | 73.06 | 69.17 | 41.67 | 95.40 | 94.38 | 74.73 | -0.12 | 26.67 | 19.72 | 13.06 | 84.85 | 69.38 | 42.73 | +1.77 |
| GRPO | ✓ | 73.89 | 69.44 | 43.61 | 95.45 | 99.38 | 76.35 | +1.50 | 25.00 | 22.50 | 15.00 | 86.20 | 80.62 | 45.86 | +4.90 |
| OPSD | ✓ | 75.28 | 68.06 | 43.06 | 95.20 | 99.38 | 76.20 | +1.35 | 32.22 | 20.83 | 16.39 | 85.75 | 76.25 | 46.29 | +5.33 |
| All stones | Kept, by vote on problem | ||||
|---|---|---|---|---|---|
| Kept | Rejected | No vote | Wrong | Correct | |
| Stones ( ) | 13,284 | 10,686 | 3,566 | 3,113 | 6,605 |
| Well-posed | 86.8 | 36.6 | 85.4 | 89.3 | 86.4 |
| Correct | 90.2 | 72.6 | 88.9 | 90.5 | 90.7 |
| Sound | 78.3 | 26.6 | 75.9 | 80.9 | 78.3 |
| Easier | 95.4 | 62.7 | 95.1 | 94.9 | 95.9 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Step 1: agreement on the problem | |
| Attempts | 8, non-thinking mode |
| Sampling | temperature 0.7, top- 0.8, top- 20 |
| Maximum length | 4,096 tokens |
| Selection | |
| Step 2: stepping stones | |
| Stones written | 3 per problem |
| H100 hours | |
|---|---|
| Qwen3-4B | |
| Step 1, both modes | 10.9 |
| Step 2, non-thinking | 9.3 |
| Step 3, non-thinking | 16.3 |
| Step 2, thinking | 127.3 |
| Step 3, thinking | 16.6 |
| Thinking mode | Without thinking mode | ||||||||||||
| Method | GT | AIME24 | AIME25 | HMMT25 | MATH500 | AMC23 | Avg. | AIME24 | AIME25 | HMMT25 | MATH500 | AMC23 | Avg. |
| Gemma 4 E4B-it | |||||||||||||
| Base | 56.94 | 41.39 | 28.06 | 91.20 | 85.62 | 60.64 | 43.06 | 26.94 | 20.83 | 73.25 | 76.25 | 48.07 | |
| OPSD | ✓ | 58.33 | 43.33 | 29.72 | 92.45 | 90.62 | 62.89 | 48.61 | 31.39 | 23.89 | 73.00 | 73.12 | 50.00 |
| u-OPSD † ‡ | ✗ | 58.33 | 42.50 | 28.33 | 91.15 | 89.38 | 61.94 | 50.56 | 28.61 | 26.11 | 70.50 | 73.75 | 49.91 |
| Stone (ours) | ✗ | 57.78 | 44.17 | 30.28 | 91.95 | 92.50 | 63.33 | 48.33 | 32.78 | 23.61 | 71.75 | 76.25 | 50.54 |
| Non-thinking | Thinking | |||
|---|---|---|---|---|
| Model | original | matched | original | matched |
| E4B-it | 47.66 | 50.00 | 61.54 | 62.89 |
| Non-thinking | Thinking | |||
|---|---|---|---|---|
| Model | reported | ours | reported | ours |
| Qwen3-4B | 46.29 | 45.60 | 76.20 | 75.78 |
| Qwen3-8B | 52.04 | 49.95 | 77.97 | 78.65 |
| u-OPSD | ||||
|---|---|---|---|---|
| Qwen 3 | Stone | reported | ours | diff. |
| 4B, non-thinking | 54.17 | 49.49 | 48.52 | |
| 8B, non-thinking | 66.29 | 54.31 | 56.88 | |
| 4B, thinking | 77.55 | 77.05 | 76.87 | |
| 8B, thinking | 79.20 | 77.99 | 77.66 | |
| Teacher context | Solve rate (%) | |
|---|---|---|
| No context | 46.1 | |
| Stones ( Stone ) | 47.0 | +0.9 |
| Own solution of | 80.2 | +34.1 |
| Ground-truth solution (OPSD) | 91.0 | +44.9 |
| With stones | ||||
|---|---|---|---|---|
| Model | Problems | Selected | non-thinking | thinking |
| Qwen3-4B | 29,434 | 9,218 | 7,645 | 7,338 |
| Qwen3-8B | 29,434 | 8,706 | 7,302 | 8,136 |
| Gemma 4 E4B | 29,434 | 15,842 | 14,057 | 15,084 |
| Thinking | Non-thinking | |||
| 4B | 8B | 4B | 8B | |
| Base model | 74.85 | 76.09 | 40.96 | 43.57 |
| Stone | 77.55 | 79.20 | 54.17 | 66.29 |
| Stones from the other mode | 77.31 | 78.83 | 51.68 | 65.46 |
| Own solution of | 77.40 | 78.90 | 45.23 | 58.01 |
| OPSD | 77.30 | 78.70 | 44.85 | 50.36 |
Self-Distillation
Self-distillation trains a model from its own predictions, representations, or experience to improve capability, efficiency, and robustness.
See all 154 papers