Shuffled OPSD: When Semantic Consistency Hurts On-Policy Self-Distillation?
Organizations: Duke University
Abstract
Knowledge distillation improves a student model's reasoning ability through supervision from a teacher. On-policy self-distillation (OPSD) uses the same model as both teacher and student under different contexts: the student generates reasoning trajectories from the question alone, while the teacher additionally receives a complete worked example and provides token-level supervision along those trajectories. Because the problem statement is shared, its content and ordering shape both student generation and teacher feedback. Preserving its semantic structure therefore seems natural, but does it always improve learning? We introduce Shuffled OPSD, which shuffles the words of the shared problem statement while keeping the teacher-only worked example intact. Across three mathematical reasoning benchmarks and three Qwen3 model scales, the method improves mean accuracy over both an intact-target control and Standard OPSD under matched training configurations. The largest gain over Standard OPSD is 3.52 percentage points on Qwen3-1.7B. Further controls show that preserving larger word blocks offers no advantage over word-level shuffling at the evaluated scales, whereas shuffling the teacher-only worked example reduces accuracy at every scale. Cross-problem mixing and target-removal experiments further distinguish the roles of problem content, word order, and teacher context. These results show that disrupting the shared problem statement can improve learning, while disrupting the teacher-only reference hurts performance. They challenge semantic coherence as a universally beneficial principle for OPSD and motivate closer study of how input structure shapes student generation and teacher supervision.
Figures & tables
| Model | Condition | AIME24 | AIME25 | HMMT25 | Avg. | Ref | OPSD |
|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | Frozen Base | 50.28 | 35.00 | 21.94 | 35.74 | -3.06 | -3.52 |
| UnrelatedRef | 52.78 | 39.17 | 24.44 | 38.80 | 0.00 | -0.46 | |
| OPSD | 53.06 | 38.89 | 25.83 | 39.26 | +0.46 | 0.00 | |
| Shuffle-1 (Ours) | 57.22 | 45.83 | 25.28 | 42.78 | +3.98 | +3.52 | |
| CrossShuffle | 53.33 | 44.72 | 25.28 | 41.11 | +2.31 | +1.85 | |
| AllShuffle-1 | 51.94 | 42.50 | 26.67 | 40.37 | +1.57 | +1.11 |
| Condition | Reference? | Teacher mode | Mean | NoTarget |
|---|---|---|---|---|
| NoTarget | Yes | On | 41.76 | +0.00 |
| NoTarget-NoRef | No | On | 37.78 | -3.98 |
| NoTarget-SameMode | Yes | Off | 38.98 | -2.78 |
| Teacher-only reference | Non-thinking | Thinking | KL |
|---|---|---|---|
| None | 0.0000 | 0.2064 | +0.2064 |
| Intact unrelated | 0.0419 | 0.1605 | +0.1186 |
| Word-shuffled | 0.0507 | 0.1869 | +0.1362 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| Backbones | Qwen3-1.7B; Qwen3-4B; Qwen3-8B |
| Training data | Full OpenThoughts mathematical-reasoning training pool (29,434 records) |
| Training seed | 42 |
| Optimizer steps | 50 |
| Global / generation batch size | 16 / 16 |
| Optimizer | AdamW; , , |
| Condition | Shared problem | Teacher-only reference | Teacher mode |
|---|---|---|---|
| Standard OPSD | Intact | Associated solution | Thinking |
| UnrelatedRef | Intact | Unrelated worked example | Thinking |
| Shuffle-1 | Word shuffled | Same as UnrelatedRef | Thinking |
| Shuffle-4 | Four-word blocks shuffled | Same as UnrelatedRef | Thinking |
| Shuffle-8 | Eight-word blocks shuffled | Same as UnrelatedRef | Thinking |
| CrossShuffle | Words redistributed across problems | Same as UnrelatedRef | Thinking |
| Setting | Value |
|---|---|
| Benchmarks | AIME 2024; AIME 2025; HMMT February 2025 |
| Problems per benchmark | 30 |
| Input | Original intact problem |
| Teacher-only reference | None |
| Thinking mode | Enabled |
| Temperature | 1.0 |
| Model | Condition | AIME24 | AIME25 | HMMT25 | Avg. |
|---|---|---|---|---|---|
| Qwen3-1.7B | Shuffle-4 | 54.17 | 41.39 | 27.50 | 41.02 |
| Shuffle-8 | 56.67 | 41.94 | 23.33 | 40.65 | |
| AllShuffle-4 | 52.78 | 38.33 | 25.28 | 38.80 | |
| AllShuffle-8 | 55.83 | 41.39 | 23.89 | 40.37 | |
| Qwen3-4B | Shuffle-4 | 76.67 | 66.94 | 43.06 | 62.22 |
| Shuffle-8 | 75.28 | 68.61 | 44.72 | 62.87 |
On-Policy Self-Distillation
On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.
See all 174 papers