Privileged Information Does Not Guarantee Privileged Guidance: Auditing On-Policy Self-Distillation for Reasoning Models
Organizations: Meta Superintelligence Labs · University of California, Los Angeles
Abstract
On-policy self-distillation (OPSD) trains a student on next-token targets from a teacher, a frozen copy of the student that additionally receives privileged information, such as a worked solution, hidden from the student. Whether student gains reflect that information remains unclear. Taking the worked solution as privileged information, we audit four stages across three Qwen3 scales on mathematics: Source, whether the teacher can use the solution; Encoding, whether the resulting token targets are useful; Transmission, whether the student learns from the targets; and Expression, whether the student applies the learned behavior without the solution. With the solution, the teacher solves hard problems and recovers from student errors, yet the teacher does not reliably recognize errors, and its token targets do not favor correct trajectories. Repeated training makes successes within the student's reach more frequent rather than unlocking new ones, and corrections the student learns to predict are not reliably expressed in free generation. To isolate the solution's contribution, we vary its availability separately from teacher and student thinking modes. The largest gains occur when a teacher in thinking mode supervises a student trained and evaluated without an explicit thinking trace, and problem-only teachers often match or exceed privileged-solution teachers: the gain tracks the teacher's thinking mode rather than the solution, a pattern we call reasoning arbitrage. Accuracy gains alone therefore do not establish transfer of reasoning from privileged solutions: bottlenecks lie after Source, across Encoding, Transmission, and Expression. Future evaluations must measure what the solution adds to otherwise identical training.
Figures & tables
Appendix figures & tables62 assets
Supplementary material from the paper’s appendix.
Appendix
| Training component | Setting |
|---|---|
| Update unit | One on-policy response per problem; global batch 32; 100 updates (one pass over 3,200 examples). |
| Student rollout | Maximum 1,024 response tokens; temperature 1.1; top-p 0.95; top-k 20; presence penalty 0. |
| Scoring | The frozen teacher and current student score the same sampled response tokens at every prefix under their respective inputs; loss temperature 1.1. |
| Objective | Standard teacher-to-student forward KL; each vocabulary summand is upper-clipped at 0.05 before summation, then averaged over response positions; no post-vocabulary soft cap. |
| Trainable parameters | LoRA rank 64, alpha 128; target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj. |
| Optimizer | AdamW; constant learning rate ; no warmup; weight decay 0; gradient-norm clip 0.1. |
| Evaluation parameter | Value |
|---|---|
| Samples per problem | 12 |
| Decoding | Temperature 1.0; top-p 1.0; top-k disabled (-1); min-p 0; presence penalty 0. |
| Maximum response length | 38,912 tokens |
| Maximum context length | 40,960 tokens |
| Inference precision | bfloat16 |
| Scoring | Last well-formed \boxed{...} expression after any closed thinking block; mathematical-equivalence grading. |
| Teacher | Student train | Eval | Step | AIME 2024 | AIME 2025 | AIME 2026 | HMMT Nov. 2025 | HMMT Feb. 2026 | Macro average | Macro 95% CI |
|---|---|---|---|---|---|---|---|---|---|---|
| Base | — | OFF | 0 | 13.89 | 8.98 | 9.81 | 3.61 | 7.07 | 8.67 | — |
| Base | — | ON | 0 | 47.78 | 34.44 | 40.28 | 29.35 | 29.46 | 36.26 | — |
| OFF | OFF | OFF | 20 | 12.59 (-1.30) | 10.83 (+1.85) | 9.44 (-0.37) | 3.70 (+0.09) | 7.49 (+0.42) | 8.81 (+0.14) | [-0.08, +0.33] |
| OFF | OFF | OFF | 40 | 13.70 (-0.19) | 11.67 (+2.69) | 9.44 (-0.37) | 3.70 (+0.09) | 6.06 (-1.01) | 8.92 (+0.24) | [-0.02, +0.63] |
| OFF | OFF | OFF | 60 | 11.85 (-2.04) | 9.07 (+0.09) | 7.41 (-2.41) | 2.78 (-0.83) | 5.05 (-2.02) | 7.23 (-1.44) | [-2.37, -0.92] |
| OFF | OFF | OFF | 80 | 9.35 (-4.54) | 7.41 (-1.57) | 7.41 (-2.41) | 3.06 (-0.56) | 4.04 (-3.03) | 6.25 (-2.42) | [-3.18, -1.88] |
| Teacher | Student train | Eval | Step | AIME 2024 | AIME 2025 | AIME 2026 | HMMT Nov. 2025 | HMMT Feb. 2026 | Macro average | Macro 95% CI |
|---|---|---|---|---|---|---|---|---|---|---|
| Base | — | OFF | 0 | 22.78 | 18.43 | 15.09 | 7.87 | 16.25 | 16.08 | — |
| Base | — | ON | 0 | 73.33 | 65.74 | 66.48 | 55.37 | 41.16 | 60.42 | — |
| OFF | OFF | OFF | 20 | 21.76 (-1.02) | 18.24 (-0.19) | 15.37 (+0.28) | 7.41 (-0.46) | 14.31 (-1.94) | 15.42 (-0.66) | [-1.07, -0.23] |
| OFF | OFF | OFF | 40 | 22.31 (-0.46) | 16.85 (-1.57) | 14.63 (-0.46) | 7.04 (-0.83) | 12.29 (-3.96) | 14.62 (-1.46) | [-1.76, -1.02] |
| OFF | OFF | OFF | 60 | 21.02 (-1.76) | 15.74 (-2.69) | 15.46 (+0.37) | 5.19 (-2.69) | 11.95 (-4.29) | 13.87 (-2.21) | [-2.66, -1.76] |
| OFF | OFF | OFF | 80 | 22.22 (-0.56) | 18.98 (+0.56) | 16.57 (+1.48) | 6.02 (-1.85) | 12.63 (-3.62) | 15.28 (-0.80) | [-1.36, -0.24] |
| Teacher | Student train | Eval | Step | AIME 2024 | AIME 2025 | AIME 2026 | HMMT Nov. 2025 | HMMT Feb. 2026 | Macro average | Macro 95% CI |
|---|---|---|---|---|---|---|---|---|---|---|
| Base | — | OFF | 0 | 27.22 | 20.65 | 17.31 | 9.54 | 17.00 | 18.35 | — |
| Base | — | ON | 0 | 75.00 | 68.98 | 69.26 | 54.26 | 45.12 | 62.52 | — |
| OFF | OFF | OFF | 20 | 21.20 (-6.02) | 17.87 (-2.78) | 14.26 (-3.06) | 6.67 (-2.87) | 13.47 (-3.54) | 14.69 (-3.65) | [-4.29, -3.02] |
| OFF | OFF | OFF | 40 | 23.89 (-3.33) | 18.24 (-2.41) | 16.11 (-1.20) | 6.48 (-3.06) | 14.90 (-2.10) | 15.92 (-2.42) | [-3.08, -1.77] |
| OFF | OFF | OFF | 60 | 25.46 (-1.76) | 21.11 (+0.46) | 17.31 (+0.00) | 8.15 (-1.39) | 15.40 (-1.60) | 17.49 (-0.86) | [-1.56, -0.11] |
| OFF | OFF | OFF | 80 | 24.81 (-2.41) | 20.09 (-0.56) | 16.76 (-0.56) | 9.17 (-0.37) | 14.90 (-2.10) | 17.15 (-1.20) | [-1.71, -0.63] |
| Teacher input | Eval | Step | AIME 2024 | AIME 2025 | AIME 2026 | HMMT Nov. 2025 | HMMT Feb. 2026 | Macro average | Macro 95% CI |
|---|---|---|---|---|---|---|---|---|---|
| Base | OFF | 0 | 13.89 | 8.98 | 9.81 | 3.61 | 7.07 | 8.67 | — |
| Base | ON | 0 | 47.78 | 34.44 | 40.28 | 29.35 | 29.46 | 36.26 | — |
| Problem only | OFF | 20 | 8.43 (-5.46) | 6.67 (-2.31) | 7.87 (-1.94) | 3.52 (-0.09) | 4.55 (-2.53) | 6.21 (-2.47) | [-2.59, -2.34] |
| Problem only | OFF | 40 | 8.98 (-4.91) | 6.20 (-2.78) | 6.30 (-3.52) | 3.24 (-0.37) | 4.97 (-2.10) | 5.94 (-2.74) | [-3.12, -2.16] |
| Problem only | OFF | 60 | 12.78 (-1.11) | 9.81 (+0.83) | 8.80 (-1.02) | 4.17 (+0.56) | 8.84 (+1.77) | 8.88 (+0.21) | [-0.44, +0.66] |
| Problem only | OFF | 80 | 14.17 (+0.28) | 10.74 (+1.76) | 8.52 (-1.30) | 4.35 (+0.74) | 9.01 (+1.94) | 9.36 (+0.68) | [+0.08, +1.38] |
| Teacher input | Eval | Step | AIME 2024 | AIME 2025 | AIME 2026 | HMMT Nov. 2025 | HMMT Feb. 2026 | Macro average | Macro 95% CI |
|---|---|---|---|---|---|---|---|---|---|
| Base | OFF | 0 | 22.78 | 18.43 | 15.09 | 7.87 | 16.25 | 16.08 | — |
| Base | ON | 0 | 73.33 | 65.74 | 66.48 | 55.37 | 41.16 | 60.42 | — |
| Problem only | OFF | 20 | 19.44 (-3.33) | 13.52 (-4.91) | 13.33 (-1.76) | 5.28 (-2.59) | 12.88 (-3.37) | 12.89 (-3.19) | [-3.65, -2.73] |
| Problem only | OFF | 40 | 22.96 (+0.19) | 18.24 (-0.19) | 16.57 (+1.48) | 8.80 (+0.93) | 15.15 (-1.09) | 16.35 (+0.26) | [-0.48, +1.02] |
| Problem only | OFF | 60 | 40.93 (+18.15) | 32.59 (+14.17) | 31.67 (+16.57) | 25.83 (+17.96) | 23.99 (+7.74) | 31.00 (+14.92) | [+12.88, +16.62] |
| Problem only | OFF | 80 | 53.52 (+30.74) | 43.06 (+24.63) | 47.78 (+32.69) | 33.24 (+25.37) | 30.13 (+13.89) | 41.55 (+25.46) | [+23.93, +26.97] |
| Teacher input | Eval | Step | AIME 2024 | AIME 2025 | AIME 2026 | HMMT Nov. 2025 | HMMT Feb. 2026 | Macro average | Macro 95% CI |
|---|---|---|---|---|---|---|---|---|---|
| Base | OFF | 0 | 27.22 | 20.65 | 17.31 | 9.54 | 17.00 | 18.35 | — |
| Base | ON | 0 | 75.00 | 68.98 | 69.26 | 54.26 | 45.12 | 62.52 | — |
| Problem only | OFF | 20 | 22.41 (-4.81) | 15.65 (-5.00) | 13.89 (-3.43) | 5.93 (-3.61) | 14.14 (-2.86) | 14.40 (-3.94) | [-4.49, -3.38] |
| Problem only | OFF | 40 | 26.02 (-1.20) | 22.59 (+1.94) | 20.28 (+2.96) | 9.44 (-0.09) | 17.85 (+0.84) | 19.24 (+0.89) | [+0.18, +1.66] |
| Problem only | OFF | 60 | 41.94 (+14.72) | 32.50 (+11.85) | 30.83 (+13.52) | 23.70 (+14.17) | 24.83 (+7.83) | 30.76 (+12.42) | [+10.01, +13.95] |
| Problem only | OFF | 80 | 57.96 (+30.74) | 43.24 (+22.59) | 48.70 (+31.39) | 38.43 (+28.89) | 32.07 (+15.07) | 44.08 (+25.74) | [+23.30, +27.84] |
| Model | Eval | Step | Problem-only privileged (pp) | 95% paired seed-bootstrap CI |
|---|---|---|---|---|
| Qwen3-1.7B | OFF | 20 | -1.24 | [-2.28, -0.06] |
| Qwen3-1.7B | OFF | 40 | -1.13 | [-1.41, -0.84] |
| Qwen3-1.7B | OFF | 60 | -0.25 | [-0.51, +0.10] |
| Qwen3-1.7B | OFF | 80 | +0.91 | [+0.04, +2.02] |
| Qwen3-1.7B | OFF | 100 | +1.36 | [-0.33, +2.93] |
| Qwen3-1.7B | ON | 20 | -1.09 | [-1.78, -0.58] |
| Corpus | Model | Screened | Audited source | Mixed-outcome / candidate | Consensus eligible | Selected |
|---|---|---|---|---|---|---|
| OpenThoughts | Qwen3-1.7B | 3,200 | — | 646 | 296 | 50 |
| OpenThoughts | Qwen3-4B | 3,200 | — | 183 | 113 | 50 |
| OpenThoughts | Qwen3-8B | 3,200 | — | 164 | 78 | 50 |
| Hendrycks MATH Level 5 | Qwen3-1.7B | 2,242 | 111 | 180 | 149 | 50 |
| Hendrycks MATH Level 5 | Qwen3-4B | 2,242 | 44 | 80 | 69 | 50 |
| Hendrycks MATH Level 5 | Qwen3-8B | 2,242 | 35 | 79 | 53 | 50 |
| Model | Prefixes / clusters | Problem only | Shuffled | Privileged | Privileged problem | Privileged shuffled | Trunc. P/S/V |
|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | 50 / 46 | 48.50% | 41.50% | 74.50% | +26.00 [+16.35, +36.06] | +33.00 [+21.50, +44.61] | 0.0/0.5/0.0% |
| Qwen3-4B | 50 / 46 | 66.00% | 51.00% | 88.50% | +22.50 [+11.27, +34.31] | +37.50 [+24.53, +50.00] | 0.0/0.0/0.0% |
| Qwen3-8B | 50 / 46 | 63.00% | 53.50% | 88.50% | +25.50 [+15.87, +36.00] | +35.00 [+23.21, +47.34] | 0.0/0.0/0.0% |
| Model | Pairs | Valid N/R | Error N/R | Overall N/R | Pair N/R | Pair R N | Parse N/R | Trunc. N/R |
|---|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | 200 | 56.0/65.0% | 23.0/25.5% | 39.5/45.2% | 9.5/13.0% | +3.50 [-2.00, +9.50] | 78.8/78.8% | 3.0/0.8% |
| Qwen3-4B | 200 | 78.5/82.0% | 14.5/13.0% | 46.5/47.5% | 9.0/6.5% | -2.50 [-7.50, +2.50] | 89.0/87.0% | 2.5/5.5% |
| Qwen3-8B | 200 | 86.0/89.0% | 18.5/16.5% | 52.2/52.8% | 14.0/12.5% | -1.50 [-6.50, +3.50] | 94.0/97.0% | 0.5/0.2% |
| Audit dimension | Label | Thinking ON (n = 40) | Thinking OFF (n = 66) |
|---|---|---|---|
| Mechanism | Local repair | 14 (35.0%) | 22 (33.3%) |
| Mechanism | Global strategy selection | 9 (22.5%) | 19 (28.8%) |
| Mechanism | Verification or final-answer correction | 13 (32.5%) | 8 (12.1%) |
| Mechanism | Close reference reproduction | 4 (10.0%) | 8 (12.1%) |
| Mechanism | Search pruning or supplied fact | 0 (0.0%) | 9 (13.6%) |
| Deployment | Self-contained derivation | 35 (87.5%) | 58 (87.9%) |
| Model | n | Problem only | Shuffled | Privileged | Privileged problem | Privileged shuffled |
|---|---|---|---|---|---|---|
| Qwen3-1.7B | 111 | 2.93% | 1.13% | 40.54% | +37.61 [+31.31, +44.14] | +39.41 [+33.11, +45.95] |
| Qwen3-4B | 44 | 1.14% | 1.70% | 64.20% | +63.07 [+51.14, +74.43] | +62.50 [+50.57, +73.86] |
| Qwen3-8B | 35 | 0.00% | 1.43% | 58.57% | +58.57 [+46.43, +70.71] | +57.14 [+45.71, +68.57] |
| Model | Prefixes / clusters | Problem only | Shuffled | Privileged | Privileged problem | Privileged shuffled | Trunc. P/S/V |
|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | 50 / 46 | 26.00% | 22.00% | 92.50% | +66.50 [+55.29, +76.96] | +70.50 [+60.11, +80.10] | 0.5/0.0/0.0% |
| Qwen3-4B | 50 / 41 | 39.00% | 27.50% | 97.50% | +58.50 [+46.76, +69.68] | +70.00 [+57.87, +80.80] | 1.0/0.0/0.0% |
| Qwen3-8B | 50 / 41 | 37.00% | 36.50% | 96.00% | +59.00 [+47.92, +69.39] | +59.50 [+47.73, +70.28] | 1.0/0.5/0.0% |
| Model | Pairs | Valid N/R | Error N/R | Overall N/R | Pair N/R | Pair R N | Parse N/R | Trunc. N/R |
|---|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | 50 | 42/52% | 40/42% | 41/47% | 12/22% | +10.00 [-4.00, +24.00] | 83/84% | 2/2% |
| Qwen3-4B | 50 | 72/66% | 30/38% | 51/52% | 18/16% | -2.00 [-14.00, +10.00] | 92/97% | 2/1% |
| Qwen3-8B | 50 | 72/76% | 40/20% | 56/48% | 24/12% | -12.00 [-26.00, +2.00] | 99/98% | 0/1% |
| Teacher | Matched slots | Confirmed | Local | New strategy | Supplied fact | Close reproduction | Other | Self-contained | Privilege/direct |
|---|---|---|---|---|---|---|---|---|---|
| Thinking ON | 219 | 105 | 12 (11.4%) | 23 (21.9%) | 24 (22.9%) | 35 (33.3%) | 11 (10.5%) | 48 (45.7%) | 57 (54.3%) |
| Thinking OFF | 219 | 80 | 11 (13.8%) | 21 (26.3%) | 7 (8.8%) | 34 (42.5%) | 7 (8.8%) | 39 (48.8%) | 41 (51.3%) |
| Teacher | Model | Problem-only tokens | Privileged tokens | Reduction | Confirmed |
|---|---|---|---|---|---|
| Thinking ON | Qwen3-1.7B | 17,949 | 10,927 | 39.1% | 33 |
| Thinking ON | Qwen3-4B | 18,693 | 10,817 | 42.1% | 36 |
| Thinking ON | Qwen3-8B | 17,900 | 11,033 | 38.4% | 36 |
| Thinking OFF | Qwen3-1.7B | 1,980 | 1,289 | 34.9% | 24 |
| Thinking OFF | Qwen3-4B | 2,558 | 1,537 | 39.9% | 30 |
| Thinking OFF | Qwen3-8B | 2,414 | 1,461 | 39.5% | 26 |
| Student trajectories | Teacher mode | Qwen3-1.7B | Qwen3-4B | Qwen3-8B |
|---|---|---|---|---|
| Thinking OFF | Thinking OFF | 0.0508 [0.0463, 0.0558] | 0.0369 [0.0339, 0.0400] | 0.0325 [0.0296, 0.0356] |
| Thinking OFF | Thinking ON | 0.0423 [0.0381, 0.0471] | 0.0350 [0.0320, 0.0381] | 0.0315 [0.0285, 0.0345] |
| Thinking ON | Thinking OFF | 0.1382 [0.1302, 0.1463] | 0.1042 [0.0975, 0.1109] | 0.0993 [0.0928, 0.1062] |
| Thinking ON | Thinking ON | 0.0761 [0.0678, 0.0850] | 0.0689 [0.0621, 0.0761] | 0.0583 [0.0519, 0.0650] |
| Model | Student | Teacher | Math / num. | Lexical | Epistemic | Punct. / sym. | Control / space |
|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | OFF | OFF | 0.031 | 0.090 | 0.215 | 0.031 | 0.024 |
| OFF | ON | 0.025 | 0.075 | 0.144 | 0.027 | 0.022 | |
| ON | OFF | 0.025 | 0.188 | 0.811 | 0.101 | 0.059 | |
| ON | ON | 0.016 | 0.109 | 0.518 | 0.036 | 0.024 | |
| Qwen3-4B | OFF | OFF | 0.020 | 0.065 | 0.205 | 0.028 | 0.011 |
| OFF | ON | 0.019 | 0.060 | 0.187 | 0.030 | 0.012 |
| Student trajectories | Teacher mode | Qwen3-1.7B | Qwen3-4B | Qwen3-8B |
|---|---|---|---|---|
| Thinking OFF | Thinking OFF | 0.0375 [0.0331, 0.0422] | 0.0370 [0.0324, 0.0421] | 0.0301 [0.0267, 0.0338] |
| Thinking OFF | Thinking ON | 0.0324 [0.0281, 0.0368] | 0.0360 [0.0311, 0.0414] | 0.0306 [0.0269, 0.0345] |
| Thinking ON | Thinking OFF | 0.1437 [0.1296, 0.1581] | 0.0971 [0.0883, 0.1061] | 0.0954 [0.0867, 0.1045] |
| Thinking ON | Thinking ON | 0.0863 [0.0713, 0.1026] | 0.0624 [0.0538, 0.0718] | 0.0579 [0.0486, 0.0680] |
| Model | Student | Teacher | Math / num. | Lexical | Epistemic | Punct. / sym. | Control / space |
|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | OFF | OFF | 0.023 | 0.069 | 0.132 | 0.023 | 0.012 |
| OFF | ON | 0.021 | 0.058 | 0.085 | 0.021 | 0.013 | |
| ON | OFF | 0.025 | 0.202 | 0.827 | 0.104 | 0.057 | |
| ON | ON | 0.017 | 0.128 | 0.586 | 0.041 | 0.022 | |
| Qwen3-4B | OFF | OFF | 0.019 | 0.061 | 0.105 | 0.026 | 0.013 |
| OFF | ON | 0.019 | 0.057 | 0.097 | 0.028 | 0.013 |
| Student training | Model (pairs) | Privileged-solution AUROC | Problem-only AUROC | Privileged minus problem |
|---|---|---|---|---|
| Thinking OFF | Qwen3-1.7B (174) | 0.571 [0.496, 0.644] | 0.561 [0.487, 0.634] | +0.010 [-0.033, +0.054] |
| Qwen3-4B (150) | 0.569 [0.489, 0.647] | 0.547 [0.469, 0.624] | +0.022 [-0.033, +0.076] | |
| Qwen3-8B (139) | 0.484 [0.400, 0.568] | 0.499 [0.417, 0.578] | -0.014 [-0.055, +0.024] | |
| Thinking ON | Qwen3-1.7B (156) | 0.517 [0.440, 0.592] | 0.519 [0.442, 0.596] | -0.002 [-0.075, +0.071] |
| Qwen3-4B (145) | 0.448 [0.370, 0.529] | 0.497 [0.416, 0.577] | -0.048 [-0.124, +0.028] | |
| Qwen3-8B (130) | 0.549 [0.464, 0.633] | 0.523 [0.438, 0.608] | +0.026 [-0.056, +0.108] |
| Student training | Model (pairs) | Privileged-solution AUROC | Problem-only AUROC | Privileged minus problem |
|---|---|---|---|---|
| Thinking OFF | Qwen3-1.7B (174) | 0.466 [0.391, 0.540] | 0.441 [0.368, 0.513] | +0.025 [-0.036, +0.086] |
| Qwen3-4B (150) | 0.471 [0.389, 0.553] | 0.462 [0.382, 0.540] | +0.009 [-0.051, +0.067] | |
| Qwen3-8B (139) | 0.499 [0.417, 0.580] | 0.506 [0.424, 0.590] | -0.007 [-0.053, +0.036] | |
| Thinking ON | Qwen3-1.7B (156) | 0.417 [0.342, 0.494] | 0.476 [0.400, 0.556] | -0.060 [-0.128, +0.004] |
| Qwen3-4B (145) | 0.508 [0.428, 0.589] | 0.467 [0.386, 0.547] | +0.041 [-0.021, +0.103] | |
| Qwen3-8B (130) | 0.495 [0.410, 0.579] | 0.477 [0.392, 0.562] | +0.018 [-0.054, +0.090] |
| Corpus | Model | States | Student | Privileged | Privileged student | Improve / tie / harm |
|---|---|---|---|---|---|---|
| OpenThoughts | Qwen3-1.7B | 149 | 57.38% | 63.09% | +5.70 [+1.51, +10.07] | 36 / 98 / 15 |
| OpenThoughts | Qwen3-4B | 150 | 73.67% | 74.33% | +0.67 [-1.67, +3.00] | 14 / 126 / 10 |
| OpenThoughts | Qwen3-8B | 150 | 80.50% | 79.33% | -1.17 [-3.67, +1.17] | 9 / 129 / 12 |
| Hendrycks MATH Level 5 | Qwen3-1.7B | 150 | 30.50% | 29.50% | -1.00 [-4.50, +2.50] | 26 / 97 / 27 |
| Hendrycks MATH Level 5 | Qwen3-4B | 68 | 41.18% | 45.96% | +4.78 [+0.00, +9.93] | 15 / 47 / 6 |
| Hendrycks MATH Level 5 | Qwen3-8B | 59 | 50.00% | 49.58% | -0.42 [-5.08, +4.66] | 7 / 42 / 10 |
| Stratum | Audited / non-ambiguous | False negatives | Judged correct share |
| Qwen3-1.7B | 67 / 67 | 11 | 16.4% [9.4, 27.1] |
| Qwen3-4B | 67 / 66 | 36 | 54.5% [42.6, 66.0] |
| Qwen3-8B | 66 / 65 | 42 | 64.6% [52.5, 75.1] |
| Both selectors | 50 / 49 | 20 | 40.8% [28.2, 54.8] |
| Student only | 51 / 50 | 23 | 46.0% [33.0, 59.6] |
| Privileged only | 51 / 51 | 24 | 47.1% [34.1, 60.5] |
| Student mode | Model | Unsolved avail. / used | Unsolved steps | Frontier avail. / used | Frontier steps |
|---|---|---|---|---|---|
| Thinking OFF | Qwen3-1.7B | 261 / 160 | 5 / 10 / 20 / 40 / 80 | 721 / 160 | 5 / 10 / 20 / 40 / 80 |
| Thinking OFF | Qwen3-4B | 121 / 121 | 4 / 8 / 15 / 30 / 61 | 381 / 160 | 5 / 10 / 20 / 40 / 80 |
| Thinking OFF | Qwen3-8B | 156 / 156 | 5 / 10 / 20 / 39 / 78 | 399 / 160 | 5 / 10 / 20 / 40 / 80 |
| Thinking ON | Qwen3-1.7B | 106 / 106 | 3 / 7 / 13 / 27 / 53 | 111 / 111 | 3 / 7 / 14 / 28 / 56 |
| Thinking ON | Qwen3-4B | 107 / 107 | 3 / 7 / 13 / 27 / 54 | 45 / 45 | 1 / 3 / 6 / 11 / 23 |
| Thinking ON | Qwen3-8B | 97 / 97 | 3 / 6 / 12 / 24 / 49 | 35 / 35 | 1 / 2 / 4 / 9 / 18 |
| Model | Bucket | Train | n OFF | Eval OFF: 1 / 2 / 4 / 8 / 16 | n ON | Eval ON: 1 / 2 / 4 / 8 / 16 |
|---|---|---|---|---|---|---|
| Qwen3-1.7B | Frontier | OFF | 159 | 0.6 / 1.5 / 1.5 / 3.4 / 6.1 | 34 | 23.5 / 29.4 / 30.4 / 44.1 / 42.2 |
| Qwen3-1.7B | Frontier | ON | 98 | 1.4 / 0.0 / 0.0 / 1.7 / 1.0 | 108 | 1.2 / 2.2 / 3.4 / 6.8 / 7.1 |
| Qwen3-4B | Frontier | OFF | 155 | 2.2 / 1.5 / 3.7 / 6.7 / 20.0 | 16 | 12.5 / 12.5 / 8.3 / 10.4 / 10.4 |
| Qwen3-4B | Frontier | ON | 25 | 5.3 / 4.0 / 5.3 / 6.7 / 5.3 | 44 | 2.3 / 1.5 / 3.0 / 2.3 / 5.3 |
| Qwen3-8B | Frontier | OFF | 156 | 0.6 / 1.1 / 0.4 / 4.9 / 23.7 | 16 | 10.4 / 16.7 / 8.3 / 10.4 / 20.8 |
| Qwen3-8B | Frontier | ON | 20 | 6.7 / 11.7 / 15.0 / 6.7 / 13.3 | 35 | 1.0 / 1.9 / 1.9 / 1.9 / 2.9 |
| Model | Bucket | Teacher input | Eval OFF : 1 / 2 / 4 / 8 / 16 | Eval ON : 1 / 2 / 4 / 8 / 16 |
|---|---|---|---|---|
| Qwen3-1.7B | Unsolved | Privileged | +1.8 / +2.1 / +2.8 / +4.1 / +7.7 | +0.5 / +0.7 / +2.9 / +4.0 / +5.1 |
| Qwen3-1.7B | Frontier | Privileged | +0.1 / +2.0 / +2.8 / +2.8 / +9.7 | -0.1 / +0.5 / +1.4 / +2.2 / +2.2 |
| Qwen3-4B | Unsolved | Privileged | +0.5 / +1.9 / +6.0 / +9.6 / +23.0 | -0.0 / -0.4 / -0.1 / +0.6 / +1.5 |
| Qwen3-4B | Frontier | Privileged | +1.3 / +3.1 / +5.7 / +10.2 / +27.7 | -0.5 / -0.5 / -0.8 / -0.6 / -1.2 |
| Qwen3-8B | Unsolved | Privileged | +0.7 / +1.5 / +3.7 / +15.9 / +24.3 | +0.7 / +0.0 / +0.5 / +0.6 / -0.9 |
| Qwen3-8B | Frontier | Privileged | +2.0 / +0.8 / +0.0 / +12.3 / +32.3 | -0.3 / -0.6 / +0.1 / -0.7 / -0.1 |
| Model | Bucket | Teacher input | Eval OFF : 1 / 2 / 4 / 8 / 16 | Eval ON : 1 / 2 / 4 / 8 / 16 |
|---|---|---|---|---|
| Qwen3-1.7B | Unsolved | Privileged | +0.1 / +0.9 / +0.0 / +0.7 / +1.1 | +0.2 / +0.3 / +0.4 / +0.9 / +2.8 |
| Qwen3-1.7B | Frontier | Privileged | +0.2 / +0.6 / +0.8 / +1.6 / +1.9 | -1.3 / +0.9 / +2.6 / +6.1 / +7.4 |
| Qwen3-4B | Unsolved | Privileged | +1.6 / +0.3 / +1.2 / +0.5 / +1.6 | +0.2 / -0.2 / -0.0 / +1.1 / +5.5 |
| Qwen3-4B | Frontier | Privileged | -0.3 / +1.7 / +0.5 / +2.0 / +2.0 | -1.0 / +0.4 / -0.4 / +2.1 / +5.0 |
| Qwen3-8B | Unsolved | Privileged | -1.1 / -0.4 / -1.0 / -0.9 / +0.0 | -0.5 / -0.4 / -0.3 / +0.4 / +3.2 |
| Qwen3-8B | Frontier | Privileged | +0.5 / +2.0 / +1.8 / +1.2 / +2.4 | +1.2 / +0.5 / +2.4 / +1.3 / +4.3 |
| Eval OFF (R/P) | Eval ON (R/P) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Bucket | Train | 1 2 | 2 4 | 4 8 | 8 16 | 1 2 | 2 4 | 4 8 | 8 16 |
| Qwen3-1.7B | Frontier | OFF | 0/3 | 1/7 | 1/7 | 3/16 | 17/24 | 19/30 | 27/31 | 35/45 |
| Qwen3-1.7B | Frontier | ON | 0/4 | — | — | 1/5 | 0/4 | 1/7 | 4/11 | 8/22 |
| Qwen3-4B | Frontier | OFF | 1/10 | 2/7 | 5/17 | 21/31 | 3/6 | 2/6 | 2/4 | 4/5 |
| Qwen3-4B | Frontier | ON | 1/4 | 2/3 | 2/4 | 3/5 | 1/3 | 0/2 | 0/4 | 0/3 |
| Qwen3-8B | Frontier | OFF | 0/3 | 0/5 | 1/2 | 14/23 | 4/5 | 4/8 | 4/4 | 5/5 |
| Model | Bucket | Teacher input | Train n | Eval OFF [95% CI] | Eval ON [95% CI] |
|---|---|---|---|---|---|
| Qwen3-1.7B | Unsolved | Privileged | 160 | -0.05 [-0.23, +0.19] | +4.74 [+4.21, +5.27] |
| Qwen3-1.7B | Unsolved | Problem only | 160 | +3.78 [+2.88, +4.76] | +5.21 [+4.41, +5.94] |
| Qwen3-1.7B | Frontier | Privileged | 160 | +0.43 [-0.05, +0.72] | +5.14 [+4.42, +5.78] |
| Qwen3-1.7B | Frontier | Problem only | 160 | +4.38 [+2.34, +6.20] | +5.63 [+3.95, +7.03] |
| Qwen3-4B | Unsolved | Privileged | 121 | +9.12 [+7.91, +10.15] | +1.31 [+1.11, +1.54] |
| Qwen3-4B | Unsolved | Problem only | 121 | +15.98 [+14.97, +16.80] | +1.74 [+1.34, +2.03] |
| Model | Teacher input | n | Eval OFF Avg@16 | Eval OFF stable | Eval ON Avg@16 | Eval ON stable |
|---|---|---|---|---|---|---|
| Qwen3-1.7B | Privileged | 60 | 3.5 / 3.4 / 3.8 / 4.8 / 6.4 | 0 / 0 / 0 / 0 / 0 | 55.6 / 56.3 / 56.5 / 60.7 / 63.9 | 31.7 / 33.3 / 36.1 / 43.3 / 45.6 |
| Qwen3-1.7B | Problem only | 60 | 3.4 / 3.8 / 3.3 / 3.1 / 7.2 | 0 / 0 / 0 / 0 / 0 | 55.9 / 56.5 / 55.7 / 57.8 / 61.7 | 33.3 / 31.7 / 31.1 / 36.1 / 45.6 |
| Qwen3-4B | Privileged | 24 | 3.0 / 3.6 / 2.7 / 4.6 / 5.1 | 0 / 0 / 0 / 0 / 0 | 64.0 / 64.0 / 65.3 / 64.1 / 65.6 | 47.2 / 44.4 / 51.4 / 48.6 / 48.6 |
| Qwen3-4B | Problem only | 24 | 2.5 / 3.4 / 3.9 / 3.8 / 5.0 | 0 / 0 / 0 / 0 / 0 | 64.4 / 64.1 / 63.9 / 64.2 / 65.3 | 47.2 / 50.0 / 47.2 / 47.2 / 47.2 |
| Qwen3-8B | Privileged | 11 | 4.0 / 6.6 / 4.4 / 4.2 / 3.2 | 0 / 0 / 0 / 0 / 0 | 79.2 / 77.1 / 77.7 / 78.0 / 78.2 | 72.7 / 63.6 / 72.7 / 69.7 / 72.7 |
| Qwen3-8B | Problem only | 11 | 5.9 / 5.9 / 5.5 / 4.7 / 5.9 | 0 / 0 / 0 / 0 / 0 | 77.3 / 76.7 / 78.4 / 78.8 / 78.0 | 63.6 / 69.7 / 66.7 / 69.7 / 72.7 |
| Model | Schedule | Base mean [95% CI] | Step 20 | Step 40 | Step 60 | Step 80 | Step 100 |
|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | Standard | 36.26 [35.80, 36.60] | 36.88 [36.75, 37.04] | 35.93 [35.71, 36.10] | 34.18 [34.04, 34.38] | 34.87 [33.71, 35.97] | 34.35 [33.27, 34.91] |
| Qwen3-1.7B | Purified | 36.26 [35.80, 36.60] | 36.77 [36.06, 37.66] | 37.51 [36.71, 37.92] | 36.71 [36.61, 36.78] | 37.44 [36.96, 37.99] | 38.12 [37.10, 38.79] |
| Qwen3-1.7B | S20 P80 | 36.26 [35.80, 36.60] | 36.88 [36.75, 37.04] | 36.57 [35.46, 37.59] | 37.85 [37.02, 38.44] | 38.27 [38.07, 38.57] | 37.30 [36.08, 38.30] |
| Qwen3-1.7B | P20 S80 | 36.26 [35.80, 36.60] | 36.77 [36.06, 37.66] | 38.01 [37.52, 38.59] | 34.98 [34.14, 36.21] | 34.95 [34.53, 35.24] | 35.03 [33.90, 35.63] |
| Qwen3-4B | Standard | 60.42 [60.33, 60.51] | 61.17 [60.71, 61.80] | 59.77 [59.67, 59.89] | 58.78 [58.61, 58.88] | 57.72 [57.49, 58.06] | 57.79 [57.36, 58.53] |
| Qwen3-4B | Purified | 60.42 [60.33, 60.51] | 61.30 [60.54, 62.21] | 61.31 [60.69, 62.21] | 61.04 [60.72, 61.32] | 61.10 [60.67, 61.44] | 61.16 [61.04, 61.30] |
| Model | Composition | Standard last | Purified last | P-last S-last |
|---|---|---|---|---|
| Qwen3-1.7B | 80% S + 20% P | 34.02 | 36.62 | +2.60 [+2.16, +3.11] |
| Qwen3-1.7B | 20% S + 80% P | 37.93 | 37.84 | -0.09 [-0.54, +0.39] |
| Qwen3-4B | 80% S + 20% P | 59.30 | 59.90 | +0.60 [+0.19, +1.08] |
| Qwen3-4B | 20% S + 80% P | 61.25 | 60.88 | -0.37 [-0.94, +0.36] |
| Qwen3-8B | 80% S + 20% P | 59.94 | 63.22 | +3.27 [+3.16, +3.42] |
| Qwen3-8B | 20% S + 80% P | 63.80 | 63.95 | +0.15 [-0.89, +1.19] |
| Arm | Step 20 | Step 40 | Step 60 | Step 80 | Step 100 |
|---|---|---|---|---|---|
| RawExact | 54.98 [54.92, 55.11] | 53.50 [52.22, 55.45] | 52.44 [51.37, 53.11] | 52.69 [51.97, 53.33] | 52.35 [52.06, 52.88] |
| PurifiedExact | 57.43 [56.84, 57.84] | 57.66 [57.35, 58.13] | 57.04 [56.44, 57.88] | 58.13 [57.34, 58.73] | 58.52 [58.09, 59.16] |
| Standard, clipped | 54.39 | 51.68 | 43.58 | 29.24 | 14.08 |
| Purified, canonical | 57.89 | 57.52 | 59.59 | 57.08 | 57.82 |
| Benchmark | Arm | Metric | Base | Step 20 | Step 60 | Step 100 |
|---|---|---|---|---|---|---|
| LCBv6 | Standard | Accuracy | 44.66 | 46.88 [45.99, 47.90] | 43.26 [42.37, 43.70] | 40.27 [39.69, 41.03] |
| Truncation | 3.82 | 4.20 [3.44, 4.96] | 15.33 [13.74, 16.41] | 29.39 [28.82, 30.34] | ||
| Purified | Accuracy | 44.66 | 43.00 [41.22, 43.89] | 43.00 [41.98, 44.47] | 44.72 [43.70, 45.99] | |
| Truncation | 3.82 | 5.34 [4.96, 5.73] | 4.64 [4.58, 4.77] | 3.75 [3.24, 4.39] | ||
| CodeContests | Standard | Accuracy | 42.58 | 46.35 [46.09, 46.68] | 38.48 [37.30, 39.45] | 35.29 [34.96, 35.55] |
| Truncation | 6.25 | 6.45 [6.25, 6.64] | 26.43 [24.61, 28.91] | 44.21 [43.36, 45.51] |
| Budget | Seeds | Benchmark | Arm | Steps | Metric | Checkpoint 1 | Checkpoint 2 | Checkpoint 3 |
|---|---|---|---|---|---|---|---|---|
| 16K | 3 | LCBv6 | Standard | 20/60/100 | Accuracy | 47.07 [46.18, 48.47] | 39.76 [38.55, 40.46] | 37.60 [37.02, 38.55] |
| Truncation | 2.86 [2.48, 3.24] | 25.38 [23.09, 28.05] | 45.29 [43.51, 46.37] | |||||
| 16K | 3 | LCBv6 | Purified | 20/60/100 | Accuracy | 44.53 [42.75, 46.18] | 43.58 [41.98, 44.66] | 44.78 [43.32, 46.76] |
| Truncation | 4.39 [4.01, 4.58] | 4.45 [3.63, 5.53] | 4.07 [3.82, 4.39] | |||||
| 16K | 3 | CodeContests | Standard | 20/60/100 | Accuracy | 46.35 [45.70, 47.07] | 34.11 [33.01, 35.35] | 26.17 [25.20, 26.95] |
| Truncation | 4.75 [3.52, 7.03] | 43.36 [41.41, 46.88] | 62.57 [61.91, 63.09] |
| Benchmark | Schedule | Accuracy mean [seed range] | from Base | Truncation mean [seed range] |
|---|---|---|---|---|
| LCBv6 | S20 P80 | 44.59 [41.98, 46.18] | -0.06 | 4.96 [4.39, 5.34] |
| LCBv6 | P80 S20 | 41.54 [40.46, 42.94] | -3.12 | 29.58 [28.63, 30.92] |
| LCBv6 | P20 S80 | 39.12 [37.60, 41.60] | -5.53 | 25.25 [19.85, 30.73] |
| LCBv6 | S80 P20 | 44.47 [44.08, 44.85] | -0.19 | 3.82 [3.44, 4.39] |
| CodeContests | S20 P80 | 41.15 [38.87, 42.38] | -1.43 | 6.32 [5.47, 7.23] |
| CodeContests | P80 S20 | 30.08 [28.91, 30.86] | -12.50 | 53.91 [53.32, 54.69] |
| Model | Candidates | Self-contained | Source-dependent | Invalid | Selected | Train / held out |
|---|---|---|---|---|---|---|
| Qwen3-4B | 499 | 383 | 36 | 80 | 125 | 100 / 25 |
| Qwen3-8B | 415 | 320 | 28 | 67 | 102 | 82 / 20 |
| Arm | Total updates | Correction updates | Correction share | Scheduled examples |
|---|---|---|---|---|
| Standard | 48 | 0 | 0% | 1,536 |
| Correction25 | 48 | 12 | 25% | 1,536 |
| Correction50 | 48 | 24 | 50% | 1,536 |
| Correction100 | 48 | 48 | 100% | 1,536 |
| Readout | Input | Evaluation |
|---|---|---|
| Correction-target fit | Exact audited error prefix; teacher-forced verified correction | Question-averaged held-out per-token NLL; Steps 12/24/36/48 |
| Audited-prefix generation | Exact audited error prefix; no privileged support | Avg@8 accuracy; 16,384-token cap |
| Original-question restart | Original problem prompt; no privileged support | Avg@8 accuracy; 16,384-token cap |
| Explicit repair feedback | Error prefix + answer-free repair instruction | Step 48 Avg@8 accuracy; 16,384-token cap |
| Correction onset | Error prefix + first 32 non-answer-revealing correction tokens | Step 48 Avg@8 accuracy; 16,384-token cap |
| Broad mathematics | Original benchmark prompt; no privileged support | Step 48 Avg@12; 38,912-token cap; five-benchmark macro |
| Held-out correction-target NLL | Audited-prefix Avg@8 (%) | Question-restart Avg@8 (%) | |||||
| Arm | Step | Mean [95% CI] | Mean [95% CI] | Mean [95% CI] | |||
| Frozen Base | 0 | 0.414 | 91.50 | 88.00 | |||
| Standard | 12 | 0.420 [0.420, 0.420] | +0.006 | 90.67 [89.54, 91.50] | -0.83 | 88.50 [86.37, 90.28] | +0.50 |
| 24 | 0.430 [0.429, 0.430] | +0.016 | 89.50 [87.61, 91.92] | -2.00 | 90.17 [87.87, 92.17] | +2.17 | |
| 36 | 0.438 [0.438, 0.438] | +0.024 | 92.33 [90.93, 93.67] | +0.83 | 92.50 [90.47, 94.00] | +4.50 | |
| 48 | 0.441 [0.441, 0.441] | +0.027 | 93.83 [92.43, 95.17] | +2.33 | 89.50 [88.04, 90.78] | +1.50 | |
| Held-out correction-target NLL | Audited-prefix Avg@8 (%) | Question-restart Avg@8 (%) | |||||
| Arm | Step | Mean [95% CI] | Mean [95% CI] | Mean [95% CI] | |||
| Frozen Base | 0 | 0.418 | 86.25 | 87.50 | |||
| Standard | 12 | 0.416 [0.415, 0.416] | -0.003 | 89.38 [88.40, 90.35] | +3.12 | 91.25 [89.51, 93.43] | +3.75 |
| 24 | 0.415 [0.414, 0.415] | -0.004 | 90.21 [89.51, 90.98] | +3.96 | 89.17 [88.40, 89.86] | +1.67 | |
| 36 | 0.415 [0.414, 0.415] | -0.004 | 88.54 [87.50, 89.95] | +2.29 | 89.58 [89.02, 90.00] | +2.08 | |
| 48 | 0.415 [0.415, 0.415] | -0.003 | 88.54 [86.66, 90.57] | +2.29 | 90.21 [89.02, 91.47] | +2.71 | |
| Step-48 metric | Frozen Base | Standard | Correction25 | Correction50 | Correction100 |
|---|---|---|---|---|---|
| Held-out correction-target NLL | 0.414 | 0.441 [0.441, 0.441] | 0.404 [0.404, 0.404] | 0.384 [0.384, 0.385] | 0.361 [0.361, 0.361] |
| Exact audited prefix, Avg@8 | 91.50 | 93.83 [92.43, 95.17] | 92.00 [91.61, 92.39] | 89.33 [88.11, 90.85] | 87.67 [86.43, 88.78] |
| Original-question restart, Avg@8 | 88.00 | 89.50 [88.04, 90.78] | 90.00 [88.54, 91.28] | 91.00 [88.58, 92.89] | 87.17 [85.33, 89.24] |
| Explicit repair feedback, Avg@8 | 96.50 | 94.33 [94.11, 94.50] | 95.50 [95.11, 95.89] | 92.50 [90.47, 94.00] | 91.17 [88.37, 93.85] |
| First 32 correction tokens, Avg@8 | 91.00 | 89.50 [88.72, 90.28] | 88.83 [88.22, 89.39] | 86.33 [86.11, 86.50] | 85.83 [84.54, 86.89] |
| Five-benchmark Avg@12 | 60.42 | 60.19 [59.72, 60.68] | 58.89 [58.04, 59.58] | 56.53 [56.13, 56.83] | 53.16 [52.84, 53.43] |
| Step-48 metric | Frozen Base | Standard | Correction25 | Correction50 | Correction100 |
|---|---|---|---|---|---|
| Held-out correction-target NLL | 0.418 | 0.415 [0.415, 0.415] | 0.404 [0.404, 0.404] | 0.397 [0.397, 0.397] | 0.385 [0.385, 0.385] |
| Exact audited prefix, Avg@8 | 86.25 | 88.54 [86.66, 90.57] | 87.92 [87.64, 88.12] | 87.08 [86.52, 87.50] | 86.88 [86.39, 87.36] |
| Original-question restart, Avg@8 | 87.50 | 90.21 [89.02, 91.47] | 89.58 [88.89, 90.35] | 89.58 [88.89, 90.35] | 90.62 [89.02, 92.45] |
| Explicit repair feedback, Avg@8 | 93.12 | 93.54 [92.64, 94.59] | 92.29 [91.88, 92.85] | 93.33 [92.15, 94.59] | 92.29 [91.88, 92.85] |
| First 32 correction tokens, Avg@8 | 88.16 | 89.47 [88.96, 89.99] | 87.72 [84.84, 89.99] | 87.50 [86.84, 88.39] | 87.94 [86.10, 89.70] |
| Five-benchmark Avg@12 | 62.52 | 63.49 [62.92, 64.08] | 62.48 [62.37, 62.57] | 62.56 [61.58, 63.29] | 60.10 [59.56, 60.69] |
| Arm | AIME24 | AIME25 | AIME26 | HMMT Nov25 | HMMT Feb26 | Macro |
|---|---|---|---|---|---|---|
| Frozen Base (3-seed mean) | 73.33 | 65.74 | 66.48 | 55.37 | 41.16 | 60.42 |
| Standard | 74.54 [73.26, 75.65] | 64.91 [64.23, 65.75] | 64.72 [64.13, 65.22] | 54.17 [52.74, 55.79] | 42.59 [41.58, 43.75] | 60.19 [59.72, 60.68] |
| Correction25 | 73.61 [71.24, 75.59] | 63.06 [62.19, 63.93] | 63.52 [62.05, 64.66] | 51.76 [50.36, 53.19] | 42.51 [41.74, 43.13] | 58.89 [58.04, 59.58] |
| Correction50 | 70.65 [68.56, 72.48] | 60.46 [60.12, 60.77] | 61.67 [60.54, 62.50] | 50.56 [48.66, 52.36] | 39.31 [38.44, 40.42] | 56.53 [56.13, 56.83] |
| Correction100 | 67.96 [67.40, 68.49] | 56.57 [55.52, 57.69] | 58.15 [57.84, 58.49] | 46.94 [45.82, 47.78] | 36.20 [34.81, 37.78] | 53.16 [52.84, 53.43] |
| Qwen3-4B metric | Standard | Correction100 | Matched-SFT | C100 - Matched |
|---|---|---|---|---|
| Held-out correction-target NLL | 0.441 [0.441, 0.441] | 0.361 [0.361, 0.361] | 0.416 [0.416, 0.417] | -0.055 [-0.056, -0.055] |
| Audited-prefix accuracy | 93.83 [92.43, 95.17] | 87.67 [86.43, 88.78] | 88.33 [87.50, 89.46] | -0.67 [-3.02, +1.28] |
| Question-restart accuracy | 89.50 [88.04, 90.78] | 87.17 [85.33, 89.24] | 88.67 [85.30, 91.57] | -1.50 [-4.31, +0.78] |
| Repair-instruction accuracy | 94.33 [94.11, 94.50] | 91.17 [88.37, 93.85] | 94.83 [94.22, 95.39] | -3.67 [-7.03, -0.37] |
| 32-token-onset accuracy | 89.50 [88.72, 90.28] | 85.83 [84.54, 86.89] | 86.33 [84.93, 87.67] | -0.50 [-3.13, +1.96] |
| Five-benchmark Avg@12 | 60.19 [59.72, 60.68] | 53.16 [52.84, 53.43] | 59.71 [59.29, 60.27] | -6.55 [-7.06, -6.02] |
| Qwen3-8B metric | Standard | Correction100 | Matched-SFT | C100 - Matched |
|---|---|---|---|---|
| Held-out correction-target NLL | 0.415 [0.415, 0.415] | 0.385 [0.385, 0.385] | 0.419 [0.419, 0.419] | -0.034 [-0.034, -0.033] |
| Audited-prefix accuracy | 88.54 [86.66, 90.57] | 86.88 [86.39, 87.36] | 86.88 [85.41, 88.34] | +0.00 [-0.98, +0.98] |
| Question-restart accuracy | 90.21 [89.02, 91.47] | 90.62 [89.02, 92.45] | 88.12 [86.79, 89.24] | +2.50 [+1.03, +3.97] |
| Repair-instruction accuracy | 93.54 [92.64, 94.59] | 92.29 [91.88, 92.85] | 93.33 [92.77, 93.75] | -1.04 [-1.88, +0.08] |
| 32-token-onset accuracy | 89.47 [88.96, 89.99] | 87.94 [86.10, 89.70] | 86.18 [85.30, 86.84] | +1.75 [+0.80, +2.86] |
| Five-benchmark Avg@12 | 63.49 [62.92, 64.08] | 60.10 [59.56, 60.69] | 63.38 [62.98, 63.71] | -3.28 [-3.88, -2.83] |
| Qwen3-4B metric | Frozen Base | Standard | Correction50 | Matched-SFT50 | Matched - C50 |
|---|---|---|---|---|---|
| Five-benchmark Avg@12 | 60.42 [60.33, 60.51] | 60.19 [59.62, 60.84] | 56.53 [55.95, 56.85] | 60.68 [60.28, 60.99] | +4.15 [+3.97, +4.32] |
| Truncation (%) | 2.58 [2.56, 2.61] | 7.81 [7.24, 8.55] | 1.18 [0.98, 1.36] | 3.49 [3.21, 3.76] | +2.31 [+2.23, +2.40] |
| Arm | Held-out correction-target NLL | Audited-prefix accuracy | Five-benchmark Avg@12 | Truncation |
|---|---|---|---|---|
| Base-recovery SFT | 0.418 [0.418, 0.418] | 87.50 [86.22, 88.96] | 59.54 [58.95, 59.90] | 3.21 [2.94, 3.38] |
| Privileged-correction subset | 0.363 [0.362, 0.363] | 85.17 [83.65, 86.39] | 53.17 [52.66, 53.77] | 0.22 [0.11, 0.38] |
| Condition | Step 20 | Step 40 | Step 60 | Step 80 | Step 100 | Final Base |
|---|---|---|---|---|---|---|
| Frozen Base | 60.42 | 60.42 | 60.42 | 60.42 | 60.42 | +0.00 |
| Standard, LoRA | 61.50 [60.48, 62.08] | 59.52 [59.23, 59.68] | 57.92 [57.55, 58.44] | 58.33 [57.96, 58.94] | 58.09 [57.29, 59.05] | -2.33 |
| Standard, full parameter | 60.47 [60.23, 60.75] | 58.97 [58.57, 59.53] | 58.10 [57.14, 58.96] | 57.84 [57.35, 58.37] | 56.80 [56.23, 57.35] | -3.62 |
| Purified, LoRA | 61.67 [60.97, 62.26] | 60.98 [60.16, 61.52] | 61.25 [60.47, 61.89] | 61.69 [60.83, 62.21] | 62.35 [61.99, 62.76] | +1.93 |
| Purified, full parameter | 60.56 [59.85, 61.14] | 60.82 [60.56, 61.05] | 60.66 [60.22, 61.03] | 60.59 [59.52, 61.30] | 61.00 [60.85, 61.20] | +0.58 |
| Selector | Reference-free gain (pp) | Role |
|---|---|---|
| Hindsight oracle | +11.30 [+8.15, +14.63] | Non-deployable upper bound |
| Always privileged target | +2.04 | No routing |
| Matched random router | +1.12 | Matches 54.8% target coverage |
| Held-out learned critic | +0.56 [-2.96, +4.07] | Five-fold CV grouped by problem |
| High-FKL matched router | -1.30 | Same target-action count as critic |
| Low-FKL matched router | +3.15 | Same target-action count as critic |
| Arm | Accuracy [95% CI] | Base [95% CI] | Pass-any (%) | Trunc. (%) | Mean tokens |
|---|---|---|---|---|---|
| Frozen Base | 6.25 [0.00, 12.50] | +0.00 [0.00, 0.00] | 25.00 | 39.58 | 12,928.5 |
| Oracle replay | 8.33 [0.00, 18.75] | +2.08 [-6.25, +10.42] | 25.00 | 41.67 | 12,645.2 |
| FKL replay | 2.08 [0.00, 6.25] | -4.17 [-10.42, +0.00] | 8.33 | 39.58 | 12,607.2 |
| Random 0 | 8.33 [0.00, 18.75] | +2.08 [-4.17, +8.33] | 25.00 | 45.83 | 12,713.9 |
| Random 1 | 2.08 [0.00, 6.25] | -4.17 [-10.42, +0.00] | 8.33 | 39.58 | 12,916.5 |
| Random 2 | 4.17 [0.00, 10.42] | -2.08 [-8.33, +4.17] | 16.67 | 45.83 | 13,010.1 |
| Eval OFF: seed 0 / 1 / 2 | OFF mean [95% CI] | Eval ON: seed 0 / 1 / 2 | ON mean [95% CI] | |
|---|---|---|---|---|
| 0 | 38.94 / 41.79 / 52.37 | 44.37 [38.94, 52.37] | 60.89 / 61.09 / 61.06 | 61.01 [60.89, 61.09] |
| 0.5 | 33.77 / 37.30 / 43.04 | 38.04 [33.77, 43.04] | 60.12 / 60.95 / 62.11 | 61.06 [60.12, 62.11] |
| 1 | 27.66 / 30.49 / 30.17 | 29.44 [27.66, 30.49] | 59.75 / 59.98 / 60.88 | 60.21 [59.75, 60.88] |
| 1.5 | 21.44 / 22.25 / 20.73 | 21.47 [20.73, 22.25] | 58.93 / 59.25 / 59.29 | 59.15 [58.93, 59.29] |
| Evaluation mode | Contrast | Avg@12 [95% CI] |
|---|---|---|
| Thinking OFF | - | -6.33 [-9.34, -4.49] |
| Thinking OFF | - | -14.93 [-22.20, -11.28] |
| Thinking OFF | - | -22.90 [-31.64, -17.50] |
| Thinking OFF | - | -7.97 [-9.44, -6.22] |
| Thinking ON | - | +0.05 [-0.77, +1.05] |
| Thinking ON | - | -0.80 [-1.14, -0.18] |
| Eval mode | Step 20 | Step 40 | Step 60 | Step 80 | Step 100 | |
|---|---|---|---|---|---|---|
| Thinking OFF | 0 | 12.89 | 16.35 | 31.00 | 41.55 | 43.87 |
| Thinking OFF | 0.5 | 13.99 | 16.48 | 25.53 | 33.85 | 38.04 |
| Thinking OFF | 1 | 14.80 | 16.62 | 21.89 | 27.13 | 28.66 |
| Thinking OFF | 1.5 | 16.20 | 17.30 | 20.23 | 20.84 | 21.47 |
| Thinking ON | 0 | 60.65 | 61.07 | 61.88 | 62.19 | 60.96 |
| Thinking ON | 0.5 | 60.93 | 60.95 | 61.25 | 61.93 | 61.06 |
On-Policy Self-Distillation
On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.
See all 174 papers