On-policy self-distillation (OPSD) enables a language model to learn from a Teacher instantiated from the same model but given access to privileged information, such as a reference solution. We find that access to sufficient privileged information allows the Teacher to generate highly accurate trajectories; however, accuracy alone does not guarantee that these trajectories are suitable for Student learning. Direct access to a complete solution may encourage shortened, reference-conditioned derivations that omit the exploration, verification, and self-correction required to solve problems without privileged information. We introduce TeachBack, a bidirectional preference-learning framework for on-policy self-distillation that learns not only from the Teacher but also how the Teacher should teach. TeachBack shapes the Teacher's generation behavior by controlling the length of the reference-reasoning suffix injected into the prompt and by varying the prompt style. While maintaining comparable Teacher accuracy, it encourages the Teacher to explore and re-solve problems, thereby reducing its reliance on the reference solution. At each rollout refresh, the shared policy generates both a privileged Teacher trajectory and a problem-only Student trajectory. A pairwise preference objective uses privileged Teacher trajectories to improve Student learning, while the reversed preference uses the Student's behavior to shape future Teacher supervision under the privileged condition. Experiments on mathematical and logical reasoning tasks show that TeachBack consistently improves Student performance over existing OPSD methods and related baselines.
Figures & tables
Figure 1: Motivation for TeachBack . In standard OPSD, access to privileged information can lead the Teacher to produce a short solution that omits exploration and verification. TeachBack uses partial reference reasoning and Student feedback to encourage complete, self-contained Teacher trajectories that remain useful to a problem-only Student.
Figure 2: Overview of TeachBack . (a) The Student receives the problem alone, while a reference-reasoning suffix is framed as optional working notes for the Teacher to re-solve the problem. (b) The shared policy samples paired Student and Teacher trajectories; both responses are replayed under both conditions and scored against a fixed reference policy. (c) Under the problem-only condition, the Teacher-origin response is preferred; under the PI-conditioned condition, the Student-origin response is preferred. The weighted losses update the shared policy before the next on-policy pair is generated. At test time, only the problem-only Student condition is used.
Mathematical Reasoning (mean@12)
Knight-and-Knaves (pass@1)
Model
Method
AMC23
AIME24
AIME25
Avg.
4–8 Roles
9 Roles
10 Roles
11 Roles
Avg.
Qwen3-1.7B
Base
75.7
46.3
37.5
53.2
63.8
56.0
44.0
34.0
49.4
GRPO
75.0
46.6
37.5
53.0
67.4
54.0
53.0
33.0
51.8
OPSD
74.5
47.7
35.0
52.4
64.4
55.0
52.0
32.0
50.9
SPIN
75.9
45.0
36.7
52.5
67.8
58.0
51.0
42.0
54.7
PBSD
72.8
48.3
37.5
52.9
67.0
63.0
44.0
32.0
51.5
Table 1: Main results across model scales. Mathematical reasoning is reported as mean@12. Knight-and-Knaves (KK) is reported as pass@1, with 4–8 Roles as the in-domain (ID) setting and 9–11 Roles as out-of-domain (OOD) settings. Bold and underlined entries denote the best and second-best results within each model block; parenthetical values are TeachBack gains over Base.
Figure 3: Prompt-construction diagnostics. (a) Five Teacher prompt styles are compared with the Student. The gray shaded region represents Student behavior, and the green shaded region represents Notes Teacher behavior. Reference, Notes, and History share a 50% privileged body; Answer-only and Instruction use different information budgets. (b) The Working Notes suffix sweep varies donor-reasoning injection from 0% to 100% and plots accuracy, rollout length, exploration, and reference degree. 0% suffix represents the Student with no injected reasoning. The shaded 25–50% range yields high accuracy while retaining more exploration and lower reference degree than larger suffix ratios.
Figure 4: Diagnostics from the Qwen3-1.7B run at λ=0.75 . Panels (a)–(d) show the branch and joint losses, four reference-relative scores Aθc(y) , conditional entropy for the corresponding c⊕y pairs, and diagnostic KL, respectively. Here c is the scoring condition and y the sampled response.
Figure 5: Training dynamics under different Teacher-side weights. In (a)–(b), colors denote λ , solid and dashed curves denote Student and Teacher rollouts. The panels show non-clipped accuracy, response length, policy entropy, and Math mean@12, respectively.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Condition
PI ratio
Accuracy
Length
Trunc.
Explore/1K
Breadth
Content ref.
Ref. degree
Explicit
Strong attr.
(%)
(%)
(tokens)
(%)
(/1K tokens)
(%)
(%)
(%)
(%)
(%)
Student
0
79.88
7459.2
7.42
17.19
98.46
—
—
5.66
0.39
Working notes
25
87.30
4209.9
2.15
11.64
91.58
16.09
17.23
21.78
13.96
Working notes
50
88.28
3605.7
1.95
9.93
87.21
21.42
19.52
11.91
6.54
Working notes
75
86.72
3308.0
0.98
9.30
83.54
24.91
21.45
7.62
4.88
Working notes
100
87.70
2718.7
0.20
7.86
69.18
32.04
26.14
2.54
1.27
Appendix
Table 2: Prompt-construction audit results. The privileged-information column reports the percentage of the full ground-truth reasoning body injected into the prompt; metric definitions and the complete rollout protocol are given in this subsection.
Main dimension
Subcategory
Matched words, phrases, or pattern families
Reasoning exploration
Exploration
another approach ; alternative ; another way ; different approach ; different way ; let’s try ; alternatively
Uncertainty
maybe ; perhaps ; possibly ; might ; could ; may ; probably ; likely ; unlikely ; it seems ; seems like ; I think ; not sure
Verification
check ; verify ; confirm ; double-check ; let’s check ; make sure ; ensure ; test ; examine
Backtracking
wait ; hold on ; actually ; I made a mistake ; that’s not right ; go back ; correction
Replanning
first, let’s ; we need to consider ; let’s analyze ; step by step ; approach ; consider ; instead
Reference dependence
Explicit mention
reference ; reference solution ; reference answer ; working notes ; notes say/state/indicate ; according to ; based on (the) notes ; given/provided answer, solution, or reasoning
Appendix
Table 3: Unified lexical inventory for the two text-based diagnostic dimensions. Reasoning exploration is divided into five behavioral categories. Reference dependence uses a broad explicit-mention detector and a stricter strong-attribution detector. These entries describe observable wording rather than latent reasoning states.
Table 9
Group
Hyperparameter
Value
Optimization
Optimizer
AdamW
Learning rate
1×10−6
Hardware
8 × NPU
Training
Maximum prompt length
2,048 tokens
Thinking mode
On
Training step
300
Appendix
Table 5: Hyperparameters and evaluation settings shared by the main methods. Math uses 12 samples per problem for mean@12; KK uses greedy pass@1.
Method
Hyperparameter
Value
Description
GRPO
Objective
Group-relative PPO
Outcome-reward RL
Group size G
8
Rollouts per prompt
Maximum completion length
16,384 tokens
Training rollout cap
Gradient clip
1.0
Maximum gradient norm
OPSD
Objective
Token JSD distillation
Token-level Teacher matching
Teacher
Fixed privileged
PI-conditioned target
Appendix
Table 6: Method-specific training settings. Baseline configurations follow the defaults reported in their original papers. Completion length denotes the training rollout cap; all methods are evaluated under the shared protocol in Table 5 .
Model
Method
AMC23
AIME24
AIME25
Avg.
DeepSeek-R1-Distill-Qwen-7B
Base
80.7
52.0
41.2
58.0
GRPO
83.7
56.6
37.5
59.3
OPSD
82.2
55.2
42.7
60.0
SPIN
83.7
58.3
36.6
59.5
TeachBack
82.1
57.2
43.6
61.0
DeepSeek-R1-Distill-Llama-8B
Base
82.5
53.3
36.6
57.5
Appendix
Table 7: Cross-architecture Math results on the two DeepSeek distilled models.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory. However, this interpretation conflates two effects. The reference solution not only reveals the answer to the current instance but also changes the context under which the teacher provides token-level supervision. We investigate the role of target-specific privilege with OP2SD (On-Policy Self-Distillation from Other Problems), which replaces the paired reference with a problem and solution from a different example, while preserving the student rollout, teacher, and distillation objective. Across three models and three mathematics benchmarks, OP2SD improves over the base model, remains competitive with OPSD. The success of OP2SD implies that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor.
Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar +1
Mohamed bin Zayed University of Artificial Intelligence · Nagoya University · RIKEN AIP
On-policy self-distillation trains reasoning models on their own trajectories using dense token distribution guidance from a privileged teacher with access to a verified solution. This supervision transfers what the teacher predicts without directly transferring where it attends within the preceding context. We introduce On-Policy Attention Self-Distillation (OPASD), which complements token-level supervision with solution-conditioned attention distillation. Because the privileged teacher can attend to verified solution tokens unavailable to the student, OPASD projects teacher attention onto student-visible positions and renormalizes the resulting distribution before alignment. Across three model sizes and four competition-level mathematics benchmarks, OPASD consistently outperforms token-only OPSD, improving average accuracy by 4.98 to 8.40 percentage points. OPASD also avoids the response-length inflation and performance degradation observed with token-only distillation, reducing generated rollout tokens by 73.9% and estimated model compute by 72.6% while training 1.53x faster. These results show that solution-conditioned attention provides a complementary supervision signal that makes on-policy self-distillation more accurate, stable, and compute-efficient.
Safaeid Hossain Arib, Rabeya Akter, Ismam Nur Swapnil +3
ACI PLC, Bangladesh · University of Dhaka, Bangladesh · BRAC University, Bangladesh +2
On-policy self-distillation (OPSD) trains a question-only student with token-level feedback from a teacher given training-only privileged information (PI). OPSD therefore provides dense, on-policy supervision, and is free of a larger external teacher, but its effectiveness rests on how PI is designed and utilized. Our preliminary diagnostics suggest a significant gap between teacher utility and student learnability, where a small fraction of high-disagreement tokens dominate the distillation signal, and short teacher continuations at these positions further expose more explicit PI leakage than transferable correction cues, indicating a strong intent on injecting PI-conditioned shortcuts. We propose Adaptive On-Policy Self-Distillation (AOPSD), which adapts what information the teacher receives and how strongly its feedback influences learning. AOPSD encodes each solution as a reasoning DAG, orders problems by the student's evolving capability, and reveals only the affordable subgraph and its next frontier as PI. For high-disagreement tokens, AOPSD utilizes short teacher continuations as probes to encourage useful guidance while mitigating PI-conditioned shortcuts among teacher supervisions. On HMMT25, AIME24, AIME25, and BRUMo25, AOPSD achieves 72.5% Pass@8, which is 6.7 percentage points above OPSD and 4.2 above the strongest competing baseline while reducing 15 percentage points of training time at lower cost.
Jiacheng Du, Weiwei Xie, Tianyi Du +3
National University of Singapore · Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory