On-policy self-distillation (OPSD) enables a language model to learn from a Teacher instantiated from the same model but given access to privileged information, such as a reference solution. We find that access to sufficient privileged information allows the Teacher to generate highly accurate trajectories; however, accuracy alone does not guarantee that these trajectories are suitable for Student learning. Direct access to a complete solution may encourage shortened, reference-conditioned derivations that omit the exploration, verification, and self-correction required to solve problems without privileged information. We introduce TeachBack, a bidirectional preference-learning framework for on-policy self-distillation that learns not only from the Teacher but also how the Teacher should teach. TeachBack shapes the Teacher's generation behavior by controlling the length of the reference-reasoning suffix injected into the prompt and by varying the prompt style. While maintaining comparable Teacher accuracy, it encourages the Teacher to explore and re-solve problems, thereby reducing its reliance on the reference solution. At each rollout refresh, the shared policy generates both a privileged Teacher trajectory and a problem-only Student trajectory. A pairwise preference objective uses privileged Teacher trajectories to improve Student learning, while the reversed preference uses the Student's behavior to shape future Teacher supervision under the privileged condition. Experiments on mathematical and logical reasoning tasks show that TeachBack consistently improves Student performance over existing OPSD methods and related baselines.
Figures & tables
Figure 1: Motivation for TeachBack . In standard OPSD, access to privileged information can lead the Teacher to produce a short solution that omits exploration and verification. TeachBack uses partial reference reasoning and Student feedback to encourage complete, self-contained Teacher trajectories that remain useful to a problem-only Student.
Figure 2: Overview of TeachBack . (a) The Student receives the problem alone, while a reference-reasoning suffix is framed as optional working notes for the Teacher to re-solve the problem. (b) The shared policy samples paired Student and Teacher trajectories; both responses are replayed under both conditions and scored against a fixed reference policy. (c) Under the problem-only condition, the Teacher-origin response is preferred; under the PI-conditioned condition, the Student-origin response is preferred. The weighted losses update the shared policy before the next on-policy pair is generated. At test time, only the problem-only Student condition is used.
Mathematical Reasoning (mean@12)
Knight-and-Knaves (pass@1)
Model
Method
AMC23
AIME24
AIME25
Avg.
4–8 Roles
9 Roles
10 Roles
11 Roles
Avg.
Qwen3-1.7B
Base
75.7
46.3
37.5
53.2
63.8
56.0
44.0
34.0
49.4
GRPO
75.0
46.6
37.5
53.0
67.4
54.0
53.0
33.0
51.8
OPSD
74.5
47.7
35.0
52.4
64.4
55.0
52.0
32.0
50.9
SPIN
75.9
45.0
36.7
52.5
67.8
58.0
51.0
42.0
54.7
PBSD
72.8
48.3
37.5
52.9
67.0
63.0
44.0
32.0
51.5
Table 1: Main results across model scales. Mathematical reasoning is reported as mean@12. Knight-and-Knaves (KK) is reported as pass@1, with 4–8 Roles as the in-domain (ID) setting and 9–11 Roles as out-of-domain (OOD) settings. Bold and underlined entries denote the best and second-best results within each model block; parenthetical values are TeachBack gains over Base.
Figure 3: Prompt-construction diagnostics. (a) Five Teacher prompt styles are compared with the Student. The gray shaded region represents Student behavior, and the green shaded region represents Notes Teacher behavior. Reference, Notes, and History share a 50% privileged body; Answer-only and Instruction use different information budgets. (b) The Working Notes suffix sweep varies donor-reasoning injection from 0% to 100% and plots accuracy, rollout length, exploration, and reference degree. 0% suffix represents the Student with no injected reasoning. The shaded 25–50% range yields high accuracy while retaining more exploration and lower reference degree than larger suffix ratios.
Figure 4: Diagnostics from the Qwen3-1.7B run at λ=0.75 . Panels (a)–(d) show the branch and joint losses, four reference-relative scores Aθc(y) , conditional entropy for the corresponding c⊕y pairs, and diagnostic KL, respectively. Here c is the scoring condition and y the sampled response.
Figure 5: Training dynamics under different Teacher-side weights. In (a)–(b), colors denote λ , solid and dashed curves denote Student and Teacher rollouts. The panels show non-clipped accuracy, response length, policy entropy, and Math mean@12, respectively.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Condition
PI ratio
Accuracy
Length
Trunc.
Explore/1K
Breadth
Content ref.
Ref. degree
Explicit
Strong attr.
(%)
(%)
(tokens)
(%)
(/1K tokens)
(%)
(%)
(%)
(%)
(%)
Student
0
79.88
7459.2
7.42
17.19
98.46
—
—
5.66
0.39
Working notes
25
87.30
4209.9
2.15
11.64
91.58
16.09
17.23
21.78
13.96
Working notes
50
88.28
3605.7
1.95
9.93
87.21
21.42
19.52
11.91
6.54
Working notes
75
86.72
3308.0
0.98
9.30
83.54
24.91
21.45
7.62
4.88
Working notes
100
87.70
2718.7
0.20
7.86
69.18
32.04
26.14
2.54
1.27
Appendix
Table 2: Prompt-construction audit results. The privileged-information column reports the percentage of the full ground-truth reasoning body injected into the prompt; metric definitions and the complete rollout protocol are given in this subsection.
Main dimension
Subcategory
Matched words, phrases, or pattern families
Reasoning exploration
Exploration
another approach ; alternative ; another way ; different approach ; different way ; let’s try ; alternatively
Uncertainty
maybe ; perhaps ; possibly ; might ; could ; may ; probably ; likely ; unlikely ; it seems ; seems like ; I think ; not sure
Verification
check ; verify ; confirm ; double-check ; let’s check ; make sure ; ensure ; test ; examine
Backtracking
wait ; hold on ; actually ; I made a mistake ; that’s not right ; go back ; correction
Replanning
first, let’s ; we need to consider ; let’s analyze ; step by step ; approach ; consider ; instead
Reference dependence
Explicit mention
reference ; reference solution ; reference answer ; working notes ; notes say/state/indicate ; according to ; based on (the) notes ; given/provided answer, solution, or reasoning
Appendix
Table 3: Unified lexical inventory for the two text-based diagnostic dimensions. Reasoning exploration is divided into five behavioral categories. Reference dependence uses a broad explicit-mention detector and a stricter strong-attribution detector. These entries describe observable wording rather than latent reasoning states.
Table 9
Group
Hyperparameter
Value
Optimization
Optimizer
AdamW
Learning rate
1×10−6
Hardware
8 × NPU
Training
Maximum prompt length
2,048 tokens
Thinking mode
On
Training step
300
Appendix
Table 5: Hyperparameters and evaluation settings shared by the main methods. Math uses 12 samples per problem for mean@12; KK uses greedy pass@1.
Method
Hyperparameter
Value
Description
GRPO
Objective
Group-relative PPO
Outcome-reward RL
Group size G
8
Rollouts per prompt
Maximum completion length
16,384 tokens
Training rollout cap
Gradient clip
1.0
Maximum gradient norm
OPSD
Objective
Token JSD distillation
Token-level Teacher matching
Teacher
Fixed privileged
PI-conditioned target
Appendix
Table 6: Method-specific training settings. Baseline configurations follow the defaults reported in their original papers. Completion length denotes the training rollout cap; all methods are evaluated under the shared protocol in Table 5 .
Model
Method
AMC23
AIME24
AIME25
Avg.
DeepSeek-R1-Distill-Qwen-7B
Base
80.7
52.0
41.2
58.0
GRPO
83.7
56.6
37.5
59.3
OPSD
82.2
55.2
42.7
60.0
SPIN
83.7
58.3
36.6
59.5
TeachBack
82.1
57.2
43.6
61.0
DeepSeek-R1-Distill-Llama-8B
Base
82.5
53.3
36.6
57.5
Appendix
Table 7: Cross-architecture Math results on the two DeepSeek distilled models.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.