On-Policy Self-Distillation (OPSD) improves reasoning by training a student on its own reasoning, using the same model conditioned on reference solutions as a teacher. Applying OPSD to multilingual reasoning requires the teacher to provide useful feedback across languages, even when the student makes mistakes in intermediate steps. We observe a phenomenon we call Teacher Drift: as these mistakes accumulate, the teacher may follow the student's incorrect steps instead of correcting them. Our analysis shows that this shift tends to occur earlier in lower-resource languages. To account for these differences across languages, we propose Teachable On-Policy Self-Distillation (TOPSD), which determines where and whether teacher supervision should be applied to each reasoning trajectory. TOPSD uses teacher-student likelihood to estimate where drift begins and select a prefix for supervision. It then verifies whether the teacher can recover the correct answer from that prefix, applying distillation only to prefixes that pass verification. Experiments on multilingual mathematical reasoning across ten languages and three model scales show that TOPSD consistently outperforms conventional OPSD, particularly in low- and medium-resource languages.
Figures & tables
Figure 1: Teacher Drift in a Korean reasoning example. The teacher’s next-token predictions are evaluated on prefixes of one student trajectory. At t=36 and 180 , it predicts 2 instead of the student’s 0 , correcting a dodecagon’s side count from 10 to 12 ; at t=314 , it predicts 0 , following the repeated error. English translations are shown in gray.
Figure 2: Teacher Drift in multilingual reasoning. (a) Student accuracy across languages. (b) Cumulative teacher-student likelihood ratio Ci,t indicates earlier alignment in low-resource languages. (c) Teacher recoverability Rec(m) decreases with longer student prefixes. Colors indicate language-resource groups.
Figure 3: Overview of TOPSD. The student generates a reasoning trajectory in the target language, while the privileged teacher additionally observes the English problem and reference solution. Step 1 (Teachable Prefix Selection) uses the cumulative teacher-student likelihood ratio to select a candidate teachable prefix, determining where teacher supervision may be applied. Step 2 (Teacher Recoverability Verification) checks whether the teacher can recover the correct answer from the selected prefix, determining whether supervision should be applied. Only prefixes that pass verification receive distillation on the corresponding states of the original student trajectory. English translations are shown in gray.
Model
Metric
Method
Low
Medium
High
Overall
BN
TE
SW
Avg
JA
KO
TH
Avg
EN
ES
RU
ZH
Avg
Avg
Qwen3-1.7B
Avg@3
Vanilla
2.13
0.00
0.27
0.80
11.47
6.40
6.13
8.00
17.87
15.73
14.93
17.33
16.47
9.23
OPSD
2.93
1.07
1.60
1.87
10.13
6.40
6.93
7.82
21.33
14.67
12.53
18.40
16.73
9.60
COPSD
3.47
1.07
1.07
1.87
9.87
8.53
8.27
8.89
20.27
13.87
13.07
20.00
16.80
9.95
TOPSD
4.53
2.13
1.60
2.76
13.07
10.13
9.87
11.02
20.27
17.07
17.07
20.27
18.67
11.60
Pass@3
Vanilla
4.80
0.00
0.80
1.87
18.40
11.20
10.40
13.33
26.40
28.00
25.60
28.80
27.20
15.44
Table 1: Avg@3 and Pass@3 (%) on PolyMath High across ten languages grouped by resource level. Group and Overall averages are mean performance within each group and across all ten languages, respectively. Bold and underlining indicate the best and second-best scores.
Prefix Selection
Recoverability Filter
Avg@3 (%)
ΔCOPSD
Low
Medium
High
Overall
Full trajectory (COPSD)
None
1.87
8.89
16.80
9.95
–
Teachable prefix
None (all)
2.40
10.53
17.20
10.76
+0.81
Failed only
1.96
9.87
16.67
10.21
+0.26
Successful only (Ours)
2.76
11.02
18.67
11.60
+1.65
Table 2: Ablation of Teachable Prefix Selection and Teacher Recoverability Verification. Avg@3 (%) on PolyMath High with Qwen3-1.7B. ΔCOPSD denotes the overall improvement over COPSD in percentage points.
Figure 4: Qualitative comparison of COPSD and TOPSD on Bengali mathematical reasoning. Starting from similar correct formulations, COPSD becomes stuck in repeated divisibility statements, while TOPSD continues the derivation to b=21 and b=49 and obtains the correct answer 70 . English translations are shown in gray.
Model
Cutoff Detection Rate (%)
Relative Cutoff Position (%)
Low
Medium
High
Overall
Qwen3-1.7B
95.94
4.09
13.65
13.02
10.47
Qwen3-4B
98.60
5.72
9.24
8.00
7.69
Qwen3-8B
99.28
6.36
10.19
5.03
6.98
Table 3: Teachable prefix selection statistics. Cutoff Detection Rate is the percentage of trajectories with a detected cutoff. Relative Cutoff Position reports the cutoff as a percentage of trajectory length, averaged over trajectories with detected cutoffs. Overall is the mean relative cutoff position over all trajectories with detected cutoffs across the ten languages.
Method
AIME24 Avg@3 / Pass@3
AIME25 Avg@3 / Pass@3
Qwen3-4B Vanilla
76.67 / 80.00
64.44 / 76.67
OPSD
76.67 / 86.67
67.78 / 83.33
TOPSD
76.67 / 90.00
71.11 / 86.67
Table 4: Generalization to English mathematical reasoning on AIME 2024 and AIME 2025. Results are reported in percentages. Bold indicates the best result for each metric.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Base models
Qwen3-1.7B, 4B, 8B
Optimizer
AdamW
Training precision
bfloat16
Learning rate
1×10−6
Effective batch size
32
Training steps
160
Appendix
Table 5: Training hyperparameters used for TOPSD. Unless otherwise specified, the same configuration is used across all model scales.
Table 6: Student user messages and language-specific prefills used to initialize multilingual reasoning generation.
Method
Low
Medium
High
Overall
Avg@3 (%)
Fixed Prefix (10%)
2.22
9.42
18.00
10.69
TOPSD
2.76
11.02
18.67
11.60
Pass@3 (%)
Fixed Prefix (10%)
4.80
16.27
27.80
17.44
TOPSD
5.33
18.67
28.80
18.72
Appendix
Table 7: Comparison with fixed-prefix truncation. Avg@3 and Pass@3 accuracy (%) on PolyMath High using Qwen3-1.7B. The fixed-prefix baseline retains the first 10% of each student trajectory, while TOPSD adaptively selects teachable prefixes. Both methods use the same Teacher Recoverability Verification procedure. Bold indicates the best result for each metric.
Figure 5: Qualitative comparison on a high-resource Spanish problem. Both the base model and COPSD omit the cross-term 3a12 and incorrectly use 3a2=19 , producing the answer 22. TOPSD correctly matches the coefficient using 3a12+3a2=19 and obtains the correct answer 352. English translations are shown in gray.
Figure 6: Completion-based illustration of Teacher Drift. The same privileged teacher is greedily continued from three progressively longer prefixes of a single student completion. It recovers the correct answer 78 at t=600 and t=700 , but follows the student’s incorrect reasoning and returns 6 at t=768 . This provides an outcome-level view of how prefix accumulation affects teacher recovery. English translations are shown in gray.
Figure 7: Qualitative comparison on a medium-resource Thai problem. The base model uses an incorrect asymmetric parameter range, while COPSD reports only an incomplete enumeration of five solutions. TOPSD factorizes the equation as (4x−3y)(3x+2y)=0 , counts both parameterized solution families over their complete ranges, and obtains the correct answer 117. English translations are shown in gray.
Figure 8: Qualitative comparison on a low-resource Telugu problem. The base model forms an incorrect determinant-area expression, and COPSD produces repetitive reasoning without a valid area derivation or boxed answer. TOPSD correctly derives the triangle area as 1012(i+j) , computes S(n)=1012n(n+1)2 , and obtains the correct limiting value 1012. English translations are shown in gray.
K
Cutoff Detection Rate (%)
Relative Cutoff Position (%)
Avg@3 (%)
1
96.34
9.54
10.45
2
96.22
9.87
10.94
4
95.94
10.47
11.60
8
95.56
11.55
10.27
Appendix
Table 8: Sensitivity to persistence length K with Qwen3-1.7B. Bold marks the default setting ( K=4 ). Cutoff Detection Rate and Relative Cutoff Position characterize the resulting prefix selection, and Avg@3 (%) is macro-averaged across ten languages on PolyMath High.
On-policy self-distillation (OPSD) has attracted growing interest as a promising approach to improve the reasoning ability of language models. Without external rewards nor a separate stronger teacher, the self-teacher with privileged information could provide dense signals on student's trajectories. However, its behavior in language reasoning remains unclear, with reported outcomes ranging from modest gains to behavioral collapse. In this work, we diagnose OPSD for mathematical reasoning across models spanning 0.6B--8B parameters. We conduct controlled experiments and token-level analyses to fully delve into OPSD. We point out that teacher's signal is shaped by reasoning-mode alignment and the complete teacher prefix, rather than by privileged semantics alone. OPSD improves reasoning only in narrow compatibility regimes. Otherwise, it produces ineffective length growth, stable degradation, or behavioral collapse. Token-level analysis shows that teacher's signal is not stable and does not predict downstream performance. Based on these results, we argue that OPSD is a sensitive algorithm rather than a generally reliable reasoning-improvement post-training method.
Multilingual reasoning transfer is crucial for extending reasoning capabilities of large language models (LLMs) beyond high-resource languages. On-policy self-distillation (OPSD) and its variants have emerged as a promising paradigm, providing dense token-level supervision on student-generated rollouts, yet their objectives do not explicitly prioritize reasoning signals most critical to cross-lingual transfer. We characterize that target-language reasoning comprises the generation of both surface text and reasoning pivots, which are decisions that advance or redirect the reasoning process and shape subsequent inference. This motivates concentrating privileged distillation around such pivots. We therefore propose RP-OPSD, Reasoning-Pivot-guided On-Policy Self-Distillation, using the distributional shift between matched teacher views with and without an English reference solution as an operational proxy to guide privileged distillation and reference anchoring. Experiments on mathematical reasoning benchmarks covering 17 languages and multiple difficulty levels show that our method outperforms strong multilingual reasoning baselines and OPSD variants. Further analysis reveals that RP-OPSD concentrates privileged distillation on reasoning-control and problem-condistioned state-update tokens, while downweighting it for tokens that mainly support surface realization. Our code is available at https://github.com/NJUNLP/RP-OPSD.
Xinye Wang, Junxiao Liu, Shujian Huang
National Key Laboratory for Novel Software Technology, Nanjing University
On-policy self-distillation (OPSD) enables a language model to learn from a Teacher instantiated from the same model but given access to privileged information, such as a reference solution. We find that access to sufficient privileged information allows the Teacher to generate highly accurate trajectories; however, accuracy alone does not guarantee that these trajectories are suitable for Student learning. Direct access to a complete solution may encourage shortened, reference-conditioned derivations that omit the exploration, verification, and self-correction required to solve problems without privileged information. We introduce TeachBack, a bidirectional preference-learning framework for on-policy self-distillation that learns not only from the Teacher but also how the Teacher should teach. TeachBack shapes the Teacher's generation behavior by controlling the length of the reference-reasoning suffix injected into the prompt and by varying the prompt style. While maintaining comparable Teacher accuracy, it encourages the Teacher to explore and re-solve problems, thereby reducing its reliance on the reference solution. At each rollout refresh, the shared policy generates both a privileged Teacher trajectory and a problem-only Student trajectory. A pairwise preference objective uses privileged Teacher trajectories to improve Student learning, while the reversed preference uses the Student's behavior to shape future Teacher supervision under the privileged condition. Experiments on mathematical and logical reasoning tasks show that TeachBack consistently improves Student performance over existing OPSD methods and related baselines.
Linfei Ji, Jiayun Li, Haozhao Wang +5
Huazhong University of Science and Technology · Tsinghua University · Harbin Institute of Technology +1