Knowing When to Teach: Learning from Teachable Prefixes in Multilingual Reasoning Distillation

Jaewon Cho; Minseo Yoon; Jinyoung Park; Jinwoo Seo; Jaewon Chu; Alice Oh; Hyunwoo J. Kim

cs.CLNov 5, 2026

Knowing When to Teach: Learning from Teachable Prefixes in Multilingual Reasoning Distillation

Authors: Jaewon Cho, Minseo Yoon, Jinyoung Park, Jinwoo Seo, Jaewon Chu, Alice Oh, Hyunwoo J. Kim

Organizations: KAIST · Korea University

Abstract

On-Policy Self-Distillation (OPSD) improves reasoning by training a student on its own reasoning, using the same model conditioned on reference solutions as a teacher. Applying OPSD to multilingual reasoning requires the teacher to provide useful feedback across languages, even when the student makes mistakes in intermediate steps. We observe a phenomenon we call Teacher Drift: as these mistakes accumulate, the teacher may follow the student's incorrect steps instead of correcting them. Our analysis shows that this shift tends to occur earlier in lower-resource languages. To account for these differences across languages, we propose Teachable On-Policy Self-Distillation (TOPSD), which determines where and whether teacher supervision should be applied to each reasoning trajectory. TOPSD uses teacher-student likelihood to estimate where drift begins and select a prefix for supervision. It then verifies whether the teacher can recover the correct answer from that prefix, applying distillation only to prefixes that pass verification. Experiments on multilingual mathematical reasoning across ten languages and three model scales show that TOPSD consistently outperforms conventional OPSD, particularly in low- and medium-resource languages.

Figures & tables

Figure 1: Teacher Drift in a Korean reasoning example. The teacher’s next-token predictions are evaluated on prefixes of one student trajectory. At $t=36$ and $180$ , it predicts $2$ instead of the student’s $0$ , correcting a dodecagon’s side count from $10$ to $12$ ; at $t=314$ , it predicts $0$ , following the repeated error. English translations are shown in gray.

Figure 2: Teacher Drift in multilingual reasoning. (a) Student accuracy across languages. (b) Cumulative teacher-student likelihood ratio $C_{i,t}$ indicates earlier alignment in low-resource languages. (c) Teacher recoverability $\mathrm{Rec}(m)$ decreases with longer student prefixes. Colors indicate language-resource groups.

Figure 3: Overview of TOPSD. The student generates a reasoning trajectory in the target language, while the privileged teacher additionally observes the English problem and reference solution. Step 1 (Teachable Prefix Selection) uses the cumulative teacher-student likelihood ratio to select a candidate teachable prefix, determining where teacher supervision may be applied. Step 2 (Teacher Recoverability Verification) checks whether the teacher can recover the correct answer from the selected prefix, determining whether supervision should be applied. Only prefixes that pass verification receive distillation on the corresponding states of the original student trajectory. English translations are shown in gray.

Model	Metric	Method	Low				Medium				High					Overall
			BN	TE	SW	Avg	JA	KO	TH	Avg	EN	ES	RU	ZH	Avg	Avg
Qwen3-1.7B	Avg@3	Vanilla	2.13	0.00	0.27	0.80	11.47	6.40	6.13	8.00	17.87	15.73	14.93	17.33	16.47	9.23
		OPSD	2.93	1.07	1.60	1.87	10.13	6.40	6.93	7.82	21.33	14.67	12.53	18.40	16.73	9.60
		COPSD	3.47	1.07	1.07	1.87	9.87	8.53	8.27	8.89	20.27	13.87	13.07	20.00	16.80	9.95
		TOPSD	4.53	2.13	1.60	2.76	13.07	10.13	9.87	11.02	20.27	17.07	17.07	20.27	18.67	11.60
	Pass@3	Vanilla	4.80	0.00	0.80	1.87	18.40	11.20	10.40	13.33	26.40	28.00	25.60	28.80	27.20	15.44

Table 1: Avg@3 and Pass@3 (%) on PolyMath High across ten languages grouped by resource level. Group and Overall averages are mean performance within each group and across all ten languages, respectively. Bold and underlining indicate the best and second-best scores.

Prefix Selection	Recoverability Filter	Avg@3 (%)				$\Delta_{\mathrm{COPSD}}$
Prefix Selection	Recoverability Filter	Low	Medium	High	Overall	$\Delta_{\mathrm{COPSD}}$
Full trajectory (COPSD)	None	1.87	8.89	16.80	9.95	–
Teachable prefix	None (all)	2.40	10.53	17.20	10.76	+0.81
	Failed only	1.96	9.87	16.67	10.21	+0.26
	Successful only (Ours)	2.76	11.02	18.67	11.60	+1.65

Table 2: Ablation of Teachable Prefix Selection and Teacher Recoverability Verification. Avg@3 (%) on PolyMath High with Qwen3-1.7B.

\Delta_{\mathrm{COPSD}}

denotes the overall improvement over COPSD in percentage points.

Figure 4: Qualitative comparison of COPSD and TOPSD on Bengali mathematical reasoning. Starting from similar correct formulations, COPSD becomes stuck in repeated divisibility statements, while TOPSD continues the derivation to

b=21

and

b=49

and obtains the correct answer

70

. English translations are shown in gray.

Model	Cutoff Detection Rate (%)	Relative Cutoff Position (%)
Model	Cutoff Detection Rate (%)	Low	Medium	High	Overall
Qwen3-1.7B	95.94	4.09	13.65	13.02	10.47
Qwen3-4B	98.60	5.72	9.24	8.00	7.69
Qwen3-8B	99.28	6.36	10.19	5.03	6.98

Table 3: Teachable prefix selection statistics. Cutoff Detection Rate is the percentage of trajectories with a detected cutoff. Relative Cutoff Position reports the cutoff as a percentage of trajectory length, averaged over trajectories with detected cutoffs. Overall is the mean relative cutoff position over all trajectories with detected cutoffs across the ten languages.

Method	AIME24 Avg@3 / Pass@3	AIME25 Avg@3 / Pass@3
Qwen3-4B Vanilla	76.67 / 80.00	64.44 / 76.67
OPSD	76.67 / 86.67	67.78 / 83.33
TOPSD	76.67 / 90.00	71.11 / 86.67

Table 4: Generalization to English mathematical reasoning on AIME 2024 and AIME 2025. Results are reported in percentages. Bold indicates the best result for each metric.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Base models	Qwen3-1.7B, 4B, 8B
Optimizer	AdamW
Training precision	bfloat16
Learning rate	$1\times 10^{-6}$
Effective batch size	32
Training steps	160

Appendix

Table 5: Training hyperparameters used for TOPSD. Unless otherwise specified, the same configuration is used across all model scales.

Table 6: Student user messages and language-specific prefills used to initialize multilingual reasoning generation.

Avg@3 (%)
Method	Low	Medium	High	Overall
Fixed Prefix (10%)	2.22	9.42	18.00	10.69
TOPSD	2.76	11.02	18.67	11.60
Pass@3 (%)
Fixed Prefix (10%)	4.80	16.27	27.80	17.44
TOPSD	5.33	18.67	28.80	18.72

Appendix

Table 7: Comparison with fixed-prefix truncation. Avg@3 and Pass@3 accuracy (%) on PolyMath High using Qwen3-1.7B. The fixed-prefix baseline retains the first 10% of each student trajectory, while TOPSD adaptively selects teachable prefixes. Both methods use the same Teacher Recoverability Verification procedure. Bold indicates the best result for each metric.

Figure 5: Qualitative comparison on a high-resource Spanish problem. Both the base model and COPSD omit the cross-term

3a_{1}^{2}

and incorrectly use

3a_{2}=19

, producing the answer 22. TOPSD correctly matches the coefficient using

3a_{1}^{2}+3a_{2}=19

and obtains the correct answer 352. English translations are shown in gray.

Figure 6: Completion-based illustration of Teacher Drift. The same privileged teacher is greedily continued from three progressively longer prefixes of a single student completion. It recovers the correct answer 78 at

t=600

and

t=700

, but follows the student’s incorrect reasoning and returns 6 at

t=768

. This provides an outcome-level view of how prefix accumulation affects teacher recovery. English translations are shown in gray.

Figure 7: Qualitative comparison on a medium-resource Thai problem. The base model uses an incorrect asymmetric parameter range, while COPSD reports only an incomplete enumeration of five solutions. TOPSD factorizes the equation as

(4x-3y)(3x+2y)=0

, counts both parameterized solution families over their complete ranges, and obtains the correct answer 117. English translations are shown in gray.

Figure 8: Qualitative comparison on a low-resource Telugu problem. The base model forms an incorrect determinant-area expression, and COPSD produces repetitive reasoning without a valid area derivation or boxed answer. TOPSD correctly derives the triangle area as

1012(i+j)

, computes

S(n)=1012n(n+1)^{2}

, and obtains the correct limiting value 1012. English translations are shown in gray.

$K$	Cutoff Detection Rate (%)	Relative Cutoff Position (%)	Avg@3 (%)
1	96.34	9.54	10.45
2	96.22	9.87	10.94
4	95.94	10.47	11.60
8	95.56	11.55	10.27

Appendix

Table 8: Sensitivity to persistence length

K

with Qwen3-1.7B. Bold marks the default setting (

K=4

). Cutoff Detection Rate and Relative Cutoff Position characterize the resulting prefix selection, and Avg@3 (%) is macro-averaged across ten languages on PolyMath High.

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 947 papers

Diagnosing On-Policy Self-Distillation for Reasoning Language Models
Sep 30, 2026Yang Li, Gongle Xue, Yuheng Yuan +4Language Model Distillation On-Policy Distillation
Peking University
RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer
Aug 6, 2026Xinye Wang, Junxiao Liu, Shujian HuangNumerical Reasoning in Language Models Multilingual Language Models
National Key Laboratory for Novel Software Technology, Nanjing University
TeachBack: Learning to Teach via Bidirectional Preference Optimization for On-Policy Self-Distillation
Oct 21, 2026Linfei Ji, Jiayun Li, Haozhao Wang +5Language Model Distillation On-Policy Self-Distillation
Huazhong University of Science and Technology · Tsinghua University · Harbin Institute of Technology +1

Sep 30, 2026J/K move · Enter open · S save

Diagnosing On-Policy Self-Distillation for Reasoning Language Models

Yang Li, Gongle Xue, Yuheng Yuan, Yijia Guo+3

Peking University

Language Model Distillation On-Policy Distillation LLM Reasoning

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) has attracted growing interest as a promising approach to improve the reasoning ability of language models. Without external rewards nor a separate stronger teacher, the self-teacher with privileged information could provide dense signals on student's trajectories. However, its behavior in language reasoning remains unclear, with reported outcomes ranging from modest gains to behavioral collapse. In this work, we diagnose OPSD for mathematical reasoning across models spanning 0.6B--8B parameters. We conduct controlled experiments and token-level analyses to fully delve into OPSD. We point out that teacher's signal is shaped by reasoning-mode alignment and the complete teacher prefix, rather than by privileged semantics alone. OPSD improves reasoning only in narrow compatibility regimes. Otherwise, it produces ineffective length growth, stable degradation, or behavioral collapse. Token-level analysis shows that teacher's signal is not stable and does not predict downstream performance. Based on these results, we argue that OPSD is a sensitive algorithm rather than a generally reliable reasoning-improvement post-training method.

Abstract

LLM Reasoning

Explore similar work

Diagnosing On-Policy Self-Distillation for Reasoning Language Models

RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer

TeachBack: Learning to Teach via Bidirectional Preference Optimization for On-Policy Self-Distillation