Bridged Self-Distillation for On-Policy Learning

Runsheng Chen; Yihong Liu; Raoyuan Zhao; Michael A. Hedderich; Hinrich Schütze; Matthias Grabmair

cs.CLOct 19, 2026

Bridged Self-Distillation for On-Policy Learning

Authors: Runsheng Chen, Yihong Liu, Raoyuan Zhao, Michael A. Hedderich, Hinrich Schütze, Matthias Grabmair

Organizations: Technical University of Munich, Germany · LMU Munich, Germany · Munich Center for Machine Learning, Germany

Abstract

On-policy distillation can transfer the reasoning capabilities of a strong teacher to a smaller student. However, distillation from a stronger teacher model faces two mismatches: its solutions may be difficult for a smaller student to imitate, and its token-level probabilities may be incompatible with, or even unavailable to, the student. Existing methods handle these settings through different forms of supervision. To this end, we propose Bridged Self-Distillation (BSD), a unified framework for distilling from any teacher that provides reasoning traces. The model to be distilled from, termed the external teacher, first provides a reasoning trace as privileged context, which can be generated in an offline fashion. The student generates rollouts without this context, while a copy of the student conditioned on it, serving as the self-teacher, scores the same rollouts in the student's own token space. BSD uses the resulting token-level guidance to adjust updates on unsuccessful rollouts, with a verifier providing the outcome signal. As the external teacher is queried only to produce text, the same training procedure applies to all teacher scenarios: shared-tokenizer, cross-tokenizer, and closed API teachers. Across these scenarios, BSD improves over the base model and outperforms baselines tailored to each scenario, offering a novel and unified pathway for distillation. Code is available: https://github.com/TUMLegalTech/Bridged-Self-Distillation

Figures & tables

Figure 1: Comparison between scenario-specific OPD and BSD. Top : existing methods derive a different training signal for each kind of teacher access, namely token-level distribution matching with a shared tokenizer, vocabulary mapping across tokenizers, and response-based feedback for a closed API. Bottom : BSD uses a unified framework for all three scenarios: the external teacher provides a reasoning trace

z

, and a self-teacher conditioned on

z

scores the student’s own on-policy rollouts.

Figure 2: Overview of Bridged Self-Distillation (BSD). An external teacher, whatever its access setting, provides a reasoning trace $z$ for each question. The student generates rollouts without $z$ , and the self-teacher (a frozen copy of the student conditioned on $z$ ) scores the same tokens. The token-level guidance $\delta_{t}$ reweights the penalty on unsuccessful rollouts, while the verifier determines the direction of every update.

Qwen3-4B
	Shared tokenizer ( Qwen3-32B )				Cross-tokenizer ( Nemotron-3-30B-A3B )				Closed API ( GPT-5.6 Luna )
Method	AIME24	AIME25	HMMT25	Avg	AIME24	AIME25	HMMT25	Avg	AIME24	AIME25	HMMT25	Avg
Base	72.5	62.5	42.5	59.2	72.5	62.5	42.5	59.2	72.5	62.5	42.5	59.2
+ SFT	72.5	65.0	44.2	60.6	76.7	65.0	49.2	63.6	68.3	62.5	40.8	57.2
+ OPD	73.3	67.5	40.0	60.3	70.8	62.5	45.8	59.7	70.8	64.2	44.2	59.7
+ BSD	79.2	68.3	44.2	63.9	76.7	66.7	47.5	63.6	75.8	65.0	45.0	61.9

Table 1: Avg@4 on three mathematical reasoning benchmarks; Avg is the mean across benchmarks. OPD denotes vanilla OPD, xOPD, and ROPD for shared-tokenizer, cross-tokenizer, and closed API teachers, respectively. Bold marks the best score among distillation methods within each teacher–student setting. BSD consistently improves over the base model in all twelve teacher–student pairs and achieves the best Avg in eleven pairs.

Figure 3: Training progression. (a)–(c) Mean response length on the development set at each checkpoint for four students under the shared-tokenizer, cross-tokenizer, and closed API teachers, respectively. OPD refers to vanilla OPD, xOPD, and ROPD in the three respective settings. (d) Avg@4 across the three benchmarks at each checkpoint for Qwen3-4B-Instruct distilled from GPT-5.6 Luna . SFT can substantially shorten its responses and cause performance to collapse with the closed API teacher, whereas BSD and OPD generally maintain stable response lengths and show stable or improving performance throughout training.

Panel of Figure 4 — Figure 4: Ablations with the Qwen3-4B student and Qwen3-32B external teacher. (a) Training accuracy of GRPO, answer-conditioned RLSD, and full-trace BSD. (b) Training accuracy of BSD with a hint, guide, or full trace, including a full-trace variant that also weights successful rollouts. (c) Avg@4 across the three benchmarks for these variants and OPSD; the base model scores 59.2. Despite similar training-accuracy curves, richer privileged context generally yields better results, with the full trace performing best. Appendix D reports the per-benchmark scores.

Student	$\|\bar{\delta}\|$	$\rho_{H}$	Gain
Qwen3-0.6B	0.072	0.87	$+2.2$
Qwen3-1.7B	0.115	0.90	$+3.5$
Qwen3-4B	0.114	0.90	$+3.9$
Qwen3-4B-Instruct	0.037	0.83	$+1.5$

Table 2: Token-level guidance before training, measured on unsuccessful rollouts.

|\bar{\delta}|

is the absolute value of the mean of

\delta_{t}

over all tokens,

\rho_{H}

is the Spearman correlation between

|\delta_{t}|

and the student’s token entropy, and Gain is the mean improvement of BSD over the base model shown in Table 1 .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Qwen3-32B	Nemotron-3-30B-A3B	GPT-5.6 Luna
Access	Local weights	Local weights	API
Thinking mode	Enabled	Enabled	Reasoning effort medium
Temperature	0.6	1.0	Provider default
Top-p	0.95	1.0	Provider default
Max new tokens	8,192	8,192	8,192
Traces per problem	1	1	1

Appendix

Table 3: Generation settings of the teacher traces. The open teachers follow their recommended sampling settings.

Parameter	BSD	OPD	xOPD	ROPD	SFT
Learning rate	$1\times 10^{-6}$	$1\times 10^{-6}$	$1\times 10^{-6}$	$1\times 10^{-6}$	$5\times 10^{-6}$
Training updates	150	150	150	150	150
LoRA rank ( $r$ )	32	32	32	32	32
LoRA alpha ( $\alpha$ )	64	64	64	64	64
Checkpoint interval (updates)	25	25	25	25	25
Max prompt length (student)	2,048	2,048	2,048	2,048	–

Appendix

Table 4: Training configuration of BSD and the baselines. The rollout setting of OPD follows its original implementation, and the maximum response length of ROPD follows its released code.

Parameter	Value
Weight coefficient $\lambda$	$0.5$ , decayed linearly to $0$ over training
Clipping range $\epsilon$ of the weight	$0.2$
Self-teacher synchronization	Every 10 updates
Policy clipping thresholds	$0.2$ (lower), $0.28$ (upper)
Loss aggregation	Token mean
Importance sampling	Token-level, truncated

Appendix

Table 5: Settings specific to BSD.

Teacher	Method	GPU-hours
Qwen3-32B	SFT	0.8
	OPD	29.4
	BSD	31.4
Nemotron-3-30B-A3B	xOPD	70.8
Nemotron-3-30B-A3B	BSD	48.7
GPT-5.6 Luna	ROPD	83.0

Appendix

Table 6: GPU-hours of training the Qwen3-4B student for 150 updates with each method. The value of SFT is measured with the Qwen3-32B traces.

Parameter	Value
Samples per problem	4
Seeds	$\{0,1,2,3\}$
Thinking mode	Enabled
Temperature	1.0
Top-p	1.0
Max new tokens	32,768

Appendix

Table 7: Evaluation parameters for the students on the three benchmarks.

Teacher	AIME24	AIME25	HMMT25	Avg
Qwen3-32B	80.0	75.0	52.5	69.2
Nemotron-3-30B-A3B	80.0	70.0	55.8	68.6
GPT-5.6 Luna	88.3	87.5	85.8	87.2

Appendix

Table 8: Avg@4 of the external teachers on the evaluation benchmarks and their mean (Avg).

Model	AIME24	AIME25	HMMT25	Avg
Released	60.8	49.2	29.2	46.4
Fine-tuned	56.7	49.2	27.5	44.4

Appendix

Table 9: Avg@4 of Qwen3-4B-Instruct-2507 before and after the fine-tuning that enables its reasoning block, and their mean (Avg).

Setting	GRPO	RLSD	OPSD
Privileged context	None	Answer	Full trace
Self-teacher	None	Synced every 10 updates	Frozen base model
Verifier reward	Yes	Yes	No
Token-level signal	None	Weight, $\lambda$ : $0.5\to 0$ , $\epsilon=0.2$	Log-ratio, clipped at $\pm 2$
Rollouts with the signal	–	Unsuccessful	All

Appendix

Table 10: Settings in which GRPO, RLSD, and OPSD differ in the ablation. All other settings follow Table 4 .

Figure 6: Mean response length on the three benchmarks at every saved checkpoint for the variants of Figure 4 , with the Qwen3-4B student and the Qwen3-32B external teacher. The dotted line marks the base model.

Method	AIME24	AIME25	HMMT25	Avg	Step	$\|\bar{\delta}\|$
Base	72.5	62.5	42.5	59.2	–	–
OPSD	70.8	66.7	41.7	59.7	50	–
GRPO	72.5	64.2	45.0	60.6	50	–
RLSD	73.3	69.2	42.5	61.7	125	$0.055$
BSD (w/ succ.)	75.8	65.0	43.3	61.4	125	$0.127$
BSD-hint	71.7	68.3	45.0	61.7	150	$0.062$

Appendix

Table 11: Scores of the ablation in Figure 4 , with the Qwen3-4B student and the Qwen3-32B training set. Protocol as in Table 1 ; Step is the training step of the selected checkpoint, and

|\bar{\delta}|

is the magnitude of the mean token-level guidance of each form of context, measured before training on the same unsuccessful rollouts (Section 6.2 ).

Figure 7: Token-level guidance

\delta_{t}

on one passage of an unsuccessful rollout, with the self-teacher conditioned on the Qwen3-32B trace (top) or on the answer alone (bottom). Red marks tokens to which the self-teacher assigns a lower probability than the student and green a higher one; the color saturates at

|\delta_{t}|=2

$n$ -gram	Base, incorrect	BSD, correct	$t$
but this is	0.84	0.38	$-4.3$
this is not	0.45	0.18	$-4.2$
not sure	0.58	0.29	$-3.4$
try to	1.21	0.66	$-3.4$
approach	0.80	0.58	$-3.3$
which matches	0.12	0.21	$+4.3$

Appendix

Table 12: Rates per 1,000 words of selected

n

-grams in incorrect responses of the base Qwen3-4B and correct responses of the student trained with BSD, on the 26 problems that have both.

t

is the paired

t

-statistic over the problems.

	Qwen3-32B			Nemotron-3-30B-A3B			GPT-5.6 Luna
Student	SFT	OPD	BSD	SFT	OPD	BSD	SFT	OPD	BSD
Qwen3-4B	25	75	125	25	125	25	25	75	50
Qwen3-4B-Instruct	25	150	150	50	75	100	25	25	150
Qwen3-1.7B	150	125	125	25	150	50	25	150	25
Qwen3-0.6B	25	50	25	25	150	50	25	100	50

Appendix

Table 13: Training step of the checkpoint selected on the development set for each run of Table 1 . Checkpoints are saved every 25 steps. As in Table 1 , the OPD column is vanilla OPD for Qwen3-32B , xOPD for Nemotron-3-30B-A3B , and ROPD for GPT-5.6 Luna .

Qwen3-4B
	Shared tokenizer ( Qwen3-32B )				Cross-tokenizer ( Nemotron-3-30B-A3B )				Closed API ( GPT-5.6 Luna )
Method	AIME24	AIME25	HMMT25	Avg	AIME24	AIME25	HMMT25	Avg	AIME24	AIME25	HMMT25	Avg
Base	86.7	80.0	56.7	74.4	86.7	80.0	56.7	74.4	86.7	80.0	56.7	74.4
+ SFT	80.0	76.7	56.7	71.1	83.3	80.0	63.3	75.6	83.3	80.0	50.0	71.1
+ OPD	83.3	80.0	50.0	71.1	86.7	80.0	60.0	75.6	83.3	76.7	56.7	72.2
+ BSD	80.0	80.0	60.0	73.3	86.7	83.3	56.7	75.6	83.3	76.7	53.3	71.1

Appendix

Table 14: Pass@4, the share of problems solved by at least one of the four samples, on each benchmark and averaged over the three (Avg). Checkpoints, protocol, and layout as in Table 1 , including the OPD variant of each teacher; bold marks the best score among the three methods for each teacher–student pair.

Figure 8: Avg@4 across the three benchmarks at every saved checkpoint for three further settings. Lines are linear fits over the checkpoints, and the legends give their slopes in percentage points per 100 training steps.

Teacher	Method	MMLU-Pro	LEXam
–	Base	$63.5\pm 1.9$	$45.9\pm 4.9$
Qwen3-32B	SFT	$63.8\pm 1.3$	$43.8\pm 0.8$
	OPD	$63.7\pm 0.9$	$44.9\pm 0.5$
	BSD	$64.5\pm 0.8$	$43.9\pm 0.9$
Nemotron-3-30B-A3B	SFT	$62.1\pm 1.5$	$43.8\pm 1.3$
	xOPD	$64.0\pm 0.5$	$44.8\pm 0.9$

Appendix

Table 15: Mean and standard deviation of Avg@4 over the six saved checkpoints of each run of the Qwen3-4B student on the out-of-domain questions of Figure 9 . For the base model, the standard deviation is over its four sampling seeds.

Figure 9: Avg@4 of the Qwen3-4B student on out-of-domain questions at every saved checkpoint. Rows are the MMLU-Pro and LEXam questions, and columns are the external teachers. OPD is vanilla OPD, xOPD, and ROPD for the three teachers, respectively. SFT falls below the base model over training with Nemotron-3-30B-A3B and GPT-5.6 Luna , whereas BSD and OPD stay at its level.

Figure 10: Mean response length of the Qwen3-4B student on the out-of-domain questions at every saved checkpoint, in the layout of Figure 9 . SFT shortens the responses with GPT-5.6 Luna and lengthens them on the MMLU-Pro questions with Nemotron-3-30B-A3B , whereas BSD and OPD keep the length of the base model.

Reasoning Distillation

Reasoning distillation transfers step-by-step problem-solving skills from powerful teachers to smaller, more efficient models.

See all 22 papers

Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
May 11, 2026Mohammadreza Armandpour, Fatih Ilhan, David Harrison +6On-Policy Self-Distillation Language Model Distillation
Apple
Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization
May 6, 2026Xin Yu, Liuchen Liao, Yiwen Zhang +3On-Policy Self-Distillation Language Model Distillation
1The Pennsylvania State University · 2TikTok
On-Policy Distillation with Best-of-N Teacher Rollout Selection
May 10, 2026Ke Zhang, Yunjie Tian, Dongdi Zhao +4CoT Distillation Best-of-N Sampling
Johns Hopkins University · TikTok · University of California, San Diego +1

May 11, 2026J/K move · Enter open · S save

Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why

Mohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal+5

Apple

On-Policy Self-Distillation Language Model Distillation On-Policy Distillation Teacher-Student Learning Gradient Alignment

Open paper AskPDF arXiv

On-policy distillation offers dense, per-token supervision for training reasoning models; however, it remains unclear under which conditions this signal is beneficial and under which it is detrimental. Which teacher model should be used, and in the case of self-distillation, which specific context should serve as the supervisory signal? Does the optimal choice vary from one token to the next? At present, addressing these questions typically requires costly training runs whose aggregate performance metrics obscure the dynamics at the level of individual tokens. We introduce a training-free diagnostic framework that operates at the highest resolution: per token, per question, and per teacher. We derive an ideal per-node gradient defined as the parameter update that maximally increases the student's probability of success. We then develop a scalable targeted-rollout algorithm to estimate this gradient efficiently, even for long chains of intermediate thoughts. The gradient alignment score, defined as the cosine similarity between this ideal gradient and any given distillation gradient, quantifies the extent to which a particular configuration approximates the ideal signal. Across a range of self-distillation settings and external teacher models, we observe that distillation guidance exhibits substantially higher alignment with the ideal on incorrect rollouts than on correct ones, where the student already performs well and the teacher's signal tends to become noisy. Furthermore, we find that the optimal distillation context depends jointly on the student model's capacity and the target task, and that no single universally effective configuration emerges. These findings motivate the use of per-task, per-token diagnostic analyses for distillation.

Abstract

Reasoning Distillation

Explore similar work

Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why

Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization

On-Policy Distillation with Best-of-N Teacher Rollout Selection