Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

Xin Li; Hao Jiang; Xin Gao; Annan Wang; Yuchen Xie; Jinghao Guo; Xingwei Qu; Yichi Zhang

cs.LGSep 28, 2026

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

Authors: Xin Li, Hao Jiang, Xin Gao, Annan Wang, Yuchen Xie, Jinghao Guo, Xingwei Qu, Yichi Zhang, +1 more

Organizations: Nanyang Technological University · Yale University · University of Manchester

Abstract

Reinforcement learning can turn one language model into several specialists, each excellent at a single skill such as mathematics, coding or following instructions, but users need one model with all of these skills. Multi-teacher on-policy distillation (MOPD) merges them by letting the specialists teach one student: the student answers each prompt, and the specialist for that prompt's domain gives feedback on every token. This routing decides which specialist teaches, but not how strongly its feedback moves the shared student. In Qwen3.5 models at three sizes, we find that MOPD's student does not beat one taught by the best single specialist and gains little of the mathematics specialist's advantage. The feedback is unbalanced: instruction-following feedback is several times more spread out than mathematics feedback and dominates the student's updates. We propose Domain-Normalized MOPD (DN-MOPD), which keeps the routing and rescales each domain's feedback by its measured spread. On six public benchmarks, DN-MOPD improves the average score over MOPD at every size, across three random seeds and under two answer-length limits, and recovers most of the lost mathematics gain. Controls with fixed domain weights show that the gain comes mainly from turning down instruction-following feedback rather than turning up mathematics alone, and that fixed weights close to those DN-MOPD measures perform comparably. Combining specialists therefore requires deciding not only which one teaches, but also how strongly its feedback counts.

Figures & tables

RL experts
	Math (avg@64)		Code (avg@6)		IF (avg@16)
Method	AIME25	AIME26	LCB v5	LCB v6	IFEval	IFBench	Total
Initial student	57.7	62.6	54.9	51.4	82.4	33.8	57.1
Math expert	59.7	69.4	53.4	49.4	82.3	34.9	58.2
Code expert	58.5	65.4	58.5	54.3	82.3	34.6	58.9
IF expert	54.6	60.5	54.0	51.5	86.5	42.1	58.2

Table 1: Capability integration at 9B. Qwen3.5-9B scores (%) on six public tasks. Total averages the six task scores.

RL experts
	Qwen3.5-4B				Qwen3.5-2B
Method	Math	Code	IF	Total	Math	Code	IF	Total
Initial student	52.2	38.1	52.8	47.7	17.6	11.3	43.3	24.0
Math expert	54.2	39.7	54.0	49.3	22.1	13.4	44.7	26.7
Code expert	51.6	47.4	53.6	50.8	20.7	21.6	44.4	28.9
IF expert	50.7	38.1	60.6	49.8	15.6	12.5	52.8	27.0

Table 2: Capability integration at smaller scales. Independently trained Qwen3.5-4B and 2B expert pools. Each domain averages its two tasks; Total weights the three domains equally.

Variant	Teacher rule	Signal change	9B	4B	2B
Uniform pool	Mixture	Unchanged	-0.41	-0.69	+0.30
Dynamic router	Response	Unchanged	-0.25	+0.28	-0.07
Label-routed	Domain	Unchanged	+0.00	+0.00	+0.00
Annealed injection	Domain	Early imitation	-0.24	+1.11	+0.30
Math $\times$ 2 only	Domain	Math scale	—	+1.20	+0.27
IF $\times$ 0.25 only	Domain	IF scale	—	+1.79	+2.19

Table 3: Changes to label-routed OPD. Total point differences (pp) relative to Label, evaluated at 16K. Pooling and Dynamic change the teacher assignment; annealed injection and DN-MOPD retain it and change the learning signal. The two fixed-weight rows set one domain’s weight (mathematics 2 or IF 0.25, others 1) and were run at 4B and 2B only.

Method	Total	Math tokens	Code tokens	IF tokens	At cap (%)
Qwen3.5-9B
SeqKD-SFT	62.6	5,547	5,896	422	1.5
ParamMerge-TA	60.5	5,083	5,479	430	1.7
Label-routed	58.4	8,637	6,846	450	9.8
DN-MOPD	59.6	7,032	6,427	441	5.2
Qwen3.5-4B

Table 4: Capability and generated length. Mean response tokens by domain and the share of responses reaching the 16K cap. Scores and lengths come from the same evaluation outputs.

Method	80	160	$\Delta$ Total
Qwen3.5-9B
Single teacher (IF)	57.3	57.6	+0.4
Label-routed	58.4	59.4	+1.0
DN-MOPD	59.6	60.7	+1.2
Qwen3.5-4B
Single teacher (code)	50.9	51.2	+0.4

Table 5: Training-duration comparison. Six-task Total at fixed 80- and 160-update endpoints, evaluated at 16K. Changes use unrounded scores.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Size	Math expert	Code expert	IF expert
9B	250	200	400
4B	320	300	400
2B	400	400	400

Appendix

Table A.1: Shared specialist checkpoints. Update counts for the domain experts used by every student comparison within each size. Each specialist is trained from the same initialization as its student.

RL experts
	Math (avg@64)		Code (avg@6)		IF (avg@16)
Method	AIME25	AIME26	LCB v5	LCB v6	IFEval	IFBench	Total
Initial student	48.4	56.0	39.1	37.0	78.3	27.2	47.7
Math expert	50.6	57.9	41.1	38.4	79.0	29.0	49.3
Code expert	48.5	54.6	49.5	45.2	79.2	28.0	50.8
IF expert	47.9	53.4	38.4	37.8	84.3	36.9	49.8

Appendix

Table B.1: Complete results for Qwen3.5-4B at 16K. Scores (%) on the six public tasks; all OPD students use 80 updates. Total equally averages the tasks. Boldface identifies the best result within the multi-teacher OPD block.

RL experts
	Math (avg@64)		Code (avg@6)		IF (avg@16)
Method	AIME25	AIME26	LCB v5	LCB v6	IFEval	IFBench	Total
Initial student	17.7	17.4	9.0	13.5	62.8	23.9	24.0
Math expert	20.9	23.3	10.1	16.7	64.1	25.3	26.7
Code expert	19.6	21.8	20.1	23.0	63.5	25.3	28.9
IF expert	16.0	15.2	10.0	15.0	74.6	30.9	27.0

Appendix

Table B.2: Complete results for Qwen3.5-2B at 16K. Scores (%) on the six public tasks; all OPD students use 80 updates. Total equally averages the tasks. Boldface identifies the best result within the multi-teacher OPD block.

RL experts
	Math (avg@64)		Code (avg@6)		IF (avg@16)
Method	AIME25	AIME26	LCB v5	LCB v6	IFEval	IFBench	Total
Initial student	45.5	50.7	45.0	42.2	82.0	34.0	49.9
Math expert	56.2	64.6	45.5	44.3	82.5	35.0	54.7
Code expert	48.3	55.7	52.1	46.4	82.2	34.3	53.2
IF expert	45.2	50.0	45.1	43.9	86.4	41.9	52.1

Appendix

Table B.3: Complete results for Qwen3.5-9B at 8K. Scores (%) on the six public tasks; all OPD students use 80 updates. Total equally averages the tasks. Boldface identifies the best result within the multi-teacher OPD block.

RL experts
	Math (avg@64)		Code (avg@6)		IF (avg@16)
Method	AIME25	AIME26	LCB v5	LCB v6	IFEval	IFBench	Total
Initial student	37.8	43.0	30.6	32.1	78.5	27.6	41.6
Math expert	49.4	57.1	34.0	33.2	79.4	28.9	47.0
Code expert	42.3	47.7	43.5	41.5	79.3	28.3	47.1
IF expert	37.4	41.8	31.3	32.9	84.4	36.7	44.1

Appendix

Table B.4: Complete results for Qwen3.5-4B at 8K. Scores (%) on the six public tasks; all OPD students use 80 updates. Total equally averages the tasks. Boldface identifies the best result within the multi-teacher OPD block.

RL experts
	Math (avg@64)		Code (avg@6)		IF (avg@16)
Method	AIME25	AIME26	LCB v5	LCB v6	IFEval	IFBench	Total
Initial student	11.1	8.0	7.6	12.0	63.6	24.1	21.1
Math expert	19.6	21.6	8.4	15.2	63.5	25.2	25.6
Code expert	17.8	17.3	15.7	19.9	63.9	24.8	26.6
IF expert	12.1	8.2	8.0	12.8	74.8	30.5	24.4

Appendix

Table B.5: Complete results for Qwen3.5-2B at 8K. Scores (%) on the six public tasks; all OPD students use 80 updates. Total equally averages the tasks. Boldface identifies the best result within the multi-teacher OPD block.

Size	Math	Code	IF
9B	792/900	724/900	812/900
4B	794/900	686/900	763/900
2B	599/900	363/900	771/900

Appendix

Table C.1: Teacher-answer coverage for annealed injection. Prompts with at least one usable verified-correct answer, out of the 900 training prompts in each domain. One response slot is replaced for covered groups while the imitation weight is positive; uncovered prompts retain Label training.

Qwen3.5-9B
Method	Updates	AIME25	AIME26	LCB v5	LCB v6	IFEval	IFBench	Total
Label	160	57.0	63.9	58.9	51.0	84.9	40.6	59.4
DN-MOPD	160	58.9	70.1	56.7	52.5	85.1	41.1	60.7
Single IF	160	55.8	61.7	52.4	50.1	85.1	40.6	57.6
Annealed injection	80	55.8	63.6	54.3	52.0	84.6	38.6	58.2
Qwen3.5-4B

Appendix

Table C.2: Additional controls at 16K. Scores (%); each block includes three 160-update continuations and the 80-update annealed-injection control. All runs use seed 42. The single-teacher reference is selected on the development instrument.

Qwen3.5-9B
Method	Updates	AIME25	AIME26	LCB v5	LCB v6	IFEval	IFBench	Total
Label	160	47.3	54.5	48.6	44.5	85.1	40.3	53.4
DN-MOPD	160	52.2	62.3	50.3	45.8	85.1	41.2	56.2
Single IF	160	45.9	51.3	43.7	43.2	85.3	41.0	51.8
Annealed injection	80	46.4	52.0	45.1	42.9	84.5	38.2	51.5
Qwen3.5-4B

Appendix

Table C.3: Additional controls at 8K. Scores (%); each block includes three 160-update continuations and the 80-update annealed-injection control. All runs use seed 42. The single-teacher reference is selected on the development instrument.

Qwen3.5-9B
Contrast (updates)	$\Delta$ Total	95% CI
DN $-$ Label (80)	$+1.17$	$[+0.28,+2.03]$
DN $-$ Label (160)	$+1.33$	$[+0.42,+2.22]$
Annealed $-$ Label (80)	$-0.24$	$[-0.98,+0.51]$
DN $-$ Annealed (80)	$+1.41$	$[+0.48,+2.34]$
Qwen3.5-4B

Appendix

Table C.4: Paired Total differences at 16K. Differences and intervals are in pp. Parentheses give student update counts. The 95% intervals use 10,000 paired question-bootstrap replicates, conditional on seed 42 and the shared expert pool.

Qwen3.5-9B
Contrast (updates)	$\Delta$ Total	95% CI
DN $-$ Label (80)	$+2.47$	$[+1.65,+3.27]$
DN $-$ Label (160)	$+2.79$	$[+1.93,+3.65]$
Annealed $-$ Label (80)	$-1.33$	$[-2.04,-0.65]$
DN $-$ Annealed (80)	$+3.80$	$[+2.91,+4.69]$
Qwen3.5-4B

Appendix

Table C.5: Paired Total differences at 8K. Differences and intervals are in pp. Parentheses give student update counts. The 95% intervals use 10,000 paired question-bootstrap replicates, conditional on seed 42 and the shared expert pool.

16K evaluation
Size	Strongest single	Label $-$ single	DN-MOPD $-$ single
9B	Code	$-0.09$ $[-1.00,+0.83]$	$+1.08$ $[+0.20,+1.98]$
4B	Code	$-0.55$ $[-1.58,+0.46]$	$+1.69$ $[+0.77,+2.61]$
2B	Code	$-2.23$ $[-3.17,-1.31]$	$+0.13$ $[-0.68,+0.93]$
8K evaluation
9B	Math	$-1.69$ $[-2.84,-0.58]$	$+0.78$ $[-0.13,+1.68]$

Appendix

Table C.6: Label and DN-MOPD against the strongest single-teacher student. Paired six-task Total differences (pp) with 95% question-bootstrap intervals, 80 updates. The strongest single-teacher student is chosen separately at each size and cap on the same public Total, which favors that comparator.

Cap	DN-MOPD $-$ code (80)	DN-MOPD $-$ code (160)	Label $-$ code (160)	Code: 160 $-$ 80
16K	$+1.08$ $[+0.20,+1.98]$	$+2.57$ $[+1.65,+3.46]$	$+1.24$ $[+0.22,+2.25]$	$-0.34$ $[-1.18,+0.53]$
8K	$+1.92$ $[+1.10,+2.71]$	$+2.97$ $[+2.10,+3.83]$	$+0.18$ $[-0.71,+1.05]$	$-0.19$ $[-0.92,+0.54]$

Appendix

Table C.7: Continuing the strongest 9B single-teacher student. The code-teacher student has the highest six-task Total among the 9B single-teacher students at the main 16K cap (at 8K the mathematics-teacher student is higher, 54.53 versus 53.39, and was not continued). It is continued from 80 to 160 updates with the same recipe, alongside the Label and DN-MOPD continuations of Table 5 . Paired six-task Total differences (pp) with 95% question-bootstrap intervals.

16K evaluation
Size	Updates	Math	Code	IF
9B	80	$+3.98$ $[+2.32,+5.68]$	$-0.82$ $[-2.78,+1.06]$	$+0.36$ $[-0.09,+0.82]$
9B	160	$+4.06$ $[+2.21,+5.99]$	$-0.38$ $[-2.27,+1.51]$	$+0.31$ $[-0.11,+0.74]$
4B	80	$+3.75$ $[+1.95,+5.60]$	$+2.13$ $[+0.02,+4.28]$	$+0.82$ $[+0.29,+1.38]$
4B	160	$+3.33$ $[+1.56,+5.21]$	$+0.95$ $[-1.16,+3.09]$	$+0.10$ $[-0.38,+0.59]$
2B	80	$+3.72$ $[+2.03,+5.52]$	$+2.88$ $[+1.31,+4.44]$	$+0.47$ $[-0.09,+1.03]$

Appendix

Table C.8: DN-MOPD minus Label by domain. Paired differences in each domain mean (pp) with 95% question-bootstrap intervals, at the main 80-update endpoint and after continuing both runs to 160 updates.

Method	Math tokens	Code tokens	IF tokens	Cap hits (%)
Qwen3.5-9B (16K evaluation)
Label	8,637	6,846	450	9.8
DN-MOPD	7,032	6,427	441	5.2
Qwen3.5-4B (16K evaluation)
Label	8,653	8,238	523	12.4
DN-MOPD	6,146	7,649	522	5.3

Appendix

Table C.9: Generation length and cap hits at both budgets. Mean output tokens for the 80-update students, averaged equally over the two suites in each domain. The final column is the cap-hit percentage averaged equally over all six suites. Each budget uses its own generated responses.

16K evaluation
Size	Seed 42	Seed 43	Seed 44	Three-seed mean
9B	$+1.17$ $[+0.28,+2.03]$	$+1.08$ $[+0.21,+1.96]$	$+1.10$ $[+0.23,+1.97]$	$+1.12$ $[+0.62,+1.62]$
4B	$+2.24$ $[+1.30,+3.20]$	$+1.85$ $[+0.91,+2.84]$	$+1.82$ $[+0.88,+2.77]$	$+1.97$ $[+1.39,+2.56]$
2B	$+2.36$ $[+1.58,+3.17]$	$+2.24$ $[+1.45,+3.04]$	$+2.43$ $[+1.53,+3.40]$	$+2.34$ $[+1.86,+2.84]$
8K evaluation
9B	$+2.47$ $[+1.65,+3.27]$	$+2.62$ $[+1.80,+3.44]$	$+3.05$ $[+2.14,+3.96]$	$+2.71$ $[+2.16,+3.29]$

Appendix

Table C.10: DN-MOPD minus Label across student seeds. Six-task Total differences (pp) between seed-matched runs at 80 updates; each seed changes the prompt order and rollout sampling while the teachers and initialization are shared. Per-seed intervals resample questions within suite; the three-seed mean uses a two-level bootstrap over seeds and questions. Seed 42 is the main comparison in Tables 1 – 2 .

16K evaluation
Size	Control	Weights	$-$ Label	$-$ DN-MOPD	Math $-$ Label
9B	Update 0	1.77/0.89/0.25	$+1.29$ $[+0.40,+2.16]$	$+0.12$ $[-0.69,+0.93]$	$+3.59$ $[+1.77,+5.42]$
9B	Global	2/1/0.25	$+1.32$ $[+0.36,+2.29]$	$+0.14$ $[-0.67,+0.95]$	$+4.01$ $[+2.01,+6.15]$
4B	Update 0	2.01/0.81/0.34	$+2.34$ $[+1.47,+3.24]$	$+0.10$ $[-0.62,+0.82]$	$+3.96$ $[+2.21,+5.83]$
4B	Global	2/1/0.25	$+2.86$ $[+1.90,+3.84]$	$+0.62$ $[-0.09,+1.36]$	$+4.53$ $[+2.63,+6.51]$
4B	Math only	2/1/1	$+1.20$ $[+0.34,+2.08]$	$-1.04$ $[-1.95,-0.14]$	$+1.72$ $[+0.08,+3.44]$

Appendix

Table C.11: Fixed domain weights. Each control trains with Label assignment and a constant multiplier per domain (math / code / IF), with the DN-MOPD recipe otherwise unchanged (80 updates, seed 42). Update 0 uses the multipliers DN-MOPD measured on its first batch at that size; Global uses one setting at every size. Paired differences (pp) with 95% question-bootstrap intervals; the last column is the mathematics domain mean.

Size	Initial	Label	Math-only OPD	DN-MOPD	Label $-$ initial	DN-MOPD $-$ Label
16K evaluation
9B	95.84	95.76	97.30	96.50	$-0.08$ $[-0.51,+0.35]$	$+0.74$ $[+0.31,+1.18]$
4B	94.46	94.08	95.84	95.33	$-0.39$ $[-0.91,+0.12]$	$+1.25$ $[+0.67,+1.85]$
2B	76.91	77.21	82.93	83.16	$+0.30$ $[-0.64,+1.22]$	$+5.95$ $[+4.95,+6.96]$
8K evaluation
9B	94.08	94.38	96.80	95.83	$+0.30$ $[-0.19,+0.81]$	$+1.45$ $[+0.92,+2.00]$

Appendix

Table C.12: MATH-500. Mean accuracy (%) over 500 problems with 16 samples each, graded by Math-Verify, and paired differences (pp) with 95% question-bootstrap intervals. Students use 80 updates. This benchmark is not part of the six-task Total.

Qwen3.5-9B
Domain	Update 0	$\sigma_{d}/\sigma_{\mathrm{all}}$	Raw factor	Applied factor	Clipped	Tokens (%)
Math	0.57	0.65	1.54	1.54	0/115	48.9
Code	1.13	0.64	1.57	1.57	0/115	50.1
IF	4.41	7.37	0.14	0.25	115/115	0.9
Qwen3.5-4B
Math	0.50	0.49	2.03	2.03	0/134	46.9

Appendix

Table D.1: Feedback scale, clipping and token composition. The first column is the spread ratio in the first training batch, sampled from the same initial student as Label. The other columns are medians over the recorded batches of the DN-MOPD runs, including their continuation beyond 80 updates ( App. D.2 ); the clipping column counts logged batches where the factor reaches a bound. Token share is the within-batch fraction of valid response tokens.

Qwen3.5-9B
Domain	Tokens (%)	Pooled variance (%)	Would-be under Label	Applied by DN-MOPD
Math	48.4	20.8	1.49	1.59
Code	50.6	28.3	0.99–1.00	1.53
IF	1.0	50.9	0.25	0.25
Qwen3.5-4B
Math	47.2	10.9	1.91–1.93	2.15

Appendix

Table D.2: Sources of the pooled spread. Token share and each domain's share of the pooled rollout log-ratio variance (within-domain plus mean-difference terms), averaged over updates 0–79 of the DN-MOPD runs. The last two columns give the multiplier DN-MOPD would apply, recorded but not applied during three Label runs (range over seeds of the mean over updates 0–79), and the multiplier DN-MOPD applied.

Initial student
Domain	$\lVert g_{d}\rVert$	Split-half cos.	Share, equal (%)	Share, DN (%)
Math	3.6	$0.89$	1	16
Code	10.6	$0.88$	5	20
IF	51.7	$0.41$	94	64
Label, 80 updates
Math	2.6	$0.90$	0	0

Appendix

Table D.3: Per-domain gradients of the OPD loss at 4B. Gradients over all trainable parameters for 32 prompts per domain and four sampled responses each, using the response-averaged loss of training. Split-half cosine compares the gradients of two disjoint prompt halves. Shares are each domain's contribution

\langle w_{d}g_{d},g\rangle/\lVert g\rVert^{2}

to the combined gradient

g=\sum_{d}w_{d}g_{d}

, under equal weights and under DN-MOPD's first-batch weights (2.01 / 0.81 / 0.34). Descriptive, one probe set; a replicate draw preserves the ordering.

Evaluation cap: 16K
Size	Math expert	Math-only OPD	Label	DN-MOPD
9B	$+4.45$ $[+2.55,+6.35]$	$+4.09$ $[+2.16,+6.04]$	$-0.86$ $[-2.42,+0.76]$	$+3.12$ $[+1.20,+5.03]$
4B	$+2.06$ $[-0.23,+4.53]$	$+1.69$ $[-0.42,+3.91]$	$-1.98$ $[-3.78,-0.18]$	$+1.77$ $[+0.05,+3.65]$
2B	$+4.58$ $[+2.47,+6.95]$	$+4.51$ $[+2.45,+6.74]$	$-0.60$ $[-1.88,+0.73]$	$+3.12$ $[+1.25,+5.26]$
Evaluation cap: 8K
9B	$+12.32$ $[+8.93,+15.76]$	$+10.70$ $[+7.50,+14.04]$	$+3.05$ $[+1.48,+4.66]$	$+7.55$ $[+5.26,+9.95]$

Appendix

Table E.1: Mathematics capability transferred from the expert. Mathematics gains over the shared initialization (pp) with paired 95% question-bootstrap intervals. The single-teacher student learns only from the mathematics expert. Students use 80 updates; all four gains in a row share the same initialization and evaluation cap.

Metric	Label	DN-MOPD	$\Delta$ (pp)	95% CI
Math	53.67	53.41	$-0.26$	$[-1.35,+0.76]$
Code	38.36	38.27	$-0.08$	$[-1.32,+1.16]$
IF	56.13	56.53	$+0.40$	$[-0.19,+1.04]$
Total	49.39	49.41	$+0.02$	$[-0.56,+0.59]$

Appendix

Table E.2: Earlier Qwen3-4B boundary case. Domain means and six-task Total (%) at 160 updates and a 16K evaluation cap. Differences are DN-MOPD minus Label (pp), with paired 95% question-bootstrap intervals. The intervals include zero for every domain and Total.

Explore similar work

From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation
Oct 1, 2026Siqi Zhu, Suozhi Huang, Kaixuan Zhang +4Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher
University of Illinois Urbana-Champaign · Princeton University · Westlake University
Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation
Aug 27, 2026Ziyuan Liu, Jiao Ou, Jian Liang +2Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher
Kuaishou Technology Beijing, China
D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation
Aug 25, 2026Zechen Sun, Zhiwei Zhang, Fei Zhao +7Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher-Student Distillation
AllSpark Team

Oct 1, 2026, cs.LGJ/K move · Enter open · S save

From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation

Siqi Zhu, Suozhi Huang, Kaixuan Zhang, Yuheng Yang+3

University of Illinois Urbana-Champaign · Princeton University · Westlake University

Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher Gradient

Open paper AskPDF arXiv

Multi-teacher on-policy distillation (MOPD) aims to combine the strengths of RL-trained teachers in a single student, but how teacher signals affect parameter changes remains underexplored. We study Qwen3-1.7B with four domain teachers trained with RL from the same initialization as the student, comparing gradients, optimizer updates, and task learning curves, with additional SmolLM3-3B diagnostics. We find that several factors influence teacher signals. First, loss averaging implicitly weights responses: token averaging favors longer responses, and equalizing domain contributions retains this weighting within domains. Second, Adam's first moment reduces differences in parameter updates: the cosine similarity is 0.83 between teachers and 0.96 between averaging rules, despite differences in raw gradients. Third, BF16 rounding hides small changes: about 97% of FP32 master weights differ from initialization, but only 7--11% of BF16 weights do. Finally, the top-64 intersection KL gradient closely matches Qwen's full-vocabulary gradient, but the effect on task performance depends on averaging: mathematics accuracy is 2.6 points higher than with sampled-token policy-gradient (PG) under response averaging and 2.1 points lower under global token averaging.

Abstract

Explore similar work

From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation

Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation

D3^33-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation

D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation