Continual Distillation of Teachers from Different Domains

Nicolas Michel; Maorong Wang; Jiangpeng He; Toshihiko Yamasaki

cs.LGApr 10, 2026

Continual Distillation of Teachers from Different Domains

Authors: Nicolas Michel, Maorong Wang, Jiangpeng He, Toshihiko Yamasaki

Organizations: The University of Tokyo · Japanese-French Laboratory for Informatics, CNRS · National Institute of Informatics · Indiana University Bloomington

Abstract

Deep learning models continue to scale, with some requiring more storage than many large-scale datasets. Thus, we introduce a new paradigm: Continual Distillation (CD), where a student learns sequentially from a stream of teacher models without retaining access to earlier teachers. CD faces two challenges: teacher training data is unavailable, and teachers have varying expertise. We show that external unlabeled data enables Unseen Knowledge Transfer (UKT), allowing the student to acquire information from domains not present in the training data, while known to the teacher. We also show that sequential distillation causes Unseen Knowledge Forgetting (UKF) when transferred knowledge is lost after training on later teachers. To better trade off between UKT and UKF, we propose Self External Data Distillation (SE2D), a method that preserves logits on external data to stabilize learning across heterogeneous teachers. Experiments on multiple benchmarks show that SE2D reduces UKF and improves cross-domain generalization. The code and implementation for this work are publicly available at: https://github.com/Nicolas1203/continual_distillation.

Figures & tables

Figure 1: Overview of the Continual Distillation problem. A student model $\mathcal{S}$ learns through distillation from a sequence of teachers $\{\mathcal{T}_{1},\mathcal{T}_{2},\mathcal{T}_{3}\}$ . During distillation, part of the teachers’ training data is unavailable. The objective is for the student to maintain high performance on all domains known to the teachers , but not necessarily introduced to the student.

Figure 2: Visualization $\mathcal{D}_{e}$ and $\mathcal{D}_{i}$ and different teacher training domains in our experimental setting. $\mathcal{D}_{i}$ and $\mathcal{D}_{e}$ are publicly available like ImageNet or Wikipedia. However, some private or restricted data might be included in training, such as medical data.

Figure 3: Illustration of Unseen Knowledge Transfer (UKT) and Unseen Knowledge Forgetting (UKF). When distilling only from $\mathcal{T}_{1}$ on Internal Data $\mathcal{D}_{i}$ , only Domain $0$ knowledge is transferred. When distilling from the same teacher $\mathcal{T}_{1}$ on $\mathcal{D}_{e}\cup\mathcal{D}_{i}$ , Domains $0$ and $1$ knowledge is transferred, although the student has never seen Domain $1$ . However, when continuing the sequence and distilling from $\mathcal{T}_{2}$ , while the student acquires knowledge from Domain $2$ , it forgets part of the knowlege from Domain $1$ .

Figure 4: Overview of Self External Data Distillation (SE2D). While the distillation from the teacher sequence is done on the entire distillation data, distillation from the checkpoint of the student at the end of the previous task is done only on $\mathcal{D}_{e}$ .

Domain	0	1	2	3	$\|\mathcal{D}_{e}\|/\|\mathcal{D}^{\mathcal{S}}\|$
After $\mathbb{D}(\mathcal{T}_{01},\mathcal{D}^{\mathcal{S}}_{0})$	93.25	37.80	45.30	36.90	0%
After $\mathbb{D}(\mathcal{T}_{02},\mathcal{D}^{\mathcal{S}}_{0})$	93.95	31.60	42.20	35.25	0%
After $\mathbb{D}(\mathcal{T}_{03},\mathcal{D}^{\mathcal{S}}_{0})$	92.10	32.00	39.05	33.00	0%
After $\mathbb{D}$ ( $\mathcal{T}_{012}$ , $\mathcal{D}^{\mathcal{S}}_{014}$ )	92.70	93.15	68.35	53.80	33%
After $\mathbb{D}$ ( $T_{013}$ , $\mathcal{D}^{\mathcal{S}}_{014}$ )	92.45	92.95	48.95	72.55	33%
After $\mathbb{D}$ ( $\mathcal{T}_{01}$ , $\mathcal{D}^{\mathcal{S}}_{04}$ )	96.35	77.15	48.95	48.45	50%

Table 1: Accuracy (%) of the student model on test sets, domain-wise, on CIFAR20, for various domain overlaps, after distilling for

1

epoch. The larger the ratio

|\mathcal{D}_{e}|/|\mathcal{D}^{\mathcal{S}}|

, the more the student performs on domains unseen during training (underlined values).

CIFAR20 - Internal Data Only
Method	D0	D1	D2	D3	D4 ✗	Avg. (0-3)	Gain ( $\uparrow$ )
$\mathcal{T}_{best}$ (upper bound)	97.75	95.80	96.70	95.75	-	96.5	0.00
KL-divergence	98.10 $\pm$ 0.05	41.40 $\pm$ 0.84	53.38 $\pm$ 0.35	54.87 $\pm$ 3.70	-	61.94 $\pm$ 1.24	0.00
DKD [CVPR’22]	95.70 $\pm$ 1.55	35.53 $\pm$ 1.94	45.42 $\pm$ 2.05	41.02 $\pm$ 2.55	-	54.42 $\pm$ 2.02	0.00
LS [CVPR’24]	96.50 $\pm$ 1.52	39.13 $\pm$ 2.90	49.77 $\pm$ 4.40	49.58 $\pm$ 5.78	-	58.75 $\pm$ 3.65	0.00
MDS [ICLR’25]	94.20 $\pm$ 0.73	34.30 $\pm$ 2.80	44.70 $\pm$ 1.40	41.00 $\pm$ 1.90	-	53.55 $\pm$ 1.72	0.00

Table 2: Performances (%, higher is better) of the student at the end of training on CIFAR20 for 4 scenarios. Internal Data Only (D0), Related External Data (D4), CUB as ED, and MNIST as ED. The number of runs is set to

3

. The gain columns shows the gain over using Internal Data Only. Grey : Internal Data; Blue : Domain known by the teacher; Red : External Data (ED); White: Ignored.

Digits - Internal Data Only
Method	MNIST	SVHN	MNIST-M	USPS	KMNIST ✗	Avg.	Gain ( $\uparrow$ )
$\mathcal{T}_{best}$ (upper bound)	99.2	97.84	99.25	98.8	-	98.77	0.00
KL-divergence	99.17 $\pm$ 0.04	35.80 $\pm$ 1.49	62.80 $\pm$ 2.69	95.81 $\pm$ 0.46	-	73.40 $\pm$ 1.17	0.00
DKD [CVPR’22]	98.70 $\pm$ 0.11	33.35 $\pm$ 1.28	54.50 $\pm$ 2.45	95.07 $\pm$ 0.43	-	70.40 $\pm$ 1.07	0.00
LS [CVPR’24]	99.13 $\pm$ 0.09	37.60 $\pm$ 1.40	64.10 $\pm$ 1.73	96.00 $\pm$ 0.25	-	74.21 $\pm$ 0.87	0.00
MDS [ICLR’25]	98.15 $\pm$ 1.13	34.51 $\pm$ 4.80	54.97 $\pm$ 9.49	93.21 $\pm$ 3.01	-	70.21 $\pm$ 3.80	0.00

Table 3: Performances (%, higher is better) of the student at the end of training on Digits for 2 scenarios. Internal Data Only (MNIST) and Related External Data (KMNIST). The number of runs is set to

3

. Average and standard deviations are reported.

	DomainNet - Internal Data Only
Method	Clipart	Infograph	Painting	Quickdraw	Real	Sketch ✗	Avg.	Gain ( $\uparrow$ )
$\mathcal{T}_{best}$ (upper bound)	74.35	35.83	66.66	66.19	78.90	-	64.39	0.00
KL-divergence	77.08 $\pm$ 0.45	18.16 $\pm$ 0.24	44.17 $\pm$ 0.50	17.19 $\pm$ 0.73	67.29 $\pm$ 0.24	-	44.78 $\pm$ 0.29	0.00
DKD [CVPR’22]	77.32 $\pm$ 0.22	18.04 $\pm$ 0.09	42.76 $\pm$ 0.40	17.43 $\pm$ 0.31	63.99 $\pm$ 0.14	-	43.91 $\pm$ 0.17	0.00
LS [CVPR’24]	77.21 $\pm$ 0.37	17.21 $\pm$ 0.91	40.95 $\pm$ 0.14	16.42 $\pm$ 1.01	60.39 $\pm$ 0.20	-	42.43 $\pm$ 0.44	0.00
MDS [ICLR’25]	76.92 $\pm$ 0.42	18.08 $\pm$ 0.23	42.14 $\pm$ 0.38	17.44 $\pm$ 1.03	63.29 $\pm$ 0.20	-	43.57 $\pm$ 0.34	0.00

Table 4: Performances (%, higher is better) of the student at the end of training on DomainNet for 2 scenarios. Internal Data Only (Clipart) and Related External Data (Sketch). The number of runs is set to

3

. Average and standard deviations are reported.

Figure 5: Average Accuracy of the student across all domains known by teachers at the end of training on CIFAR20.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure B.1 : Entropy distribution of the teacher (ViT-Base) trained on domains 0 and 1 of CIFAR20 for various datasets and domains.

Figure B.2 : Entropy distribution of the teacher (ViT-Base) trained on domains Real and Clipart of Domainnet for various DomainNet domains.

CIFAR20 - Internal Data Only
Method	D0	D1	D2	D3	D4 ✗	Avg. (0-3)
KL-divergence	0	11.95	2.98	0	-	3.73
DKD [CVPR’22]	0	2.38	3.47	0	-	1.46
LS [CVPR’24]	0	10.42	1.88	0	-	3.08
MDS [ICLR’25]	0	11.95	2.98	0	-	3.23
Self-Distillation	0	16.02	2.47	0	-	4.12

Appendix

Table C.1 : Forgetting (%, lower is better) of the student at the end of training on CIFAR20 for 4 scenarios. Internal Data Only (D0), Related External Data (D4), CUB as ED, and MNIST as ED. The number of runs is set to

3

Digits - Internal Data Only
Method	MNIST	SVHN	MNIST-M	USPS	KMNIST ✗	Avg.
KL-divergence	0.23	3.34	16.20	0	-	4.94
DKD [CVPR’22]	0.47	0.53	11.73	0	-	3.12
LS [CVPR’24]	0.10	4.09	12.88	0	-	4.40
MDS [ICLR’25]	0.23	3.34	16.20	0	-	4.94
Self-Distillation	0.13	4.06	7.46	0	-	3.54

Appendix

Table C.2 : Forgetting (%, lower is better) of the student at the end of training on Digits for 2 scenarios. Internal Data Only (D0), Related External Data (D4). The number of runs is set to 3.

	DomainNet - Internal Data Only
Method	Clipart	Infograph	Painting	Quickdraw	Real	Sketch ✗	Avg.
KL-divergence	4.63	9.91	13.70	20.31	0	-	12.14
DKD [CVPR’22]	3.85	10.34	13.89	22.38	0	-	12.61
LS [CVPR’24]	3.50	11.48	15.39	23.73	0	-	13.52
MDS [ICLR’25]	5.20	10.70	15.69	21.46	0	-	13.26
Checkpoint	1.27	5.86	11.30	20.68	0	-	9.78

Appendix

Table C.3 : Forgetting (%, lower is better) on DomainNet for 2 scenarios: Internal Data Only (Seq -1) and Related External Data (Seq 0). Averaged across 3 runs.

CIFAR20 - Internal Data Only
Method	D0	D1	D2	D3	D4 ✗	Avg. (0-3)
$\mathcal{T}_{best}$ (upper bound)	97.75	95.80	96.70	95.75	-	96.5
KL-divergence	96.27 $\pm$ 0.21	37.85 $\pm$ 0.78	52.28 $\pm$ 0.42	49.48 $\pm$ 1.28	-	58.97 $\pm$ 0.67
DKD [CVPR’22]	95.40 $\pm$ 1.05	33.27 $\pm$ 1.60	45.38 $\pm$ 0.90	40.45 $\pm$ 1.34	-	53.62 $\pm$ 1.22
LS [CVPR’24]	96.25 $\pm$ 0.85	38.95 $\pm$ 2.17	51.58 $\pm$ 2.47	50.15 $\pm$ 4.40	-	59.23 $\pm$ 2.47
MDS [ICLR’25]	94.45 $\pm$ 0.44	35.08 $\pm$ 0.88	45.35 $\pm$ 0.41	41.98 $\pm$ 3.10	-	54.22 $\pm$ 1.21

Appendix

Table C.4 : Performances (%, higher is better) of the student at the end of training on CIFAR20 for 2 scenarios with a ViT-tiny. Internal Data Only (D0), Related External Data (D4). The number of runs is set to 3. Average and standard deviations are reported.

	DomainNet - Internal Data Only
Method	Clipart	Infograph	Painting	Quickdraw	Real	Sketch ✗	Avg.
$\mathcal{T}_{best}$ (upper bound)	86.33	53.89	79.74	69.62	89.86	-	76.23
KL-divergence	78.33 $\pm$ 0.23	19.00 $\pm$ 0.30	39.25 $\pm$ 0.81	35.36 $\pm$ 0.49	51.51 $\pm$ 0.74	-	44.69 $\pm$ 0.32
DKD [CVPR’22]	78.55 $\pm$ 0.03	18.89 $\pm$ 0.41	39.69 $\pm$ 0.19	34.77 $\pm$ 0.38	52.45 $\pm$ 0.49	-	44.87 $\pm$ 0.06
LS [CVPR’24]	74.39 $\pm$ 0.39	15.54 $\pm$ 0.66	33.92 $\pm$ 0.56	20.09 $\pm$ 1.11	46.62 $\pm$ 0.76	-	38.11 $\pm$ 0.64
MDS [ICLR’25]	75.51 $\pm$ 0.10	17.52 $\pm$ 0.20	35.67 $\pm$ 1.39	26.22 $\pm$ 0.70	47.71 $\pm$ 0.69	-	40.53 $\pm$ 0.27

Appendix

Table C.5 : Domain Accuracy (%, higher is better) on DomainNet with CLIP-based teachers. Mean and standard deviation are reported.

	DomainNet - Sequence 1 (Quickdraw is used as ED and Infograph is ignored)
Method	Clipart	Infograph	Painting	Quickdraw	Real	Sketch	Avg.
KL-divergence	74.66 $\pm$ 0.14	-	33.01 $\pm$ 0.21	-	46.17 $\pm$ 0.62	48.68 $\pm$ 0.15	50.63
DKD [CVPR’22]	76.18 $\pm$ 0.20	-	36.20 $\pm$ 0.49	-	49.70 $\pm$ 0.62	54.85 $\pm$ 0.32	54.23
MDS [ICLR’25]	70.76 $\pm$ 0.40	-	30.01 $\pm$ 0.78	-	43.52 $\pm$ 1.09	51.56 $\pm$ 0.68	48.96
Self-Distillation	80.43 $\pm$ 0.10	-	47.47 $\pm$ 0.71	-	61.10 $\pm$ 0.59	58.15 $\pm$ 0.31	61.79
SE2D (ours)	76.89 $\pm$ 0.25	-	38.72 $\pm$ 0.49	-	52.83 $\pm$ 0.20	58.84 $\pm$ 0.25	56.82

Appendix

Table C.6 : Accuracy per domain (%). Grey : Internal Data; Blue : Domain known by the teacher (active); Red : External Data (ED); White : Ignored. Avg computed on Internal Data + Active domains.

Figure D.1 : Accuracy of the student on all domains at all steps for the considered methods on CIFAR20, training with Internal Data only.

Figure D.2 : Accuracy of the student on all domains at all steps for the considered methods on CIFAR20, training with External Data.

Figure D.3 : Accuracy of the student on all domains at all steps for the considered methods on Digits, training with Internal Data only.

Figure D.4 : Accuracy of the student on all domains at all steps for the considered methods on Digits, training with External Data.

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 819 papers

Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation
Sep 6, 2026Jingang Zhou, Yuyi Zhou, Haiyang Guo +6Language Model Distillation On-Policy Distillation
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Science · Ant Group
Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training
Jul 2, 2026Meng Wang, Haohan Zhao, Wenzhuo Liu +7Continual Learning for LLMs On-Policy Self-Distillation
Centre for Artificial Intelligence and Robotics, HKISI, CAS · Institute of Automation, CAS · University of Chinese Academy of Sciences +1
Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails
Jun 22, 2026Xin Liu, Simin Ma, Shujian Liu +5Data-Free Knowledge Distillation Language Model Distillation
University of Michigan · 2Zoom Video Communications

Sep 6, 2026J/K move · Enter open · S save

Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation

Jingang Zhou, Yuyi Zhou, Haiyang Guo, Xukai Wang+5

School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Science · Ant Group

Language Model Distillation On-Policy Distillation Teacher-Student Learning Uncertainty-Aware Knowledge Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) is a promising approach for transferring knowledge between language models, where a student receives dense token-level supervision along its own generated trajectories. However, teacher supervision can be unreliable when conditioned on incomplete or low-quality student prefixes. We identify Teacher Uncertainty Contraction (TUC), a systematic phenomenon whereby the teacher's predictive uncertainty decreases as it continues from a student-generated prefix. We theoretically characterize this trade-off through a variance-bias decomposition of teacher-branch gradients, showing that uncertainty contraction reduces variance while teacher-student path divergence increases bias, thereby favoring a finite continuation. Guided by this insight, we propose Adaptive-Continuations On-Policy Distillation (AC-OPD), which augments informative states along student rollouts with teacher continuations and adaptively selects their effective supervision horizons. Experiments on mathematical reasoning and code generation across model scales demonstrate that AC-OPD consistently improves over standard OPD. Controlled-continuations and matched-budget analyses further validate the adaptive-continuations design, highlighting adaptive teacher continuations as an effective principle for reliable on-policy distillation.The code will be made publicly available upon publication.

Abstract

Knowledge Distillation

Explore similar work

Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation

Denser ≠\neq= Better: Limits of On-Policy Self-Distillation for Continual Post-Training

Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails

Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training