Scaling Multi-Teacher Distillation for Digital Pathology

Sofiène Boutaj; Pierre Marza; Varun Belagali; Dimitris Samaras; Maria Vakalopoulou; Stergios Christodoulidis

cs.CVOct 8, 2026

Scaling Multi-Teacher Distillation for Digital Pathology

Authors: Sofiène Boutaj, Pierre Marza, Varun Belagali, Dimitris Samaras, Maria Vakalopoulou, Stergios Christodoulidis

Organizations: Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, IHU PRISM, Cancer Data Science Unit, France · Université Paris-Saclay, CentraleSupélec, MICS Laboratory, France · Department of Computer Science, Stony Brook University, USA

Abstract

Multi-teacher distillation allows transferring knowledge from multiple "teacher" networks to a single "student" network. This method is promising in fields such as digital pathology, where many powerful foundation models were proposed recently. However, as we show in this paper, the standard multi-teacher distillation approach reacts poorly to an increase in the number of teachers used to train a student encoder. We demonstrate that learning teacher-specific representations is a key to scaling in MTD. Importantly, different design choices, such as learnable teacher tokens, a tailored attention scheme, additional mixture-of-experts layers and a contrastive loss, are proposed to better learn such teacher-specific representations. We show that our method better scales with respect to the number of considered teachers, allowing us to train compact student encoders, up to 10x more efficient than larger teacher foundation models, while matching their performance and even outperforming them on a large set of tile-level and slide-level tasks. We conduct a thorough empirical validation, evaluating more than 10 foundation models on 39 tasks spanning tile and slide levels. As a result, we release a new collection of strong and efficient foundation models, named OMNI, trained from the knowledge of 10 state-of-the-art foundation models. We finally conduct an analysis of the learned teacher-specific representations, highlighting their complementarity and explaining why they can be easily aggregated at downstream time.

Figures & tables

Figure 1: OMNI . We propose a new method to scale Multi-Teacher Distillation (MTD) to more teachers. (a) Unlike vanilla MTD, we learn to extract teacher-specific representations. (b) Downstream performance (tile-level linear probing averaged across 13 tasks) better scales when increasing the diversity of teachers for our method. For a fair evaluation, we rank teachers by their individual performance on the THUNDER benchmark and add them from strongest to weakest. (c) Overall we provide a collection of light vision encoder ( $7-330M$ parameters) models that are competitive with and even outperform all state-of-the-art larger teacher foundation models. In particular, we report here the average of the tile-level and slide-level linear probing performances (see Table 1 for more details) as a function of GFLOPs. Circle sizes indicate the number of parameters of each model.

Figure 2: Method overview . (a) We train teacher-specific embeddings along with the student model with a token loss ( $\mathcal{L}^{\mathrm{cls}}_{\mathrm{MTD},i}$ for each teacher $T_{i}$ ), a dense patch loss ( $\mathcal{L}^{\mathrm{patch}}_{\mathrm{MTD},i}$ ) and a contrastive loss ( $\mathcal{L}^{\mathrm{ctr}}_{i}$ ). (b) The student architecture is composed of a sequence of transformer and Mixture-of-Experts (MoE) layers. (c) Each MoE layer is based on an expert-selection mechanism where top-2 experts are picked conditioned on the input. An additional regularization loss ( $\mathcal{L}_{\mathrm{MoE}}$ ) ensures a diversity in the selection of the experts. (d) All attention operations in the student follow a structured masking. Each teacher token attends to itself and to all patch tokens, but not to other teacher tokens, while patch tokens attend to all tokens.

				Tile-level		Slide-level		Mean
Model	Params.	Dim	GFLOPs	F1 (%)	Rank $(\downarrow)$	Bacc (%)	Rank $(\downarrow)$	Score	Rank $(\downarrow)$
				13 ds.	13 ds.	25 tasks	7 datasets
Foundation models
UNI2-h [ 12 ]	681M	1536	180.4	87.4	4.6	61.5 ∗∗	9.0	74.5	6.8
Virchow2 [ 80 ]	631M	1280	164.6	85.9	6.6	61.5 ∗	8.3	73.7	7.4
H-optimus-1 [ 56 ]	1.1B	1536	296.0	86.3 ∗	6.2	62.5	4.7	74.4	5.4

Table 1: Linear probing performance at tile level (F1 %, 13 datasets), slide level (Bacc %, 7 CPTAC datasets / 25 tasks), and their mean. Mean rank is computed per dataset then averaged. * indicates significance of paired Wilcoxon signed-rank test vs. OMNI-L:

{}^{*}p<0.05

;

{}^{**}p<0.01

. Bold : best; underline : second best per column.

Panel of Figure 3 — Figure 3: Analysis of teacher representations . (a) Cosine reconstruction quality of UNI2-H as the number of teachers increases. As teachers compete for a single shared token, vanilla MTD reconstruction quality degrades significantly, whereas our approach is substantially more robust to the growing number of teachers. (b) UMAP of the $M{=}10$ teacher-specific tokens from OMNI-B on 3 $k$ images. Well-separated clusters confirm each token specializes toward its assigned teacher. (c) Smooth latent interpolation between two teacher-specific tokens (averaged over all teacher pairs), showing a consistent X-shaped crossover pattern as the conditioning vector is linearly interpolated.

Table 5

Model	F1 (%) $(\uparrow)$
Vanilla MTD	85.4
+ Teacher drop. [ 58 ] (1)	85.6
+ Transf. decoder [ 57 ] (2)	85.9
OMNI-B (ours)	87.1
+ (1) + (2)	87.0

Table 5: Main contributions of natural-domain MTD methods (1) UNIC and (2) DUNE, applied on top of vanilla MTD or OMNI-B. F1 averaged over 13 THUNDER tasks.

Table 7

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

				KNN		16-shot
Model	Params.	Dim	GFLOPs	F1 (%)	Rank $(\downarrow)$	F1 (%)	Rank $(\downarrow)$
				13 ds.	13 ds.	13 ds.	13 ds.
UNI2-h	681M	1536	180.4	85.0	4.7	81.9	3.4
Virchow2	631M	1280	164.6	84.7	5.0	75.9	9.6
H-optimus-1	1.1B	1536	296.0	83.9	5.3	78.3	7.1
KEEP	414M	1024	59.7	83.4	7.7	79.3	7.4

Appendix

Table 8: KNN and 16-shot few-shot classification results (F1 %, 13 datasets). Mean rank is computed per dataset then averaged. Bold : best; underline : second best per column.

Figure 4: Scaling with a reverse teacher ordering. Tile-level linear probing F1 (%) averaged across 13 THUNDER tasks as a function of the number of teachers, when adding teachers from weakest to strongest. Our method keeps improving as teachers are added, while vanilla MTD saturates.

Model	GFLOPs	Bacc (CI)
H0-mini [ 18 ]	22.3	94.0 (92.3–95.4)
UNI2-h [ 12 ]	180.4	95.0 (94.0–96.5)
Vanilla MTD	21.97	93.7 (92.3–95.5)
OMNI-B (ours)	21.8	94.9 (93.3–96.1)

Appendix

Table 9: ABMIL slide-level classification on the IMP colorectal dataset (PathoBench). Balanced accuracy (%) with 95% CI and efficiency (GFLOPs).

Figure 5: MoE routing diversity of OMNI-B over

5

K random training tiles (top-2 routing,

E{=}5

). (a) Fraction of images by the maximum number of teacher tokens sharing the same ordered top-2 expert set. (b) Fraction of images by the maximum percentage of patch tokens sharing the same ordered top-2 expert set. Shaded bins mark full routing collapse.

Teacher	Mean grad. cosine w/ others	Min	Max
UNI2-h	0.229	0.049	0.418
Virchow2	0.137	0.035	0.206
H-optimus-1	0.225	0.046	0.423
Prov-GigaPath	0.201	0.050	0.344
Kaiko-ViT-B/8	0.135	0.039	0.209
Hibou-L	0.168	0.062	0.250

Appendix

Table 10: Gradient conflict analysis for vanilla MTD (across 500k iterations).

Figure 6: UNI2-h teacher reconstruction loss. (a) Loss during training with

N{=}10

teachers. Both methods converge smoothly, with OMNI reaching a lower value. (b) Converged loss as a function of the number of teachers

N

. The single-CLS bottleneck of vanilla MTD degrades sharply as

N

grows. With a single teacher, both methods reduce to the same model.

$\lambda_{ctr}=0$	$\lambda_{ctr}=0.1$	$\lambda_{ctr}=0.25$	$\lambda_{ctr}=0.5$
86.6	87.1	87.1	86.8

Appendix

Table 11: Sensitivity to the contrastive loss weight

\lambda_{ctr}

. F1 (%) averaged across 13 THUNDER tasks.

Optimization
Hyperparameter	OMNI-T	OMNI-S	OMNI-B	OMNI-L
Optimizer	Adam
Learning rate	$1\times 10^{-4}$
Training iterations	700k
Batch size	128
Architecture

Appendix

Table 12: Training hyperparameters for OMNI models.

Cohort	Cancer Type	Task	Task Type
CPTAC-BRCA	Breast carcinoma	Immune class	Subtyping
		PIK3CA mutation	Mutation prediction
		TP53 mutation	Mutation prediction
CPTAC-CCRCC	Clear cell renal cell carcinoma	BAP1 mutation	Mutation prediction
		VHL mutation	Mutation prediction
CPTAC-COAD	Colon adenocarcinoma	ACVR2A mutation	Mutation prediction

Appendix

Table 13: Slide-level tasks from the PathoBench benchmark [ 77 ] on CPTAC cohorts. All tasks are evaluated using balanced accuracy (BACC) with a 50-fold train/test split.

Explore similar work

DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance
Aug 18, 2026Ramon Kaspar, Andrey Ignatov, Valentina BoevaPathology Foundation Models Efficient ViTs
Department of Computer Science, ETH Zürich, Switzerland
Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models
Jul 6, 2026Tim Lenz, Maurice Heide, Marco Gustav +2Computational Pathology Pathology Foundation Models
EKFZ for Digital Health, TU Dresden, Dresden, Germany · Pathology, University of Augsburg, Augsburg, Germany · Bavarian Cancer Research Center (BZKF), Augsburg, Germany +3
LaGuadia: Language-Guided Adaptive Distillation from Pathology Foundation Models
Jul 13, 2026Gangsu Kim, Won-Ki JeongComputational Pathology Pathology Foundation Models
Department of Computer Science and Engineering, College of Informatics, Korea University, Republic of Korea

Aug 18, 2026J/K move · Enter open · S save

DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance

Ramon Kaspar, Andrey Ignatov, Valentina Boeva

Department of Computer Science, ETH Zürich, Switzerland

Pathology Foundation Models Efficient ViTs Knowledge Distillation Vision Transformer

Open paper AskPDF arXiv

Many high-performing pathology tile encoders are now foundation models with hundreds of millions to over a billion parameters. Encoding and storing the thousands of tiles in each whole-slide image with such models is costly on commodity hardware, so compact encoders that retain useful downstream performance are a valuable alternative. We present DistillPath-KS16, which starts from the existing 22M kaiko ViT-S/16 encoder and improves it by distilling from released pathology encoders used as frozen teachers. The recipe reads only the teachers' final class and patch tokens and trains on 6,000 public slides, needing neither their DINO nor iBOT pretraining heads nor a billion-tile corpus, so it applies to any released encoder that exposes backbone tokens. We distill four teachers spanning 86M to 1.1B parameters into the same student. Every variant improves the kaiko baseline on all three benchmarks we use, EVA, HEST, and PLISM, and the strongest teacher is task-dependent. On the seven-task EVA mean, DistillPath-KS16-Virchow2 reaches

0.795

, within

0.015

points of Virchow2, the top-scoring model in our evaluation, at about

29\times

fewer parameters; it also scores above H0-mini and GPFM on this aggregate metric, though that advantage is task-concentrated rather than uniform. Because it remains a 22M ViT-S/16 with 384-dimensional features, DistillPath-KS16 runs more than

25\times

faster than Virchow2. Code is available at https://github.com/RamonKaspar/DistillPath, and released model weights are available at https://huggingface.co/collections/RamonK/distillpath.

Abstract

Explore similar work

DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance

Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models

LaGuadia: Language-Guided Adaptive Distillation from Pathology Foundation Models