DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance

Ramon Kaspar; Andrey Ignatov; Valentina Boeva

cs.CVAug 18, 2026

DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance

Authors: Ramon Kaspar, Andrey Ignatov, Valentina Boeva

Organizations: Department of Computer Science, ETH Zürich, Switzerland

Abstract

Many high-performing pathology tile encoders are now foundation models with hundreds of millions to over a billion parameters. Encoding and storing the thousands of tiles in each whole-slide image with such models is costly on commodity hardware, so compact encoders that retain useful downstream performance are a valuable alternative. We present DistillPath-KS16, which starts from the existing 22M kaiko ViT-S/16 encoder and improves it by distilling from released pathology encoders used as frozen teachers. The recipe reads only the teachers' final class and patch tokens and trains on 6,000 public slides, needing neither their DINO nor iBOT pretraining heads nor a billion-tile corpus, so it applies to any released encoder that exposes backbone tokens. We distill four teachers spanning 86M to 1.1B parameters into the same student. Every variant improves the kaiko baseline on all three benchmarks we use, EVA, HEST, and PLISM, and the strongest teacher is task-dependent. On the seven-task EVA mean, DistillPath-KS16-Virchow2 reaches $0.795$ , within $0.015$ points of Virchow2, the top-scoring model in our evaluation, at about $29\times$ fewer parameters; it also scores above H0-mini and GPFM on this aggregate metric, though that advantage is task-concentrated rather than uniform. Because it remains a 22M ViT-S/16 with 384-dimensional features, DistillPath-KS16 runs more than $25\times$ faster than Virchow2. Code is available at https://github.com/RamonKaspar/DistillPath, and released model weights are available at https://huggingface.co/collections/RamonK/distillpath.

Figures & tables

Figure 1: Downstream score versus encoder size. Four teachers are distilled into the same 22M kaiko ViT-S/16. DistillPath-KS16-Virchow2 improves the EVA mean from

0.764

0.795

, within

0.015

of Virchow2 using about

29\times

fewer parameters. (a) EVA mean over seven tasks; (b) mean HEST Pearson

r

over nine tasks.

Figure 2: Method overview. Tiles are streamed online from whole-slide images and encoded by a frozen teacher and the trainable kaiko ViT-S/16 student. We read only the final class and patch tokens. The class token is supervised by a cosine loss (per tile) and a relational RKD loss (across the batch). The patch tokens are supervised by a cosine loss after the teacher grid is resized to the student grid. A shared projector $g$ maps student features to the teacher dimension for the cosine losses.

Teacher	$\gamma_{T}$	$\lambda_{T}$	RKD / CLS	Patch / total
H0-mini	24	0.16	25.2%	28.3%
UNI2-h	32	0.31	24.4%	24.0%
H-optimus-0	84	0.28	25.4%	24.6%
Virchow2	56	0.30	24.5%	24.5%

Table 1: Teacher-specific loss coefficients for the DistillPath-KS16 recipe. The class-token coefficient

\gamma_{T}

scales RKD, and

\lambda_{T}

scales the patch-token cosine loss. The last two columns report realized weighted-loss fractions averaged over steps 40,000–50,000.

Teachers, reference encoders, and baseline
	EVA classification						EVA segmentation
Model	BACH	PCam	CRC	MHIST	BrHis	Gleas.	CoNSeP	MoNu.	EVA ${}_{\text{mean}}$
Virchow2 (632M)	0.879	0.939	0.966	0.861	0.821	0.778	0.640	0.667	0.810
UNI2-h (681M)	0.917	0.951	0.966	0.821	0.859	0.772	0.630	0.643	0.806
H-optimus-0 (1.1B)	0.756	0.942	0.956	0.843	0.806	0.752	0.642	0.681	0.803
Midnight-12k (1.1B)	0.900	0.929	0.966	0.799	0.816	0.799	0.624	0.658	0.799

Table 2: EVA results for DistillPath-KS16 variants, distilled into the same 22M kaiko ViT-S/16. EVA

{}_{\text{mean}}

averages the seven non-BACH tasks; BrHis, Gleas., and MoNu. denote BreakHis, Gleason, and MoNuSAC. Bold marks the best value in each column across all rows; underline marks the best DistillPath-KS16 variant in each column.

Teachers, reference encoders, and baseline
Model	IDC	PRAD	PAAD	SKCM	COAD	READ	ccRCC	LUNG	LYMPH-IDC	HEST ${}_{\text{mean}}$
Virchow2 (632M)	0.592	0.348	0.472	0.619	0.259	0.209	0.274	0.553	0.256	0.398
UNI2-h (681M)	0.590	0.357	0.500	0.659	0.301	0.223	0.264	0.558	0.272	0.414
H-optimus-0 (1.1B)	0.598	0.385	0.491	0.645	0.309	0.222	0.268	0.559	0.259	0.415
Midnight-12k (1.1B)	0.582	0.337	0.490	0.636	0.291	0.185	0.213	0.558	0.264	0.395
GPFM (303M)	0.566	0.342	0.460	0.589	0.248	0.165	0.259	0.547	0.237	0.379

Table 3: HEST gene-expression prediction results. Values are Pearson correlations for the nine HEST tasks, followed by the mean. Bold marks the best value in each column across all rows; underline marks the best DistillPath-KS16 variant in each column.

Teachers, reference encoders, and baseline
Model	PLISM score	All pairs cosine	All pairs top-5	Scanner top-5	Stain top-5	Scanner+stain top-5
Virchow2 (632M)	0.447	0.744	0.094	0.516	0.203	0.076
UNI2-h (681M)	0.333	0.592	0.033	0.421	0.123	0.023
H-optimus-0 (1.1B)	0.480	0.686	0.124	0.668	0.240	0.103
Midnight-12k (1.1B)	0.337	0.743	0.060	0.276	0.111	0.051
GPFM (303M)	0.264	0.594	0.009	0.253	0.049	0.006

Table 4: PLISM robustness results on the 8,139-tile protocol. PLISM score is the benchmark aggregate: the mean of all-pairs median cosine similarity and median top-10 retrieval accuracy for scanner-only, staining-only, and paired scanner-plus-staining changes. The remaining columns are diagnostic summaries: median cosine similarity and median top-5 retrieval accuracy. Bold marks the best value in each column across all rows; underline marks the best DistillPath-KS16 variant in each column.

Model	Params	Dim	EVA ${}_{\text{mean}}$	GFLOPs/tile	RTX 4090		M4 Pro		CUDA peak	MPS alloc.	Storage
					tiles/s	slowdown	tiles/s	slowdown	(MiB)	(MiB)	(GiB/1M)
DistillPath-KS16-Virchow2	21.7M	384	0.795	9.2	7994	$1.0\times$	396.1	$1.0\times$	374	138	1.43
H0-mini	85.7M	768	0.784	47.1	1807	$4.4\times$	80.7	$4.9\times$	1082	592	2.86
GPFM	303.2M	1024	0.789	162.0	602	$13.3\times$	26.6	$14.9\times$	2388	1666	3.81
Virchow2	631.2M	1280	0.810	340.1	300	$26.6\times$	12.8	$30.9\times$	4551	2662	4.77
H-optimus-0	1134.8M	1536	0.803	608.3	181	$44.2\times$	7.4	$53.5\times$	7618	5146	5.72

Table 5: Encoder-forward efficiency on TCGA tiles at batch size 64. The RTX 4090 benchmark uses CUDA/bfloat16 and 500 timed batches; the MacBook Pro benchmark uses Apple M4 Pro MPS/fp32 and 200 timed batches. Slowdown is relative to DistillPath-KS16 on the same device, so larger values are worse. CUDA peak reports peak allocated CUDA memory, while MPS alloc. reports allocated MPS memory recorded by the benchmark. FLOPs are estimates for linear, convolution, and ViT attention matmuls, excluding normalization, softmax, activation, residual, and preprocessing operations. Storage is fp32 feature storage per one million tile embeddings.

Figure 3: Distillation trajectories for the four teachers into kaiko ViT-S/16, across five checkpoints. (a) EVA mean over seven tasks; (b) mean HEST Pearson

r

over nine tasks. The dashed line is the kaiko baseline. Virchow2 leads on EVA at later checkpoints, while H0-mini leads on HEST, so the best student depends on the target task.

	EVA classification						EVA segmentation
Model	BACH	PCam	CRC	MHIST	BrHis	Gleas.	CoNSeP	MoNu.	EVA ${}_{\text{mean}}$	HEST	PLISM
kaiko ViT-S/16 baseline	0.832	0.901	0.939	0.830	0.720	0.723	0.600	0.633	0.764	0.349	0.307
ViT-S/16 IN21K baseline	0.612	0.856	0.904	0.815	0.723	0.709	0.515	0.581	0.729	0.311	0.383
DistillPath-IS16-Virchow2	0.828	0.919	0.959	0.773	0.754	0.762	0.586	0.587	0.763	0.358	0.490
DistillPath-IS16-HOpt0	0.705	0.917	0.951	0.796	0.765	0.736	0.599	0.611	0.768	0.363	0.526
DistillPath-IS16-H0mini	0.764	0.920	0.952	0.784	0.688	0.735	0.605	0.597	0.754	0.377	0.543

Table 6: Student initialization comparison after the same 50,000-step distillation setup. IS16 denotes an ImageNet-21k ViT-S/16 student; DistillPath-IS16-X denotes the same ImageNet-initialized student distilled from teacher X. EVA

{}_{\text{mean}}

averages the seven non-BACH EVA tasks as in Table 2 ; BrHis, Gleas., and MoNu. denote BreakHis, Gleason, and MoNuSAC. HEST is the mean Pearson correlation across nine tasks, and PLISM is the aggregate robustness score.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: TCGA cohort distribution in the 6,000-slide distillation set, ordered by share. The six slides without cohort labels are omitted from the bars; percentages use all 6,000 slides as the denominator.

Figure 5: Share of accepted tile views per magnification among the 12.81 million views logged at step 50,000 of the DistillPath-KS16-Virchow2 run. Scanner-reported microns-per-pixel values are grouped into nominal bins spanning

[0.10,0.35)

[0.35,0.70)

[0.70,1.35)

, and

[1.35,2.35)

, respectively; values outside these intervals and missing metadata form the other / unknown bin.

Setting	Value
Slide reader	TiffSlide
Tissue mask	CLAM detector with Otsu thresholding on a thumbnail bounded to $2048\times 2048$ px
CLAM contour filtering	median-blur kernel $7$ , closing kernel $4$ , $a_{t}=16$ , $a_{h}=4$ , at most 8 holes per contour, reference patch size 512
Magnification sampling	$0.25$ , $0.5$ , $1.0$ , or $2.0$ microns per pixel, selected uniformly
Pyramid-level selection	nearest available level to the target scale; level 0 when slide-scale metadata are unavailable
Spatial sampling	random coordinates sampled with replacement; at least $40\%$ tissue within the candidate tile

Appendix

Table 7: wsistream 0.1.5 configuration used for DistillPath.

Role	Model	Dim	Patch	Registers	Normalization preset
Student	kaiko ViT-S/16	384	16	0	kaiko
Student	ViT-S/16 IN21K	384	16	0	ImageNet
Teacher	H0-mini	768	14	4	H-optimus
Teacher	Virchow2	1280	14	4	ImageNet
Teacher	UNI2-h	1536	14	8	ImageNet
Teacher	H-optimus-0	1536	14	4	H-optimus

Appendix

Table 8: Encoder configurations. All encoders use

224\times 224

inputs during distillation and evaluation. The same augmented tile is passed to teacher and student, and each model wrapper applies its own normalization statistics. Register tokens are excluded from the distillation loss.

Setting	Value
Class-token loss	cosine $+$ RKD
Class-token cosine coefficient	$1.0$
RKD coefficient $\gamma_{T}$	run-specific (Table 10 )
RKD distance term weight	$1.0$
RKD angle term weight	$2.0$
RKD penalty	Huber (smooth $L_{1}$ ), $\delta=1$

Appendix

Table 9: Distillation objective and projector. The projector maps student tokens into the teacher dimension

d_{t}

for the class- and patch-token cosine losses and is discarded after training. RKD is computed within the original student and teacher spaces and does not use the projector.

Student initialization	Teacher	$\gamma_{T}$	$\lambda_{T}$
kaiko ViT-S/16	H0-mini	24	0.16
	Virchow2	56	0.30
	UNI2-h	32	0.31
	H-optimus-0	84	0.28
ViT-S/16 IN21K	H0-mini	24	0.16
	Virchow2	28	0.39

Appendix

Table 10: Loss coefficients used in each teacher–student run. The class-token coefficient

\gamma_{T}

scales RKD, and

\lambda_{T}

scales the patch-token cosine loss.

Setting	Value
Optimizer	AdamW
$\beta_{1},\ \beta_{2}$	$0.9,\ 0.999$ (PyTorch default)
$\epsilon$	$10^{-8}$ (PyTorch default)
Peak learning rate	$10^{-4}$
Final learning rate	$10^{-6}$
Weight decay	$0.05$

Appendix

Table 11: Optimization settings and schedule.

Model	Step	BACH	PCam	CRC	MHIST	BrHis	Gleas.	CoNSeP	MoNu.	EVA ${}_{\text{mean}}$
Reference encoders and baselines
Virchow2 (632M)	–	0.879	0.939	0.966	0.861	0.821	0.778	0.640	0.667	0.810
UNI2-h (681M)	–	0.917	0.951	0.966	0.821	0.859	0.772	0.630	0.643	0.806
H-optimus-0 (1.1B)	–	0.756	0.942	0.956	0.843	0.806	0.752	0.642	0.681	0.803
Midnight-12k (1.1B)	–	0.900	0.929	0.966	0.799	0.816	0.799	0.624	0.658	0.799
GPFM (303M)	–	0.829	0.945	0.953	0.813	0.764	0.763	0.637	0.650	0.789

Appendix

Table 12: Full EVA results across all checkpoints. EVA

{}_{\text{mean}}

averages the seven non-BACH tasks.

Model	Step	IDC	PRAD	PAAD	SKCM	COAD	READ	ccRCC	LUNG	LYMPH-IDC	HEST ${}_{\text{mean}}$
Reference encoders and baselines
Virchow2 (632M)	–	0.592	0.348	0.472	0.619	0.259	0.209	0.274	0.553	0.256	0.398
UNI2-h (681M)	–	0.590	0.357	0.500	0.659	0.301	0.223	0.264	0.558	0.272	0.414
H-optimus-0 (1.1B)	–	0.598	0.385	0.491	0.645	0.309	0.222	0.268	0.559	0.259	0.415
Midnight-12k (1.1B)	–	0.582	0.337	0.490	0.636	0.291	0.185	0.213	0.558	0.264	0.395
GPFM (303M)	–	0.566	0.342	0.460	0.589	0.248	0.165	0.259	0.547	0.237	0.379

Appendix

Table 13: Full HEST gene-expression results (Pearson

r

) across all checkpoints.

Model	Step	PLISM	Cosine	Top-5	Scanner	Stain	Scan+stain
Reference encoders and baselines
Virchow2 (632M)	–	0.447	0.744	0.094	0.516	0.203	0.076
UNI2-h (681M)	–	0.333	0.592	0.033	0.421	0.123	0.023
H-optimus-0 (1.1B)	–	0.480	0.686	0.124	0.668	0.240	0.103
Midnight-12k (1.1B)	–	0.337	0.743	0.060	0.276	0.111	0.051
GPFM (303M)	–	0.264	0.594	0.009	0.253	0.049	0.006

Appendix

Table 14: Full PLISM robustness results on the 8,139-tile protocol across all checkpoints. PLISM is the mean of all-pairs median cosine similarity and median top-10 retrieval accuracy under scanner-only, staining-only, and paired changes. The remaining columns report median cosine similarity and median top-5 retrieval accuracy.

Explore similar work

Scaling Multi-Teacher Distillation for Digital Pathology
Oct 8, 2026Sofiène Boutaj, Pierre Marza, Varun Belagali +3Multi-Teacher Knowledge Distillation Pathology Foundation Models
Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, IHU PRISM, Cancer Data Science Unit, France · Université Paris-Saclay, CentraleSupélec, MICS Laboratory, France · Department of Computer Science, Stony Brook University, USA
Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models
Jul 6, 2026Tim Lenz, Maurice Heide, Marco Gustav +2Computational Pathology Pathology Foundation Models
EKFZ for Digital Health, TU Dresden, Dresden, Germany · Pathology, University of Augsburg, Augsburg, Germany · Bavarian Cancer Research Center (BZKF), Augsburg, Germany +3
TAP-Path: Task-Adaptive Structural and Token Pruning for Efficient and Trustworthy Pathology Foundation Models
Sep 3, 2026Mehedi Hasan, Ashfak Yeafi, Md Khairul IslamComputational Pathology Pathology Foundation Models
Department of Computer Science and Engineering, Brac University, Dhaka, Bangladesh · EEE, Khulna University of Engineering & Technology, Khulna, Bangladesh · Mathematics and Computer Science, Hobart and William Smith Colleges, Geneva, NY, USA

Oct 8, 2026J/K move · Enter open · S save

Scaling Multi-Teacher Distillation for Digital Pathology

Sofiène Boutaj, Pierre Marza, Varun Belagali, Dimitris Samaras+2

Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, IHU PRISM, Cancer Data Science Unit, France · Université Paris-Saclay, CentraleSupélec, MICS Laboratory, France · Department of Computer Science, Stony Brook University, USA

Multi-Teacher Knowledge Distillation Pathology Foundation Models Computational Pathology

Open paper AskPDF arXiv

Multi-teacher distillation allows transferring knowledge from multiple "teacher" networks to a single "student" network. This method is promising in fields such as digital pathology, where many powerful foundation models were proposed recently. However, as we show in this paper, the standard multi-teacher distillation approach reacts poorly to an increase in the number of teachers used to train a student encoder. We demonstrate that learning teacher-specific representations is a key to scaling in MTD. Importantly, different design choices, such as learnable teacher tokens, a tailored attention scheme, additional mixture-of-experts layers and a contrastive loss, are proposed to better learn such teacher-specific representations. We show that our method better scales with respect to the number of considered teachers, allowing us to train compact student encoders, up to 10x more efficient than larger teacher foundation models, while matching their performance and even outperforming them on a large set of tile-level and slide-level tasks. We conduct a thorough empirical validation, evaluating more than 10 foundation models on 39 tasks spanning tile and slide levels. As a result, we release a new collection of strong and efficient foundation models, named OMNI, trained from the knowledge of 10 state-of-the-art foundation models. We finally conduct an analysis of the learned teacher-specific representations, highlighting their complementarity and explaining why they can be easily aggregated at downstream time.

Abstract

Explore similar work

Scaling Multi-Teacher Distillation for Digital Pathology

Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models

TAP-Path: Task-Adaptive Structural and Token Pruning for Efficient and Trustworthy Pathology Foundation Models