Learning Where to Focus: Self-Supervised Multi-Scale ViTs for Histopathology

Anabel Stammer; Valay Bundele; Mehran Hosseinzadeh; Hendrik P. A. Lensch

cs.CVSep 16, 2026

Learning Where to Focus: Self-Supervised Multi-Scale ViTs for Histopathology

Authors: Anabel Stammer, Valay Bundele, Mehran Hosseinzadeh, Hendrik P. A. Lensch

Organizations: Eberhard Karls Universität Tübingen, Germany

Abstract

Pathologists diagnose diseases by first locating suspicious tissue and then examining it at higher magnification, whereas self-supervised vision transformers (ViTs) allocate the same spatial resolution to every image region despite diagnostic evidence being sparse and spanning multiple biological scales. Recent pathology foundation models have substantially improved representation quality by scaling training data and model capacity, but largely retain uniform tokenization. We instead investigate whether pathology representations can be improved by learning where to allocate spatial resolution during self-supervised learning. To this end, we propose CRAFT (Coarse-to-fine Region-Adaptive Feature Tokenization), a DINO-based framework that learns image-dependent mixed-scale representations by using self-supervised attention to selectively refine informative regions while preserving coarse context, together with a symmetric cross-scale regularization objective that encourages complementary coarse and fine representations. Across CAMELYON16, TCGA-Lung subtype classification, and TCGA-LUAD survival prediction, CRAFT consistently outperforms comparable-scale self-supervised methods while requiring lower inference computation. Despite using only a compact 22M parameter backbone trained on comparatively small pathology datasets, CRAFT remains competitive with, and often surpasses, substantially larger pathology foundation models.

Figures & tables

Figure 1: (a) Class attention maps guide mixed-scale tokenization by refining salient regions while keeping others coarse. (b) Tumor recall via linear probing on CAMELYON16 after SSL pretraining at two token sizes (16 vs 32).

Figure 2: (a) Mixed-scale tokenization (green, Ours ) is inserted into both DINO branches before the ViT; the rest is standard DINO, with gradients through the student only. Coarse and refined predictions $P_{c}^{s}$ , $P_{r}^{s}$ are aligned by $\mathcal{L}_{\mathrm{KL\text{-}sym}}$ (Eq. 5 ). (b) Coarse-to-fine pipeline: Stage 1 encodes coarse tokens and selects key regions by class attention; Stage 2 re-splits and fuses them ( $\oplus$ ).

Method	Encoder	Pretrain	DS	AB	Trans	Max	Avg	#P	GMACs
Reproduced under our 5-seed MIL protocol
UNI † [ 10 ]	ViT-L/16	DINOv2	96.06 $\pm$ 1.85	90.04 $\pm$ 2.62	97.26 $\pm$ 0.74	97.10 $\pm$ 1.19	95.12	307M	59.70
Virchow † [ 47 ]	ViT-H/14	DINOv2	95.00 $\pm$ 1.74	93.95 $\pm$ 2.60	94.90 $\pm$ 1.68	93.59 $\pm$ 0.63	94.36	632M	161.99
CONCH † [ 34 ]	ViT-B/16	iBOT+CoCa	92.47 $\pm$ 0.98	92.34 $\pm$ 0.89	95.77 $\pm$ 1.16	90.49 $\pm$ 0.33	92.77	86M	16.87
H0-mini † [ 18 ]	ViT-B/14	DINOv2	98.23 $\pm$ 0.81	97.58 $\pm$ 0.34	97.87 $\pm$ 0.37	99.22 $\pm$ 0.00	98.23	86M	22.31
CTransPath † [ 50 ]	Swin-T	SRCL	85.62 $\pm$ 2.34	88.08 $\pm$ 1.95	96.31 $\pm$ 0.56	90.45 $\pm$ 2.22	90.12	28M	4.51

Table 1: WSI classification AUC (%) on CAMELYON16 (mean

\pm

std over 5 runs). Among reproduced rows, best bold , second underlined ;

\dagger

external pretraining; Avg = mean over DS/AB/Trans/Max MILs.

	Method	Encoder	Pretraining	Mean AUC	#P
Frozen public Encoder trained on Massive Data
Massive Data	UNI [ 10 ]	ViT-L/16	DINOv2, Mass-100K	96.81	307M
	Virchow [ 47 ]	ViT-H/14	DINOv2, MSK-1.5M	96.67	632M
	H0-mini ¶ [ 18 ]	ViT-B/14	DINOv2 distill, TCGA	97.79	86M
	CTransPath ¶ [ 50 ]	Swin-T	SRCL, TCGA+PAIP	92.81	28M
	RetCCL ¶	ResNet-50	TCGA+PAIP (CCL)	$85.61$	23.5M

Table 2: WSI subtype classification on TCGA-Lung (LUAD vs. LUSC), AUC (%) on the fixed 256-slide test set, averaged over four MIL aggregators. Each aggregator value is the mean over five model-selection folds. Top: public encoders trained on massive data; bottom: low-data 22M encoders. Best bold , second underlined among 22M encoders.

\lx@paragraphsign

pretraining overlaps our TCGA test split, so AUC may be inflated.

Figure 3: Survival prediction on TCGA-LUAD (selected encoders). Bars show mean $\pm$ std across 5 folds; color encodes compute (GMACs).

Table 6

Panel of Figure 4 — Figure 4: (a) Mixed-scale tokenization on CAMELYON16 patches: key regions are refined to fine tokens while background and stroma stay coarse. (b) Self-attention maps for selected heads, comparing DINO (fine) with CRAFT under selective and coarse inference; input shown left.

Encoder	Arch. (dim)	KRAS	EGFR	TP53	STK11	Macro
Virchow	ViT-H/14 (2560)	54.3 $\pm 5.9$	54.0 $\pm 4.3$	60.6 $\pm 5.8$	49.7 $\pm 11.0$	54.65
H0-mini	ViT-B/14 (1536)	56.4 $\pm 5.1$	54.6 $\pm 7.6$	69.4 $\pm 6.2$	64.1 $\pm 6.7$	61.13
UNI	ViT-L/16 (1024)	56.3 $\pm 4.7$	55.6 $\pm 3.7$	63.7 $\pm 5.1$	55.2 $\pm 5.4$	57.71
CRAFT-mixed	ViT-S/16 (384)	57.8 $\pm 4.5$	56.8 $\pm 6.0$	65.5 $\pm 3.4$	61.3 $\pm 8.9$	60.35

Table 5: Driver-gene mutation prediction on TCGA-LUAD. Weakly-supervised ROC-AUC (%), DSMIL [ 31 ] aggregation; encoder is the only variable. Mean

\pm

std over 5 patient-level, label-stratified folds; Macro = mean over genes. Bold and Underlined show the first and second best methods per column.

	Method	Encoder	Pretraining	DS	AB	Trans	Max	Avg	#P
Frozen public Encoder trained on Massive Data
Massive Data	UNI [ 10 ]	ViT-L/16	DINOv2, Mass-100K	97.82 $\pm 0.49$	95.47 $\pm 1.87$	98.10 $\pm 1.15$	95.83 $\pm 0.34$	96.81	307M
	Virchow [ 47 ]	ViT-H/14	DINOv2, MSK-1.5M	97.53 $\pm 0.21$	95.37 $\pm 1.12$	97.80 $\pm 1.37$	95.96 $\pm 0.28$	96.67	632M
	CTransPath ¶ [ 50 ]	Swin-T	SRCL, TCGA+PAIP	93.80 $\pm 0.28$	91.21 $\pm 2.20$	$96.88_{\pm 0.99}$	89.34 $\pm 0.46$	92.81	28M
	RetCCL ¶	ResNet-50	TCGA+PAIP (CCL)	$87.80_{\pm 1.26}$	$83.57_{\pm 1.38}$	$95.14_{\pm 1.97}$	$75.92_{\pm 2.11}$	$85.61$	23.5M
	H0-mini ¶ [ 18 ]	ViT-B/14	DINOv2 distill, TCGA	98.35 $\pm 0.19$	96.90 $\pm 1.10$	98.18 $\pm 0.90$	97.71 $\pm 0.24$	97.79	86M

Table 6: Per-aggregator WSI subtype classification on TCGA-NSCLC (LUAD vs. LUSC), AUC (%). DS/AB/Trans/Max = DSMIL, ABMIL, TransMIL, Max-pooling; Avg = mean of the four. Among 22M encoders ( Low Data ), best bold , second underlined .

\lx@paragraphsign

TCGA overlap may inflate AUC.

Model	Encoder	C-index	GMACs
DINO-HistoPretrain † [ 55 ]	ViT-B/16	56.9 $\pm 5.5$	17.6
GigaPath † [ 53 ]	ViT-G/14	$56.2\pm 0.52$	228.1
UNI † [ 10 ]	ViT-L/16	$\textbf{57.1}\pm 9.1$	61.55
H0-mini † [ 18 ]	ViT-B/14	$56.5\pm 2.43$	23.5
HIPT † [ 9 ]	Hier. ViT	$53.8\pm 4.0$	4.6
CTransPath † [ 50 ]	Swin-T	$51.2\pm 5.0$	4.5

Table 7: Survival prediction (C-index) on TCGA-LUAD.

\dagger

External pretraining. All methods use CLAMSurvival [ 55 ] ; C-indices except Ours are quoted from [ 55 ] . GMACs: single

224\times 224

forward pass (ResNet-50/ViT-B/Swin-T from literature; HIPT dominated by its cell-level ViT-S/16,

\approx

4.6).

	Coarse–fine		Val.	Test (mixed-scale [CLS] )		Label-free
$\lambda$ (KL ${}_{\text{sym}}$ )	cls_sim	KL ( $\times 10^{-3}$ )	Bal. Acc.	Bal. Acc.	Macro-F1	Eff. rank $\uparrow$	Part. ratio $\uparrow$
0.5 (default)	0.993	0.16	98.81	90.97	90.46	13.3	6.7
2.0	0.975	0.35	98.09	89.64	89.55	10.4	5.5
5.0	0.824	54.3	89.47	77.87	77.32	6.1	3.6

Table 8:

\lambda

-sensitivity on NCT-CRC-HE-100K (mixed-scale [CLS] ; single run per row). Row shading marks the regime: safe

\to

caution

\to

over-regularized collapse . cls_sim/KL are model-level (final epoch). ∗ Operating point selected on validation Bal. Acc.

Panel of Figure 5 — Figure 5: $\lambda$ -sensitivity of CRAFT (ViT-S/16; single run per $\lambda$ ) on NCT-CRC-HE-100K / CRC-VAL-HE-7K.

Linear Probing

MIL

Model

\mathcal{L}_{\mathrm{KL\text{-}asym}}

\mathcal{L}_{\mathrm{KL\text{-}sym}}

ACC

Prec

Recall

Avg AUC

Avg Recall

Max

GMACs

CRAFT

✓

\times

98.16

92.3

80.7

96.74

97.67

\pm

0.48

5.76

CRAFT

\times

✓

\times

97.97

94.2

75.6

95.59

94.80

95.66

\pm

1.18

5.76

MF effect (mixed,

16/32

)

CRAFT

16/32

✓

\times

97.80

93.8

73.2

95.65

94.19

95.66

\pm

2.76

4.01

CRAFT

16/32

\times

✓

\times

97.73

92.9

73.0

92.76

91.02

94.37

\pm

0.63

4.01

Table 9: Component ablation of CRAFT. Each block removes one component from a CRAFT operating point (full

\to

ablated);

\Delta

is the change on the key metric of that block. LP Recall is tumor recall.

Figure 6: Self-attention maps from the six final-layer teacher heads for three tissue regions (input on the left). Top row of each pair: selective grid; bottom row: coarse grid. Brighter is higher attention.

Figure 7: Visualization of adaptive mixed-scale tokenization applied to histopathology patches. High-attention regions, typically rich in cellular structures, are refined into fine-grained tokens, while background and less informative stromal areas remain at a coarse resolution. This hierarchical tokenization strategy preserves diagnostically relevant morphological features while optimizing computational efficiency.

Figure 8: Teacher–student selection agreement. Top: teacher branch; bottom: student branch. Left: global CLS attention. Center: selected tokens overlaid on the input. Right: CLS attention. Both branches attend to and select the same regions, showing the two-branch design preserves a consistent token selection.

Figure 9: Mixed-scale tokenization vs. tumor annotations on CAMELYON16. Each pair (a–d): CRAFT tokenization grid (left) and pathologist annotation of tumor outlined in blue (right). Finer tokens follow dense/high-contrast morphology, not the annotated tumor extent.

Explore similar work

From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathology
Aug 4, 2026Basit Alawode, Moshira Ali Abdalla, Dwarikanath Mahapatra +2Representation Learning Vision Transformer
Khalifa University of Science and Technology, UAE · University of Western Australia, Australia
DaX: Learning General Pathology Representations Across Scales
Jun 5, 2026Bokai Zhao, Yiyang Zhang, Long Bai +3Pathology Foundation Models Medical Image Representation Learning
DAMO Academy, Alibaba Group · Institute of Automation, Chinese Academy of Sciences · Hupan Lab
TVT-PAPD: Pathology-Aware Prototype Distillation for Self-Supervised Whole Slide Image Classification
Jul 11, 2026Ramesh Naidu Laveti, Jaya Sreevalsan-Nair, T K SrikanthPrototype Learning Histopathology Image Classification
E-Health Research Center, International Institute of Information Technology Bangalore, Karnataka 560100, India.

Aug 4, 2026J/K move · Enter open · S save

From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathology

Basit Alawode, Moshira Ali Abdalla, Dwarikanath Mahapatra, Muzammal Naseer+1

Khalifa University of Science and Technology, UAE · University of Western Australia, Australia

Representation Learning Vision Transformer Pathology Foundation Models Histopathology Image Classification Whole-Slide Image Analysis

Open paper AskPDF arXiv

Vision Transformers (ViTs) and their hierarchical variants have achieved strong performance in Computational Pathology (CPath). However, most are pre-trained on single-resolution Whole Slide Images (WSIs), limiting their generalization across arbitrary resolutions. Gigapixel WSIs inherently contain diagnostic patterns at multiple scales, including cellular morphologies, tissue architectures, and global context, mirroring how expert pathologists examine WSIs. We introduce Multi-Resolution Pyramid Transformer (MRPT), a model that hierarchically aggregates multi-resolution information from cellular to tissue and WSI levels. MRPT employs a biologically meaningful Consecutive Cross-Resolution Attention (CCRA) mechanism to capture scale-independent interactions and enforces multi-resolution semantic consistency by aligning embeddings across resolutions, yielding robust and generalizable WSI representations. Pre-trained in a multi-resolution self-supervised manner on 624M patches, 2.4M regions, and 36K WSIs, MRPT learns rich coarse-to-fine histopathology features. Extensive experiments on 34 diverse datasets show that MRPT surpasses recent foundation models and Multimodal Large Language Models (MLLMs) in cancer subtype classification, tissue phenotyping, and Visual Question Answering (VQA) for WSI understanding.

Abstract

Explore similar work

From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathology

DaX: Learning General Pathology Representations Across Scales

TVT-PAPD: Pathology-Aware Prototype Distillation for Self-Supervised Whole Slide Image Classification