MuViT: Multi-Resolution Vision Transformers for Learning Across Scales in Microscopy

Albert Dominguez Mantes; Gioele La Manno; Martin Weigert

cs.CVFeb 27, 2026

MuViT: Multi-Resolution Vision Transformers for Learning Across Scales in Microscopy

Authors: Albert Dominguez Mantes, Gioele La Manno, Martin Weigert

Organizations: Swiss Federal Institute of Technology Lausanne, Switzerland · Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Technische Universität Dresden, Germany

Abstract

Modern microscopy routinely produces gigapixel images that contain structures across multiple spatial scales, from fine cellular morphology to broader tissue organization. Many analysis tasks require combining these scales, yet most vision models operate at a single resolution or derive multi-scale features from one view, limiting their ability to exploit the inherently multi-resolution nature of microscopy data. We introduce MuViT, a transformer architecture built to fuse true multi-resolution observations from the same underlying image. MuViT embeds all patches into a shared world-coordinate system and extends rotary positional embeddings to these coordinates, enabling attention to integrate wide-field context with high-resolution detail within a single encoder. Across synthetic benchmarks, kidney histopathology, and high-resolution mouse-brain microscopy, MuViT delivers consistent improvements over strong ViT and CNN baselines. Multi-resolution MAE pretraining further produces scale-consistent representations that enhance downstream tasks. These results demonstrate that explicit world-coordinate modelling provides a simple yet powerful mechanism for leveraging multi-resolution information in large-scale microscopy analysis.

Figures & tables

Figure 1: Overview of the MuViT architecture. a) MuViT processes multiple crops of the same scene at different physical resolutions and encodes them jointly using a shared transformer with resolution-specific RoPE position encodings. As a decoder we use b) a lightweight masked autoencoding decoder that reconstructs the masked patches for each resolution level, or c) a semantic segmentation decoder that predicts the class of each pixel at the target resolution ( e.g . level 1).

Method	Hierarchical	Physical	Joint	World Coordinate
	Stages	Multi-Res Input	Attention	Alignment
MultiMAE	✗	✗	✓	✗
Scale-MAE	✗	✗	✗	✗
PVT / Swin	✓	✗	✗	✗
HIPT	✓	✓	✗	✗
MuViT (Ours)	✗	✓	✓	✓

Table 1: Conceptual comparison of relevant related work.

Figure 2: MuViT -MAE reconstruction results on a) Mouse and b) KPIS . Columns show masked input and reconstruction at multiple resolution levels ( $l_{1}=1$ top, $l_{2}=8$ middle, $l_{3}=32$ bottom). The same overall masking ratio ( $\rho=0.75$ ) is used across examples, but each column demonstrates different Dirichlet-sampled per-level masking distributions.

Dataset	Method	Level 1	Level 8	Level 32	Level 64
Mouse	MuViT [l]	$5.192\times 10^{-3}$	$1.628\times 10^{-2}$	$1.671\times 10^{-2}$	-
	MuViT [1,8,32] (naive)	$\mathbf{4.848\times 10^{-3}}$	$\mathbf{1.589\times 10^{-2}}$	$\mathbf{1.373\times 10^{-2}}$	-
	MuViT [1,8,32]	$8.734\times 10^{-3}$	$2.274\times 10^{-2}$	$1.464\times 10^{-2}$	-
KPIS	MuViT [l]	$4.352\times 10^{-3}$	$1.122\times 10^{-2}$	$1.013\times 10^{-2}$	$1.021\times 10^{-2}$
	MuViT [1,8,32,64] (naive)	$3.410\times 10^{-3}$	$9.998\times 10^{-3}$	$8.816\times 10^{-3}$	$7.302\times 10^{-3}$
	MuViT [1,8,32,64]	$\mathbf{3.268\times 10^{-3}}$	$\mathbf{9.387\times 10^{-3}}$	$\mathbf{7.941\times 10^{-3}}$	$\mathbf{6.913\times 10^{-3}}$

Table 2: MAE ablation study on Mouse and KPIS . Reconstruction mean squared error is reported on 100 nested multi-resolution samples per image (test set). MuViT [l] denotes an MAE model that only receives the evaluated level.

Method	Input Size	Epoch
		E1	E10	E25	E50
U-Net	$1024\!\times\!1024$	0.086	0.110	0.288	0.412
DeepLabV3	$1024\!\times\!1024$	0.234	0.582	0.634	0.704
SegFormer	$1024\!\times\!1024$	0.045	0.159	0.266	0.332
SwinUNETRV2	$1024\!\times\!1024$	0.049	0.232	0.305	0.388
MuViT [1] +Mask2Former	$256\!\times\!256$	0.033	0.142	0.266	0.310

Table 3: Mouse training progress results. Validation mDSC (11 classes) is reported at different epochs during training.

Figure 3: Semantic Segmentation results on Synthetic . Columns show: input image, ground truth, and semantic predictions for different architectures (U-Net, DeepLabV3, MuViT [1] , MuViT [1,4] +UNETR (naive, i.e . without cross-scale consistent positional encodings), and MuViT [1,4] +UNETR). Top: full image ( $2k\!\times\!2k$ px). Bottom: magnified regions (white boxes) showing detail comparison.

Method	outer	inner	mDSC
U-Net	0.5433	0.2600	0.4016
DeepLabV3	0.5473	0.4317	0.4895
MuViT [1] +UNETR	0.5391	0.4615	0.5003
MuViT [1,4] +UNETR (naive)	0.4662	0.3065	0.3864
MuViT [1,4] +UNETR	0.9499	0.9577	0.9538

Table 4: Semantic segmentation results on Synthetic . Reported are Dice scores for each class (outer, inner) and mDSC. All baselines operate at the highest resolution (corresponding to

l=1

Figure 4: Semantic segmentation results on Mouse . Columns show: input image, ground truth, U-Net, DeepLabV3, and MuViT [1,8,32] +UNETR semantic predictions. Top: full image ( $13k\!\times\!10k$ px). Bottom: magnified regions (white boxes) showing detail comparison.

Method	Input Size	Semantic class											mDSC
		Hippocampus	Thalamus	Hypothalamus	Septal complex	Caudoputamen	Dorsal cortex	Ventral cortex	Lateral cortex	Corpus callosum	Ventricular system	Midbrain
U-Net [ 26 ]	$1024\times 1024$	0.777	0.489	0.515	0.112	0.802	0.780	0.546	0.216	0.533	0.480	0.149	0.491
	$512\times 512$	0.651	0.310	0.409	0.000	0.742	0.677	0.378	0.189	0.447	0.268	0.000	0.370
	$256\times 256$	0.027	0.000	0.121	0.000	0.274	0.234	0.141	0.000	0.135	0.000	0.000	0.085
DeepLabV3 [ 4 ]	$1024\times 1024$	0.941	0.863	0.809	0.734	0.908	0.943	0.874	0.756	0.782	0.857	0.807	0.843
	$512\times 512$	0.894	0.675	0.660	0.278	0.851	0.878	0.721	0.472	0.670	0.747	0.585	0.676

Table 5: Semantic segmentation results on Mouse (test set). Shown are DSC for each anatomical class as well as averaged (mDSC). All baselines operate at the highest resolution (corresponding to

l=1

Figure 5: Semantic segmentation results on KPIS . Depicted in different columns are the input image, ground truth, U-Net, MuViT [1] +UNETR, and MuViT [1,8] +UNETR predictions. Top: full image ( $52k\!\times\!56k$ px); bottom: magnified regions (white boxes) showing detail comparison.

Method	Input size	DSC
nnU-Net ∗ [ 16 ]	$1024\!\times\!1024$	0.6219
UNETR ∗ [ 12 ]	$1024\!\times\!1024$	0.6425
SegFormer ∗ [ 35 ]	$1024\!\times\!1024$	0.6501
DeepLabV3 ∗ [ 4 ]	$1024\!\times\!1024$	0.6822
SwinUNETR-V2 ∗ [ 14 ]	$1024\!\times\!1024$	0.7158
SAM-ViT-B ∗ [ 18 ]	$1024\!\times\!1024$	0.7519

Table 6: Binary semantic segmentation results on KPIS (test set). * indicates the score was taken from [ 32 ] . All baselines operate at the highest resolution (corresponding to

l=1

Method	FOV size	ROC-AUC
HIPT [ 5 ] (off-the-shelf)	$4096\!\times\!4096$	0.907
MuViT [1]	$256\!\times\!256$	0.958
MuViT [1,8]	$2\mkern-2.0mu\times\mkern-2.0mu256\mkern-2.0mu\times\mkern-2.0mu256$	0.963
MuViT [1,8,32,64]	$4\mkern-2.0mu\times\mkern-2.0mu256\mkern-2.0mu\times\mkern-2.0mu256$	0.988

Table 7: Linear probing results on KPIS test set (4 classes) for HIPT [ 5 ] and MuViT encoders operating at different resolution sets. Reported is one-vs.-all (OvA) averaged ROC-AUC. Note that HIPT was used off-the-shelf and thus was not pre-trained on the KPIS dataset.

Module	Parameters
MuViT [1] (encoder)	25.24M
MuViT [1,8,32] (encoder)	25.31M
MuViT [1] +MAE decoder	26.44M
MuViT [1,8,32] +MAE decoder	28.88M
MuViT [1] +Mask2Former decoder	33.44M
MuViT [1,8,32] +Mask2Former decoder	36.80M

Table S1 : Number of parameters for MuViT components and variants.

Explore similar work

MuRF: Unlocking the Multi-Scale Potential of Vision Foundation Models
Mar 26, 2026Bocheng Zou, Mu Cai, Mark Stanley +2Multi-Scale Feature Fusion Vision Foundation Models
University of Wisconsin-Madison
From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathology
Aug 4, 2026Basit Alawode, Moshira Ali Abdalla, Dwarikanath Mahapatra +2Representation Learning Vision Transformer
Khalifa University of Science and Technology, UAE · University of Western Australia, Australia
Learning Where to Focus: Self-Supervised Multi-Scale ViTs for Histopathology
Sep 16, 2026Anabel Stammer, Valay Bundele, Mehran Hosseinzadeh +1Computational Pathology Self-Supervised Visual Representation Learning
Eberhard Karls Universität Tübingen, Germany

Mar 26, 2026J/K move · Enter open · S save

MuRF: Unlocking the Multi-Scale Potential of Vision Foundation Models

Bocheng Zou, Mu Cai, Mark Stanley, Dingfu Lu+1

University of Wisconsin-Madison

Multi-Scale Feature Fusion Vision Foundation Models Visual Representation Learning Vision Foundation Model Adaptation

Open paper AskPDF arXiv

Vision Foundation Models (VFMs) have become the cornerstone of modern computer vision, offering robust representations across a wide array of tasks. While recent advances allow these models to handle varying input sizes during training, inference typically remains restricted to a single, fixed scale. This prevalent single-scale paradigm overlooks a fundamental property of visual perception: varying resolutions offer complementary inductive biases, where low-resolution views excel at global semantic recognition and high-resolution views are essential for fine-grained refinement. In this work, we propose Multi-Resolution Fusion (MuRF), a simple yet universally effective strategy to harness this synergy at inference time. Instead of relying on a single view, MuRF constructs a unified representation by processing an image at multiple resolutions through a frozen VFM and fusing the resulting features. The universality of MuRF is its most compelling attribute. It is not tied to a specific architecture, serving instead as a fundamental, training-free enhancement to visual representation. We empirically validate this by applying MuRF to a broad spectrum of critical computer vision tasks across multiple distinct VFM families - primarily DINOv2, but also demonstrating successful generalization to contrastive models like SigLIP2.

Abstract

Explore similar work

MuRF: Unlocking the Multi-Scale Potential of Vision Foundation Models

From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathology

Learning Where to Focus: Self-Supervised Multi-Scale ViTs for Histopathology