Multi-scale Fusion of Fast Vision Mamba and Foundation Model Features for Patch-Level Gleason Grading in Computational Pathology

Zihe Zhang; Weimiao Yu; Piumi Don Simonge

cs.CVOct 19, 2026

Multi-scale Fusion of Fast Vision Mamba and Foundation Model Features for Patch-Level Gleason Grading in Computational Pathology

Authors: Zihe Zhang, Weimiao Yu, Piumi Don Simonge

Organizations: Nanyang Technological University, Singapore · Bioinformatics institute, Agency for Science Technology and Research, Singapore

Abstract

Automated Gleason grading of prostate whole slide images (WSIs) enables faster and more consistent cancer risk stratification. However, it remains a challenging task due to the subtle morphological differences between grades, the severe class imbalance arising from rare patterns, and the diagnostic features that span multiple spatial scales. Pathology foundation models (FMs) extract strong patch representations but often operate at a single-scale, which limits spatial context and disproportionately affects classes that depend on broader context. The challenge of fusing multiple scales and complementary models, with minimal data and computational resources, remains unresolved. We propose a multi-scale classifier built on state-space models. Our method combines two complementary feature sources: a frozen foundation model, and a lightweight FastVim, pre-trained on ImageNet and distilled from the foundation model, that reads the raw image crops. At each scale, a Mamba model encodes the features from both sources with linear-time spatial context, and a jointly trained head fuses the encoded representations into one prediction. On the PANDA dataset, our method reaches macro F1 of 0.894 and Gleason 5 F1 of 0.802 under 5-fold cross-validation, a 12% and 21% relative improvement over the best single-scale model. A controlled study over 5 foundation models, 5 base classifiers and 3 scales confirms that the gain holds across encoders, with larger improvements observed at smaller scales.

Figures & tables

Figure 1: Overview of the proposed method. Since Virchow2 accepts only its native $256$ px input, the $512$ px and $1024$ px scale representations are formed by averaging the embeddings of the $2\times 2$ and $4\times 4$ constituent tiles, respectively. In contrast, the FastVim branch encodes native crops at all three scales directly. The representations from both branches are each encoded by a Mamba encoder and fused into one patch-level prediction, and a per-scale classification head supplies an auxiliary training signal (deep supervision). Only the Mamba encoders, the per-scale classification heads and the fusion head are trained, and the two feature encoders are kept frozen.

Model	Macro F1	F1 (G5)	Bal. Acc
Single-scale, $256$	$0.794{\pm}0.012$	$0.636{\pm}0.044$	$0.916{\pm}0.003$
Single-scale, $512$	$0.796{\pm}0.016$	$0.661{\pm}0.036$	$0.922{\pm}0.004$
Single-scale, $1024$	$0.768{\pm}0.028$	$0.626{\pm}0.080$	$\mathbf{0.929}{\pm}0.003$
Multi-scale (uniform average)	$0.747{\pm}0.003$	$0.535{\pm}0.019$	$0.923{\pm}0.001$
Multi-scale (ours)	$0.874{\pm}0.001$	$0.784{\pm}0.005$	$0.860{\pm}0.005$
Multi-scale + FastVim (ours)	$\mathbf{0.894}{\pm}0.002$	$\mathbf{0.802}{\pm}0.003$	$0.926{\pm}0.004$

Table 1: Performance of our multi-scale method compared with single-scale baselines and a multi-scale uniform average. All experiments use the best-performing foundation model, Virchow2, where applicable. Results are mean

\pm

standard deviation over the 5 folds on the held-out test set. Best value per column in bold.

Figure 2: Confusion matrix showing that our method classifies most patches correctly and that the remaining errors fall between adjacent Gleason patterns. Each cell is the proportion of a true class (row) predicted as each class (column), so the diagonal gives per-class recall.

Backbone	Macro F1	QWK	Throughput	#Params
DeiT-S (ViT) [ 17 ]	0.621	0.597	47	22M
ViM-S [ 20 ]	0.625	0.598	43	26M
FastVim-S [ 8 ]	0.665	0.631	63	26M
FastVim-S with distillation	0.700	0.672	63	26M

Table 2: Backbones for the FastVim branch at

1024

crops (

70/15/15

split, not comparable with Table 1 ). Throughput is measured in patches per second on an A100 GPU.

	Macro F1			F1 (G5)
Encoder	Single-scale	Fusion	$\Delta$ (abs)	Single-scale	Fusion	$\Delta$ (abs)
Virchow2 [ 21 ]	$0.717$	$\mathbf{0.852}$	$+0.135$	$0.514$	$\mathbf{0.730}$	$+0.216$
DeiT-S (ViT) [ 17 ]	$0.635$	$\mathbf{0.799}$	$+0.164$	$0.466$	$\mathbf{0.661}$	$+0.195$
ResNet-50 (CNN) [ 6 ]	$0.547$	$\mathbf{0.781}$	$+0.234$	$0.401$	$\mathbf{0.592}$	$+0.191$
FastVim-S (ViM) [ 8 ]	$0.649$	$\mathbf{0.824}$	$+0.175$	$0.524$	$\mathbf{0.656}$	$+0.132$

Table 3: Multi-scale fusion applied to different image encoders, on the

70/15/15

split. Best results per metric in bold.

Figure 3: Change in evaluation metrics across different scales and foundation models, for two base classifiers. (a) Mamba base classifier. (b) Patch-level attention base classifier. Each line is one foundation model, with the mean across models shown in black excluding CONCH, which lacks a $256$ px run.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Data splits and their class distribution. (a) The main split, used for the 5-fold cross-validation experiments presented in Sec. 3.2 and tested on held-out test split. The ablation split, the single $70/15/15$ train/validation/test split used in Sec. 3.4 . (b) Class distribution within each split, relative to the whole cohort, on a log scale.

		$256$ px			$512$ px			$1024$ px
FM	Base classifier	BA	MF1	G5	BA	MF1	G5	BA	MF1	G5
Virchow2	Single-layer MLP	0.869	0.753	0.622	0.897	0.754	0.613	0.917	0.767	0.658
	3-layer MLP	0.872	0.741	0.600	0.898	0.760	0.646	0.915	0.776	0.683
	Mamba	0.916	0.794	0.636	0.922	0.796	0.661	0.929	0.768	0.626
	Mamba3	0.912	0.772	0.572	0.923	0.780	0.636	0.928	0.725	0.565
	Patch-level attention	0.909	0.775	0.623	0.925	0.794	0.684	0.928	0.771	0.681

Appendix

Table S1: Evaluation metrics across foundation models, base classifiers, and input scales. Results report the mean held-out test performance across 5-fold cross-validation experiments. BA is balanced accuracy, MF1 is macro F1, and G5 is the F1 of Gleason 5. Since CONCH requires an input size of at least

512

px, results are reported only for patch sizes of

512

px and above. No multi-scale fusion is applied in these experiments.

Histopathology Image Classification

Histopathology image classification uses computational models to identify tissue types, tumors, grades, and biomarkers from stained pathology images.

See all 139 papers

Sep 20, 2026

WILSON - Multi-magnification composites as a foundation for patient-level pathology analysis and diagnostic text generation

Pathologists examine tissue sections across multiple magnifications and slides, integrating morphological information to establish a diagnosis. In contrast, most pathology foundation models encode thousands of image patches from a single representative ("best") slide and aggregate their features, limiting multiscale and patient-level contextual integration. Here we introduce WILSON, a vision-language foundation model that represents whole-slide images and multi-slide cases through multi-magnification composites. Trained on approximately 189,000 slides spanning 42 organs and 838 diagnostic entities, WILSON enables patient-level analysis, multimodal retrieval and diagnostic text generation. Without task-specific training, WILSON outperformed a dedicated case-level model (macro-F1, 0.52 versus 0.38) and matched substantially larger slide-level models while requiring 400-1,350 times fewer computational operations. End-to-end fine-tuning improved breast cancer subtyping and stromal lymphocyte grading. WILSON also surpassed PRISM in image-text retrieval (recall@1, 75.6% versus 58.1%) and generated diagnostic descriptions that received higher ratings from five board-certified pathologists than competing models. These findings demonstrate that compact, multiscale representations can support computationally efficient, clinically grounded pathology foundation models.

Saghir Alfasly, Wataru Uegami, Sobhan Hemati +17

KIMIA Lab, Department of AI & Informatics, Mayo Clinic, Rochester, MN, USA. · Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA. · Department of Quantitative Health Sciences, Mayo Clinic, Rochester, MN, USA. +3

Pathology Foundation Models Computational Pathology Medical VLMs Whole-Slide Image Analysis Histopathology Image Classification Medical Report Generation

Ask PDF

Aug 4, 2026

From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathology

Vision Transformers (ViTs) and their hierarchical variants have achieved strong performance in Computational Pathology (CPath). However, most are pre-trained on single-resolution Whole Slide Images (WSIs), limiting their generalization across arbitrary resolutions. Gigapixel WSIs inherently contain diagnostic patterns at multiple scales, including cellular morphologies, tissue architectures, and global context, mirroring how expert pathologists examine WSIs. We introduce Multi-Resolution Pyramid Transformer (MRPT), a model that hierarchically aggregates multi-resolution information from cellular to tissue and WSI levels. MRPT employs a biologically meaningful Consecutive Cross-Resolution Attention (CCRA) mechanism to capture scale-independent interactions and enforces multi-resolution semantic consistency by aligning embeddings across resolutions, yielding robust and generalizable WSI representations. Pre-trained in a multi-resolution self-supervised manner on 624M patches, 2.4M regions, and 36K WSIs, MRPT learns rich coarse-to-fine histopathology features. Extensive experiments on 34 diverse datasets show that MRPT surpasses recent foundation models and Multimodal Large Language Models (MLLMs) in cancer subtype classification, tissue phenotyping, and Visual Question Answering (VQA) for WSI understanding.

Basit Alawode, Moshira Ali Abdalla, Dwarikanath Mahapatra +2

Khalifa University of Science and Technology, UAE · University of Western Australia, Australia

Representation Learning Vision Transformer Pathology Foundation Models Histopathology Image Classification Whole-Slide Image Analysis

Ask PDF

Jul 9, 2026

ProsMAE: Multi-Source MAE Pretraining for ISUP Grade Classification

Whole slide images (WSIs) provide rich diagnostic information for computational pathology, but their gigapixel scale, stain variation, scanner differences, tissue artifacts, and limited expert annotation make robust model training challenging. This paper presents a multi-source Masked Autoencoder (MAE) framework, named ProsMAE, for histopathology representation learning. Tiles from Prostate cANcer graDe Assessment (PANDA), CAncer MEtastases in LYmph nOdes challeNge 2017 (CAMELYON17), and BReAst Carcinoma Subtyping (BRACS) are used for ProsMAE pretraining to expose the encoder to diverse tissue morphology and acquisition conditions. The learned encoder is transferred for International Society of Urological Pathology (ISUP) grade classification through ProsCLS, using a frozen encoder and a linear classification head. ProsMAE achieved a higher mean validation quadratic weighted kappa (QWK) than the vanilla MAE frozen linear-probe baseline under the evaluated disjoint PANDA split. Repeated-split evaluation remains necessary to further establish robustness across split compositions.

Anna Jung, Kyeonghun Kim, Youngung Han +6

SNU · OUTTA · GIST +1

Computational Pathology Masked Autoencoders Oncology Histopathology Image Classification Prostate Cancer Grading Medical Image Representation Learning+1

Ask PDF

Abstract

Histopathology Image Classification

Explore similar work

WILSON - Multi-magnification composites as a foundation for patient-level pathology analysis and diagnostic text generation

From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathology

ProsMAE: Multi-Source MAE Pretraining for ISUP Grade Classification