MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding

Basit Alawode; Arif Mahmood; Muaz Khalifa Al-Radi; Shahad Albastaki; Asim Khan; Muhammad Bilal; Moshira Ali Abdalla; Mohammed Bennamoun

cs.CVMar 24, 2026

MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding

Authors: Basit Alawode, Arif Mahmood, Muaz Khalifa Al-Radi, Shahad Albastaki, Asim Khan, Muhammad Bilal, Moshira Ali Abdalla, Mohammed Bennamoun, +1 more

Organizations: Department of Computer Science, Khalifa University of Science and Technology, UAE. · Information Technology University, Pakistan. · KAU, KSA. · University of the Western Australia.

Abstract

Whole Slide Images (WSIs) exhibit hierarchical structure, where diagnostic information emerges from cellular morphology, regional tissue organization, and global context. Existing Computational Pathology (CPath) Multimodal Large Language Models (MLLMs) typically compress an entire WSI into a single embedding, which hinders fine-grained grounding and ignores how pathologists synthesize evidence across different scales. We introduce MLLM-HWSI, a Hierarchical WSI-level MLLM that aligns visual features with pathology language at four distinct scales, cell as word, patch as phrase, region as sentence, and WSI as paragraph to support interpretable evidence-grounded reasoning. MLLM-HWSI decomposes each WSI into multi-scale embeddings with scale-specific projectors and jointly enforces (i) a hierarchical contrastive objective and (ii) a cross-scale consistency loss, preserving semantic coherence from cells to the WSI. We compute diagnostically relevant patches and aggregate segmented cell embeddings into a compact cellular token per-patch using a lightweight Cell-Cell Attention Fusion (CCAF) transformer. The projected multi-scale tokens are fused with text tokens and fed to an instruction-tuned LLM for open-ended reasoning, VQA, report, and caption generation tasks. Trained in three stages, MLLM-HWSI achieves new SOTA results on 13 WSI-level benchmarks across six CPath tasks. By aligning language with multi-scale visual evidence, MLLM-HWSI provides accurate, interpretable outputs that mirror diagnostic workflows and advance holistic WSI understanding. Code is available at: GitHub.

Figures & tables

Figure 1: Our proposed MLLM-HWSI model aligns WSIs across multiple scales e.g., cells, patches, regions, and WSI enabling fine-grained, context-aware, and interpretable pathology reasoning.

Figure 2: Comparison of MLLM-HWSI with SOTA methods.

Figure 3: Overview of the proposed MLLM-HWSI. (A) Hierarchical decomposition of WSI into cell, patch, and region-level embeddings aligned with MLLM. (B) MLLM-HWSI three stage pre-training paradigm for multimodal reasoning.

Models	Cell	Patch	Region	WSI	PANDA	EBRAINS	WSI-VQA	SlideBench-VQA
Models	Feat.	Feat.	Feat.	Feat.	[ 13 ] (BA)	[ 69 ] (BA)	[ 17 ] (A)	(BCNB) [ 20 ] (A)
WSI-LLaVA [ 52 ]	$\times$	$\times$	$\times$	$\checkmark$	0.644	0.501	0.546	0.553
SlideChat [ 20 ]	$\times$	$\times$	$\times$	$\checkmark$	0.633	0.466	0.601	0.541
MLLM-HWSI 1	$\times$	$\times$	$\times$	$\checkmark$	0.661	0.519	0.616	0.576
MLLM-HWSI 2	$\times$	$\times$	$\checkmark$	$\checkmark$	0.686	0.534	0.611	0.592
MLLM-HWSI 3	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	0.711	0.566	0.661	0.621

Table 1: Ablation 1: Effect of hierarchical representations in MLLM-HWSI. Progressive inclusion of cell-, patch-, region-, and WSI-level features improves performance across all benchmarks. The full MLLM-HWSI achieves the highest scores, confirming the importance of hierarchical multi-scale alignment. Feat. stands for “ Features ”, BA stands for “ Balanced Accuracy ”, and A stands for “ Accuracy ”.

Variants	$\mathcal{L}_{HCA}$ Loss ( 9 )			PANDA	EBRAINS	WSI-VQA	SlideBench-VQA
Variants	$\mathcal{L}_{s}$	$\mathcal{L}_{c}$	$\mathcal{L}_{\textrm{WSI}}$	(BA)	(BA)	(A)	(BCNB) (A)
MLLM-HWSI	$\checkmark$	$\checkmark$	$\checkmark$	0.748	0.612	0.692	0.687
MLLM-HWSI	$\checkmark$	$\checkmark$	$\times$	0.716	0.592	0.668	0.654
MLLM-HWSI	$\checkmark$	$\times$	$\times$	0.705	0.582	0.655	0.636
MLLM-HWSI	$\times$	$\times$	$\checkmark$	0.661	0.519	0.616	0.576
MLLM-HWSI	$\times$	$\checkmark$	$\checkmark$	0.677	0.526	0.624	0.590

Table 2: Ablation 7: Effect of loss components in

\mathcal{L}_{HCA}

. Removing any of the semantic (

\mathcal{L}_{s}

), cross-scale (

\mathcal{L}_{c}

), or WSI-level (

\mathcal{L}_{WSI}

) losses leads to notable performance drops, confirming their complementary contributions to hierarchical cross-modal alignment.

Panel of Figure 4 — Figure 4: Performance comparison of the proposed MLLM-HWSI with SOTA CPath models. MLLM-HWSI achieves the highest overall scores across all benchmarks, underscoring the benefits of hierarchical multi-scale visual encoding and cross-modal alignment.

General MLLMs	SlideBench-VQA (TCGA)				WSI-Bench				SlideBench-	WSI-VQA
General MLLMs	Micro.	Diag.	Clinical	Average	MA	Diag.	TP	Average	VQA(BCNB)	WSI-VQA
InstructBLIP-FLAN	0.366	0.186	0.221	0.257	0.198	0.221	0.389	0.269	0.189	0.102
LLaVA-1.5	0.451	0.219	0.389	0.353	0.232	0.271	0.677	0.393	0.201	0.121
Qwen-VL-MAX	0.496	0.288	0.405	0.396	0.288	0.322	0.706	0.438	0223	0.133
GeminiProV	0.506	0.304	0.587	0.465	0.403	0.433	0.821	0.552	0.282	0.167
GPT-4V	0.628	0.466	0.667	0.587	0.471	0.530	0.875	0.625	0.414	0.304

Table 3: Comparison of MLLM-HWSI with SOTA general-purpose and CPath-specific MLLMs on multi-domain VQA benchmarks. We evaluate MLLM-HWSI across four datasets, two external (SlideBench-VQA (BCNB), WSI-VQA) and two TCGA-based (SlideBench-VQA (TCGA), WSI-Bench), covering Microscopy Micro , Diagnosis Diag. , Morphological Analysis MA , and Treatment Panning TP –related questions. Performance is reported in terms of accuracy. MLLM-HWSI achieves superior accuracy across all datasets and sub-tasks, demonstrating its strong generalization and diagnostic reasoning capabilities.

Models	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE	METEOR
GPT-4V	0.202 $\|$ 0.158	0.069 $\|$ 0.420	0.030 $\|$ 0.240	0.016 $\|$ 0.100	0.132 $\|$ 0.128	0.167 $\|$ 0.144
Quilt-LLaVA	0.474 $\|$ 0.384	0.351 $\|$ 0.364	0.282 $\|$ 0.301	0.236 $\|$ 0.254	0.475 $\|$ 0.441	0.460 $\|$ 0.462
MIGen	0.403 $\|$ 0.402	0.306 $\|$ 0.466	0.248 $\|$ 0.266	0.209 $\|$ 0.234	0.446 $\|$ 0.322	0.407 $\|$ 0.412
WSICaption	0.203 $\|$ 0.218	0.156 $\|$ 0.137	0.183 $\|$ 0.460	0.130 $\|$ 0.540	0.265 $\|$ 0.251	0.317 $\|$ 0.322
TITAN	0.399 $\|$ 0.382	0.321 $\|$ 0.318	0.223 $\|$ 0.247	0.203 $\|$ 0.314	0.304 $\|$ 0.341	0.385 $\|$ 0.329
HistGen	0.406 $\|$ 0.413	0.307 $\|$ 0.297	0.248 $\|$ 0.229	0.208 $\|$ 0.184	0.448 $\|$ 0.344	0.416 $\|$ 0.182

Table 4: Report generation comparison on two benchmarks WSI-Bench

|

HistGen. MLLM-HWSI outperforms all SOTA models across BLEU, ROUGE-L, and METEOR metrics, highlighting its ability to produce accurate and clinically coherent diagnostic reports.

Models	R $@$ 1	R $@$ 3	R $@$ 5	R $@$ 10	Average
PRISM	0.542 $\|$ 0.634	0.674 $\|$ 0.768	0.724 $\|$ 0.817	0.767 $\|$ 0.880	0.669 $\|$ 0.765
TITAN	0.732 $\|$ 0.754	0.818 $\|$ 0.896	0.841 $\|$ 0.908	0.876 $\|$ 0.922	0.811 $\|$ 0.866
SlideChat	0.766 $\|$ 0.771	0.823 $\|$ 0.855	0.838 $\|$ 0.884	0.852 $\|$ 0.907	0.819 $\|$ 0.854
WSI-LLaVA	0.786 $\|$ 0.770	0.824 $\|$ 0.867	0.844 $\|$ 0.890	0.858 $\|$ 0.912	0.828 $\|$ 0.859
MLLM-HWSI	0.822 $\|$ 0.842	0.856 $\|$ 0.942	0.897 $\|$ 0.952	0.926 $\|$ 0.946	0.875 $\|$ 0.920

Table 5: Cross-modal retrieval on TCGA-Slide-Reports. MLLM-HWSI consistently outperforms SOTA models in both report-to-slide and slide-to-report tasks across all recall metrics.

Models	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	METEOR
GPT-4V	0.100	0.030	0.010	0.010	0.110	0.131
Quilt-LLaVA	0.230	0.090	0.040	0.010	0.160	0.420
MIGen	0.370	0.240	0.150	0.100	0.250	0.381
HistGen	0.300	0.181	0.110	0.090	0.171	0.288
SlideChat	0.370	0.210	0.120	0.080	0.240	0.488
TITAN	0.377	0.233	0.169	0.176	0.302	0.506

Table 6: Captioning performance on SlideBench-Caption. MLLM-HWSI outperforms all SOTA models showing strong capability in producing morphology-aware and accurate captions.

Cell Segmentation	Cell	PANDA	EBRAINS	WSI-VQA	SlideBench-VQA
Backbone	Feat.	[ 13 ] (BA)	[ 69 ] (BA)	[ 17 ] (A)	(BCNB) [ 20 ] (A)
CellViT [ 40 ]	$\textrm{ViT}_{\textrm{cell-cell}}$	0.748	0.612	0.692	0.687
NuHTC [ 49 ]	$\textrm{ViT}_{\textrm{cell-cell}}$	0.733	0.600	0.689	0.685
STRARDIST [ 83 ]	$\textrm{ViT}_{\textrm{cell-cell}}$	0.701	0.585	0.665	0.670
MicroNet [ 68 ]	$\textrm{ViT}_{\textrm{cell-cell}}$	0.698	0.561	0.672	0.671
HoverNet [ 33 ]	$\textrm{ViT}_{\textrm{cell-cell}}$	0.725	0.591	0.690	0.673

Table 7: Effect of cell segmentation backbones in

\textrm{ViT}_{\textrm{cell-cell}}

. Results show Balanced Accuracy (BA) for PANDA and EBRAINS, and Accuracy (A) for WSI-VQA and SlideBench-VQA (BCNB). CellViT achieves the highest scores, confirming the benefit of SAM-based segmentation for cell-level feature extraction.

Models	Cell	Patch	Region	WSI	PANDA	EBRAINS	WSI-VQA	SlideBench-VQA
Models	Encoder	Encoder	Encoder	Encoder	[ 13 ] (BA)	[ 69 ] (BA)	[ 17 ] (A)	(BCNB) [ 20 ] (A)
MLLM-HWSI	$\textrm{ViT}_{\textrm{cell-cell}}$	$\mathcal{F}_{\textrm{CONCH}}$	$\textrm{ViT}_{r}$	$\textrm{ViT}_{\textrm{WSI}}$	0.748	0.612	0.692	0.687
MLLM-HWSI	$\textrm{ViT}_{\textrm{cell-cell}}$	UNI	UNI	UNI	0.721	0.589	0.653	0.664
MLLM-HWSI	$\textrm{ViT}_{\textrm{cell-cell}}$	CONCH	CONCH	CONCH	0.712	0.581	0.644	0.657
MLLM-HWSI	$\textrm{ViT}_{\textrm{cell-cell}}$	CONCH	CONCH	LongNet	0.702	0.575	0.681	0.665
MLLM-HWSI	$\textrm{ViT}_{\textrm{cell-cell}}$	GigaPath	GigaPath	LongNet	0.692	0.562	0.673	0.681

Table 8: Influence of visual encoder selection across hierarchical levels. Different combinations of patch-, region-, and WSI-level encoders (UNI, CONCH, GigaPath, LongNet) are evaluated, all fine-tuned with the proposed loss. The

\mathcal{F}_{\textrm{CONCH}}

\textrm{ViT}_{r}

, and

\textrm{ViT}_{\textrm{WSI}}

configuration yields the best overall results, highlighting the importance of heterogeneous multi-scale encoders.

Variants	$\textrm{ViT}_{\textrm{cell-cell}}$			PANDA	EBRAINS	WSI-VQA	SlideBench-VQA
Variants	# Encoder (n)	# heads (h)	Dimension (d)	(BA)	(BA)	(A)	(BCNB) (A)
a. MLLM-HWSI	2	2	768	0.748	0.612	0.692	0.687
b. MLLM-HWSI	4	4	768	0.726	0.592	0.681	0.677
c. MLLM-HWSI	6	6	768	0.727	0.590	0.682	0.675
d. MLLM-HWSI	2	2	384	0.741	0.595	0.688	0.671
e. MLLM-HWSI	2	2	192	0.723	0.596	0.690	0.676

Table 9: Effect of

\textrm{ViT}_{\textrm{cell-cell}}

architecture on performance. Variants (a–e) modify the number of encoders (

n

), heads (

h

), and embedding dimensions (

d

), while (f–h) use max, min, and average pooling instead of attention. Results (BA for PANDA/EBRAINS, A for WSI-VQA/SlideBench-VQA) show that the

n=2

h=2

d=768

configuration performs best, emphasizing the value of attention-based cell-level modeling.

Models	Cell	Patch	Region	WSI	PANDA	EBRAINS	WSI-VQA	SlideBench-VQA
Models	Feat.	Feat.	Feat.	Feat.	[ 13 ] (BA)	[ 69 ] (BA)	[ 17 ] (A)	(BCNB) [ 20 ] (A)
WSI-LLaVA [ 52 ]	$\times$	$\times$	$\times$	$\checkmark$	0.644	0.501	0.546	0.553
SlideChat [ 20 ]	$\times$	$\times$	$\times$	$\checkmark$	0.633	0.466	0.601	0.541
MLLM-HWSI 1	$\times$	$\times$	$\times$	$\checkmark$	0.661	0.519	0.616	0.576
MLLM-HWSI 2	$\times$	$\times$	$\checkmark$	$\checkmark$	0.686	0.534	0.611	0.592
MLLM-HWSI 3	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	0.711	0.566	0.661	0.621

Table 10: Effect of hierarchical representations in MLLM-HWSI. Progressive inclusion of cell-, patch-, region-, and WSI-level features in MLLM-HWSI 1-3 improves performance across all benchmarks. The full MLLM-HWSI achieves the highest scores, confirming the importance of hierarchical multi-scale alignment. PANDA and EBRAINS datasets are used for zero-shot classification while WSI-VQA and SlideBench-VQA (BCNB) datasets are used for VQA task. Feat. stands for “ Features ”, BA stands for “ Balanced Accuracy ”, and A stands for “ Accuracy ”.

Figure 5: Pathologically meaningful patches and discarded patch.

LLMs	SlideBench	WSI-	PANDA
LLMs	(BCNB) (A)	VQA (A)	(BA)
Vicuna-7B-v1.5 [ 22 ]	0.682	0.683	0.725
Phi-3-Mini-4k-Instruct [ 1 ]	0.664	0.674	0.706
Llama3-8B-Instruct [ 34 ]	0.677	0.673	0.725
Internlm2-Chat-7B [ 14 ]	0.681	0.684	0.738
Qwen2-2.5 7B-Instruct [ 86 ]	0.687	0.692	0.748

Table 11: Effect of LLM choice on VQA performance. Comparison of five instruction-tuned LLMs integrated into MLLM-HWSI across SlideBench (BCNB), WSI-VQA, and PANDA datasets. Qwen2.5-7B-Instruct yields the highest accuracy, highlighting its stronger multimodal reasoning capability.

Variants	Semantic Patch Filtering		PANDA	EBRAINS	WSI-VQA	SlideBench-VQA
Variants	HPS	DPS	(BA)	(BA)	(A)	(BCNB)(A)
a. MLLM-HWSI	$\checkmark$	$\checkmark$	0.748	0.612	0.692	0.687
b. MLLM-HWSI	$\checkmark$	$\times$	0.731	0.592	0.674	0.663
c. MLLM-HWSI	$\times$	$\checkmark$	0.741	0.606	0.684	0.676
d. MLLM-HWSI	$\times$	$\times$	0.711	0.566	0.664	0.657
Variants	Semantic Patch Filtering		PANDA	EBRAINS	WSI-VQA	SlideBench-VQA

Table 12: Effect of Semantic Patch Filtering. Comparison of different combinations of Heterogeneous Patch Selection (HPS), Diagnostically Relevant Patch Selection (DPS), and K-means clustering in MLLM-HWSI. The combination of HPS and DPS yields the best overall accuracy, highlighting their complementary roles in selecting diverse and diagnostic patches.

top- $k$	PANDA	EBRAINS	WSI-VQA	SlideBench-VQA
Value	[ 13 ] (BA)	[ 69 ] (BA)	[ 17 ] (A)	(BCNB) [ 20 ] (A)
16	0.711	0.588	0.676	0.641
32	0.731	0.605	0.681	0.663
48	0.748	0.612	0.692	0.687
64	0.743	0.604	0.686	0.682
96	0.740	0.608	0.685	0.678

Table 13: Influence of top-

k

\textrm{ViT}_{\textrm{cell-cell}}

. Performance with different top-

k

values in the Diagnostically Relevant Patch Selection (DPS) module. The best results are achieved at top-

k=48

, indicating optimal diagnostic coverage and compactness.

Method	PANDA		EBRAINS		BRACS		UBC-Ocean		TCGA-OT		IMP-CRC
	F	BA	F	BA	F	BA	F	BA	F	BA	F	BA
PLIP	0.288	0.235	0.013	0.080	0.214	0.203	0.376	0.345	0.203	0.223	0.523	0.655
PathCLIP	0.461	0.455	0.223	0.187	0.281	0.309	0.657	0.612	0.304	0.334	0.560	0.700
CPLIP	0.445	0.420	0.253	0.233	0.294	0.288	0.706	0.653	0.431	0.405	0.591	0.733
CONCH	0.596	0.566	0.304	0.278	0.336	0.344	0.786	0.807	0.488	0.532	0.637	0.833
QuiltNet	0.532	0.509	0.229	0.201	0.321	0.312	0.753	0.776	0.486	0.506	0.608	0.788

Table 14: WSI-level Zero-shot classification performance comparison results with SOTA CPath models across six datasets.

Method	PANDA		EBRAINS30		BRACS		UBC-Ocean		TCGA-OT		IMP-CRC
	F	BA	F	BA	F	BA	F	BA	F	BA	F	BA
HIPT	0.687	0.654	0.702	0.677	0.334	0.288	0.766	0.706	0.512	0.488	0.718	0.801
CTransPath	0.752	0.691	0.597	0.514	0.398	0.355	0.788	0.733	0.566	0.544	0.749	0.833
REMEDIS	0.766	0.711	0.471	0.382	0.367	0.331	0.733	0.706	0.504	0.455	0.772	0.843
CHIEF	0.745	0.724	0.706	0.688	0.413	0.387	0.823	0.789	0.640	0.528	0.701	0.781
DinoPath	0.682	0.706	0.771	0.755	0.394	0.361	0.844	0.821	0.586	0.556	0.792	0.855

Table 15: WSI-level classification results and comparisons using linear probe evaluation and weakly supervised MIL-based classification with SOTA CPath models across six datasets.

WSI
Question	What are the main observations in the whole slide histopathology image, and what is the likely diagnosis?
WSI-LLaVA	Microscopic observation of the pathology slide reveals moderately differentiated squamous cell carcinoma characterized by nests and sheets of atypical squamous epithelial cells with intercellular bridges and keratinization. The tumor cells exhibit moderate nuclear pleomorphism with enlarged, hyperchromatic nuclei and prominent nucleoli. Mitotic figures are present, indicating active cell division. There is no evidence of vascular invasion, necrosis, or calcification within the examined tissue. Based on these observations, the diagnosis is moderately differentiated keratinizing squamous carcinoma.
MI-Gen	Microscopic observation of the pathology slide reveals a poorly differentiated squamous cell carcinoma characterized by a lack of squamous cell carcinoma. The tumor exhibits a lack of squamous differentiation with a lack of squamous differentiation. The tumor cells exhibit moderate nuclear pleomorphism with variability in nuclear size and shape and there is a moderate rate of cell division. Vascular invasion is not identified. Necrosis is not observed within the tumor. Based on these observations, the diagnosis is squamous cell carcinoma.
Hist-Gen	Microscopic observation of the pathology slide reveals a poorly differentiated squamous cell carcinoma characterized by irregular nests and sheets of atypical squamous cells with intercellular bridges and keratinization. The tumor cells exhibit a high degree of nuclear pleomorphism. There is no evidence of vascular invasion or necrosis within the tumor tissue. The surrounding lung parenchyma shows areas of chronic inflammation and chronic inflammation. Based on these observations, the diagnosis is squamous cell carcinoma, poorly differentiated.
Quilt-LLaVA	Microscopic observation of the pathology slide reveals sheets and nests of atypical squamous cells with moderate differentiation. There is evidence of focal keratinization, characterized by the presence of keratin pearls within the tumor tissue. Intercellular bridges are noted among the squamous cells, indicating differentiation towards squamous epithelium. The cellular morphology shows moderate pleomorphism, with variability in nuclear size and shape, and occasional prominent nucleoli. Mitotic figures are present but not excessively numerous. Vascular invasion is not specified in the observations, and necrosis is not prominently noted. Based on these observations, the diagnosis is squamous cell carcinoma, moderately differentiated.
GPT-4o	This histopathology image shows a well-circumscribed, nodular lesion with a dense, cellular stroma. There are areas of necrosis and hemorrhage, and the tissue architecture appears disrupted. The lesion has a heterogeneous appearance with varying cellular densities and some fibrotic regions. Diagnosis: The features are consistent with a uterine fibroid (leiomyoma), which is characterized by a smooth muscle cell growth forming a circumscribed mass with areas of degeneration.

Table 16: Qualitative comparison of pathology report generation across SOTA CPath models. The qualitative analysis illustrates how MLLM-HWSI produces reports that closely match expert-annotated ground truth, capturing both fine-grained cellular morphology and global architectural context. Compared with prior models (e.g., WSI-LLaVA, MI-Gen, Hist-Gen, Quilt-LLaVA, and GPT-4o), MLLM-HWSI generates linguistically coherent and diagnostically accurate descriptions that mirror expert reasoning, demonstrating superior grounding between visual evidence and clinical language. Green: matched ground-truth content; Red: deviations; Orange: ground truth content missing in model response; Underlined: template language.

WSI
Question	Can you provide an overall description of the whole slide image (WSI)?
WSI-LLaVA	The whole slide image reveals exuberant granulation tissue characterized by notable fibroblastic proliferation and myxoid changes. The cellular component is primarily composed of spindle-shaped cells within a myxoid matrix. There is an absence of normal tissue architecture as the granulation tissue dominates the observed section. Importantly, no residual sarcoma tissue is identifiable within this examined section.
Quilt-LLaVA	The whole slide image reveals a tumor with a diffuse architectural pattern, characterized by a mixture of alveolar formations and glandular cell differentiation. There is a notable presence of large cells with prominent nucleoli and moderate nuclear atypia. The mitotic activity is moderate, with 8 mitoses per 10 high power fields. The tumor shows significant vascular invasion, with tumor cells infiltrating the renal vein. Additionally, there is focal necrosis within the tumor tissue.
WSI-VQA	The whole slide image reveals a biphasic tumor composed of spindle-shaped cells arranged in a storiform pattern. The spindle cells are arranged in a storiform pattern. The spindle cells are arranged in a storiform pattern. There is a notable absence of necrosis or calcification within the tumor.
GPT-4o	The whole slide image appears to be a tissue section stained with hematoxylin and eosin (H&E), commonly used in histopathology. The sample is divided into several sections, likely representing different orientations or parts of the tissue. – Coloration: The pink and purple hues suggest typical H&E staining, where eosin stains cytoplasmic components pink and hematoxylin stains nuclear material blue or purple. – Tissue Architecture: The sections show varying thickness with folded tissue smooth and irregular contours, indicating possible different tissue types or regions. – Other Observations: There are areas that seem to contain more densely packed cells and some regions with more open or less cellular areas, suggesting different tissue densities or types. A detailed examination at higher magnification would be necessary for further analysis, such as identifying specific pathological features or tissues.
MLLM-HWSI	The WSI demonstrates exuberant granulation tissue with extensive fibroblastic proliferation embedded in a myxoid stroma. The spindle-shaped fibroblasts are loosely arranged within the extracellular matrix, accompanied by delicate capillary networks and scattered inflammatory infiltrates. No evidence of residual malignant or sarcomatous components is observed. The overall architecture is dominated by reactive granulation tissue replacing normal parenchymal organization, consistent with post-treatment reparative changes.

Table 17: A comparative example of global morphology description outputs from different CPath models. Green: matched ground-truth content; Red: deviations; Orange: ground truth content missing in model response; Underlined: template language.

WSI
Question	Which observed features are most significant for making a diagnosis?
WSI-LLaVA	The most significant features include the papillary architecture with fibrovascular cores, the presence of psammoma bodies, and the nuclear characteristics such as overlapping, grooves, and intranuclear cytoplasmic inclusions. The absence of vascular invasion, necrosis, and calcification also plays a crucial role in the diagnostic process.
Quilt-LLaVA	The most significant features include the irregular clusters of epithelial cells with poor glandular differentiation, marked nuclear pleomorphism, and the presence of perineural invasion. The high mitotic activity and evidence of lymphovascular invasion are also critical, as they indicate aggressive tumor behavior.
WSI-VQA	The most significant features include the papillary architecture with fibrovascular cores and the presence of fibrovascular cores. The absence of vascular invasion and necrosis further supports the diagnosis. The absence of necrosis and calcification further supports the diagnosis.
GPT-4o	The whole slide image appears to be a tissue section stained with hematoxylin and eosin (H&E), commonly used in histopathology. The sample is divided into several sections, likely representing different orientations or parts of the tissue. – Coloration: The pink and purple hues suggest typical H&E staining, where eosin stains cytoplasmic components pink and hematoxylin stains nuclear material blue or purple. – Tissue Architecture: The sections show varying thickness with folded tissue smooth and irregular contours, indicating possible different tissue types or regions. – Other Observations: There are areas that seem to contain more densely packed cells and some regions with more open or less cellular areas, suggesting different tissue densities or types. A detailed examination at higher magnification would be necessary for further analysis, such as identifying specific pathological features or tissues.
MLLM-HWSI	The diagnostic features most critical for interpretation include a prominent papillary architecture with well-formed fibrovascular cores lined by epithelial cells showing characteristic nuclear overlapping, grooves, and intranuclear cytoplasmic inclusions. Multiple psammoma bodies are present within the papillary stalks. There is no evidence of vascular, lymphatic, or perineural invasion, and no areas of necrosis are identified. These cytologic and architectural features are diagnostic of a papillary-patterned neoplasm consistent with papillary carcinoma morphology.

Table 18: A comparative example of key diagnostic description outputs from different CPath models. Green: matched ground-truth content; Red: deviations; Orange: ground truth content missing in model response; Underlined: template language.

WSI
Question	Based on the IHC results showing P16 negativity and HR, HPV ISH status negativity, what is the molecular subtype in this head and neck squamous cell carcinoma?
WSI-LLaVA	The molecular subtype is HNSC HPV.
Quilt-LLaVA	The molecular subtype is HNSC HPV.
WSI-VQA	the molecular subtype is hnsc unk .
GPT-4o	The molecular subtype of head and neck squamous cell carci noma with P16 negativity and HR HPV ISH negativity is typ ically HPV-negative. This subtype is generally associated with non-viral risk factors such as smoking and alcohol use.
MLLM-HWSI	Given the P16 negativity and HR HPV ISH negativity, the molecular subtype is correctly classified as HNSC HPV-negative. This reflects a non-viral, keratinizing phenotype often linked to tobacco and alcohol-related carcinogenesis. These molecular features are characteristic of the HPV-independent pathway in head and neck squamous cell carcinoma.

Table 19: A comparative example of molecular subtyping outputs from different CPath models.

WSI
Question	What is the TNM staging for a patient with a 7 cm tumor, several affected lymph nodes, but no distant metastasis?
WSI-LLaVA	The TNM staging for this patient is T3 N2 M0.
Quilt-LLaVA	The TNM staging for this patient is T3 N2 M0.
WSI-VQA	the tnm staging for this patient is t2 n2 m0.
GPT-4o	To determine the TNM staging, we need to consider the following components:- T (Tumor size): The tumor is 7 cm. This generally corresponds to T2 or T3, depending on the specific cancer type, but often T3 for many cancers. GPT-4o- N(Nodes): ¨ Several affected lymph nodes¨ typically indicates N1 or N2, depending on the exact number and extent of nodal involvement.- M (Metastasis): No distant metastasis is indicated as M0. Without specific details about the type of cancer, a general staging based on the information given could be T3N1M0 or T3N2M0. However, the exact stage can vary based on cancer type and guidelines, so consultation with specific cancer staging resources is advised for precise staging.
MLLM-HWSI	According to TNM criteria, the 7 cm primary tumor meets the definition of T3 due to its size exceeding 5 cm. The description of “several affected lymph nodes” corresponds to N2, indicating regional nodal spread to multiple ipsilateral nodes. No evidence of distant metastasis defines M0. Therefore, the overall TNM staging for this patient is accurately classified as T3 N2 M0.

Table 20: A comparative example of staging outputs from different CPath models.

Explore similar work

PathAgentBench: Benchmarking Evidence-Seeking Vision-Language Models on Whole-Slide Pathology Image
Jul 21, 2026Dankai Liao, Tianyi Zhang, Yufeng Wu +6VLM Evaluation Medical VLMs
Department of Biomedical Engineering, National University of Singapore, Singapore 117417, Singapore
PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology
Dec 19, 2025Fengchun Liu, Songhan Jiang, Linghan Cai +2Computational Pathology Vision-Language Grounding
Harbin Institute of Technology, Shenzhen, School of Computer Science and Technology · National University of Singapore, Department of Electronic and Computer Engineering
From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathology
Aug 4, 2026Basit Alawode, Moshira Ali Abdalla, Dwarikanath Mahapatra +2Representation Learning Vision Transformer
Khalifa University of Science and Technology, UAE · University of Western Australia, Australia

Jul 21, 2026J/K move · Enter open · S save

PathAgentBench: Benchmarking Evidence-Seeking Vision-Language Models on Whole-Slide Pathology Image

Dankai Liao, Tianyi Zhang, Yufeng Wu, Xinyue Zhang+5

Department of Biomedical Engineering, National University of Singapore, Singapore 117417, Singapore

VLM Evaluation Medical VLMs Medical Image Localization Medical Image Benchmarks Whole-Slide Image Analysis

Open paper AskPDF arXiv

Whole-slide image (WSI) diagnosis requires identifying diagnostically relevant regions, examining them across magnifications, and integrating multi-scale evidence. However, most existing pathology benchmarks evaluate models on pre-cropped patches or pre-extracted slide features, leaving their ability to acquire evidence directly from gigapixel WSIs largely untested. We introduce PathAgentBench, a benchmark for evaluating evidence-seeking vision-language models (VLMs) across four complementary capabilities: image-to-text matching for evidence interpretation, text-to-image retrieval for evidence verification, diagnostic-region localization for evidence acquisition, and multi-scale reasoning for evidence integration. The benchmark is organized as a diagnostic tree that links nested regions across magnifications with scale-specific findings and path-level diagnoses. It contains 1,822 TCGA WSIs and 17,135 diagnostic paths annotated by ten board-certified pathologists. An additional private cohort of 190 breast cancer WSIs with detailed annotations is used to evaluate autonomous whole-slide exploration. We evaluate 20 general-purpose, medical, and pathology-specialized models. Leading open-weight models achieve over 93% accuracy in multi-scale reasoning and over 50% accuracy in both cross-modal matching tasks. In contrast, diagnostic-region localization remains challenging: the best text-guided mean intersection-over-union is below 0.09, underperforming a simple center-based heuristic. During autonomous exploration, the unconditional hit rate decreases from 0.522 at low magnification to 0.185 at intermediate magnification and 0.020 at high magnification. These results reveal a pronounced gap between reasoning over curated evidence and acquiring that evidence directly from WSIs. PathAgentBench provides a unified framework for measuring and improving evidence-seeking pathology models.

Models	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE	METEOR
GPT-4V	0.202 $\|$ 0.158	0.069 $\|$ 0.420	0.030 $\|$ 0.240	0.016 $\|$ 0.100	0.132 $\|$ 0.128	0.167 $\|$ 0.144
Quilt-LLaVA	0.474 $\|$ 0.384	0.351 $\|$ 0.364	0.282 $\|$ 0.301	0.236 $\|$ 0.254	0.475 $\|$ 0.441	0.460 $\|$ 0.462
MIGen	0.403 $\|$ 0.402	0.306 $\|$ 0.466	0.248 $\|$ 0.266	0.209 $\|$ 0.234	0.446 $\|$ 0.322	0.407 $\|$ 0.412
WSICaption	0.203 $\|$ 0.218	0.156 $\|$ 0.137	0.183 $\|$ 0.460	0.130 $\|$ 0.540	0.265 $\|$ 0.251	0.317 $\|$ 0.322
TITAN	0.399 $\|$ 0.382	0.321 $\|$ 0.318	0.223 $\|$ 0.247	0.203 $\|$ 0.314	0.304 $\|$ 0.341	0.385 $\|$ 0.329
HistGen	0.406 $\|$ 0.413	0.307 $\|$ 0.297	0.248 $\|$ 0.229	0.208 $\|$ 0.184	0.448 $\|$ 0.344	0.416 $\|$ 0.182

Models	R $@$ 1	R $@$ 3	R $@$ 5	R $@$ 10	Average
PRISM	0.542 $\|$ 0.634	0.674 $\|$ 0.768	0.724 $\|$ 0.817	0.767 $\|$ 0.880	0.669 $\|$ 0.765
TITAN	0.732 $\|$ 0.754	0.818 $\|$ 0.896	0.841 $\|$ 0.908	0.876 $\|$ 0.922	0.811 $\|$ 0.866
SlideChat	0.766 $\|$ 0.771	0.823 $\|$ 0.855	0.838 $\|$ 0.884	0.852 $\|$ 0.907	0.819 $\|$ 0.854
WSI-LLaVA	0.786 $\|$ 0.770	0.824 $\|$ 0.867	0.844 $\|$ 0.890	0.858 $\|$ 0.912	0.828 $\|$ 0.859
MLLM-HWSI	0.822 $\|$ 0.842	0.856 $\|$ 0.942	0.897 $\|$ 0.952	0.926 $\|$ 0.946	0.875 $\|$ 0.920

Abstract

Explore similar work

PathAgentBench: Benchmarking Evidence-Seeking Vision-Language Models on Whole-Slide Pathology Image

PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology

From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathology