Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoning

Wentao Huang; Weimin Lyu; Peiliang Lou; Qingqiao Hu; Xiaoling Hu; Shahira Abousamra; Wenchao Han; Ruifeng Guo

cs.CVFeb 28, 2026

Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoning

Authors: Wentao Huang, Weimin Lyu, Peiliang Lou, Qingqiao Hu, Xiaoling Hu, Shahira Abousamra, Wenchao Han, Ruifeng Guo, +3 more

Organizations: Stony Brook University, NY, USA · Mayo Clinic, MN, USA · Harvard Medical School, MA, USA · Stanford University, CA, USA

Abstract

Computational pathology has advanced rapidly in recent years, driven by domain-specific image encoders and growing interest in using vision-language models to answer natural-language questions about diseases. Yet, the core problem behind pathology question-answering remains unsolved, considering that a gigapixel slide contains far more information than necessary for a given question. Pathologists naturally navigate tissue and morphology complexity by scanning broadly, and zooming in selectively according to the clinical questions. Current models, in contrast, rely on uniform patch sampling or broad attention maps, often attending equally to irrelevant regions while overlooking key visual evidence. In this work, we try to bring models closer to how humans actually examine slides. We propose a question-guided, tissue-aware, and coarse-to-fine retrieval framework, HistoSelect, that consists of two key components: a group sampler that identifies question-relevant tissue regions, followed by a patch selector that retrieves the most informative patches within those regions. By selecting only the most informative patches, our method becomes significantly more efficient: reducing visual token usage by 70% on average, while improving accuracy across three pathology QA tasks. Evaluated on 356,000 question-answer pairs, our approach outperforms existing methods and produces answers grounded in interpretable, pathologist-consistent regions. Our results suggest that bringing human-like search and attention patterns into WSI reasoning is a promising direction for building practical and reliable pathology VLMs. Code is available at https://github.com/winston52/HistoSelect.

Figures & tables

Figure 1: Illustration of our HistoSelect framework. (a) The baseline method feeds a large number of patches indiscriminately into the VLM, leading to high redundancy and question-irrelevance. (b) Our question-guided tissue-aware selection method. The question guides the model to select a relevant and sparse subset of informative patches, which are then fed to the VLM for reasoning.

Figure 2: Visualization and quantitative pre-analysis of patch relevance for a VQA sample (from TCGA-BRCA). (a) Reference WSI. (b) Tissue segmentation, with tumor region shown in red. (c) Patch-level relevance heatmap based on question-patch similarity. High-relevance regions (light region) align with the tumor region from (b). (d) F1 score comparison for retrieving tumor patches using different sampling methods. The question-guided (red) sampling strategy vastly outperforms question-agnostic methods like diversity sampling [ 2 ] (blue) and random sampling (gray), demonstrating the limited efficacy of non-guided selection.

Figure 3: Overview of HistoSelect. The framework operates in two stages: Tissue Segmentation partitions the WSI into $M$ tissue types (e.g., tumor, stromal, lymphocyte) using pathologist-designed prompts. The Hierarchical Selector then uses the question feature to dynamically select the top $K$ most relevant patch tokens, which are subsequently passed to the LLM for multi-modal answer generation.

		SlideBench-VQA (TCGA)			WSI-Bench (Close)			In-house Ovarian	Average
Method	Input	Microscopy	Diagnosis	Clinical	Morphology	Diagnosis	Treatment	Diagnosis	Average
GPT-4o	Thumbnail	39.24	24.12	44.67	47.07	53.06	87.50	—	49.28
Quilt-LLaVA [ 31 ]	Thumbnail	52.39	30.19	49.33	94.13	84.13	97.92	70.67	68.39
LLaVA-Med [ 19 ]	Thumbnail	52.15	29.97	47.33	91.04	81.32	95.83	70.67	66.90
SlideChat [ 8 ]	WSI	83.15	71.36	75.33	91.34	82.15	93.75	69.33	80.88
HistoSelect	WSI	84.62	73.09	77.30	94.57	85.79	97.92	73.33	83.80

Table 1: Close-ended VQA accuracy (%) across three benchmarks: SlideBench-VQA, WSI-Bench, and our In-house Ovarian dataset. Our method consistently achieves the highest accuracy across all task categories and obtains the best overall average performance.

	WSI-Bench
	Report Generation					Morphology		Diagnosis		Treatment
Method	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	WSI-P	WSI-R	WSI-P	WSI-R	WSI-P	WSI-R
GPT-4o	0.202	0.069	0.030	0.016	0.132	0.220	0.204	0.472	0.457	0.513	0.704
WSI-VQA [ 6 ]	0.301	0.225	0.181	0.155	0.343	0.395	0.462	0.436	0.525	0.591	0.595
MI-Gen [ 5 ]	0.403	0.306	0.248	0.209	0.446	—	—	—	—	—	—
Histo-Gen [ 12 ]	0.406	0.307	0.248	0.208	0.448	—	—	—	—	—	—

Table 2: Open-ended VQA performance on the WSI-Bench dataset. We evaluate two tasks: Report Generation and domain-specific VQA . Our method achieves the highest performance across all report-generation metrics and the best results on 5 of 6 domain-specific metrics.

Figure 4: Visualization of tissue segmentation and selection process. (a) Original WSI. (b) Tissue segmentation mask. (c)Visualization before selection (a randomly selected subset is shown for clarity). (d) Visualization after selection. Compared to (c), the patches selected by our model in (d) significantly remove non-tumor patches, demonstrating an improved focus on informative tumor-related regions.

Category	Question ID	Avg. Score (P1)	Avg. Score (P2)
Tissue Seg.	Q1 Accuracy	4.17	3.67
Patch Selection	Q2.1 Question-relevant	4.80	3.87
Patch Selection	Q2.2 Answer-relevant	4.67	3.73

Table 3: Average scores from the pathologist evaluation survey.

Table 8

Figure 5: The user interface for the Tissue Segmentation Survey. The central area shows a side-by-side comparison of the original WSI and the tissue segmentation result. The right legend clarifies the tissue classes, and the bottom section collects the pathologists’ rating.

Figure 6: The user interface for the Patch Selection Survey. The view visualizes the “Before” and “After” states of patch selection. Note that the input question and ground truth answer are displayed below the images to provide necessary context for the pathologists’ evaluation.

	Microscopy				Diagnosis				Clinical
Method	Acc.	Macro-P	Macro-R	Macro-F1	Acc.	Macro-P	Macro-R	Macro-F1	Acc.	Macro-P	Macro-R	Macro-F1
GPT-4o	39.24	36.12	39.45	35.54	24.12	22.83	24.34	21.72	44.67	42.93	44.95	43.58
Quilt-LLaVA [ 31 ]	52.39	46.75	50.09	46.84	30.19	27.94	30.34	27.15	49.33	47.01	48.91	47.54
LLaVA-Med [ 19 ]	52.15	46.30	49.80	46.51	29.97	27.84	29.93	26.86	47.33	45.12	47.19	45.58
SlideChat [ 8 ]	83.15	81.77	78.50	79.70	71.36	71.80	65.22	67.52	75.33	73.84	72.74	72.98
Ours	84.62	80.79	80.47	80.33	73.09	72.10	68.08	69.22	77.30	73.97	74.24	73.94

Table 7: Quantitative results on SlideBench-VQA (TCGA). Performance is evaluated using accuracy and macro-averaged metrics.

	Morphology				Diagnosis				Treatment (Binary)
Method	Acc.	Macro-P	Macro-R	Macro-F1	Acc.	Macro-P	Macro-R	Macro-F1	Acc.	Macro-P	Macro-R	Macro-F1
GPT-4o	47.07	42.89	47.29	43.19	53.06	49.02	53.37	49.27	87.50	79.12	83.76	81.05
Quilt-LLaVA [ 31 ]	94.13	91.74	91.19	91.42	84.13	81.35	78.68	79.63	97.92	98.75	94.44	96.42
LLaVA-Med [ 19 ]	91.04	86.21	87.59	86.83	81.32	77.80	74.81	76.02	95.83	93.16	93.16	93.16
SlideChat [ 8 ]	91.34	86.11	87.92	86.97	82.15	79.15	75.58	77.02	93.75	88.68	91.88	90.16
Ours	94.57	92.66	91.34	91.85	85.79	85.03	79.45	81.70	97.92	95.00	98.72	96.72

Table 8: Quantitative results on WSI-Bench (Close-ended). Performance is evaluated using accuracy and macro-averaged metrics.

Figure 7: The sample distribution for the test set.

Figure 8: Additional visualization of the selection process on the public WSI-Bench dataset. (a) Original WSI. (b) The tissue segmentation mask. (c) A visualization of candidate patches extracted from tissue regions prior to selection. (d) The sparse set of patches retained by our model. As observed in (d), the model effectively suppresses irrelevant regions, focusing the attention solely on the informative patches required for the VQA task.

Figure 9: Visualization of the selection process on the private Ovarian dataset. The figure follows the same pipeline as the main manuscript and Figure 8 : (a) Original WSI. (b) Tissue segmentation mask. (c) Patches before selection. (d) Patches after selection. These results demonstrate the robustness of our method against domain shifts common in private clinical data. The model successfully filters out non-informative tissue, preserving only the regions essential for accurate question answering.

Figure 10: Mean sampling rate distribution bar charts for identified clusters. We report the average 13-dimensional sampling vectors for the four clusters discovered in Fig. 11 . Each cluster exhibits a unique sampling rate pattern.

Figure 11: Visualization of question-aware sampling distributions. We visualize the 13-dimensional sampling rate vectors from the WSI-Bench Diagnosis and Morphology test set using t-SNE.

$\beta_{g}$	Morphology	Diagnosis	Treatment
0	91.78	81.82	95.83
0.1	93.39	84.13	95.83
0.2	94.57	85.79	97.92
0.3	93.10	83.25	93.75

Table 9: Ablation Study on the weight

\beta_{g}

for the group sampler.

$\beta_{p}$	Morphology	Diagnosis	Treatment
0	92.07	81.32	93.75
0.05	93.25	84.23	95.83
0.10	94.57	85.79	97.92
0.15	93.83	83.74	95.83

Table 10: Ablation study on the weight

\beta_{p}

for the patch selector.

Training Strategy	Morphology	Diagnosis	Treatment	Avg.
Joint Training	94.57	85.79	97.92	92.76
Joint + Patch Selector	94.71	85.95	97.92	92.86
Joint + Group Sampler	95.15	86.11	97.92	93.06

Table 11: Ablation on training strategies.

Method	ACC	Macro-P	Macro-R	Macro-F1
Gemini 3 Flash	59.5	57.0	60.0	57.0
Gemini 3 Flash + HistoSelect	62.5	58.0	64.0	59.0

Table 12: Performance comparison on WSI-Bench (200 samples).

Question Type	Base	Base + Ideal Group	Gain ( $\Delta$ )
Single-tissue (Easy)	85.0	87.0	+2.0
Multi-tissue (Hard)	73.0	79.0	+6.0

Table 13: Ablation on group selection.

Explore similar work

PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQA
May 22, 2026Chunze Yang, Qidong Liu, Wenjie Zhao +10Healthcare Medical VQA
School of Comp. Science & Technology, Xi’an Jiaotong University · Tencent Jarvis Lab · University of Cambridge
Agentic Visual Reasoning in Whole-Slide Pathology Images via Active Perception
Aug 9, 2026Jingyun Chen, Fengchun Liu, Linghan Cai +5Active Perception Medical VQA
College of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · School of Computing and Data Science, The University of Hong Kong
PathAgentBench: Benchmarking Evidence-Seeking Vision-Language Models on Whole-Slide Pathology Image
Jul 21, 2026Dankai Liao, Tianyi Zhang, Yufeng Wu +6VLM Evaluation Medical VLMs
Department of Biomedical Engineering, National University of Singapore, Singapore 117417, Singapore

May 22, 2026J/K move · Enter open · S save

PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQA

Chunze Yang, Qidong Liu, Wenjie Zhao, Yue Tang+9

School of Comp. Science & Technology, Xi’an Jiaotong University · Tencent Jarvis Lab · University of Cambridge

Healthcare Medical VQA Tool-Using Vision-Language Agents Visual Search Whole-Slide Image Analysis

Open paper AskPDF arXiv

Whole-slide image visual question answering (WSI-VQA) frames pathology as an extreme-context search problem: to answer a free-form clinical query, a system must first navigate a gigapixel slide under a strict inspection budget to locate sparse, high-resolution evidence. Existing approaches largely fall into two paradigms: i) supervised pathology multimodal large language models (MLLMs) and agents can absorb localization and reasoning into learned modules, but they often couple navigation to task-specific supervision and retraining, limiting their practicality; ii) training-free pathology agents avoid this cost by keeping core models frozen, but often follow a question-first design, constructing the initial candidate set mainly from query-conditioned relevance. This can miss decisive morphology that is not named in the question, and force heavier inference-time scaffolding. To address this challenge, we introduce PathNavigate, a training-free pathology agent built around a scan-search-readout routine. Before question matching, PathNavigate scans the current slide at low magnification with a shared online memory module over frozen pathology features, producing a slide-specific surprise field that marks an abnormal-region pool. It then applies question-conditioned PLIP relevance only within this pool to select high-magnification search targets. Finally, it extracts local high-magnification evidence and answers with a frozen perceptor-adjudicator stack, using the same online memory as slide-level context. Experiments on WSI-VQA and SlideBench-BCNB show that the proposed scan-search-readout design improves answer accuracy and yields more interpretable evidence-selection trajectories with higher efficiency.The code is available online.

Abstract

Explore similar work

PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQA

Agentic Visual Reasoning in Whole-Slide Pathology Images via Active Perception

PathAgentBench: Benchmarking Evidence-Seeking Vision-Language Models on Whole-Slide Pathology Image