Look Beyond the Question: Preserve-Focus-Fuse for Whole-Slide Image Question Answering

Xiangyu Li; Jincheng Li; Tianle Du; Yanheng Li; Haoxuan Che; Yuxin Sun; Long Bai; Liangliang Liu

cs.CVOct 21, 2026

Look Beyond the Question: Preserve-Focus-Fuse for Whole-Slide Image Question Answering

Authors: Xiangyu Li, Jincheng Li, Tianle Du, Yanheng Li, Haoxuan Che, Yuxin Sun, Long Bai, Liangliang Liu,

Organizations: Tianjin University · Shanghai AI Laboratory · University of Science and Technology of China · Nantong University ·

Abstract

A question about a whole-slide image rarely names every morphological finding needed to answer it. Question-guided patch selection can therefore overlook supporting morphology, while slide-level compression can dilute focal evidence. Inspired by how pathologists combine broad slide review, question-directed examination, and integration of their findings, we introduce BEYONDPATH, a Preserve-Focus-Fuse framework for whole-slide image question answering. Preserve learns a question-agnostic Morphology Bank through answer supervision across multiple questions about each training slide. Focus retrieves questionspecific evidence from all foreground patches, including regions outside the Bank. Fuse integrates the preserved morphology at the retrieved evidence positions through Morphology-on-Visual-Evidence Fusion, producing a fixed 96-token visual prefix. On WSI-Bench, BEYONDPATH achieves an Avg-9 of 0.769 and leads all eight reported report-generation metrics using the same checkpoint. It also achieves 0.639 zero-shot overall accuracy on SlideBench-BCNB. Controlled ablations show complementary benefits from morphology preservation and question-directed retrieval, and support cross-question supervision and fusion at the evidence positions. These results support looking beyond the question's explicit cues: preserving broader slide morphology while retaining the ability to seek evidence for each question.

Figures & tables

Figure 1: Preserve–Focus–Fuse looks beyond explicit question cues. (a) Question-guided selection can miss supporting morphology; (b) broad compression can dilute focal evidence; (c) BeyondPath preserves a question-agnostic Morphology Bank, retrieves evidence from all foreground patches, and fuses both at the retrieved positions. Cross-question answer supervision trains the Bank; the output is a 96-token visual prefix. Snowflakes and flames mark frozen and trainable components.

Figure 2: Overview of BeyondPath . Preserve–Focus–Fuse draws inspiration from broad slide review, question-directed examination, and integration of findings. Cross-question answer supervision trains Preserve to construct the Morphology Bank. Focus retrieves evidence from all foreground patches, and MoV Fusion integrates the Bank at the retrieved evidence positions, producing a 96-token visual prefix.

Figure 3: Preserve, Focus, and MoV implement Preserve–Focus–Fuse. Preserve selects four semantic and two free expert banks with relaxed-backward training followed by hard selection. Focus retrieves from all foreground patches. MoV uses evidence tokens to query the banks and position-specific gates while retaining 96 output positions.

(a) Preserve & Focus		(b) Bank & Fusion		(c) Configuration
Setting	Avg-9	Setting	Avg-9	Setting	Avg-9
Focus only	0.721	Random same-slide Bank	0.713	Budget $=64$	0.760
Preserve only	0.713	Wrong-slide Bank	0.677	Budget $=192$	0.772
Single-stream	0.739	Concat. + Resampler (96)	0.727	MoV blocks $=1$	0.760
Bank-restricted Focus	0.718	Direct concatenation (192)	0.739	MoV blocks $=4$	0.765
Single-QA Preserve	0.736	Uniform expert gate	0.756	Experts $=4/0$	0.759

Table 4: WSI-Bench ablations (Avg-9). LLM prefixes use 96 tokens unless specified. The configuration tuple denotes (tokens, MoV blocks, semantic/free experts). Bank-restricted Focus limits retrieval to patches selected by Preserve. Best and second-best results are shown in red and blue , respectively.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Stage I	Stage II	Inference
CONCH image encoder	Frozen	Frozen	Frozen
CONCH text encoder	Frozen	Frozen	Frozen
Feature adapter	Train	Frozen	Frozen
Morphology router	Train	Frozen	Frozen
Expert FFNs and identities	Train	Frozen	Frozen
Semantic null banks	Train	Frozen	Frozen

Appendix

Table 6: Module states during training and inference.

Expert	Prompt ensemble
Tumor	a histopathology image of tumor tissue ; malignant neoplastic cells ; invasive tumor morphology ; a tumor-rich histology region
Stroma	a histopathology image of stromal tissue ; fibrous connective tissue stroma ; desmoplastic tumor-associated stroma ; a stroma-rich histology region
Immune infiltration	a histopathology image with immune-cell infiltration ; lymphocyte-rich tissue ; tumor-infiltrating lymphocytes ; an inflammatory cell-rich histology region
Necrosis	a histopathology image of necrotic tissue ; tumor necrosis ; coagulative necrosis and cellular debris ; a necrosis-rich histology region

Appendix

Table 7: Prompt ensembles for the four semantic morphology experts.

Symbol	Meaning	Shape
$N_{s}$	foreground patches in slide $s$	scalar
$X_{s}$	adapted content features	$N_{s}\times 512$
$Z_{s}^{\mathrm{vl}}$	normalized CONCH features	$N_{s}\times 512$
$q_{c}$	normalized question embedding	$512$
$B_{s,e}$	bank of expert $e$	$k_{e}\times 512$
$M_{s}$	concatenated Morphology Bank	$K_{m}\times 512$

Appendix

Table 8: Principal notation and tensor shapes.

Tokens per stream	Avg-9	Memory (GB)	TFLOPs
32	0.742	15.72	2.580
64	0.760	15.83	2.920
96	0.769	15.95	3.304
128	0.771	16.06	3.580
192	0.772	16.27	4.150

Appendix

Table 11: Joint evidence and Bank budget. Memory is peak GPU allocation. Higher Avg-9 and lower memory/TFLOPs are better. Best and second-best results are shown in red and blue , respectively.

Panel of Figure 6 — Figure 6: Qualitative evidence for complementary preservation and retrieval. (a) Case 3 asks for the Nottingham histologic grade. The full model selects Grade 2, whereas Focus-only and random-patch variants favor Grade 3 and Preserve-only favors Grade 1; the full model also has the lowest displayed NLL. (b) Case 4 asks for the histological classification. The full model gives an invasive ductal carcinoma interpretation and cites both invasive glands in desmoplastic stroma and an associated cribriform DCIS component.

	BeyondPath		SlideChat
$N_{s}$	Time	95% interval	Time	95% interval
1,024	2.05	[1.96, 2.17]	1.68	[1.60, 1.80]
2,048	2.07	[1.98, 2.19]	1.94	[1.83, 2.08]
4,096	2.11	[2.01, 2.24]	2.47	[2.31, 2.67]
8,192	2.20	[2.08, 2.34]	3.55	[3.31, 3.84]
16,384	2.36	[2.22, 2.52]	5.92	[5.50, 6.43]

Appendix

Table 13: Single-question inference latency as the number of retained foreground patches increases. Times are in seconds with 95% intervals and exclude CONCH feature extraction. Lower latency is better. Best and second-best results are shown in red and blue , respectively.

Visual Question Answering

Visual Question Answering (VQA) develops systems that answer natural-language questions about images, videos, and visual documents.

See all 253 papers

PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQA
May 22, 2026Chunze Yang, Qidong Liu, Wenjie Zhao +10Healthcare Medical VQA
School of Comp. Science & Technology, Xi’an Jiaotong University · Tencent Jarvis Lab · University of Cambridge
Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoning
Feb 28, 2026Wentao Huang, Weimin Lyu, Peiliang Lou +8Pathology VQA Medical VQA
Stony Brook University, NY, USA · Mayo Clinic, MN, USA · Harvard Medical School, MA, USA +1
SlideBank: A Persistent Hierarchical Evidence Bank for Consistent Whole-Slide Reasoning
Aug 31, 2026Beidi Zhao, Gexin Huang, Ciro Zhang +6Memory-Augmented VLMs Medical VQA
University of British Columbia · Vector Institute · Harvard University +3

May 22, 2026J/K move · Enter open · S save

PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQA

Chunze Yang, Qidong Liu, Wenjie Zhao, Yue Tang+9

School of Comp. Science & Technology, Xi’an Jiaotong University · Tencent Jarvis Lab · University of Cambridge

Healthcare Medical VQA Tool-Using Vision-Language Agents Visual Search Whole-Slide Image Analysis Pathology VQA

Open paper AskPDF arXiv

Whole-slide image visual question answering (WSI-VQA) frames pathology as an extreme-context search problem: to answer a free-form clinical query, a system must first navigate a gigapixel slide under a strict inspection budget to locate sparse, high-resolution evidence. Existing approaches largely fall into two paradigms: i) supervised pathology multimodal large language models (MLLMs) and agents can absorb localization and reasoning into learned modules, but they often couple navigation to task-specific supervision and retraining, limiting their practicality; ii) training-free pathology agents avoid this cost by keeping core models frozen, but often follow a question-first design, constructing the initial candidate set mainly from query-conditioned relevance. This can miss decisive morphology that is not named in the question, and force heavier inference-time scaffolding. To address this challenge, we introduce PathNavigate, a training-free pathology agent built around a scan-search-readout routine. Before question matching, PathNavigate scans the current slide at low magnification with a shared online memory module over frozen pathology features, producing a slide-specific surprise field that marks an abnormal-region pool. It then applies question-conditioned PLIP relevance only within this pool to select high-magnification search targets. Finally, it extracts local high-magnification evidence and answers with a frozen perceptor-adjudicator stack, using the same online memory as slide-level context. Experiments on WSI-VQA and SlideBench-BCNB show that the proposed scan-search-readout design improves answer accuracy and yields more interpretable evidence-selection trajectories with higher efficiency.The code is available online.

Abstract

Visual Question Answering

Explore similar work

PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQA

Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoning

SlideBank: A Persistent Hierarchical Evidence Bank for Consistent Whole-Slide Reasoning