PathPilot: An Adaptive Question-Conditioned Multi-Magnification Navigation Agent for Whole-Slide Image Analysis
Authors: Zhengyang Xu, Han Li, Jingsong Liu, Ayako Ura, Shihui Zu, Shaohua Kevin Zhou, Nassir Navab, Peter Schüffler
Organizations: Institute of Pathology, Technical University of Munich, Germany · Munich Center for Machine Learning, Munich, Germany · Chair of Computer Aided Medical Procedures, Technical University of Munich, Germany · Department of Human Pathology, Juntendo University Graduate School of Medicine, Tokyo, Japan · School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, China · University of Science and Technology of China, China · Munich Data Science Institute, Technical University of Munich, Germany
The diagnostic capability of WSI agents relies on effective slide navigation to identify diagnostically relevant regions and efficiently integrate evidence across the slide. While in clinical practice, pathologists dynamically examine slides across multiple scales, integrating global and cellular evidence in a sequential manner, existing AI navigation approaches mostly operate at a single fixed magnification or rely on predefined magnification traversal. This mismatch prevents existing methods from modeling cross-magnification interactions and adaptive magnification selection inherent to real diagnostic workflows. We propose PathPilot, a clinically consistent multi-magnification WSI navigation agent that explicitly models cross-magnification localization and evidence-adaptive exploration. Specifically, we introduce a CrossMagLoc that learns magnification-specific diagnostic localization from pathologist viewing behavior and integrates information from adjacent magnifications to identify the diagnostically most relevant region within the current ROI, thereby providing a diagnostically informed starting point for subsequent question-conditioned exploration. We further introduce an EviRegionSearch that starts question-conditioned exploration from this region, progressively accumulating question-relevant evidence and adaptively determining the next navigation action according to the remaining evidence requirement. Their iterative interaction enables PathPilot to sequentially explore across ROIs and magnifications until sufficient evidence is acquired. Extensive QA evaluations across five cohorts demonstrate consistent superiority over competing SOTA methods on inter/external cohorts, respectively, validating question-conditioned evidence acquisition across regions and magnifications that mirrors the sequential, knowledge-driven navigation of pathologists.
Figures & tables
Fig. 1: Overview of PathPilot for adaptive WSI exploration. For each navigation step (e.g., Step t ), CrossMagLoc identifies the diagnostically most relevant region within the current ROI, while EviRegionSearch accumulates question-relevant evidence and adaptively selects the next navigation action (e.g., Step t+1 ). Their iterative interaction enables sequential movement across ROIs and magnifications until evidence is sufficient (e.g., Step t+N ), where Stop is selected and the final answer is generated.
Fig. 2: Overview of PathPilot and its two complementary navigation components. Given a WSI and pathology question, PathPilot starts from the thumbnail and iteratively alternates between CrossMagLoc and EviRegionSearch until sufficient evidence has been accumulated for answering the question. CrossMagLoc performs question-agnostic localization of diagnostically relevant candidate ROIs at each magnification by combining magnification-specific representations with spatially aligned information from adjacent scales. EviRegionSearch performs question-conditioned exploration over the ROI identified by CrossMagLoc, integrates morphological observations from the selected ROI with the accumulated evidence, identifies the remaining evidence requirement, and adaptively determines the next navigation action (Move, Zoom In, Zoom Out, or Stop) to select the next region.
Fig. 3: Overview of the datasets and evaluation cohorts used in this study. Top: Three navigation datasets provide pathologist viewing supervision for CrossMagLoc, including the Skin Eye-Tracking dataset and the TUM Liver and TUM Uterine Viewpoint datasets. The Skin dataset additionally provides pixel-level tumor annotations for region-level localization evaluation. Bottom: Five open-ended QA cohorts are used for end-to-end PathPilot evaluation, comprising the internal TUM Liver and TUM-Uteria cohorts and the external TUM Skin, WSI-Bench TCGA-SKCM, and WSI-Bench TCGA-LIHC cohorts. For the TUM cohorts, the circular plots summarize the predefined H&E-visible findings grouped into tissue-level, architectural, and cellular categories; for the WSI-Bench cohorts, they summarize the distributions of question categories.
Fig. 4: QA performance of PathPilot across internal and external validation cohorts. Comparisons include pathology-specialized thumbnail-based VLMs (PathGen-LLaVA and Patho-R1-7B), patch-based WSI VLMs (WSI-LLaVA and ANTONI-Alpha), and pathology agents (PathAgent and PathNavigate). The left radar plot summarizes average performance across all questions on two internal cohorts (TUM-Uteria and TUM Liver) and three external cohorts (TUM Skin, WSI-Bench TCGA-SKCM, and WSI-Bench TCGA-LIHC). The four right-hand plots further assess tissue-level, architectural-level, and cellular-level QA on the three TUM cohorts, and morphology-focused QA on the two WSI-Bench cohorts, distinguishing overall QA performance from performance on specific pathological assessment tasks. Each cohort is evaluated using GACC, PBR, and 1−IAR , with higher values indicating better performance on all axes. Colored outer arcs identify cohorts; solid and dashed arcs denote internal and external validation, respectively. The plots summarize the quantitative results in Tables 2 – 5 .
Fig. 5: Pathologist-in-the-loop interpretation of CrossMagLoc across five magnifications (thumbnail, 1.25 × , 2.5 × , 5 × , and 10 × ; top to bottom). The columns show the pathology images (column 1), CrossMagLoc predictions (column 2), test-set navigation annotations (column 3), and independent annotations with morphological explanations from pathologists trained in Europe (column 4) and Asia (column 5). Across magnifications, CrossMagLoc consistently prioritizes diagnostically relevant regions identified by the pathologists while deprioritizing non-primary or artifact-limited areas. At 2.5 × , it assigns higher attention to well-preserved tumor regions than to the crush-artifact region, while at higher magnifications it continues to emphasize morphologically informative tumor regions. Overall, its localization broadly agrees with pathologist viewing behavior while additionally highlighting clinically relevant regions identified during independent review.
Metric
Expected
WSI-Bench TCGA-SKCM
WSI-Bench TCGA-LIHC
Pooled
Variant-level judge scores
Paraphrase ( SP )
High
0.957 [0.923, 0.991]
0.903 [0.855, 0.952]
0.932 [0.900, 0.959]
Omission ( SO )
≈SP
0.829 [0.769, 0.886]
0.845 [0.787, 0.894]
0.836 [0.797, 0.876]
Attribute error ( SA )
Lower
0.611 [0.560, 0.654]
0.348 [0.252, 0.439]
0.488 [0.427, 0.545]
Core contradiction ( SC )
Lowest
0.000 [0.000, 0.000]
0.000 [0.000, 0.000]
0.000 [0.000, 0.000]
Overall ordered sensitivity
Table 6: Validation of the LLM-based evaluator on the morphology subsets of WSI-Bench TCGA-SKCM and TCGA-LIHC using four controlled answer variants: paraphrase (P), omission (O), attribute error (A), and core contradiction (C). The first block reports the evaluator scores for the four variants, the second block evaluates whether the expected ordering ( P≥O≥A≥C ) is preserved, and the final block reports score differences between adjacent variants. Scores are reported as mean [95% CI]. Overall, paraphrasing has little effect on the score, omission causes only a small decrease, factual attribute errors lead to a clear score reduction, and core contradictions are consistently assigned a score of 0.
Fig. 6: Quantitative evaluation of CrossMagLoc navigation performance. The radar plot summarizes pathologist-prioritization alignment ( ρN , AP5N , and 1−JSN ) and tumor-region precision/recall at the thumbnail, 1.25 × , and 2.5 × magnifications, with PathFinder [ 14 ] and PathAttFormer [ 6 ] as navigation baselines and pathologist navigation as a clinical reference for tumor-region alignment. CrossMagLoc achieves the strongest agreement with pathologist prioritization while maintaining high tumor recall across magnifications, indicating a similar balance between diagnostic focus and broad tumor coverage.
Fig. 7: Visualization of a representative question-conditioned navigation workflow. CrossMagLoc-based ROI localization and EviRegionSearch-based evidence reasoning are iteratively coupled to support adaptive exploration across regions and magnifications, evidence requirement reasoning, and final answer generation once sufficient evidence is acquired.
Recent AI navigation approaches aim to improve Whole-Slide Image (WSI) diagnosis by modeling spatial exploration and selecting diagnostically relevant regions, yet most operate at a single fixed magnification or rely on predefined magnification traversal. In clinical practice, pathologists examine slides across multiple magnifications and selectively inspect only necessary scales, dynamically integrating global and cellular evidence in a sequential manner. This mismatch prevents existing methods from modeling cross-magnification interactions and adaptive magnification selection inherent to real diagnostic workflows. To these, we propose a clinically consistent Multi-Magnification WSI Navigation Agent (MMNavAgent) that explicitly models multi magnification interaction and adaptive magnification selection. Specifically, we introduce a Cross-Magnification navigation Tool (CMT) that aggregates contextual information from adjacent magnifications to enhance discriminative representations along the navigation path. We further introduce a Magnification Selection Tool (MST) that leverages memory-driven reasoning within the agent framework to enable interactive and adaptive magnification selection, mimicking the sequential decision process of pathologists. Extensive experiments on a public dataset demonstrate improved diagnostic performance, with 1.45% gain of AUC and 2.93% gain of BACC over a non-agent baseline. Code will be public upon acceptance.
Zhengyang Xu, Han Li, Jingsong Liu +10
Institute of Pathology, Technical University of Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Computer Aided Medical Procedures (CAMP), TU Munich, Munich, Germany +7
Whole-slide image visual question answering (WSI-VQA) frames pathology as an extreme-context search problem: to answer a free-form clinical query, a system must first navigate a gigapixel slide under a strict inspection budget to locate sparse, high-resolution evidence. Existing approaches largely fall into two paradigms: i) supervised pathology multimodal large language models (MLLMs) and agents can absorb localization and reasoning into learned modules, but they often couple navigation to task-specific supervision and retraining, limiting their practicality; ii) training-free pathology agents avoid this cost by keeping core models frozen, but often follow a question-first design, constructing the initial candidate set mainly from query-conditioned relevance. This can miss decisive morphology that is not named in the question, and force heavier inference-time scaffolding. To address this challenge, we introduce PathNavigate, a training-free pathology agent built around a scan-search-readout routine. Before question matching, PathNavigate scans the current slide at low magnification with a shared online memory module over frozen pathology features, producing a slide-specific surprise field that marks an abnormal-region pool. It then applies question-conditioned PLIP relevance only within this pool to select high-magnification search targets. Finally, it extracts local high-magnification evidence and answers with a frozen perceptor-adjudicator stack, using the same online memory as slide-level context. Experiments on WSI-VQA and SlideBench-BCNB show that the proposed scan-search-readout design improves answer accuracy and yields more interpretable evidence-selection trajectories with higher efficiency.The code is available online.
Chunze Yang, Qidong Liu, Wenjie Zhao +10
School of Comp. Science & Technology, Xi’an Jiaotong University · Tencent Jarvis Lab · University of Cambridge
Whole-slide visual reasoning requires identifying sparse diagnostic evidence in gigapixel pathology slides and integrating observations across spatial scales. Existing WSI methods either compress densely sampled patches into global representations or use pretrained vision-language models with heuristic region selection, weakening links between predictions and morphology or lacking pathology-trained observation policies. We present AdaptivePath, an active-perception framework that formulates WSI evidence acquisition as sequential decision making. The Navigator learns question-agnostic abnormality-driven navigation from pathologist-reviewed labels to select observation locations and spatial extents, avoiding costly question-specific trajectory annotations. We train this policy through alternating representation learning and proximal policy optimization, followed by fine-tuning with geometric and appearance consistency objectives to stabilize focus trajectories. During inference, the Navigator hierarchically acquires sparse observations from low to high magnification under a limited ROI budget. A Morphology Interpreter converts observations into question-conditioned evidence, while the Deliberator evaluates evidence and revises intermediate answers across magnifications. The Arbiter integrates deliberation history to produce final answers. AdaptivePath achieves state-of-the-art zero-shot performance on WSI and region pathology VQA benchmarks and reaches 80.14% accuracy for cancer subtype classification across six TCGA cohorts. In a blinded diagnostic-utility study, pathologists using AdaptivePath-selected observation sequences achieve 82.9% accuracy. These results demonstrate that learned active perception enables effective and traceable visual reasoning over gigapixel pathology slides.
Jingyun Chen, Fengchun Liu, Linghan Cai +5
College of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · School of Computing and Data Science, The University of Hong Kong