Towards Scalable Diagnostic-Driven Whole-Slide Image Retrieval for Pathology Archives

Jingsong Liu; Youssef Wally Han Li; Weiwei Zhou; Jing Dai; Elisabeth Wetzer; Peter J. Schüffler

cs.CVNov 5, 2026

Towards Scalable Diagnostic-Driven Whole-Slide Image Retrieval for Pathology Archives

Authors: Jingsong Liu, Youssef Wally Han Li, Weiwei Zhou, Jing Dai, Elisabeth Wetzer, Peter J. Schüffler

Organizations: Institute of Pathology, Technical University of Munich, TUM School of Medicine and Health, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Munich Data Science Institute (MDSI), Munich, Germany · Department of Physics and Technology, UiT The Arctic University of Norway, Tromsø, Norway ·

Abstract

The growing adoption of digital pathology has rapidly expanded collections of hematoxylin-and-eosin whole-slide images, yet efficient navigation through these archives remains challenging, as metadata is often inconsistent or unavailable. Content-based image retrieval addresses this challenge by identifying similar cases directly from visual representations instead of manual annotations or metadata. Existing retrieval systems have demonstrated promising performance but largely rely on handcrafted indexing pipelines and predefined sampling strategies, constraining scalability and generalization. Here, we present CLEAR-WSI (Constant-Length Embedding for Automatic Retrieval of Whole-Slide Images) as a scalable, diagnostic-driven slide-retrieval framework, validated on both public and clinical data. Using a fixed-length slide embedding, CLEAR-WSI refines retrieved candidates with either a predicted diagnostic label or the ground-truth label, depending on the retrieval setting. Newly processed cases are subsequently added to the expanding memory bank, allowing the retrieval archive to grow over time. On a comprehensive benchmark consisting of two public and one private clinical datasets representing four retrieval tasks, CLEAR-WSI improves Precision@5 over the strongest evaluated baseline, with gains on CAMELYON16 (3.10), BRACS-1 (2.75), BRACS-2 (1.95), and X-HER2 (3.75). Additional ablation and latency studies further examine the contribution of each component and highlight its potential for scalable deployment.

Figures & tables

Figure 1: Overview of the CLEAR-WSI framework for scalable, diagnosis-aligned retrieval from growing clinical archives. Left: (1) A growing clinical archive contains WSIs from heterogeneous sources. (2) Each WSI is processed by Diagnostic-driven Aggregation (DX-Agg), implemented with Attention-based MIL, to identify diagnostically relevant evidence and capture global slide morphology. (3) DX-Agg maps each variable-size patch set to a fixed-length slide embedding $Z_{i}$ and diagnostic prediction $Y_{i}$ , which are stored in an ever-growing memory bank, providing constant per-slide storage and predictable index expansion. (4) Given a query WSI, CLEAR-WSI retrieves the top- $k$ similar cases and refines the ranked list using either Self-Retrieval (SR) filtering, when the query diagnosis is unknown, or Class-Informed (CI) filtering, when it is available. Right: Key features of CLEAR-WSI , including fixed-length storage, automated diagnosis-focused retrieval, and a scalable, ever-growing memory bank.

Figure 2: Comparative retrieval performance of CLEAR-WSI across CAMELYON16, BRACS-1, BRACS-2, and X-HER2, together with available results from existing WSI retrieval methods. Each task sector reports normalized discounted cumulative gain (NDCG@

k

) and precision at rank

k

(P@

k

, i.e., Precision@

k

) for

k\in\{1,3,5\}

. Self-Retrieval (SR) is the fully automated CLEAR-WSI configuration and does not require the query ground-truth label. Class-Informed (CI) uses the query ground-truth label and is therefore shown as a dashed oracle reference. For visualization, CAMELYON16, BRACS-1, and X-HER2 use a radial maximum of 100, whereas BRACS-2 uses a radial maximum of 80; values should therefore be compared using the labeled scale within each task sector.

Medical Image Retrieval

See all 21 papers

Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment
Oct 27, 2025Hongyi Wang, Zhengjie Zhu, Junlin Hou +13Multimodal IR Computational Pathology
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China. · School of Medicine, Wake Forest University, Winston-Salem, NC, USA. +11
Reducing Redundancy in Whole-Slide Image Patching for Scalable Indexing and Retrieval
Jun 23, 2026Jialiang Geng, Ghazal Alabtah, Saghir Alfasly +2Whole-Slide Image Analysis
KIMIA Lab, Dept. of Artificial Intelligence & Informatics, Mayo Clinic, Rochester, MN, USA
CRISP -- Clustering-Based Redundancy-Reduced Instance Sampling for Pathology Case Representation and Retrieval
May 22, 2026Zahra Rahimi Afzal, Wataru Uegami, Saghir Alfasly +6Computational Pathology Medical Image Representation Learning
Kimia Lab, Department of Artificial Intelligence & Informatics, Mayo Clinic, Rochester, MN, USA · DICE Lab, Department of Electrical and Computer Engineering, University of Illinois Chicago, IL, USA · Division of Computational Pathology and Informatics, Mayo Clinic, Rochester, MN, USA +4

Oct 27, 2025J/K move · Enter open · S save

Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment

Hongyi Wang, Zhengjie Zhu, Junlin Hou, Jiabo Ma+12

Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China. · School of Medicine, Wake Forest University, Winston-Salem, NC, USA.+11

Multimodal IR Computational Pathology Image-Text Retrieval Medical Image Retrieval

Open paper AskPDF arXiv

The rapid digitization of histopathology slides has opened new opportunities for computational tools in clinical and research workflows. Content-based slide retrieval can help pathologists identify morphologically and semantically related precedent cases, supporting expert diagnosis and example-based education. Effective retrieval of whole-slide images (WSIs), however, remains challenging because gigapixel slides contain abundant irrelevant content, focal diagnostic patterns and slide-level semantic information that must be represented at a practicable search cost. Here we present PathSearch, a retrieval framework that combines fine-grained attentive mosaics with slide-level embeddings aligned through vision-language contrastive learning. Trained on 6,926 slide-report pairs, PathSearch captures both fine-grained morphological cues and high-level semantic patterns to enable accurate and flexible retrieval. The framework supports two key functionalities: (1) mosaic-based image-to-image (I2I) retrieval, ensuring accurate and efficient slide search; and (2) multimodal retrieval, where text queries can directly retrieve relevant slides. PathSearch was evaluated on eight tasks comprising 5,021 evaluation slides, spanning malignancy assessment on frozen and hematoxylin and eosin (H&E)-stained slides, lymph-node metastasis detection, tumor subtyping, mixed-gallery rare-cancer retrieval, and hepatocellular carcinoma (HCC) risk stratification. Internal and external experimental results demonstrate that PathSearch consistently outperforms the strongest existing methods without compromising multimodal accuracy. A multi-center reader study further demonstrated increases in task-level mean diagnostic accuracy, confidence, and inter-observer agreement with PathSearch's support. Together, these results support the effectiveness of PathSearch across diverse retrieval tasks and evaluation settings.

Abstract

Medical Image Retrieval

Explore similar work

Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment

Reducing Redundancy in Whole-Slide Image Patching for Scalable Indexing and Retrieval

CRISP -- Clustering-Based Redundancy-Reduced Instance Sampling for Pathology Case Representation and Retrieval