Composed Vision-Language Retrieval for Skin Cancer Case Search via Joint Alignment of Global and Local Representations

Yuheng Wang; Yuji Lin; Jiayue Cai; Z. Jane Wang; Tim K. Lee

cs.CVMar 10, 2026

Composed Vision-Language Retrieval for Skin Cancer Case Search via Joint Alignment of Global and Local Representations

Authors: Yuheng Wang, Yuji Lin, Jiayue Cai, Z. Jane Wang, Tim K. Lee

Abstract

Medical image retrieval aims to identify clinically relevant lesion cases to support diagnostic decision making, education, and quality control. In practice, retrieval queries often combine a reference lesion image with textual descriptors such as dermoscopic features. We study composed vision-language retrieval for skin cancer, where each query consists of an image to text pair and the database contains biopsy-confirmed, multi-class disease cases. We propose a transformer based framework that learns hierarchical composed query representations and performs joint global-local alignment between queries and candidate images. Local alignment aggregates discriminative regions via multiple spatial attention masks, while global alignment provides holistic semantic supervision. The final similarity is computed through a convex, domain-informed weighting that emphasizes clinically salient local evidence while preserving global consistency. Experiments on the public Derm7pt dataset demonstrate consistent improvements over state-of-the-art methods. The proposed framework enables efficient access to relevant medical records and supports practical clinical deployment.

Figures & tables

Fig. 1: Types of clinical skin cancer case search and query. Conventional retrieval uses image-only queries or text-only clinical descriptors separately, whereas our composed retrieval pairs a reference lesion image with its associated text to form a vision-language query. All settings aim to retrieve visually similar, biopsy-confirmed cases from an image-only database to support clinical decision making.

Fig. 2: Overall workflow of composed vision-language retrieval for skin cancer. The query image and text are fused via cross-modal Transformers on top of a hierarchical vision backbone to form multi-level composed query representations (a), while each database image is encoded by the same backbone (b). Multi-level global and local alignment jointly compute query-target similarity for retrieval ranking (c).

Method	Acc@1	Acc@2	Acc@4
ResNet50-CosSim [ 19 ]	77.6 $\pm$ 3.5	80.9 $\pm$ 2.6	83.2 $\pm$ 1.9
SNF-DCA [ 21 ]	77.8 $\pm$ 2.3	82.8 $\pm$ 1.4	83.3 $\pm$ 2.5
MaskRCNN-Fusion [ 8 ]	62.5 $\pm$ 1.2	77.3 $\pm$ 3.1	87.3 $\pm$ 2.5
DAHNET [ 9 ]	74.4 $\pm$ 2.1	74.9 $\pm$ 1.8	75.6 $\pm$ 2.7
DEML [ 24 ]	76.3 $\pm$ 1.3	80.2 $\pm$ 1.5	84.3 $\pm$ 2.2
WalshHash [ 4 ]	77.5 $\pm$ 2.0	81.6 $\pm$ 1.6	84.9 $\pm$ 1.2

TABLE I: Accuracy@K overall performance across 5-fold cross-validation (Mean

\pm

Std, in %).

Method	Fold 1	Fold 2	Fold 3	Fold 4	Fold 5	Mean $\pm$ Std
ResNet50-CosSim [ 19 ]	78.5	81.7	84.0	77.2	80.6	80.4 $\pm$ 2.7
SNF-DCA [ 21 ]	79.4	81.6	80.4	80.1	78.4	80.0 $\pm$ 1.2
MaskRCNN-Fusion [ 8 ]	61.3	57.8	60.4	61.5	61.2	60.5 $\pm$ 1.5
DAHNET [ 9 ]	79.1	80.5	81.6	80.2	81.5	80.6 $\pm$ 1.0
DEML [ 24 ]	80.1	81.4	82.2	78.4	79.8	80.4 $\pm$ 1.5
WalshHash [ 4 ]	80.6	82.3	80.4	79.9	80.6	80.8 $\pm$ 0.9

TABLE II: Mean Average Precision (mAP) for each fold and overall performance across 5-fold cross-validation (in %).

Fig. 3: Qualitative composed retrieval examples on Derm7pt. Each row shows one query (left: query image; middle: associated clinical text) and the top-5 retrieved images (right, R@1–R@5). Rows 1–3 correspond to mel (red borders), bkl (yellow borders), and nevus (blue borders), respectively.

Medical Image Retrieval

See all 18 papers

Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment
Oct 27, 2025Hongyi Wang, Zhengjie Zhu, Junlin Hou +13Multimodal IR Computational Pathology
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China. · School of Medicine, Wake Forest University, Winston-Salem, NC, USA. +11
SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis
Mar 22, 2026Zhixiang Lu, Shijie Xu, Kaicheng Yan +6Medical VLMs VLM Adaptation
Xi’an Jiaotong-Liverpool University, China · University of Liverpool, United Kingdom
Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT
Mar 2, 2026Simon Ging, Philipp Arnold, Sebastian Walter +6Medical Image Retrieval Medical VLMs
Computer Vision Group, University of Freiburg, Germany · Adaptive & Agentic AI · Adaptive & Agentic AI (A3) Lab, Aarhus University, Denmark +5

Oct 27, 2025J/K move · Enter open · S save

Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment

Hongyi Wang, Zhengjie Zhu, Junlin Hou, Jiabo Ma+12

Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China. · School of Medicine, Wake Forest University, Winston-Salem, NC, USA.+11

Multimodal IR Computational Pathology Image-Text Retrieval Medical Image Retrieval

Open paper AskPDF arXiv

The rapid digitization of histopathology slides has opened new opportunities for computational tools in clinical and research workflows. Content-based slide retrieval can help pathologists identify morphologically and semantically related precedent cases, supporting expert diagnosis and example-based education. Effective retrieval of whole-slide images (WSIs), however, remains challenging because gigapixel slides contain abundant irrelevant content, focal diagnostic patterns and slide-level semantic information that must be represented at a practicable search cost. Here we present PathSearch, a retrieval framework that combines fine-grained attentive mosaics with slide-level embeddings aligned through vision-language contrastive learning. Trained on 6,926 slide-report pairs, PathSearch captures both fine-grained morphological cues and high-level semantic patterns to enable accurate and flexible retrieval. The framework supports two key functionalities: (1) mosaic-based image-to-image (I2I) retrieval, ensuring accurate and efficient slide search; and (2) multimodal retrieval, where text queries can directly retrieve relevant slides. PathSearch was evaluated on eight tasks comprising 5,021 evaluation slides, spanning malignancy assessment on frozen and hematoxylin and eosin (H&E)-stained slides, lymph-node metastasis detection, tumor subtyping, mixed-gallery rare-cancer retrieval, and hepatocellular carcinoma (HCC) risk stratification. Internal and external experimental results demonstrate that PathSearch consistently outperforms the strongest existing methods without compromising multimodal accuracy. A multi-center reader study further demonstrated increases in task-level mean diagnostic accuracy, confidence, and inter-observer agreement with PathSearch's support. Together, these results support the effectiveness of PathSearch across diverse retrieval tasks and evaluation settings.

Abstract

Medical Image Retrieval

Explore similar work

Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment

SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis

Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT