MApLe: Multi-instance Alignment of Diagnostic Reports and Large Medical Images

Felicia Bader; Philipp Seeböck; Anastasia Bartashova; Ulrike Attenberger; Georg Langs

cs.CVApr 15, 2026

MApLe: Multi-instance Alignment of Diagnostic Reports and Large Medical Images

Authors: Felicia Bader, Philipp Seeböck, Anastasia Bartashova, Ulrike Attenberger, Georg Langs

Organizations: Computational Imaging Research Lab, Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Austria · Comprehensive Center for Artificial Intelligence in Medicine, Medical University of Vienna, Austria · Medical Anomaly Detection (MANO) Group, Computational Imaging Research (CIR), Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Austria · Department of Biomedical Imaging and Image-Guided Therapy, Medical University of Vienna, Austria

Abstract

In diagnostic reports, experts encode complex imaging data into clinically actionable information. They describe subtle pathological findings that are meaningful in their anatomical context. Reports follow relatively consistent structures, expressing diagnostic information with few words that are often associated with tiny but consequential image observations. Standard vision language models struggle to identify the associations between these informative text components and small locations in the images. Here, we propose "MApLe", a multi-task, multi-instance vision language alignment approach that overcomes these limitations. It disentangles the concepts of anatomical region and diagnostic finding, and links local image information to sentences in a patch-wise approach. Our method consists of a text embedding trained to capture anatomical and diagnostic concepts in sentences, a patch-wise image encoder conditioned on anatomical structures, and a multi-instance alignment of these representations. We demonstrate that MApLe can successfully align different image regions and multiple diagnostic findings in free-text reports. We show that our model improves the alignment performance compared to state-of-the-art baseline models when evaluated on several downstream tasks. The code is available at https://github.com/cirmuw/MApLe.

Figures & tables

Figure 1: Multi-instance alignment of diagnostic reports and image volumes. (a) During training the report is decomposed into sentences, each describing a finding, and its corresponding class (e.g., present or not). Anatomical regions in images are segmented, and patches are extracted. A finding specific multi-instance alignment of sentence- and image representations maps textual finding classes and associated image patches into a joint embedding space. (b) During test time, all patches of an anatomical region are first encoded and than pooled together using the corresponding finding network. The resulting profile describes the diagnostically relevant findings.

Figure 2: Boxplots of the mean pairwise cosine-similarities of sentences within each diagnostic finding between and across different states of this finding for (a) the BioClinical BERT model Alsentzer et al. (2019) and (b) our fine-tuned text encoder.

Method	Zero-shot	Acc.	F1	Sens.	Spec.	PPV	NPV	AUC
Calcification
CLIP Radford et al. (2021a)	$\times$	0.5139	0.6789	1.0	0.0	0.5139	0.0	0.5
ConVIRT Zhang et al. (2022)	$\times$	0.5833	0.7000	0.9459	0.2000	0.5556	0.7778	0.5730
GLoRIA Huang et al. (2021)	$\checkmark$	0.4722	0.4493	0.4189	0.5286	0.4844	0.4625	0.4738
LSE+NL Wang et al. (2023)	$\checkmark$	0.5139	0.6789	1.0	0.0	0.5139	0.0	0.5
MApLe (ours)	$\checkmark$	0.5731	0.6078	0.6515	0.4922	0.5695	0.5780	0.5719

Table 1: Classification results of our zero-shot approach on our internal 3D heart CT dataset compared to SOTA alignment methods CLIP Radford et al. (2021a) , ConVIRT Zhang et al. (2022) , GLoRIA Huang et al. (2021) and LSE+NL Wang et al. (2023) . Zero-shot methods are models that don’t have to be fine-tuned after the alignment, in contrast to the others that have to be fine-tuned for every single prediction task separately. The highest accuracy, F1-score and AUC among the zero-shot methods are marked.

Radiology VLMs

Radiology vision-language models (VLMs) align medical images and text to support grounded diagnosis, reporting, retrieval, and clinical reasoning.

See all 109 papers

HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction
Mar 20, 2026Ruicheng Yuan, Zhenxuan Zhang, Anbang Wang +5Computational Pathology Medical Report Generation
College of Computer Science and Electronic Engineering, Hunan University, Changsha, Hunan, China · Department of Bioengineering and Imperial-X, Imperial College London, London, UK · Department of Pathology, Xiangtan Maternal and Child Health Hospital, Xiangtan, Hunan, China +1
A4-MR: Abnormality-aware Anatomical Alignment for MR
Oct 9, 2026Noel D'souza, Zhijian Yang, Nishank Singla +14Medical Imaging Foundation Models Radiology VLMs
GE Healthcare, Seattle, USA · GE Healthcare, Budapest, Hungary · Amazon, USA
Learning to See Locally and Align Clinically with Pathology Semantics for Radiology Report Generation
Jul 31, 2026Xuan Cuong NgoRadiology Report Generation VLM Robustness

Mar 20, 2026J/K move · Enter open · S save

HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction

Ruicheng Yuan, Zhenxuan Zhang, Anbang Wang, Liwei Hu+4

College of Computer Science and Electronic Engineering, Hunan University, Changsha, Hunan, China · Department of Bioengineering and Imperial-X, Imperial College London, London, UK · Department of Pathology, Xiangtan Maternal and Child Health Hospital, Xiangtan, Hunan, China+1

Computational Pathology Medical Report Generation Healthcare Medical VLMs Vision-Language Alignment

Open paper AskPDF arXiv

Pathology reports are structured, multi-granular documents encoding diagnostic conclusions, histological grades, and ancillary test results across one or more anatomical sites; yet existing pathology vision-language models (VLMs) reduce this output to a flat label or free-form text. We present HiPath, a lightweight VLM framework built on frozen UNI2 and Qwen3 backbones that treats structured report prediction as its primary training objective. Three trainable modules totalling 15M parameters address complementary aspects of the problem: a Hierarchical Patch Aggregator (HiPA) for multi-image visual encoding, Hierarchical Contrastive Learning (HiCL) for cross-modal alignment via optimal transport, and Slot-based Masked Diagnosis Prediction (Slot-MDP) for structured diagnosis generation. Trained on 749K real-world Chinese pathology cases from three hospitals, HiPath achieves 68.9% strict and 74.7% clinically acceptable accuracy with a 97.3% safety rate, outperforming all baselines under the same frozen backbone. Cross-hospital evaluation confirms generalisation with only a 3.4pp drop in strict accuracy while maintaining 97.1% safety.

Abstract

Radiology VLMs

Explore similar work

HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction

A4-MR: Abnormality-aware Anatomical Alignment for MR

Learning to See Locally and Align Clinically with Pathology Semantics for Radiology Report Generation