cs.CVSep 17, 2025

Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays

Authors: Hanbin Ko, Gihun Cho, Inhyeok Baek, Donguk Kim, Joonbeom Koo, Changi Kim, Dongheon Lee, Chang Min Park

Organizations: Interdisciplinary Program in Bioengineering, Seoul National University Graduate School, Seoul, Republic of Korea · Integrated Major in Innovative Medical Science, Seoul National University Graduate School, Seoul, Republic of Korea · Seoul National University College of Medicine, Seoul, Republic of Korea · Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital, Seoul, Republic of Korea · 

Abstract

Multimodal learning from paired medical images and clinical text is a central challenge in medical data-driven informatics, where effective cross-modal alignment is critical for scalable analysis and retrieval. In chest radiography, vision-language pretraining is constrained by heterogeneous radiology reports that contain abbreviations, impression-only notes, and institution-specific writing styles. Unlike general-domain settings, naively aggregating large collections of noisy reports can plateau or even degrade multimodal learning when reporting styles differ substantially. We propose a domain-adapted bidirectional large language model text encoder for chest radiograph reports, trained with masked token prediction and supervised contrastive learning on stylistically diverse but clinically equivalent report variants to produce robust, generalizable text embeddings. We then integrate this encoder into a dual-tower contrastive vision-language framework using parameter-efficient adaptation to improve image-text alignment. Across 1.6 million paired studies from public datasets and a de-identified hospital cohort, the proposed models improve bidirectional retrieval accuracy and external generalization, achieving GREEN scores of 0.308 on MIMIC-CXR and 0.618 on Open-I, while reducing the degradation observed when abbreviation-rich, impression-only hospital reports are added to training.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture

    Jan 22, 2026Anas Anwarul Haq Khan, Mariam Husain, Pratik Jalan +1Radiology Report GenerationMedical Imaging

  2. MedProbCLIP: Probabilistic Adaptation of Vision-Language Foundation Model for Reliable Radiograph-Report Retrieval

    Feb 17, 2026Ahmad Elallaf, Yu Zhang, Yuktha Priya Masupalli +4Radiology VLMsVision-Language Models

  3. CXR-Retrieve: Compositional Text-to-Image Retrieval in Chest Radiography

    Jul 30, 2026Tomer Erez, Moshe Kimhi, Chaim Baskin +1Contrastive LearningMedical Imaging