Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays
Authors: Hanbin Ko, Gihun Cho, Inhyeok Baek, Donguk Kim, Joonbeom Koo, Changi Kim, Dongheon Lee, Chang Min Park
Organizations: Interdisciplinary Program in Bioengineering, Seoul National University Graduate School, Seoul, Republic of Korea · Integrated Major in Innovative Medical Science, Seoul National University Graduate School, Seoul, Republic of Korea · Seoul National University College of Medicine, Seoul, Republic of Korea · Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital, Seoul, Republic of Korea · Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea · Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul, Republic of Korea
Multimodal learning from paired medical images and clinical text is a central challenge in medical data-driven informatics, where effective cross-modal alignment is critical for scalable analysis and retrieval. In chest radiography, vision-language pretraining is constrained by heterogeneous radiology reports that contain abbreviations, impression-only notes, and institution-specific writing styles. Unlike general-domain settings, naively aggregating large collections of noisy reports can plateau or even degrade multimodal learning when reporting styles differ substantially. We propose a domain-adapted bidirectional large language model text encoder for chest radiograph reports, trained with masked token prediction and supervised contrastive learning on stylistically diverse but clinically equivalent report variants to produce robust, generalizable text embeddings. We then integrate this encoder into a dual-tower contrastive vision-language framework using parameter-efficient adaptation to improve image-text alignment. Across 1.6 million paired studies from public datasets and a de-identified hospital cohort, the proposed models improve bidirectional retrieval accuracy and external generalization, achieving GREEN scores of 0.308 on MIMIC-CXR and 0.618 on Open-I, while reducing the degradation observed when abbreviation-rich, impression-only hospital reports are added to training.
Figures & tables
Fig. 1: Overview of the proposed framework. (a) Training and evaluation datasets combining public and private data for 1.6M CXR image–report pairs. (b) LLM2VEC4CXR : domain-adapted LLM trained with masked token prediction and contrastive learning. (c) Compared to BERT, LLM-based embeddings cluster clinically similar phrases, yielding more robust representations for multimodal learning.
Pair Type
Count
Original reports
319,564
Split reports
178,993
Prior-omitted reports
169,491
Anatomical-partitioned reports
178,096
Clinically similar reports
272,230
Total
1,118,374
TABLE I: Counts of preprocessed reports used for LLM2VEC4CXR training.
Dataset
Train
Tune
Int. Val.
Ext. Val.
MIMIC-CXR [ 36 ]
247,648
2,032
3,394
–
CheXpert-plus [ 37 ]
190,668
202
–
–
PadChest [ 38 ]
60,261
1,604
–
–
Open-I [ 39 ]
–
–
–
1,825
SNUH
1,136,410
1,328
–
–
TABLE II: Overview of the datasets.
Task
Name
Description
Metric(s)
1
Prior-omitted - Original matching
Given a prior-omitted report ( ro ), the model retrieves the original report to evaluate sensitivity to omitted context.
Top- k retrieval
2
Report summarization
Given the Findings section, the model retrieves the corresponding Impression , assessing summarization and abstraction ability.
Top- k retrieval
3
Report error discrimination
Following ReXErr [ 46 ] , three erroneous impressions (false negation, severity change, etc.) are synthesized per true report. The model must identify the correct impression given the Findings section.
Accuracy
4
Medical acronym understanding
Reports containing acronyms (e.g., “BLLF”, “PTX”) are paired with expert-refined, fully expanded versions (e.g., “bilateral lower lung field”). Tests robustness to lexical variation.
Top- k retrieval
5
Clinical similarity matching
Given a findings section from Open-I, the model retrieves the most similar MIMIC-CXR findings report to assess clinical equivalence.
RadGraph F1, CheXbert F1, RaTEScore, GREEN
TABLE III: Overview of text-only evaluation tasks for assessing LLM2VEC4CXR. Top- k retrieval is used unless otherwise noted.
Prior-omitted Original matching
Report summarization
Report error discrimination
Medical acronym understanding
Clinical similarity matching
Model
@1
@5
@10
@1
@5
@10
Acc
@1
@3
@5
RadGraph
MF1
RaTE
GREEN
Base-BERT [ 40 ]
0.520
0.664
0.708
0.016
0.039
0.054
0.092
0.137
0.384
0.513
0.324
0.206
0.693
0.632
PubMedBERT [ 20 ]
0.447
0.585
0.638
0.030
0.066
0.097
0.223
0.274
0.470
0.598
0.309
0.227
0.696
0.638
BioClinicalBERT [ 11 ]
0.543
0.690
0.743
0.017
0.038
0.054
0.100
0.231
0.385
0.556
0.298
0.207
0.681
0.616
ClinicalBERT [ 10 ]
0.546
0.711
0.767
0.016
0.039
0.054
0.110
0.376
0.692
0.795
0.336
0.284
0.711
0.665
CXR-BERT-Specialized [ 8 ]
0.395
0.537
0.600
0.082
0.155
0.196
0.484
0.248
0.513
0.650
0.284
0.297
0.678
0.664
TABLE IV: Text-only evaluation of LLM2VEC4CXR and baselines. Tasks 1–2: Top- k retrieval, Task 3: accuracy, Task 4: Top- k retrieval, Task 5: clinical efficacy metrics.
Model
Text
Ground Truth (Query)
Cardiomegaly is noted, with central pulmonary vascular prominence and coarsened interstitial markings, suspicious for developing interstitial pulmonary edema . No focal consolidation, pneumothorax, or definite effusion identified. No acute bony abnormality seen.
Base BERT
Patient is status post median sternotomy and CABG. Heart size is normal. Mediastinal contour is unremarkable. Fullness of the right hilum likely reflects known lung cancer. Mild upper zone pulmonary vascular redistribution suggests mild pulmonary vascular congestion. Coarse interstitial opacities are noted in the lung bases which may reflect chronic interstitial abnormality. Additionally, more focal opacity is seen within the periphery of the right lung base, a nonspecific finding. No pleural effusion or pneumothorax is identified. No acute osseous abnormalities seen.
ClinicalBERT
Heart size is normal. Mediastinal and hilar contours are unchanged with architectural distortion, superior bilateral hilar retraction, traction bronchiectasis, nodularity and fibrosis involving both upper lobes. No new focal consolidation, pleural effusion or pneumothorax is present. No pulmonary edema is detected, though there may be mild pulmonary vascular congestion. No acute osseous abnormality is visualized.
LLM2VEC (1B)
The heart is severely enlarged . There are perihilar hazy opacities with vascular indistinctness compatible with mild to moderate pulmonary edema . Assessment of the lung bases is limited due to technique, but there may be atelectasis. No large pleural effusion or pneumothorax is seen.
LLM2VEC4CXR+
The heart size is borderline enlarged . Mediastinal contours are unremarkable. Hilar contours are similar compared to the prior exam. Diffuse increased interstitial markings bilaterally suggest mild interstitial pulmonary edema. No pleural effusion or pneumothorax is identified. No acute osseous abnormality seen.
TABLE VI: Multimodal image–text retrieval results on MIMIC and Open-I.
Model
Student Mean Rank ↓
LLM Mean Rank ↓
Overall Mean Rank ↓
LLM2CLIP4CXR all a
1.85
2.46
2.19
LLM2CLIP4CXR MC
2.46
2.75
2.62
MAIRA2 [ 22 ]
2.50
3.55
3.08
CLIP4CXR
3.54
4.00
3.80
CXR-CLIP [ 13 ]
3.71
4.22
4.03
BC2C [ 45 ]
3.47
3.89
3.71
TABLE VII: Qualitative evaluation on 200 Open-I cases. Student values are averaged across three medical-student raters, and LLM values are averaged across four LLM judges.
Model
Expert mean rank ↓
Expert top-rank votes ↑
Student mean rank ↓
Student top-rank votes ↑
LLM mean rank ↓
LLM top-rank votes ↑
LLM2CLIP4CXR all
1.29
55/72 (76%)
1.14
193/216 (89%)
1.43
191/288 (66%)
MAIRA2 [ 22 ]
2.21
11/72 (15%)
2.07
62/216 (29%)
2.39
48/288 (17%)
BC2C [ 45 ]
2.50
6/72 (8%)
2.35
35/216 (16%)
2.17
53/288 (18%)
TABLE VIII: Thoracic-radiologist-reference ranking on the 72-case subset. Ranks were computed within the three representative systems only; lower mean rank indicates better clinical match to the ground-truth report.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Task type
Prompt template
Semantic similarity
Retrieve semantically similar sentences.
Summarization
Summarize the CXR report.
Finding-level classification
Determine the change or status of the {finding}.
Appendix
TABLE IX: Instruction prompt templates used for LLM2CLIP4CXR training.
Configuration
Trainable parameters
Peak GPU memory
Step time
1B + LoRA (r=16)
16M (1.3%)
∼ 10 GiB
∼ 6 s
1B full FT
1,236M (100%)
∼ 14 GiB
∼ 7 s
8B + LoRA (r=16)
59M (0.7%)
∼ 31 GiB
∼ 44 s
Appendix
TABLE X: Computational cost of LLM2VEC4CXR variants. Peak GPU memory and step time are reported under the same four-A6000 training setup.
Model
Text
Ground Truth (Query)
A tiny calcified granuloma, RULF. No active lung disease.
Base BERT
In comparison with the study of _, the heart remains at the upper limits of normal in size and there is some tortuosity of the aorta . No vascular congestion. No evidence of acute focal pneumonia at this time.
ClinicalBERT
In comparison with the study of _, there is no evidence of acute cardiopulmonary disease at this time. No vascular congestion, pleural effusion, or acute pneumonia. No evidence of old granulomatous disease.
LLM2VEC(1B)
Mild cardiomegaly. Lung volumes are low. There is no focal consolidation. No pneumothorax.
LLM2VEC4CXR+
There is a small calcified granuloma in the right upper lung field. This is of no clinical significance. There is no lung consolidation, pleural effusion, or pneumothorax. Cardiomediastinal silhouette and hilar structures are normal.
Appendix
TABLE XI: Top-1 retrieved report for Task 5. A report from private dataset is used as the query, and the model retrieves the most similar report from the MIMIC pool. Blue = correct findings; red = incorrect/hallucinated; orange = uncertain/partially wrong statements.
Excluded subset
Task 1 ( ↑ )
Task 2 ( ↑ )
Task 3 ( ↑ )
Task 4 ( ↑ )
Task 5 ( ↑ )
@1
@5
@1
@5
Acc
@1
@3
RadGraph
MF1
GREEN
Full (no exclusion)
0.918
0.984
0.203
0.290
0.826
0.607
0.854
0.373
0.715
0.686
– Summarization reports
0.912
0.978
0.115
0.182
0.812
0.611
0.854
0.353
0.698
0.668
– Prior-omitted reports
0.867
0.899
0.182
0.271
0.812
0.594
0.820
0.361
0.664
0.658
– Anatomical-partitioned reports
0.901
0.912
0.196
0.278
0.790
0.607
0.795
0.386
0.712
0.662
– Clinically similar reports
0.910
0.978
0.156
0.206
0.728
0.417
0.783
0.342
0.652
0.646
Appendix
TABLE XII: Ablation of dataset composition for training LLM2VEC4CXR+ .
Pooling method
Task 1 ( ↑ )
Task 2 ( ↑ )
Task 3 ( ↑ )
Task 4 ( ↑ )
Task 5 ( ↑ )
@1
@5
@1
@5
Acc
@1
@3
RadGraph
MF1
GREEN
Latent attention
0.918
0.984
0.203
0.290
0.826
0.607
0.854
0.373
0.715
0.686
Mean pooling
0.901
0.960
0.188
0.271
0.812
0.556
0.762
0.381
0.704
0.670
Appendix
TABLE XIII: Ablation of pooling strategy for training LLM2VEC4CXR+ .
Vision-language pretraining has driven progress in medical image representation learning, but it depends on paired image-text data and can inherit reporting bias from clinical narratives. We study whether language-free predictive pretraining can produce an image encoder that transfers effectively to radiology report generation. RadJEPA is a chest-X-ray adaptation of I-JEPA, pretrained on approximately 840K unlabeled radiographs using latent context-to-target prediction. Our primary contribution is an extensive empirical evaluation of this language-free encoder for report generation: the frozen image encoder is coupled to a trainable two-layer projector and language decoder, and is also substituted into four established vision-language backbones. Across MIMIC-CXR and IU-Xray, RadJEPA matches or exceeds the evaluated image-only and image-text baselines on lexical, entity-relation, and clinical-label metrics. Controlled MIMIC-only comparisons provide evidence that the predictive objective contributes beyond domain-specific pretraining, while broader comparisons also reflect differences in pretraining data, model capacity, and input resolution. Complementary classification and segmentation experiments assess transfer beyond report generation.
Anas Anwarul Haq Khan, Mariam Husain, Pratik Jalan +1
Department of Computer Science (CSE), Indian Institute of Technology Bombay, India · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Department of Biomedical Engineering, Johns Hopkins University, USA +1
Vision-language foundation models have emerged as powerful general-purpose representation learners with strong potential for multimodal understanding, but their deterministic embeddings often fail to provide the reliability required for high-stakes biomedical applications. This work introduces MedProbCLIP, a probabilistic vision-language learning framework for chest X-ray and radiology report representation learning and bidirectional retrieval. MedProbCLIP models image and text representations as Gaussian embeddings through a probabilistic contrastive objective that explicitly captures uncertainty and many-to-many correspondences between radiographs and clinical narratives. A variational information bottleneck mitigates overconfident predictions, while MedProbCLIP employs multi-view radiograph encoding and multi-section report encoding during training to provide fine-grained supervision for clinically aligned correspondence, yet requires only a single radiograph and a single report at inference. Evaluated on the MIMIC-CXR dataset, MedProbCLIP outperforms deterministic and probabilistic baselines, including CLIP, CXR-CLIP, and PCME++, in both retrieval and zero-shot classification. Beyond accuracy, MedProbCLIP demonstrates superior calibration, risk-coverage behavior, selective retrieval reliability, and robustness to clinically relevant corruptions, underscoring the value of probabilistic vision-language modeling for improving the trustworthiness and safety of radiology image-text retrieval systems.
Ahmad Elallaf, Yu Zhang, Yuktha Priya Masupalli +4
Texas A&M University-San Antonio · Boise State University
Large chest radiography archives are difficult to search because most studies are paired only with free-text reports rather than structured clinical annotations. Vision-language models offer a natural interface for text-to-image retrieval, but current biomedical models are primarily optimized for report-to-image matching rather than for satisfying short clinical search queries. This creates an objective mismatch: a model may retrieve images related to words in the query while failing to satisfy the full clinical constraint, especially for conjunctions and negations such as ``atelectasis and no pneumonia.'' We introduce CXR-Retrieve, a structured benchmark for compositional chest X-ray text-to-image retrieval. The benchmark contains 5,159 test images from the official test-split of MIMIC-CXR-JPG and 145 textual queries spanning single and conjunction findings, both positive and negative. Relevance is defined by whether a retrieved image satisfies all asserted pathology constraints, rather than by whether it matches a paired report. We further propose a label-aware contrastive fine-tuning objective for clinical retrieval. Our method attracts image-text pairs with compatible asserted pathology constraints, including shared confirmed absences, while explicitly repelling contradictory pairs. Starting from the in-domain CXR-CLIP checkpoint, our method improves Precision@5 over CXR-CLIP by 8.5 percentage points on two-pathology conjunctions and by 22.0 percentage points on negation queries. These results show that reliable chest X-ray retrieval requires training objectives that model not only which findings are mentioned, but also how they are clinically asserted.
Tomer Erez, Moshe Kimhi, Chaim Baskin +1
Technion – Israel Institute of Technology · Ben-Gurion University of the Negev