Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays
Authors: Hanbin Ko, Gihun Cho, Inhyeok Baek, Donguk Kim, Joonbeom Koo, Changi Kim, Dongheon Lee, Chang Min Park
Organizations: Interdisciplinary Program in Bioengineering, Seoul National University Graduate School, Seoul, Republic of Korea · Integrated Major in Innovative Medical Science, Seoul National University Graduate School, Seoul, Republic of Korea · Seoul National University College of Medicine, Seoul, Republic of Korea · Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital, Seoul, Republic of Korea · Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea · Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul, Republic of Korea
Multimodal learning from paired medical images and clinical text is a central challenge in medical data-driven informatics, where effective cross-modal alignment is critical for scalable analysis and retrieval. In chest radiography, vision-language pretraining is constrained by heterogeneous radiology reports that contain abbreviations, impression-only notes, and institution-specific writing styles. Unlike general-domain settings, naively aggregating large collections of noisy reports can plateau or even degrade multimodal learning when reporting styles differ substantially. We propose a domain-adapted bidirectional large language model text encoder for chest radiograph reports, trained with masked token prediction and supervised contrastive learning on stylistically diverse but clinically equivalent report variants to produce robust, generalizable text embeddings. We then integrate this encoder into a dual-tower contrastive vision-language framework using parameter-efficient adaptation to improve image-text alignment. Across 1.6 million paired studies from public datasets and a de-identified hospital cohort, the proposed models improve bidirectional retrieval accuracy and external generalization, achieving GREEN scores of 0.308 on MIMIC-CXR and 0.618 on Open-I, while reducing the degradation observed when abbreviation-rich, impression-only hospital reports are added to training.
Figures & tables
Fig. 1: Overview of the proposed framework. (a) Training and evaluation datasets combining public and private data for 1.6M CXR image–report pairs. (b) LLM2VEC4CXR : domain-adapted LLM trained with masked token prediction and contrastive learning. (c) Compared to BERT, LLM-based embeddings cluster clinically similar phrases, yielding more robust representations for multimodal learning.
Pair Type
Count
Original reports
319,564
Split reports
178,993
Prior-omitted reports
169,491
Anatomical-partitioned reports
178,096
Clinically similar reports
272,230
Total
1,118,374
TABLE I: Counts of preprocessed reports used for LLM2VEC4CXR training.
Dataset
Train
Tune
Int. Val.
Ext. Val.
MIMIC-CXR [ 36 ]
247,648
2,032
3,394
–
CheXpert-plus [ 37 ]
190,668
202
–
–
PadChest [ 38 ]
60,261
1,604
–
–
Open-I [ 39 ]
–
–
–
1,825
SNUH
1,136,410
1,328
–
–
TABLE II: Overview of the datasets.
Task
Name
Description
Metric(s)
1
Prior-omitted - Original matching
Given a prior-omitted report ( ro ), the model retrieves the original report to evaluate sensitivity to omitted context.
Top- k retrieval
2
Report summarization
Given the Findings section, the model retrieves the corresponding Impression , assessing summarization and abstraction ability.
Top- k retrieval
3
Report error discrimination
Following ReXErr [ 46 ] , three erroneous impressions (false negation, severity change, etc.) are synthesized per true report. The model must identify the correct impression given the Findings section.
Accuracy
4
Medical acronym understanding
Reports containing acronyms (e.g., “BLLF”, “PTX”) are paired with expert-refined, fully expanded versions (e.g., “bilateral lower lung field”). Tests robustness to lexical variation.
Top- k retrieval
5
Clinical similarity matching
Given a findings section from Open-I, the model retrieves the most similar MIMIC-CXR findings report to assess clinical equivalence.
RadGraph F1, CheXbert F1, RaTEScore, GREEN
TABLE III: Overview of text-only evaluation tasks for assessing LLM2VEC4CXR. Top- k retrieval is used unless otherwise noted.
Prior-omitted Original matching
Report summarization
Report error discrimination
Medical acronym understanding
Clinical similarity matching
Model
@1
@5
@10
@1
@5
@10
Acc
@1
@3
@5
RadGraph
MF1
RaTE
GREEN
Base-BERT [ 40 ]
0.520
0.664
0.708
0.016
0.039
0.054
0.092
0.137
0.384
0.513
0.324
0.206
0.693
0.632
PubMedBERT [ 20 ]
0.447
0.585
0.638
0.030
0.066
0.097
0.223
0.274
0.470
0.598
0.309
0.227
0.696
0.638
BioClinicalBERT [ 11 ]
0.543
0.690
0.743
0.017
0.038
0.054
0.100
0.231
0.385
0.556
0.298
0.207
0.681
0.616
ClinicalBERT [ 10 ]
0.546
0.711
0.767
0.016
0.039
0.054
0.110
0.376
0.692
0.795
0.336
0.284
0.711
0.665
CXR-BERT-Specialized [ 8 ]
0.395
0.537
0.600
0.082
0.155
0.196
0.484
0.248
0.513
0.650
0.284
0.297
0.678
0.664
TABLE IV: Text-only evaluation of LLM2VEC4CXR and baselines. Tasks 1–2: Top- k retrieval, Task 3: accuracy, Task 4: Top- k retrieval, Task 5: clinical efficacy metrics.
Model
Text
Ground Truth (Query)
Cardiomegaly is noted, with central pulmonary vascular prominence and coarsened interstitial markings, suspicious for developing interstitial pulmonary edema . No focal consolidation, pneumothorax, or definite effusion identified. No acute bony abnormality seen.
Base BERT
Patient is status post median sternotomy and CABG. Heart size is normal. Mediastinal contour is unremarkable. Fullness of the right hilum likely reflects known lung cancer. Mild upper zone pulmonary vascular redistribution suggests mild pulmonary vascular congestion. Coarse interstitial opacities are noted in the lung bases which may reflect chronic interstitial abnormality. Additionally, more focal opacity is seen within the periphery of the right lung base, a nonspecific finding. No pleural effusion or pneumothorax is identified. No acute osseous abnormalities seen.
ClinicalBERT
Heart size is normal. Mediastinal and hilar contours are unchanged with architectural distortion, superior bilateral hilar retraction, traction bronchiectasis, nodularity and fibrosis involving both upper lobes. No new focal consolidation, pleural effusion or pneumothorax is present. No pulmonary edema is detected, though there may be mild pulmonary vascular congestion. No acute osseous abnormality is visualized.
LLM2VEC (1B)
The heart is severely enlarged . There are perihilar hazy opacities with vascular indistinctness compatible with mild to moderate pulmonary edema . Assessment of the lung bases is limited due to technique, but there may be atelectasis. No large pleural effusion or pneumothorax is seen.
LLM2VEC4CXR+
The heart size is borderline enlarged . Mediastinal contours are unremarkable. Hilar contours are similar compared to the prior exam. Diffuse increased interstitial markings bilaterally suggest mild interstitial pulmonary edema. No pleural effusion or pneumothorax is identified. No acute osseous abnormality seen.
TABLE VI: Multimodal image–text retrieval results on MIMIC and Open-I.
Model
Student Mean Rank ↓
LLM Mean Rank ↓
Overall Mean Rank ↓
LLM2CLIP4CXR all a
1.85
2.46
2.19
LLM2CLIP4CXR MC
2.46
2.75
2.62
MAIRA2 [ 22 ]
2.50
3.55
3.08
CLIP4CXR
3.54
4.00
3.80
CXR-CLIP [ 13 ]
3.71
4.22
4.03
BC2C [ 45 ]
3.47
3.89
3.71
TABLE VII: Qualitative evaluation on 200 Open-I cases. Student values are averaged across three medical-student raters, and LLM values are averaged across four LLM judges.
Model
Expert mean rank ↓
Expert top-rank votes ↑
Student mean rank ↓
Student top-rank votes ↑
LLM mean rank ↓
LLM top-rank votes ↑
LLM2CLIP4CXR all
1.29
55/72 (76%)
1.14
193/216 (89%)
1.43
191/288 (66%)
MAIRA2 [ 22 ]
2.21
11/72 (15%)
2.07
62/216 (29%)
2.39
48/288 (17%)
BC2C [ 45 ]
2.50
6/72 (8%)
2.35
35/216 (16%)
2.17
53/288 (18%)
TABLE VIII: Thoracic-radiologist-reference ranking on the 72-case subset. Ranks were computed within the three representative systems only; lower mean rank indicates better clinical match to the ground-truth report.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Task type
Prompt template
Semantic similarity
Retrieve semantically similar sentences.
Summarization
Summarize the CXR report.
Finding-level classification
Determine the change or status of the {finding}.
Appendix
TABLE IX: Instruction prompt templates used for LLM2CLIP4CXR training.
Configuration
Trainable parameters
Peak GPU memory
Step time
1B + LoRA (r=16)
16M (1.3%)
∼ 10 GiB
∼ 6 s
1B full FT
1,236M (100%)
∼ 14 GiB
∼ 7 s
8B + LoRA (r=16)
59M (0.7%)
∼ 31 GiB
∼ 44 s
Appendix
TABLE X: Computational cost of LLM2VEC4CXR variants. Peak GPU memory and step time are reported under the same four-A6000 training setup.
Model
Text
Ground Truth (Query)
A tiny calcified granuloma, RULF. No active lung disease.
Base BERT
In comparison with the study of _, the heart remains at the upper limits of normal in size and there is some tortuosity of the aorta . No vascular congestion. No evidence of acute focal pneumonia at this time.
ClinicalBERT
In comparison with the study of _, there is no evidence of acute cardiopulmonary disease at this time. No vascular congestion, pleural effusion, or acute pneumonia. No evidence of old granulomatous disease.
LLM2VEC(1B)
Mild cardiomegaly. Lung volumes are low. There is no focal consolidation. No pneumothorax.
LLM2VEC4CXR+
There is a small calcified granuloma in the right upper lung field. This is of no clinical significance. There is no lung consolidation, pleural effusion, or pneumothorax. Cardiomediastinal silhouette and hilar structures are normal.
Appendix
TABLE XI: Top-1 retrieved report for Task 5. A report from private dataset is used as the query, and the model retrieves the most similar report from the MIMIC pool. Blue = correct findings; red = incorrect/hallucinated; orange = uncertain/partially wrong statements.
Excluded subset
Task 1 ( ↑ )
Task 2 ( ↑ )
Task 3 ( ↑ )
Task 4 ( ↑ )
Task 5 ( ↑ )
@1
@5
@1
@5
Acc
@1
@3
RadGraph
MF1
GREEN
Full (no exclusion)
0.918
0.984
0.203
0.290
0.826
0.607
0.854
0.373
0.715
0.686
– Summarization reports
0.912
0.978
0.115
0.182
0.812
0.611
0.854
0.353
0.698
0.668
– Prior-omitted reports
0.867
0.899
0.182
0.271
0.812
0.594
0.820
0.361
0.664
0.658
– Anatomical-partitioned reports
0.901
0.912
0.196
0.278
0.790
0.607
0.795
0.386
0.712
0.662
– Clinically similar reports
0.910
0.978
0.156
0.206
0.728
0.417
0.783
0.342
0.652
0.646
Appendix
TABLE XII: Ablation of dataset composition for training LLM2VEC4CXR+ .
Pooling method
Task 1 ( ↑ )
Task 2 ( ↑ )
Task 3 ( ↑ )
Task 4 ( ↑ )
Task 5 ( ↑ )
@1
@5
@1
@5
Acc
@1
@3
RadGraph
MF1
GREEN
Latent attention
0.918
0.984
0.203
0.290
0.826
0.607
0.854
0.373
0.715
0.686
Mean pooling
0.901
0.960
0.188
0.271
0.812
0.556
0.762
0.381
0.704
0.670
Appendix
TABLE XIII: Ablation of pooling strategy for training LLM2VEC4CXR+ .
Department of Computer Science (CSE), Indian Institute of Technology Bombay, India · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Department of Biomedical Engineering, Johns Hopkins University, USA +1