Seeing Like Radiologists: Context- and Gaze-Guided Vision-Language Pretraining for Chest X-rays

Kang Liu; Zhuoqi Ma; Siyu Liang; Yunan Li; Xiyue Gao; Chao Liang; Kun Xie; Qiguang Miao

cs.CVMar 27, 2026

Seeing Like Radiologists: Context- and Gaze-Guided Vision-Language Pretraining for Chest X-rays

Authors: Kang Liu, Zhuoqi Ma, Siyu Liang, Yunan Li, Xiyue Gao, Chao Liang, Kun Xie, Qiguang Miao

Organizations: Xidian University, Xi’an, China · Wuhan University, Wuhan, China

Abstract

Despite recent advances in medical vision-language pretraining, existing models still struggle to capture the diagnostic workflow: radiographs are typically treated as context-agnostic images, while radiologists' gaze -- a crucial cue for visual reasoning -- remains largely underexplored by existing methods. These limitations hinder the modeling of disease-specific patterns and weaken cross-modal alignment. To bridge this gap, we introduce CoGaze, a Context- and Gaze-guided vision-language pretraining framework for chest X-rays. We first propose a context-infused vision encoder that models how radiologists integrate clinical context -- including patient history, symptoms, and diagnostic intent -- to guide diagnostic reasoning. We then present a multi-level supervision paradigm that (1) enforces intra- and inter-modal semantic alignment through hybrid-positive contrastive learning, (2) injects diagnostic priors via disease-aware cross-modal representation learning, and (3) leverages radiologists' gaze as probabilistic priors to guide attention toward diagnostically salient regions. Extensive experiments demonstrate that CoGaze consistently outperforms state-of-the-art methods across diverse tasks, achieving up to +2.0% CheXbertF1 and +1.2% BLEU2 for free-text and structured report generation, +23.2% AUROC for zero-shot classification, and +12.2% Precision@1 for image-text retrieval. Code is available at https://github.com/mk-runner/CoGaze.

Figures & tables

Figure 1. Overview of the pretraining framework. (a) Image-report pairs with optional clinical context. (b) Heatmap-transcript pairs capturing physicians’ visual attention. (c-d) Comparison between existing methods and ours. (e) Image-text retrieval results on the MIMIC-5x200 dataset ( Zhou et al., 2024 ) . Clinical context and gaze data are available for a subset of cases. “TS” denotes transcript.

Figure 2. (A) Overview of our CoGaze. (B) Context-infused vision encoder. Gaze supervision is used only during pretraining.

Figure 3. Workflow of downstream tasks. All tasks are performed without requiring gaze heatmaps.

Method	Venue	NLG Metrics $\uparrow$						CE Metrics $\uparrow$
Method	Venue	BLEU1	BLEU2	BLEU3	BLEU4	METEOR	R-L	P	R	14 Mi-F1
KiUT ( Huang et al., 2023 )	CVPR’23	0.393	0.243	0.159	0.113	0.160	0.285	0.371	0.318	0.321
METransformer ( Wang et al., 2023a )	CVPR’23	0.386	0.250	0.169	0.124	0.152	0.291	0.364	0.309	0.311
MAN ( Shen et al., 2024 )	AAAI’24	0.396	0.244	0.162	0.115	0.151	0.274	0.411	0.398	0.389
R2GenGPT ( Wang et al., 2023b )	Meta-Radio’23	0.411	0.267	0.186	0.134	0.160	0.297	0.392	0.387	0.389
Med-LLM ( Liu et al., 2024c )	MM’24	-	-	-	0.128	0.161	0.289	0.412	0.373	0.395

Table 1. Free-text report generation on MIMIC-CXR ( Johnson et al., 2019 ) dataset.

\bigtriangleup

denotes CoGaze’s gain over the strongest baseline. EGMA ( Ma et al., 2024 ) is reproduced using its official code with DistilGPT2, while other results are from original papers ( Best , Second Best ).

Model	Base Metrics $\uparrow$			F1-SRR ( Delbrouck et al., 2025 ) $\uparrow$
Model	BLEU	R-L	RG	P	R	F1
CheXagent ( Chen et al., 2024 )	1.80	19.65	15.41	77.12	82.56	77.90
RaDialog ( Pellegrini et al., 2025 )	1.28	17.53	13.82	69.48	70.12	69.76
CoGaze ♠	2.80	20.23	14.23	75.82	85.61	78.32
CoGaze ♡	3.00	21.64	15.53	74.83	85.56	78.07

Table 2. Structured report generation on SRRG-Findings ( Delbrouck et al., 2025 ) dataset. “RG” denotes F1-RadGraph ( Jain et al., 2021 ) . ♠ and ♡ are DistilGPT2 and Llama-3B variants of CoGaze ( Best , Second Best ).

Figure 4. Zero-shot image-text retrieval on MIMIC-5x200 dataset. P@K: precision at top-K; R@K: recall at top-K.

Model	NIH (AUROC $\uparrow$ )			SIIM (F1 $\uparrow$ )			RSNA (F1 $\uparrow$ )
Model	1%	10%	100%	1%	10%	100%	1%	10%	100%
MedCLIP-ViT ( Wang et al., 2022b )	76.1 $\pm$ 0.3	81.4 $\pm$ 0.25	84.5 $\pm$ 0.17	68.6 $\pm$ 0.8	71.5 $\pm$ 1.1	75.7 $\pm$ 0.2	63.5 $\pm$ 0.5	65.3 $\pm$ 1.0	66.2 $\pm$ 0.8
MedKLIP ( Wu et al., 2023 )	75.2 $\pm$ 0.1	80.3 $\pm$ 0.08	83.9 $\pm$ 0.08	61.4 $\pm$ 0.3	64.4 $\pm$ 2.1	72.7 $\pm$ 1.4	60.4 $\pm$ 0.6	61.9 $\pm$ 1.4	66.0 $\pm$ 0.6
M-FLAG ( Liu et al., 2023 )	66.5 $\pm$ 0.5	78.4 $\pm$ 0.55	84.0 $\pm$ 0.04	47.1 $\pm$ 0.3	61.8 $\pm$ 1.5	72.1 $\pm$ 1.6	56.0 $\pm$ 0.9	60.3 $\pm$ 1.4	64.4 $\pm$ 0.3
MGCA-ViT ( Wang et al., 2022a )	78.2 $\pm$ 0.1	82.4 $\pm$ 0.03	84.4 $\pm$ 0.05	66.3 $\pm$ 0.3	68.6 $\pm$ 0.9	73.3 $\pm$ 0.8	61.0 $\pm$ 1.3	64.3 $\pm$ 0.4	66.9 $\pm$ 1.4
MRM ( Zhou et al., 2023 )	80.1 $\pm$ 0.1	83.5 $\pm$ 0.10	85.3 $\pm$ 0.05	65.0 $\pm$ 0.5	69.3 $\pm$ 1.0	75.6 $\pm$ 0.7	62.6 $\pm$ 1.1	66.6 $\pm$ 0.3	66.5 $\pm$ 0.2

Table 3. Classification results (%) on NIH, SIIM, and RSNA datasets, reported as mean

\pm

std over three seeds ( Best , Second Best ).

Figure 5. Comparison of supervised classification performance in terms of AUROC on the Shenzhen ( Jaeger et al., 2013 ) dataset.

Method	RSNA		Shenzhen
Method	F1 $\uparrow$	AUROC $\uparrow$	F1 $\uparrow$	AUROC $\uparrow$
MedCLIP-ViT ( Wang et al., 2022b )	34.9	50.3	50.7	51.1
MedKLIP ( Wu et al., 2023 )	23.2	72.1	51.5	48.3
M-FLAG ( Liu et al., 2023 )	77.4	59.1	49.3	27.6
MGCA-R50 ( Wang et al., 2022a )	27.5	52.3	50.7	49.1
MGCA-ViT ( Wang et al., 2022a )	22.8	51.6	51.5	48.3

Table 4. Zero-shot classification performance (%) on RSNA ( Shih et al., 2019 ) and Shenzhen ( Jaeger et al., 2013 ) dataset. Since CheXWorld ( Yue et al., 2025 ) lacks a text encoder, its results are omitted ( Best , Second Best ).

Method	RSNA (Dice $\uparrow$ )	TBX11K (Dice $\uparrow$ )
MedCLIP-R50 ( Wang et al., 2022b )	75.45 $\pm$ 0.11	85.52 $\pm$ 0.17
MedCLIP-ViT ( Wang et al., 2022b )	73.29 $\pm$ 1.41	85.62 $\pm$ 0.07
MedKLIP ( Wu et al., 2023 )	74.68 $\pm$ 0.42	87.06 $\pm$ 0.31
M-FLAG ( Liu et al., 2023 )	67.86 $\pm$ 0.63	79.12 $\pm$ 0.16
MGCA-R50 ( Wang et al., 2022a )	75.04 $\pm$ 0.59	87.05 $\pm$ 0.19
MGCA-ViT ( Wang et al., 2022a )	75.48 $\pm$ 0.28	86.89 $\pm$ 0.39

Table 5. Segmentation performance (%) on the RSNA ( Shih et al., 2019 ) and TBX11K ( Liu et al., 2020 ) dataset ( Best , Second Best ).

Model	BLEU2	BS	14 Mi-F1	14 Ma-F1	5 Mi-F1	5 Ma-F1
Effect of context-infused vision encoder
w/o CC	0.210	0.535	0.499	0.358	0.540	0.467
Effect of multi-level supervision paradigm
${\mathcal{L}_{con}}$	0.280	0.589	0.507	0.369	0.553	0.477
${\mathcal{L}_{con}}$ + ${\mathcal{L}_{G}}$	0.286	0.595	0.522	0.380	0.568	0.496
Effect of varying eye gaze ratio

Table 6. Ablation study on supervision strategies, context integration, and gaze ratio for the MIMIC-CXR report generation task. “BS” and “CC” denotes BERTScore ( Zhang* et al., 2020 ) and clinical context, respectively. n Mi-F1 and n Ma-F1 refer to the micro- and macro-F1 scores computed by CheXbert, with

n

observations. Higher is better for all metrics.

Model	ZSC	Retrieval			Report Generation
Model	F1	P@1	P@5	P@10	BLEU2	14 Mi-F1	14 Ma-F1
MSE	69.8	72.8	60.1	56.0	0.289	0.513	0.375
IoU	59.2	59.4	52.7	49.6	0.287	0.503	0.368
JSD	77.0	75.5	61.6	57.2	0.290	0.525	0.388

Table 7. Ablation study on gaze-guidance losses, including mean squared error (MSE), intersection over union (IoU ( Sultana et al., 2024 ) ), and Jensen-Shannon divergence (JSD, i.e., CoGaze). “ZSC” denotes zero-shot classification on the RSNA dataset.

Figure 6. t-SNE ( van der Maaten and Hinton, 2008 ) visualization (left) and cosine similarity distribution (right) of paired visual features extracted from the same image, with and without clinical context.

Model	CLS (AUROC $\uparrow$ )				SEG (Dice $\uparrow$ )		Retrieval			Report Generation
Model	SIIM	RSNA	Shenzhen	NIH	RSNA	TBX11K	P@1 $\uparrow$	P@5 $\uparrow$	P@10 $\uparrow$	BLEU2 $\uparrow$	F1 $\uparrow$
CoGaze w/ single-positive	97.9	85.4	99.1	85.4	77.8	96.3	73.9	60.9	55.9	0.287	0.512
CoGaze w/ gaze mask (EGMA ( Ma et al., 2024 ) )	97.6	89.6	99.4	85.1	78.3	96.3	70.5	58.1	54.3	0.285	0.509
CoGaze (Ours)	98.5	90.1	99.6	85.9	80.7	96.7	75.5	61.6	57.2	0.290	0.525

Table 8. Ablation study on hybrid-positive contrastive learning and gaze modeling strategies ( Best ).

Figure 7. Qualitative results. (A) Free-text radiology reports generated from PromptMRG ( Jin et al., 2024 ) , MLRG ( Liu et al., 2025a ) , and CoGaze (DistilGPT2). (B-C) Attention maps of CoGaze, CoGaze w/o Gaze, and CoGaze w/o Context on pneumothorax and tuberculosis cases. (D) Comparison between radiologist’ annotations and CoGaze-predicted heatmaps.

Split	#Image	#Report	Context	Gaze
Train	240,422	150,957	234,568 (97.57%)	1,711 (0.71%)
Val	2,117	1,182	2,063 (97.45%)	10 (0.47%)

Table A9. Statistics of the MIMIC-CXR dataset used for pretraining. Gaze annotations are sourced from EGD ( Karargyris et al., 2020 ) and REFLACX ( Bigolin Lanfredi et al., 2022 ) . Since REFLACX ( Bigolin Lanfredi et al., 2022 ) provides up to five gaze recordings per image, we retain all of them to preserve radiologists’ prior knowledge, resulting in a sample count mismatch between Appendix Tab. A9 and Tab. A10 .

Dataset	Task	Train	Val	Test	Split
MIMIC-CXR ( Johnson et al., 2019 )	Free-text Report Generation	240,197	2,113	3,852	official split
SRRG-Findings ( Delbrouck et al., 2025 )	Structured Report Generation	181,874	976	1,459	official split
NIH ( Wang et al., 2017 )	14-class Classification	78,468	11,219	22,433	BenchX ( Zhou et al., 2024 )
SIIM ( Zawacki et al., 2019 )	Binary Classification	9,303	1,372	1,372	BenchX ( Zhou et al., 2024 )
Shenzhen ( Jaeger et al., 2013 )	Binary Classification	463	65	134	CheXWorld ( Yue et al., 2025 )
RSNA ( Shih et al., 2019 )	Binary Classification & Segmentation	18,678	4,003	4,003	BenchX ( Zhou et al., 2024 )

Table A10. Data distribution of downstream tasks.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A8. Prompts in CoGaze (Llama-3B) for free-text and structured report generation. For free-text report generation, a similar case is retrieved from the MIMIC-CXR training set based on vision latent similarity, and disease predictions are obtained from the vision-based classifier shown in Fig. 2.

Figure A9. Category-specific prompts used for pneumonia, tuberculosis, and normal cases in zero-shot classification.

Dataset	Task	Batch Size	LR	Patience	Epochs
RSNA	SEG	16	1e-4	5	100
TBX11K	SEG	16	5e-5	10	100
NIH	CLS	16	1e-5	5	20
SIIM	CLS	32	5e-6	2	50
RSNA	CLS	16	1e-4	5	20
Shenzhen	CLS	16	1e-5	5	50

Appendix

Table A11. Implementation details for segmentation (SEG) and supervised classification (CLS) tasks. “LR” denotes the learning rate. “Patience” indicates the number of consecutive epochs without improvement in validation loss before the learning rate scheduler decreases the learning rate.

Model	#Param	View Position	Gaze	ROUGE-L
CXR-VLM-EyeGaze	7B	PA	✓	0.298
CoGaze	321M	PA/AP/Lateral	✗	0.326

Appendix

Table A12. Comparison of CXR-VLM-EyeGaze ( Kim et al., 2024 ) and CoGaze in terms of model characteristics and report generation performance. “View Position” indicates the supported imaging views, and “Gaze” denotes whether eye-tracking data is required during free-text report generation.

Figure A10. t-SNE ( van der Maaten and Hinton, 2008 ) visualization of the learned visual feature space on the MIMIC-5×200 dataset.

Model	CLS (AUROC $\uparrow$ )		SEG (Dice $\uparrow$ )
Model	SIIM	Shenzhen	RSNA	TBX11K
PriorRG ( Liu et al., 2026 )	96.3 $\pm$ 1.6	98.40 $\pm$ 0.24	78.55 $\pm$ 0.29	96.07 $\pm$ 0.07
CoGaze	97.4 $\pm$ 0.1	99.47 $\pm$ 0.09	80.22 $\pm$ 0.41	96.56 $\pm$ 0.11

Appendix

Table A13. Comparison between context-based method and CoGaze in supervised classification (CLS) and segmentation (SEG) performance.

Figure A12. Attention visualizations of CoGaze, CoGaze w/o Gaze, and CoGaze w/o Context on pneumothorax (SIIM ( Zawacki et al., 2019 ) ) and tuberculosis (Shenzhen ( Jaeger et al., 2013 ) ) cases.

Figure A13. Examples of free-text radiology reports generated by PromptMRG ( Jin et al., 2024 ) , MLRG ( Liu et al., 2025a ) , and CoGaze (DistilGPT2). Words in the generated reports that match the reference are highlighted in the same color. Greater color diversity reflects broader coverage of clinical findings, while longer color spans suggest more detailed descriptions. Incorrect predictions are underlined .

Figure A14. Examples of structured radiology reports generated by CoGaze (Llama-3B). Correctly generated words are highlighted in green, while acceptable words are highlighted in orange. Incorrect or missing findings are underlined in red.

Figure A15. Ablation study on the impact of hyperparameters

\lambda

and

\rho

across classification (AUROC), segmentation (Dice), and retrieval (P@K) tasks on multiple datasets. The retrieval task is conducted on the MIMIC-5x200 ( Johnson et al., 2019 ; Zhou et al., 2024 ) dataset. Our CoGaze achieves optimal performance at

\lambda=0.8

and

\rho=0.25

Explore similar work

Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning
Apr 15, 2026Kinhei Lee, Peiyuan Jing, Zhenxuan Zhang +5Radiology VLMs Explainable Medical Image Analysis
Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, UK · College of physics and information engineering, Fuzhou University, Fuzhou, 350108, China · Department of Surgery and Cancer, Imperial College London, London W12 0NN, UK +3
CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation
Apr 29, 2026Sonali Sharma, Jin Long, George Shih +7Chest X-Ray Classification Medical Image Analysis
Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, CA, USA · Department of Radiology, Stanford University, Stanford, CA, USA · Department of Pediatrics, Stanford University School of Medicine, Stanford, California, USA +6
A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretation
Apr 1, 2026Yabin Zhang, Chong Wang, Yunhe Gao +19Radiology VLMs VLM Reasoning
Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA. · Department of Radiology, Stanford University, Stanford, CA, USA. · Department of Computer Science, Stanford University, Stanford, CA, USA. +4

Apr 15, 2026J/K move · Enter open · S save

Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning

Kinhei Lee, Peiyuan Jing, Zhenxuan Zhang, Yue Yang+4

Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, UK · College of physics and information engineering, Fuzhou University, Fuzhou, 350108, China · Department of Surgery and Cancer, Imperial College London, London W12 0NN, UK+3

Radiology VLMs Explainable Medical Image Analysis Radiology Report Generation Medical VQA

Open paper AskPDF arXiv

Large scale vision language models have shown promise in automating chest Xray interpretation, yet their clinical utility remains limited by a gap between model outputs and radiologist reasoning. Most systems optimize for semantic information without emulating how experts visually examine medical images, often overlooking critical findings or diverging from established diagnostic workflows. Radiologists follow structured protocols (e.g., the ABCDEF approach) that ensure all clinically relevant regions are systematically examined, reducing missed findings and supporting reliable diagnostic reasoning. We introduce GazeX, a vision language model that leverages radiologists' eye tracking data as a behavioral prior to model expert diagnostic reasoning. By incorporating gaze trajectories and fixation patterns into pretraining, GazeX learns to follow the spatial and temporal structure of radiologist attention and integrates observations in a clinically meaningful sequence. Using a curated dataset of over 30,000 gaze key frames from five radiologists, we demonstrate that GazeX produces more accurate, interpretable, and expert consistent outputs across radiology report generation, disease grounding, and visual question answering, utilizing 231,835 radiographic studies, 780,014 question answer pairs, and 1,162 image sentence pairs with bounding boxes. Unlike autonomous reporting systems, GazeX produces verifiable evidence artifacts, including inspection trajectories and finding linked localized regions, enabling efficient human verification and safe human AI collaboration. Learning through expert eyes provides a practical route toward more trustworthy, explainable, and diagnostically robust AI systems for radiology and beyond.

Abstract

Explore similar work

Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning

CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation

A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretation