A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretation
Authors: Yabin Zhang, Chong Wang, Yunhe Gao, Jiaming Liu, Maya Varma, Justin Xu, Sophie Ostmeier, Jin Long, +14 more
Organizations: Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA. · Department of Radiology, Stanford University, Stanford, CA, USA. · Department of Computer Science, Stanford University, Stanford, CA, USA. · Big Data Institute, University of Oxford, Oxford, UK. · Department of Pediatrics, Stanford University, Stanford, CA, USA. · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA. · Department of Medicine, Stanford University, Stanford, CA, USA.
Chest X-rays (CXRs) are among the most frequently performed imaging examinations worldwide, yet rising imaging volumes increase radiologist workload and the risk of diagnostic errors. Although artificial intelligence (AI) systems have shown promise for CXR interpretation, most generate only final predictions, without making explicit how visual evidence is translated into radiographic findings and diagnostic predictions. We present CheXOne, a reasoning-enabled vision-language model for CXR interpretation. CheXOne jointly generates diagnostic predictions and explicit, clinically grounded reasoning traces that connect visual evidence, radiographic findings, and these predictions. The model is trained on 14.7 million instruction and reasoning samples curated from 30 public datasets spanning 36 CXR interpretation tasks, using a two-stage framework that combines instruction tuning with reinforcement learning to improve reasoning quality. We evaluate CheXOne in zero-shot settings across visual question answering, report generation, visual grounding and reasoning assessment, covering 17 evaluation settings. CheXOne outperforms existing medical and general-domain foundation models and achieves strong performance on independent public benchmarks. A clinical reader study demonstrates that CheXOne-drafted reports are comparable to or better than resident-written reports in 55% of cases, while effectively addressing clinical indications and enhancing both report writing and CXR interpretation efficiency. Further analyses involving radiologists reveal that the generated reasoning traces show high clinical factuality and provide causal support for the final predictions, offering a plausible explanation for the performance gains. These results suggest that explicit reasoning can improve model performance, interpretability and clinical utility in AI-assisted CXR interpretation.
Figures & tables
Figure 1: Training data . a , Construction of the CheXinstruct-v2 dataset from 30 public datasets, covering 36 CXR interpretation tasks and 10.2 million instruction samples. b , Generation of the CheXReason dataset, comprising over 4.5 million LLM-generated reasoning traces. c , Illustration of training data example. d , Overview of the training data.
Figure 2: Training and inference workflow. a, Initial instruction tuning . A pre-trained VLM undergoes instruction tuning using the CheXinstruct-v2 and CheXReason datasets to establish foundational CXR interpretation and reasoning capabilities. b, Reasoning enhancement via Reinforcement Learning. The model’s reasoning logic is further refined using Group Relative Policy Optimization (GRPO), guided by task-specific reward functions. c, Multi-task zero-shot inference , CheXOne is evaluated across 17 subtasks within four categories. Performance is quantified using domain-specific metrics: accuracy for VQA, 1/RadCliQ for report generation, IoU for visual grounding, and specialized scores for factuality ( Sf ) and self-consistency ( Ssc ).
Figure 3: Technical evaluation of VQA across eight radiological skills , where bar graphs show mean accuracy with 95% confidence intervals. a , Performance of presence assessment on the ReXVQA dataset. b , Performance of anatomical localization on the ReXVQA dataset. c , Performance of negation detection on the ReXVQA dataset. d , Performance of differential diagnosis on the ReXVQA dataset. e , Performance of geometric reasoning on the ReXVQA dataset. f , Performance of view classification on the MIMIC-CXR dataset. g , Performance of temporal classification on the Chest ImaGenome dataset. h , Performance of long-tail disease identification on the MIMIC-CXR Long-tail dataset. These diseases were excluded from explicit training, serving as an OOD task to evaluate model generalization.
Figure 4: Technical evaluation on report generation. a , Findings generation performance on the public ReXRank benchmark, evaluated over ReXGradient, MIMIC-CXR, CheXpert Plus, and IU Xray datasets. Notably, the IU Xray dataset was not included in the training data and therefore serves to assess generalization to an unseen data distribution. b , Progression generation performance evaluated on the MIMIC-CXR dataset, where models are asked to generate the Findings section with comparison to a previous study.
Figure 5: Technical evaluation of visual grounding tasks. Performance is quantified using mean intersection-over-union (mIoU) and mean average precision (mAP) with 95% confidence intervals (CIs). Qualitative examples compare CheXOne’s predicted bounding boxes with expert-annotated ground truth. a , Phrase Grounding. Evaluation conducted on the MS-CXR dataset. b , Abnormality Grounding. Evaluation conducted on the VinDr-CXR dataset.
Figure 6: Technical evaluation of reasoning traces. a , Factuality evaluation, assessing whether entities extracted from generated reasoning traces are semantically supported by the corresponding reference reports. b , Self-consistency evaluation, measuring whether the model converges on a stable conclusion despite variations in sampled reasoning traces. c , Causal support and clinical factuality evaluation, in which radiologists assess the factuality of reasoning traces and their causal support for the final predictions.
Figure 7
Figure 8: Ablation and design analyses of CheXOne. a , Comparative results of the model after the first training stage (instruction tuning; CheXOne-Stage1) and the second training stage (reinforcement learning; CheXOne-Stage2). b , Performance of CheXOne under two inference modes: “CheXOne-Reason”, where the model generates an explicit reasoning trace prior to the final answer, and “CheXOne-Instruct”, where the model outputs the answer directly without intermediate reasoning steps. c , Evaluation of different sample filtering strategies employed during the reinforcement learning phase.
Figure 9: Prompt design for automated reasoning trace synthesis. Overview of the instruction template used to transform reference radiology reports, diagnostic questions, and ground-truth answers into synthesized reasoning traces using an LLM.
Task Group
Task Name
Dataset
Size
VQA
Presence Assessment
ReXVQA [ 23 ]
14698
Anatomical Localization
ReXVQA [ 23 ]
2404
Negation Detection
ReXVQA [ 23 ]
15007
Differential Diagnosis
ReXVQA [ 23 ]
8578
Geometric Reasoning
ReXVQA [ 23 ]
171
View Classification
MIMIC-CXR [ 15 ]
900
Table 1: Evaluation data partitioning and integrity. For standardized benchmarks (e.g., ReXVQA, MIMIC-CXR), we strictly adhere to official test splits. Targeted analyses, such as the expert reader study, utilize random subsamples derived from these independent partitions. All custom evaluation tasks are sourced exclusively from official test sets to preclude data leakage. Specifically, long-tail disease classification serves as an evaluation of performance on unseen task formulations , while the IU Xray dataset is utilized to assess robustness against unseen data distributions , collectively validating the OOD performance of CheXOne.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
1/RadCliQ
BLEU
BertScore
SembScore
RadGraph
RaTEScore
Findings Generation: ReXGradient
CheXOne
1.116
0.229
0.483
0.498
0.210
0.535
MedGemma
1.008
0.200
0.427
0.479
0.223
0.617
MAIRA-2
0.963
0.205
0.436
0.462
0.187
0.559
RadFM
0.775
0.157
0.365
0.392
0.135
0.504
CheXagent
0.674
0.093
0.305
0.366
0.080
0.428
Appendix
Table S2: Benchmarking report generation performance. Comparative evaluation of CheXOne and baseline models on findings and progression generation across four datasets. Performance is assessed using a comprehensive set of metrics including 1/RadCliQ, BLEU, BertScore, SembScore, RadGraph, and RaTEScore.
Figure S10: Fine-grained Report Analysis using GREEN. Comparison of average error counts per report between CheXOne and MedGemma on a randomly sampled subset of the ReXGradient test set. Error categories are defined as follows: ‘False Finding’ (False report of a finding in the candidate); ‘Missed Finding’ (Missing a finding present in the reference); ‘Location Error’ (Misidentification of a finding’s anatomic location/position); ‘Severity Error’ (Misassessment of the severity of a finding); ‘Spurious Comparison’ (Mentioning a comparison that isn’t in the reference); and ‘Missed Comparison’ (Omitting a comparison detailing a change from a prior study). While MedGemma makes fewer precision-related errors (for example, False Finding), CheXOne shows stronger recall, with fewer Missed Findings and more Matched Findings.
Figure S11: Visualization of the VQA examples with generated reasoning traces. Representative test samples are shown for a , Presence Assessment; b , Anatomical localization; c , Negation Detection; d , Differential Diagnosis (Part I).
Figure S11: Visualization of the VQA examples with generated reasoning traces. Representative test samples are shown for e , Geometric Reasoning; f , View Classification; g , Temporal Classification; h , Long-tail Disease Identification (Part II).
Figure S12: Visualization of the report generation examples with generated reasoning traces. Representative test samples are shown for a , findings generation and b , progression generation tasks. Each panel displays the input CXR(s) and the user-specified question, followed by the model-generated reasoning trace and the final free text report.
Figure S13: Visualization of the visual grounding examples with generated reasoning traces. Representative test samples are shown for a , phrase grounding and b , abnormality grounding tasks. Each panel displays the input CXR and the user-specified question, followed by the model-generated reasoning trace and the final localized answer (bounding box).
Task Group
Task name
Dataset
Sample Number
With R
GRPO
QA
Open-Ended VQA
VQA-RAD
713
✗
✗
SLAKE
1093
✗
✗
MedVQA-2019
78
✗
✗
PMC-VQA
747
✗
✗
Rad-Restruct
142340
✗
✗
MIMIC-CXR-VQA
253487
✓
✗
Appendix
Table S3: Composition and characteristics of CheXinstruct-v2 and CheXReason datasets. Detailed breakdown of the datasets utilized for instruction-following and reasoning-based training. ‘With R’ denotes subsets that include explicit reasoning traces and are incorporated into CheXReason. ‘GRPO’ identifies the specific partitions reserved for reinforcement learning in Stage 2. (Part I)
Task Group
Task name
Dataset
Sample Number
With R
GRPO
Text Generation
Findings Generation
CheXpert Public
48109
✗
✓
MIMIC-CXR
152173
✓
✓
ReXGradient
140000
✗
✓
Findings Generation with Indication
CheXpert Public
46327
✗
✓
MIMIC-CXR
148090
✓
✓
ReXGradient
140000
✗
✓
Appendix
Table S3: Composition and characteristics of CheXinstruct-v2 and CheXReason datasets. Detailed breakdown of the datasets utilized for instruction-following and reasoning-based training. ‘With R’ denotes subsets that include explicit reasoning traces and are incorporated into CheXReason. ‘GRPO’ identifies the specific partitions reserved for reinforcement learning in Stage 2. (Part II)
Chest X-ray interpretation is one of the most frequently performed diagnostic tasks in medicine and a primary target for AI development, yet current vision-language models are primarily trained on datasets of paired images and reports, not the cognitive processes and visual attention that underlie clinical reasoning. Here, we present CheXthought, a global, multimodal resource containing 103,592 chain-of-thought reasoning traces and 6,609,082 synchronized visual attention annotations across 50,312 multi-read chest X-rays from 501 radiologists in 71 countries. Our analysis reveals clinical reasoning patterns in how experts deploy distinct visual search strategies, integrate clinical context, and communicate uncertainty. We demonstrate the clinical utility of CheXthought across four dimensions. First, CheXthought reasoning significantly outperforms state-of-the-art vision-language model chain-of-thought in factual accuracy and spatial grounding. Second, visual attention data used as an inference-time hint recovers missed findings and significantly reduces hallucinations. Third, vision-language models trained on CheXthought data achieve significantly stronger pathology classification, visual faithfulness, temporal reasoning and uncertainty communication. Fourth, leveraging CheXthought's multi-reader annotations, we predict both human-human and human-AI disagreement directly from an image, enabling transparent communication of case difficulty, uncertainty and model reliability. These findings establish CheXthought as a resource for advancing multimodal clinical reasoning and the development of more transparent, interpretable vision-language models.
Sonali Sharma, Jin Long, George Shih +7
Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, CA, USA · Department of Radiology, Stanford University, Stanford, CA, USA · Department of Pediatrics, Stanford University School of Medicine, Stanford, California, USA +6
Chest X-ray plays a central role in thoracic diagnosis, and its interpretation inherently requires multi-step, evidence-grounded reasoning. However, large vision-language models (LVLMs) often generate plausible responses that are not faithfully grounded in diagnostic evidence and provide limited visual evidence for verification, while also requiring costly retraining to support new diagnostic tasks, limiting their reliability and adaptability in clinical settings. To address these limitations, we present CXReasonAgent, a diagnostic agent that integrates a large language model (LLM) with clinically grounded diagnostic tools to perform evidence-grounded diagnostic reasoning using image-derived diagnostic and visual evidence. To evaluate these capabilities, we introduce CXReasonDial, a multi-turn dialogue benchmark with 1,946 dialogues across 12 diagnostic tasks, and show that CXReasonAgent produces faithfully grounded responses, enabling more reliable and verifiable diagnostic reasoning than LVLMs. These findings highlight the importance of integrating clinically grounded diagnostic tools, particularly in safety-critical clinical settings. The demo is available here.
Large scale vision language models have shown promise in automating chest Xray interpretation, yet their clinical utility remains limited by a gap between model outputs and radiologist reasoning. Most systems optimize for semantic information without emulating how experts visually examine medical images, often overlooking critical findings or diverging from established diagnostic workflows. Radiologists follow structured protocols (e.g., the ABCDEF approach) that ensure all clinically relevant regions are systematically examined, reducing missed findings and supporting reliable diagnostic reasoning. We introduce GazeX, a vision language model that leverages radiologists' eye tracking data as a behavioral prior to model expert diagnostic reasoning. By incorporating gaze trajectories and fixation patterns into pretraining, GazeX learns to follow the spatial and temporal structure of radiologist attention and integrates observations in a clinically meaningful sequence. Using a curated dataset of over 30,000 gaze key frames from five radiologists, we demonstrate that GazeX produces more accurate, interpretable, and expert consistent outputs across radiology report generation, disease grounding, and visual question answering, utilizing 231,835 radiographic studies, 780,014 question answer pairs, and 1,162 image sentence pairs with bounding boxes. Unlike autonomous reporting systems, GazeX produces verifiable evidence artifacts, including inspection trajectories and finding linked localized regions, enabling efficient human verification and safe human AI collaboration. Learning through expert eyes provides a practical route toward more trustworthy, explainable, and diagnostically robust AI systems for radiology and beyond.
Kinhei Lee, Peiyuan Jing, Zhenxuan Zhang +5
Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, UK · College of physics and information engineering, Fuzhou University, Fuzhou, 350108, China · Department of Surgery and Cancer, Imperial College London, London W12 0NN, UK +3