Benchmarking Classification Tasks of Chest X-ray AI Across Multinational Public and Private Datasets: a retrospective diagnostic accuracy study
Authors: Qinmei Xu, Yiheng Li, Wenya Zhao, Zihao Zhao, Chuanjun Xu, Ahmet Gorkem Er, Xianghao Zhan, Brittany Dashevsky, Mohammed Alawad, Marc von der Stück, Marvin Gazibarić, Daniel Truhn, Shunsuke Shibata, Ryosuke Horino, Mio Ito, Akio Hiwatashi, Suyu Li, Mengya Yang, Ya Liu, Zhiyuan Sun, Changsheng Zhou, Haruka Itakura, Xiao Li, Olivier Gevaert
Organizations: Stanford Center for Biomedical Informatics Research (BMIR), Department of Medicine, Stanford University, Stanford, CA, USA · Department of Radiology, the Second Hospital of Nanjing, Nanjing University of Chinese Medicine, Nanjing, China · Stanford Breast Imaging, Department of Radiology, Stanford University, Stanford, CA, USA · National Center for AI (NCAI), Saudi Data and AI Authority (SDAIA), Riyadh, Saudi Arabia · Department of Radiology, Jinling Hospital, Nanjing, Jiangsu, China · Division of Oncology, Department of Medicine, Stanford University, Stanford, CA, USA
Chest X-ray (CXR) AI models are increasingly proposed for triage, screening, and rule-out, where the operating point is fixed to miss few abnormalities. Most evaluations instead report threshold-free discrimination on a few public benchmarks, leaving clinically usable performance and generalisability to unseen institutional cohorts poorly characterised. We retrospectively evaluated 10 published CXR classification models on 12 datasets from six countries: seven public benchmarks and five institutional cohorts from China, Japan, and Germany. Models spanned convolutional networks, vision-language models, a question-answering foundation model, and a repurposed encoder. Only image-level classification was assessed; segmentation, detection, and report generation were out of scope. Source labels were harmonised to a common vocabulary. The primary outcome was specificity at 95% sensitivity; PPV at 95% sensitivity and AUROC were secondary outcomes. Estimates weighted each diagnostic task equally, with 95% CIs. MAVL and MedKLIP had the highest overall specificity at 95% sensitivity (0.37 [95% CI 0.31-0.44] and 0.37 [0.30-0.44]) and kept that ranking on tasks evaluable for every model. Their PPV was 0.18 (0.13-0.24) and 0.19 (0.13-0.25): most flagged examinations would be false positives. Specificity was lower on institutional cohorts than on public benchmarks for most models, by 0.13 for MAVL, 0.15 for DenseNet, and 0.09 for MedKLIP. No model was best for every finding, country, lesion extent, or prevalence stratum; specificity fell from 0.34 for diffuse to 0.15 for focal findings. At 95% sensitivity, the best current models clear only about a third of truly negative examinations, and public-benchmark performance overstates what they achieve on unseen institutional data. Operating-point-aware evaluation and external validation on institutional cohorts should precede deployment.
Figures & tables
Figure 1: Study design: cohorts and model paradigms. Top , geographic distribution of the 12 chest-radiograph cohorts; each bar denotes one dataset, bar height is proportional to log10 (number of images), and colour denotes public (blue) versus private (orange) access. Bottom , the four model paradigms evaluated: supervised CNN classifiers trained directly on labelled radiographs; vision-language models scored zero-shot by image–text embedding similarity; a QA-based foundation model that emits a free-text answer and therefore no continuous score; and a repurposed encoder in which frozen CheXagent embeddings are passed to a supervised tabular learner. Solid arrows denote data flow, dashed arrows denote supervision by labels, and the lock denotes frozen weights.
Figure 2: Model ranking across all units and shared tasks. Raincloud distributions of per-unit performance for each model, with models ordered by median within each panel. Rows correspond to the three metrics—specificity at 95% sensitivity ( A , B ), PPV at 95% sensitivity ( C , D ), and AUROC ( E , F ); the left column ( A , C , E ) covers all available dataset-task units and the right column ( B , D , F ) the fair shared-task set available across all probabilistic models. MAVL and MedKLIP achieved the strongest operating-point performance overall and retained leading performance in the shared-task analysis. AUROC showed less clinically informative separation, emphasizing the importance of operating-point evaluation for screening and rule-out applications.
Figure 3: Public versus private generalization. Specificity at 95% sensitivity on public benchmarks and private cohorts, with the generalization gap calculated as private minus public performance. Most models showed lower specificity on private data than on public benchmarks. MedKLIP achieved the highest private-cohort specificity, whereas MAVL and DenseNet showed larger public-private decreases. These results indicate that public benchmark performance can overestimate real-world rule-out performance.
Figure 4: Task-specific model heterogeneity. Model-by-task heatmap of specificity at 95% sensitivity. Rows represent models ordered by overall performance, and columns represent diagnostic tasks ordered by average difficulty. No single model dominated all findings, demonstrating substantial task-level heterogeneity and supporting task-aware benchmarking.
Figure 5: Country-level, diagnostic-category, spatial-extent, and prevalence-related heterogeneity in model performance. A , Country-level specificity at 95% sensitivity across six countries, ordered from the best- to the worst-performing country, for the three models with the highest specificity when data were pooled across all cohorts; the same three models are shown for every country, so panel A displays their country-to-country ordering rather than the top three models within each country. B , Specificity at 95% sensitivity by diagnostic category. C , Specificity at 95% sensitivity by spatial extent, showing lower performance for focal findings than for regional or diffuse findings in seven of nine models. D , Specificity at 95% sensitivity by prevalence, showing reduced performance for rare findings. Across panels, model ranking varied by clinical and dataset context, indicating that no single model was uniformly optimal across all strata.
Dataset
Original label / column
Canonical finding
Mapping type
MIMIC-CXR
Pleural Effusion
Effusion
rename
MIMIC-CXR
uncertain ( −1 )
missing
recode → NaN
Nagoya
Lung_lesion
Lung Lesion
rename
Nagoya
PleuralEffusion
Pleural Effusion
rename
Nagoya
LungOpacity
Lung Opacity
rename
Nagoya
PleuralThickening
Pleural_Thickening
rename
Table S1: Harmonization of heterogeneous dataset labels to the standardized finding vocabulary (label side). Only non-identity mappings are shown; renames that differ solely in capitalization or spacing are omitted. “OR-merge” denotes a logical OR across the listed source columns; “recode” denotes a numeric or string re-encoding to binary presence with uncertain/unknown values set to missing.
Model(s)
Canonical finding
Model output heads / concepts
Canonical-name alias table (metric-time; prediction side only)
Table S2: Prediction-side alignment of model outputs to the standardized finding vocabulary. Multiple model heads mapped to one finding are aggregated by their maximum. Ground-truth labels are always read from the true label column; this alignment applies only to model outputs.
Figure S1: Per-category performance profiles across diagnostic tasks. Radar charts of specificity at 95% sensitivity across the individual tasks within each anatomy category: A , pulmonary; B , cardiovascular and mediastinal; C , pleural and skeletal; D , medical devices and diaphragmatic abnormalities. The pulmonary category ( A ) contains enough tasks for all nine models to share more than five gap-free spokes, so the full model fleet is shown; the remaining categories ( B – D ) have too few tasks for the fixed-vocabulary models to complete a shared spoke set, so only the three open-vocabulary models (CheXzero, BioViL-T, PsPG) are drawn, and full-model comparison for these categories is given in Figure 4 and Figure 5B. Each spoke is a canonical finding; categories with more than twelve tasks show the twelve best-covered spokes. Within each spoke, values are scaled to the maximum across the plotted models, so the radars show relative profiles rather than absolute specificity; the annotated value gives the top score on that spoke.
Task
# Datasets
Common
Rare
Prev. (%)
Class
Non-pulmonary Mass
1
0
1
0.04
Rare
Incomplete Expansion of the Lung
1
0
1
0.12
Rare
Pulmonary Arterial Hypertension
1
0
1
0.17
Rare
Bone Destruction
1
0
1
0.34
Rare
Pneumoperitoneum
1
0
1
0.43
Rare
Flattened Diaphragm
1
0
1
0.50
Rare
Table S4: Diagnostic tasks classified as common versus rare at a 1% positive-prevalence threshold. Prevalence is the fraction of positive examinations, computed once per (task, cohort) and averaged with equal weight across the cohorts evaluating the task (Nanjing Multi-Hospital subsets pooled). # Datasets is the number of cohorts evaluating the task; Common / Rare cells count how many of those cohorts are ≥ 1% vs < 1%. Class is the task-level label used in the prevalence-stratified panel (Figure 5D): rare if the cohort-averaged prevalence is below 1%.
Task
Dataset
n
Positives
Prevalence (%)
Infiltration
Jinling
161,056
21
0.013
Hernia
Jinling
161,056
55
0.034
Non-pulmonary Mass
Jinling
161,056
57
0.035
Fibrosis
Jinling
161,056
148
0.092
Incomplete Expansion of the Lung
Jinling
161,056
186
0.115
Lung Lesion
IU-Xray
2,561
3
0.117
Table S5: The 32 rare dataset–task cells (cohort prevalence < 1%). These are the long-tail cells that motivate the threshold; they are concentrated in the large low-prevalence cohorts (chiefly Jinling and PadChest).
Chest X-ray classification uses machine learning to detect and categorize diseases and findings in radiographs for screening, triage, and clinical decision support.
Public datasets of Chest X-Rays (CXRs) have long been a popular benchmark for developing machine learning (ML) computer vision models in healthcare. However, the reported strong average-case performance of these models do not necessarily reflect their actual utility when used in heterogeneous clinical settings, potentially masking weaker performance in medically significant scenarios. In this work we use clinical context to provide a more holistic evaluation of models for CXR diagnosis. In particular, we use discharge summaries, recorded prior to each CXR, to derive a ``pre-CXR'' probability of each CXR label, as a proxy for existing contextual knowledge available to clinicians when interpreting CXRs. We use this measure to probe model performance along two dimensions: First, using a stratified analysis, we show that models tend to have lower performance (as measured by AUROC and other metrics) among individuals with higher pre-CXR probability. Second, by controlling for pre-CXR probability via matching and re-weighting, we demonstrate that performance degrades when the correlation is broken between prior context and the current CXR label, suggesting that model performance is highly sensitive to the underlying distribution of clinical context. Specifically, cases with high pre-test probabilities present a fundamentally more difficult visual classification task, highlighting a gap in clinical utility when models are applied to high-risk cohorts.
Chest X-ray (CXR) machine learning relies heavily on automated evaluation using reference standards that aim to approximate clinical judgment. However, commonly used report-derived labels for pathology classification or generic image quality metrics for reconstruction may not reliably reflect clinical judgment. We systematically investigate how evaluation-reference choices affect model performance and ranking in both pathology classification and image quality assessment (IQA). To enable controlled comparison across evaluation references, we collected paired expert image- and report-derived labels for thoracic findings from a clinical cohort at Cambridge University Hospitals (CUH) and curated a subset of the public MIMIC-CXR dataset, along with expert ratings of diagnostic image quality. We show that for supervised image classifiers (ResNet, DenseNet), several zero-shot and fine-tuned vision-language models (e.g., MedKLIP, GLoRIA, and ConVIRT), changing the label source leads to substantial differences not only in performance estimates but also in model rankings. In parallel, alignment of IQA measures with expert judgment depends heavily on the choice of measure, and commonly used IQA metrics such as SSIM and PSNR often fail to align with expert assessments of diagnostic usability. Our results demonstrate that evaluation choices are crucial: they can determine which models and methods appear best and are therefore selected for further development or deployment. The selection of evaluation references should therefore be treated as a central component of clinical validity in CXR machine learning, and justified with respect to the pathology, imaging task, and intended downstream clinical use.
Panagiotis Fytas, Ian Selby, Clemens Karner +14
Language Technology Lab (LTL), University of Cambridge, United Kingdom · Cambridge University Hospitals NHS Foundation Trust, United Kingdom · Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Austria +3
Chest X-rays (CXRs) are among the most frequently performed imaging examinations worldwide, yet rising imaging volumes increase radiologist workload and the risk of diagnostic errors. Although artificial intelligence (AI) systems have shown promise for CXR interpretation, most generate only final predictions, without making explicit how visual evidence is translated into radiographic findings and diagnostic predictions. We present CheXOne, a reasoning-enabled vision-language model for CXR interpretation. CheXOne jointly generates diagnostic predictions and explicit, clinically grounded reasoning traces that connect visual evidence, radiographic findings, and these predictions. The model is trained on 14.7 million instruction and reasoning samples curated from 30 public datasets spanning 36 CXR interpretation tasks, using a two-stage framework that combines instruction tuning with reinforcement learning to improve reasoning quality. We evaluate CheXOne in zero-shot settings across visual question answering, report generation, visual grounding and reasoning assessment, covering 17 evaluation settings. CheXOne outperforms existing medical and general-domain foundation models and achieves strong performance on independent public benchmarks. A clinical reader study demonstrates that CheXOne-drafted reports are comparable to or better than resident-written reports in 55% of cases, while effectively addressing clinical indications and enhancing both report writing and CXR interpretation efficiency. Further analyses involving radiologists reveal that the generated reasoning traces show high clinical factuality and provide causal support for the final predictions, offering a plausible explanation for the performance gains. These results suggest that explicit reasoning can improve model performance, interpretability and clinical utility in AI-assisted CXR interpretation.
Yabin Zhang, Chong Wang, Yunhe Gao +19
Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA. · Department of Radiology, Stanford University, Stanford, CA, USA. · Department of Computer Science, Stanford University, Stanford, CA, USA. +4