Benchmarking Classification Tasks of Chest X-ray AI Across Multinational Public and Private Datasets: a retrospective diagnostic accuracy study

Qinmei Xu; Yiheng Li; Wenya Zhao; Zihao Zhao; Chuanjun Xu; Ahmet Gorkem Er; Xianghao Zhan; Brittany Dashevsky

eess.IVMay 21, 2025

Benchmarking Classification Tasks of Chest X-ray AI Across Multinational Public and Private Datasets: a retrospective diagnostic accuracy study

Authors: Qinmei Xu, Yiheng Li, Wenya Zhao, Zihao Zhao, Chuanjun Xu, Ahmet Gorkem Er, Xianghao Zhan, Brittany Dashevsky,

Organizations: Stanford Center for Biomedical Informatics Research (BMIR), Department of Medicine, Stanford University, Stanford, CA, USA · Department of Radiology, the Second Hospital of Nanjing, Nanjing University of Chinese Medicine, Nanjing, China · Stanford Breast Imaging, Department of Radiology, Stanford University, Stanford, CA, USA · National Center for AI (NCAI), Saudi Data and AI Authority (SDAIA), Riyadh, Saudi Arabia ·

Abstract

Chest X-ray (CXR) AI models are increasingly proposed for triage, screening, and rule-out, where the operating point is fixed to miss few abnormalities. Most evaluations instead report threshold-free discrimination on a few public benchmarks, leaving clinically usable performance and generalisability to unseen institutional cohorts poorly characterised. We retrospectively evaluated 10 published CXR classification models on 12 datasets from six countries: seven public benchmarks and five institutional cohorts from China, Japan, and Germany. Models spanned convolutional networks, vision-language models, a question-answering foundation model, and a repurposed encoder. Only image-level classification was assessed; segmentation, detection, and report generation were out of scope. Source labels were harmonised to a common vocabulary. The primary outcome was specificity at 95% sensitivity; PPV at 95% sensitivity and AUROC were secondary outcomes. Estimates weighted each diagnostic task equally, with 95% CIs. MAVL and MedKLIP had the highest overall specificity at 95% sensitivity (0.37 [95% CI 0.31-0.44] and 0.37 [0.30-0.44]) and kept that ranking on tasks evaluable for every model. Their PPV was 0.18 (0.13-0.24) and 0.19 (0.13-0.25): most flagged examinations would be false positives. Specificity was lower on institutional cohorts than on public benchmarks for most models, by 0.13 for MAVL, 0.15 for DenseNet, and 0.09 for MedKLIP. No model was best for every finding, country, lesion extent, or prevalence stratum; specificity fell from 0.34 for diffuse to 0.15 for focal findings. At 95% sensitivity, the best current models clear only about a third of truly negative examinations, and public-benchmark performance overstates what they achieve on unseen institutional data. Operating-point-aware evaluation and external validation on institutional cohorts should precede deployment.

Figures & tables

Figure 1: Study design: cohorts and model paradigms. Top , geographic distribution of the 12 chest-radiograph cohorts; each bar denotes one dataset, bar height is proportional to $\log_{10}$ (number of images), and colour denotes public (blue) versus private (orange) access. Bottom , the four model paradigms evaluated: supervised CNN classifiers trained directly on labelled radiographs; vision-language models scored zero-shot by image–text embedding similarity; a QA-based foundation model that emits a free-text answer and therefore no continuous score; and a repurposed encoder in which frozen CheXagent embeddings are passed to a supervised tabular learner. Solid arrows denote data flow, dashed arrows denote supervision by labels, and the lock denotes frozen weights.

Figure 2: Model ranking across all units and shared tasks. Raincloud distributions of per-unit performance for each model, with models ordered by median within each panel. Rows correspond to the three metrics—specificity at 95% sensitivity ( A , B ), PPV at 95% sensitivity ( C , D ), and AUROC ( E , F ); the left column ( A , C , E ) covers all available dataset-task units and the right column ( B , D , F ) the fair shared-task set available across all probabilistic models. MAVL and MedKLIP achieved the strongest operating-point performance overall and retained leading performance in the shared-task analysis. AUROC showed less clinically informative separation, emphasizing the importance of operating-point evaluation for screening and rule-out applications.

Figure 3: Public versus private generalization. Specificity at 95% sensitivity on public benchmarks and private cohorts, with the generalization gap calculated as private minus public performance. Most models showed lower specificity on private data than on public benchmarks. MedKLIP achieved the highest private-cohort specificity, whereas MAVL and DenseNet showed larger public-private decreases. These results indicate that public benchmark performance can overestimate real-world rule-out performance.

Figure 4: Task-specific model heterogeneity. Model-by-task heatmap of specificity at 95% sensitivity. Rows represent models ordered by overall performance, and columns represent diagnostic tasks ordered by average difficulty. No single model dominated all findings, demonstrating substantial task-level heterogeneity and supporting task-aware benchmarking.

Figure 5: Country-level, diagnostic-category, spatial-extent, and prevalence-related heterogeneity in model performance. A , Country-level specificity at 95% sensitivity across six countries, ordered from the best- to the worst-performing country, for the three models with the highest specificity when data were pooled across all cohorts; the same three models are shown for every country, so panel A displays their country-to-country ordering rather than the top three models within each country. B , Specificity at 95% sensitivity by diagnostic category. C , Specificity at 95% sensitivity by spatial extent, showing lower performance for focal findings than for regional or diffuse findings in seven of nine models. D , Specificity at 95% sensitivity by prevalence, showing reduced performance for rare findings. Across panels, model ranking varied by clinical and dataset context, indicating that no single model was uniformly optimal across all strata.

Dataset	Original label / column	Canonical finding	Mapping type
MIMIC-CXR	Pleural Effusion	Effusion	rename
MIMIC-CXR	uncertain ( $-1$ )	missing	recode $\rightarrow$ NaN
Nagoya	Lung_lesion	Lung Lesion	rename
Nagoya	PleuralEffusion	Pleural Effusion	rename
Nagoya	LungOpacity	Lung Opacity	rename
Nagoya	PleuralThickening	Pleural_Thickening	rename

Table S1: Harmonization of heterogeneous dataset labels to the standardized finding vocabulary (label side). Only non-identity mappings are shown; renames that differ solely in capitalization or spacing are omitted. “OR-merge” denotes a logical OR across the listed source columns; “recode” denotes a numeric or string re-encoding to binary presence with uncertain/unknown values set to missing.

Model(s)	Canonical finding	Model output heads / concepts
Canonical-name alias table (metric-time; prediction side only)
–	Pleural Effusion	Effusion
–	Lung Lesion	Lesion
–	Nodule or mass	Nodule/Mass, Nodule, Mass
Per-model concept maps (representative multi-head aggregations)
Resnet, DenseNet	Nodule/Mass	bulla, cavity, parenchymal_lesion, apical_changes

Table S2: Prediction-side alignment of model outputs to the standardized finding vocabulary. Multiple model heads mapped to one finding are aggregated by their maximum. Ground-truth labels are always read from the true label column; this alignment applies only to model outputs.

Figure S1: Per-category performance profiles across diagnostic tasks. Radar charts of specificity at 95% sensitivity across the individual tasks within each anatomy category: A , pulmonary; B , cardiovascular and mediastinal; C , pleural and skeletal; D , medical devices and diaphragmatic abnormalities. The pulmonary category ( A ) contains enough tasks for all nine models to share more than five gap-free spokes, so the full model fleet is shown; the remaining categories ( B – D ) have too few tasks for the fixed-vocabulary models to complete a shared spoke set, so only the three open-vocabulary models (CheXzero, BioViL-T, PsPG) are drawn, and full-model comparison for these categories is given in Figure 4 and Figure 5B. Each spoke is a canonical finding; categories with more than twelve tasks show the twelve best-covered spokes. Within each spoke, values are scaled to the maximum across the plotted models, so the radars show relative profiles rather than absolute specificity; the annotated value gives the top score on that spoke.

Task	# Datasets	Rare	Prev. (%)	Class
Non-pulmonary Mass	1	1	0.04	Rare
Incomplete Expansion of the Lung	1	1	0.12	Rare
Pulmonary Arterial Hypertension	1	1	0.17	Rare
Bone Destruction	1	1	0.34	Rare
Pneumoperitoneum	1	1	0.43	Rare
Flattened Diaphragm	1	1	0.50	Rare

Table S4: Diagnostic tasks classified as common versus rare at a 1% positive-prevalence threshold. Prevalence is the fraction of positive examinations, computed once per (task, cohort) and averaged with equal weight across the cohorts evaluating the task (Nanjing Multi-Hospital subsets pooled). # Datasets is the number of cohorts evaluating the task; Common / Rare cells count how many of those cohorts are

\geq

1% vs

<

1%. Class is the task-level label used in the prevalence-stratified panel (Figure 5D): rare if the cohort-averaged prevalence is below 1%.

Task	Dataset	$n$	Positives	Prevalence (%)
Infiltration	Jinling	161,056	21	0.013
Hernia	Jinling	161,056	55	0.034
Non-pulmonary Mass	Jinling	161,056	57	0.035
Fibrosis	Jinling	161,056	148	0.092
Incomplete Expansion of the Lung	Jinling	161,056	186	0.115
Lung Lesion	IU-Xray	2,561	3	0.117

Table S5: The 32 rare dataset–task cells (cohort prevalence

<

1%). These are the long-tail cells that motivate the threshold; they are concentrated in the large low-prevalence cohorts (chiefly Jinling and PadChest).

Chest X-Ray Classification

Chest X-ray classification uses machine learning to detect and categorize diseases and findings in radiographs for screening, triage, and clinical decision support.

See all 100 papers

Revisiting Performance Claims for Chest X-Ray Models Using Clinical Context
Sep 24, 2025Andrew Wang, Jiashuo Zhang, Michael OberstChest X-Ray Classification Distribution Shift
Brown University · Johns Hopkins University
Rethinking Clinical Relevance in Chest X-ray Machine Learning: How Evaluation References Define Performance
Jul 28, 2026Panagiotis Fytas, Ian Selby, Clemens Karner +14Chest X-Ray Classification Medical VLMs
Language Technology Lab (LTL), University of Cambridge, United Kingdom · Cambridge University Hospitals NHS Foundation Trust, United Kingdom · Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Austria +3
A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretation
Apr 1, 2026Yabin Zhang, Chong Wang, Yunhe Gao +19Radiology VLMs VLM Reasoning
Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA. · Department of Radiology, Stanford University, Stanford, CA, USA. · Department of Computer Science, Stanford University, Stanford, CA, USA. +4

Sep 24, 2025J/K move · Enter open · S save

Revisiting Performance Claims for Chest X-Ray Models Using Clinical Context

Andrew Wang, Jiashuo Zhang, Michael Oberst

Brown University · Johns Hopkins University

Chest X-Ray Classification Distribution Shift Healthcare Medical Image Benchmarks

Open paper AskPDF arXiv

Public datasets of Chest X-Rays (CXRs) have long been a popular benchmark for developing machine learning (ML) computer vision models in healthcare. However, the reported strong average-case performance of these models do not necessarily reflect their actual utility when used in heterogeneous clinical settings, potentially masking weaker performance in medically significant scenarios. In this work we use clinical context to provide a more holistic evaluation of models for CXR diagnosis. In particular, we use discharge summaries, recorded prior to each CXR, to derive a ``pre-CXR'' probability of each CXR label, as a proxy for existing contextual knowledge available to clinicians when interpreting CXRs. We use this measure to probe model performance along two dimensions: First, using a stratified analysis, we show that models tend to have lower performance (as measured by AUROC and other metrics) among individuals with higher pre-CXR probability. Second, by controlling for pre-CXR probability via matching and re-weighting, we demonstrate that performance degrades when the correlation is broken between prior context and the current CXR label, suggesting that model performance is highly sensitive to the underlying distribution of clinical context. Specifically, cases with high pre-test probabilities present a fundamentally more difficult visual classification task, highlighting a gap in clinical utility when models are applied to high-risk cohorts.

Abstract

Chest X-Ray Classification

Explore similar work

Revisiting Performance Claims for Chest X-Ray Models Using Clinical Context

Rethinking Clinical Relevance in Chest X-ray Machine Learning: How Evaluation References Define Performance

A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretation