A Clinical Evaluation of Conformal Prediction for Chest Radiograph Report Generation
Authors: Nikolaos Kontogeorgis, Pascal Krezer, Klaus-Rudolf Kladny, Arne Estler, Christian F. Baumgartner, Lisa Koch, Saif Afat, Michael Muehlebach
Organizations: Max Planck Institute for Intelligent Systems, Tübingen, Germany · University Hospital Tübingen, Tübingen, Germany · University of Lucerne, Lucerne, Switzerland · University of Bern, Bern, Switzerland · Vienna University of Economics and Business
While multimodal large language models (MLLMs) have improved automated radiology report generation, they remain prone to hallucinations, necessitating clinically meaningful risk-control methods. Conformal prediction (CP) provides statistical guarantees, yet existing CP approaches rely on uncalibrated automated extractors and fail to account for the clinical relevance of errors. We introduce clinical admissibility criteria (CLARITY), a framework for evaluating radiology reports that explicitly distinguishes clinically consequential errors from minor deviations. To eliminate reliance on automatic judges, we collected radiologist annotations of MLLM-generated reports using CLARITY. By benchmarking CP strategies and automatic evaluation metrics against these expert annotations, we demonstrate that commonly used admissibility proxies systematically fail to detect clinically relevant errors. Our findings reveal that automated admissibility judgments are largely decoupled from true clinical admissibility.
Figures & tables
Figure 1: Qualitative comparison of admissibility assessments. Reference-based automatic assessment maps a candidate and reference report to clinical labels and then compares those labels. This procedure can penalize a correct finding that was omitted from the reference and treats discrepancies equally without accounting for their clinical consequence. CLARITY instead uses image-grounded radiologist judgments of both relevance and accuracy. The labels and arrows, rather than color alone, distinguish the assessment pathways.
Figure 2: Comparison of CheXbert and CLARITY. Distribution of CheXbert positive-label F1 scores for 970 generated reports from 97 studies, grouped by radiologist-assessed CLARITY admissibility. Points show individual reports.
Metric
n
ρ
95% CI
PHolm
GREEN [ 15 ]
97/970
0.700
[0.562, 0.798]
2.05×10−14
− RadCliQ-v1 [ 14 ]
97/970
0.479
[0.295, 0.630]
8.87×10−6
− RadCliQ-v0 [ 14 ]
97/970
0.476
[0.288, 0.630]
9.77×10−6
BERTScore [ 13 ]
97/970
0.467
[0.282, 0.624]
1.61×10−5
FORTE pooled F1 (custom) [ 17 ]
97/970
0.432
[0.247, 0.586]
9.69×10−5
RaTE-Score [ 16 ]
97/970
0.402
[0.212, 0.572]
4.02×10−4
Table 1: Association of automatic metrics with CLARITY. Spearman correlations are computed after study-level averaging; 95% confidence intervals use 2,000 study-cluster bootstrap samples and two-sided P values use Holm correction across 14 metric variants. RadCliQ scores are sign-inverted so higher values consistently indicate better reports. The sample-size column gives studies/reports after metric-specific exclusion of undefined values.
Method
Metric
CheXbert (full set)
CheXbert (disjoint CLARITY eval.)
CheXbert (labeled set)
CLARITY
ncalibration
234
196
77
77
ntest
59
97
20
20
SCOPE-Gen (count)
Coverage
0.72±0.06
0.67
0.72±0.11
0.73±0.11
Pred. set size
5.63±0.58
5.00
6.56±1.27
8.63±1.67
nvalid
100
1
100
100
SCOPE-Gen (max)
Coverage
0.71±0.06
0.68
0.69±0.09
0.62±0.07
Table 2: Empirical coverage and prediction-set size for conformal miscoverage α=0.30 . Random-split columns report the mean ± standard deviation (s.d.) over the finite runs among 100 attempted study-level splits and the number of finite runs. The fixed disjoint CheXbert-to-CLARITY evaluation uses the primary F1 threshold of 0.4 and reports one run.
Method
Metric
CheXbert (full set)
CheXbert (disjoint CLARITY eval.)
CheXbert (labeled set)
CLARITY
ncalibration
234
196
77
77
ntest
59
97
20
20
SCOPE-Gen (count)
Coverage
0.65±0.06
0.53
0.61±0.10
0.60±0.10
Pred. set size
2.98±0.14
2.00
3.04±0.20
3.05±0.36
nvalid
100
1
100
100
SCOPE-Gen (max)
Coverage
0.57±0.08
0.54
0.58±0.11
0.56±0.11
Table 3: Empirical coverage and prediction-set size for conformal miscoverage α=0.45 . Random-split columns report the mean ± standard deviation (s.d.) over the finite runs among 100 attempted study-level splits and the number of finite runs. The fixed disjoint CheXbert-to-CLARITY column uses the primary F1 threshold of 0.4 and reports one run.
α
Method
Score
F1≥0.4
F1≥0.5
F1≥0.6
0.30
SCOPE-Gen
count
0.67 / 5.00
0.71 / 7.00
–
SCOPE-Gen
max
0.68 / 4.70
0.71 / 6.41
–
CLM
count
0.75 / 9.59
–
–
CLM
max
0.76 / 9.91
–
–
0.45
SCOPE-Gen
count
0.53 / 2.00
0.60 / 3.00
0.70 / 6.00
SCOPE-Gen
max
0.54 / 2.23
0.54 / 2.44
0.71 / 5.42
Table 4: Sensitivity of the fixed disjoint CheXbert-to-CLARITY evaluation to the CheXbert positive-label F1 admissibility threshold. Each cell reports coverage / prediction-set size for the single fixed split; “–” indicates that no finite prediction set could be calibrated.
Computer Aided Medical Procedures, Technical University of Munich, Germany · Munich Center for Machine Learning (MCML), Germany · Department of Diagnostic and Interventional Radiology, TUM Klinikum rechts der Isar, Germany +3
School of Computing Science, University of Glasgow · School of Electrical and Computer Engineering, University of Sydney · Language Technology Lab, University of Cambridge
Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Radiation Oncology, Mass General Brigham, Boston, MA · King Fahad Hospital, Al-Ahsa Health Cluster, Al Hofuf, Saudi Arabia +2