cs.LGNov 6, 2026

A Clinical Evaluation of Conformal Prediction for Chest Radiograph Report Generation

Authors: Nikolaos Kontogeorgis, Pascal Krezer, Klaus-Rudolf Kladny, Arne Estler, Christian F. Baumgartner, Lisa Koch, Saif Afat, Michael Muehlebach

Organizations: Max Planck Institute for Intelligent Systems, Tübingen, Germany · University Hospital Tübingen, Tübingen, Germany · University of Lucerne, Lucerne, Switzerland · University of Bern, Bern, Switzerland · 

Abstract

While multimodal large language models (MLLMs) have improved automated radiology report generation, they remain prone to hallucinations, necessitating clinically meaningful risk-control methods. Conformal prediction (CP) provides statistical guarantees, yet existing CP approaches rely on uncalibrated automated extractors and fail to account for the clinical relevance of errors. We introduce clinical admissibility criteria (CLARITY), a framework for evaluating radiology reports that explicitly distinguishes clinically consequential errors from minor deviations. To eliminate reliance on automatic judges, we collected radiologist annotations of MLLM-generated reports using CLARITY. By benchmarking CP strategies and automatic evaluation metrics against these expert annotations, we demonstrate that commonly used admissibility proxies systematically fail to detect clinically relevant errors. Our findings reveal that automated admissibility judgments are largely decoupled from true clinical admissibility.

Figures & tables

Explore similar work

CardsList
  1. Calibrated Confidence Expression for Radiology Report Generation

    Mar 31, 2026David Bani-Harouni, Chantal Pellegrini, Julian Lüers +6Radiology Report GenerationMedical VLMs

  2. CCS: Clinical Consensus Selection for Radiology Report Generation

    May 28, 2026Xi Zhang, Yingshu Li, Zaiqiao Meng +2Radiology Report GenerationRadiology VLMs

  3. CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation

    Mar 6, 2026Mohammed Baharoon, Thibault Heintz, Siavash Raissi +6Radiology Report EvaluationLLM Evaluation