cs.LGNov 9, 2026

RadRef: A Reference-Free Metric for Radiology Report Generation

Authors: Shelly Schwartz, Alan Ropp, Michael B. Sneider, Nur Yildirim, Cree M. Gaskin, Chirag Agarwal

Organizations: University of Virginia, Charlottesville, VA, USA

Abstract

AI-generated radiology reports are commonly evaluated against radiologist-authored references, limiting the applicability of standard reference-based metrics when no reference is available. We introduce RadRef, a reference-free metric that scores report claims against image-derived beliefs over a clinical finding vocabulary, weighted by relevance to the clinical indication. By evaluating findings beyond the report's stated claims, RadRef assesses omissions as well as unsupported assertions. On ReXVal, RadRef achieves 73.1%73.1\% agreement with radiologist rankings among non-tied same-radiograph report pairs, with 61.8%61.8\% pair coverage and 64.3%64.3\% tie-inclusive agreement. Its non-tied agreement is close to GREEN (73.9%73.9\%) and RadFact (72.6%72.6\%), with its strongest relative results on omissions. We further evaluate 100100 model-generated and 100100 radiologist-authored reports from 100100 MIMIC-CXR studies, independently annotated by two board-certified radiologists. On same-image omission rankings, RadRef attains Kendall's τb=0.295τ_b = 0.295, compared with 0.3860.386 between the readers. In separate paired comparisons across generated reports, no evaluated comparator significantly outperforms RadRef on omissions. The same image-grounded representation provides a false-finding readout, RadRef-FF, with τb=0.32τ_b = 0.32 on ReXVal and 0.290.29 in the reader study. Controlled perturbations confirm that RadRef depends on the radiograph, report, and clinical indication. RadRef enables image-grounded evaluation before a radiologist-authored reference exists, supporting quality monitoring of report-generation systems in clinical settings, and complements reference-based metrics when one is available.

Figures & tables

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation

    Sep 16, 2026Daniel P. Jeong, Charles Q. Li, Hossein Hosseiny +5Radiology Report GenerationGenerative AI Evaluation

  2. Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality

    Sep 23, 2026Jiaju Huang, Hao Yang, Xinyu Ma +6Radiology Report GenerationLLM-as-a-Judge

  3. ReportQA: QA-Based Radiology Report Evaluation

    Jun 13, 2026Yiming Shi, Shaoshuai Yang, Xi Chen +10Radiology Report GenerationMedical VLMs