RadRef: A Reference-Free Metric for Radiology Report Generation
Organizations: University of Virginia, Charlottesville, VA, USA
Abstract
AI-generated radiology reports are commonly evaluated against radiologist-authored references, limiting the applicability of standard reference-based metrics when no reference is available. We introduce RadRef, a reference-free metric that scores report claims against image-derived beliefs over a clinical finding vocabulary, weighted by relevance to the clinical indication. By evaluating findings beyond the report's stated claims, RadRef assesses omissions as well as unsupported assertions. On ReXVal, RadRef achieves agreement with radiologist rankings among non-tied same-radiograph report pairs, with pair coverage and tie-inclusive agreement. Its non-tied agreement is close to GREEN () and RadFact (), with its strongest relative results on omissions. We further evaluate model-generated and radiologist-authored reports from MIMIC-CXR studies, independently annotated by two board-certified radiologists. On same-image omission rankings, RadRef attains Kendall's , compared with between the readers. In separate paired comparisons across generated reports, no evaluated comparator significantly outperforms RadRef on omissions. The same image-grounded representation provides a false-finding readout, RadRef-FF, with on ReXVal and in the reader study. Controlled perturbations confirm that RadRef depends on the radiograph, report, and clinical indication. RadRef enables image-grounded evaluation before a radiologist-authored reference exists, supporting quality monitoring of report-generation systems in clinical settings, and complements reference-based metrics when one is available.
Figures & tables
| Agreement, same radiograph | ||||
| Metric | False findings | Omissions | Combined | |
| GREEN | 71.4 | 68.5 | 73.9 | +0.399 |
| CRIMSON | 77.7 | 68.7 | 76.9 | +0.480 |
| RadFact | 76.9 | 55.8 † | 72.6 | +0.358 |
| RadGraph-F1 | 67.2 | 49.7 † | 65.3 | +0.284 |
| CheXbert-F1 | 56.2 † | 63.6 | 59.3 | +0.152 |
| Metric | Radiograph | Report |
|---|---|---|
| GREEN | – | |
| CRIMSON | – | |
| RadFact | – | |
| RadGraph-F1 | – | |
| CheXbert-F1 | – | |
| CheXbert-emb | – |
| Omissions | False findings | |||
| Metric | Reader 1 | Reader 2 | Reader 1 | Reader 2 |
| CRIMSON | ||||
| RadFact | ||||
| GREEN | ||||
| RadGraph-F1 | ||||
| CheXbert-F1 | ||||
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Contribution | ||||
|---|---|---|---|---|
| Finding | Image belief | Radiologist | MAIRA-2 | (Radiologist / MAIRA-2) |
| Pleural effusion † | present | omitted | / | |
| Mediastinal widening | not mentioned | present | / | |
| Pneumothorax | absent | absent | / |
| RadRef baseline (95% CI) | |||
|---|---|---|---|
| Metric | False findings | Omissions | Combined |
| CRIMSON | ( , ) | ( , ) | ( , ) |
| GREEN | ( , ) | ( , ) | ( , ) |
| RadFact | ( , ) | ( , ) | ( , ) |
| RadGraph-F1 | ( , ) | ( , ) | ( , ) |
| CheXbert-F1 | ( , ) | ( , ) | ( , ) |
| Metric | Empty wins (%) |
|---|---|
| BLEU-4 | 1.0 |
| BERTScore | 90.7 |
| CheXbert-F1 | 21.0 |
| CheXbert-emb | 13.3 |
| RadGraph-F1 | 6.7 |
| GREEN | 10.7 |
| Contrast | Median (95% CI) | Mean (95% CI) | Positive (%) |
|---|---|---|---|
| Radiologist empty | ( , ) | ( , ) | |
| LLaVA-Rad empty | ( , ) | ( , ) | |
| MAIRA-2 empty | ( , ) | ( , ) | |
| Radiologist LLaVA-Rad | ( , ) | ( , ) | |
| Radiologist MAIRA-2 | ( , ) | ( , ) |
| RadRef | + calib. | + robust | |
|---|---|---|---|
| assertion forecast | |||
| denial forecast | |||
| gate strength | |||
| probe-belief confidence | |||
| uncertainty band | |||
| belief-offset correction |
| Item | Definition |
|---|---|
| Task | Grade each report independently against the radiograph |
| Omission | Finding visible on the radiograph but absent from the report |
| False finding | Report claim unsupported by the radiograph, including calling an abnormal structure normal |
| Attribute error | Wrong side, severity, or position, recorded as an omission with the error stated |
| Ignored | Comparisons to prior studies, boilerplate, and anything beyond the findings themselves |
| Ranking compared with | 95% CI | |
|---|---|---|
| RadRef | – | |
| Inter-reader | – |
| Metric | Omissions (95% CI) |
|---|---|
| CRIMSON | ( , ) |
| RadFact | ( , ) |
| GREEN | ( , ) |
| RadGraph-F1 | ( , ) |
| CheXbert-F1 | ( , ) |
| CheXbert-emb | ( , ) |
| Probe | AUC | Conf. (%) | FN (%) | FP (%) | Agreement (%) |
|---|---|---|---|---|---|
| MedGemma-27B | .824 | 55.2 | 17.5 | 4.2 | 73.1 (60.9–84.4) |
| Lingshu-32B | .696 | 36.2 | 9.5 | 12.5 | 68.6 (57.1–79.6) |
| RadVLM-7B | .878 | 6.5 | 0.0 | 0.0 | 60.0 (47.3–73.6) |
| MedGemma-4B | .587 | 37.8 | 0.0 | 45.8 | 43.3 (32.6–54.1) |
| Setting | Agreement (%) | Change |
|---|---|---|
| Primary RadRef | 73.1 | — |
| Description resampling (8 draws) | ||
| SNOMED CT terminology | 69.3 | |
| Lingshu-32B probe | 68.6 | |
| calibration | 76.1 | |
| robust | 70.1 |
| Component changed | Change in below-empty rate (pp) |
|---|---|
| Chest ImaGenome SNOMED CT | |
| Parser model (four alternatives) | to |
| Deliberately poor parser prompt | |
| Image probe (MedGemma vs. Lingshu) | difference |
| Statistic | Reference draw | Mean SD |
|---|---|---|
| RadRef | ||
| Combined agreement | 73.13 | |
| False-finding agreement | 69.77 | |
| Omission agreement | 70.18 | |
| RadRef-FF | ||
| RadRef-FF false-finding agreement | 86.67 |
| Image probe | MAIRA-2 pref. | LLaVA-Rad pref. | empty |
|---|---|---|---|
| MedGemma-27B | |||
| Lingshu-32B |
| Inventory | Terms | Combined | Omissions | |
|---|---|---|---|---|
| Chest ImaGenome | 69 | 73.1 | 70.2 | |
| grouped to 62 | 62 | 72.4 | 69.3 | |
| SNOMED CT, re-probed | 62 | 69.3 | 55.7 |
| all terms | most-specific | |
|---|---|---|
| Within-study agreement | 68.6 | 73.1 |
| Omissions | 67.4 | 70.2 |
| False findings | 66.7 | 69.8 |
| +0.305 | +0.353 | |
| Coverage | 70.5 | 61.8 |
| Ablation | Preference (%) | Change |
|---|---|---|
| Full RadRef | 66.5 | — |
| Remove distinct unmentioned state | 48.6 | |
| Remove zero-centering of unmentioned score | 52.0 | |
| Use binary rated/unrated relevance | 52.4 | |
| Remove description-based belief floor | 59.8 | |
| Remove salience/confidence weighting | 62.2 |
| Rule | Combined (%) | Omissions (%) | Length |
|---|---|---|---|
| 50.9 | 61.5 | +0.296 | |
| accuracy | 57.0 | 60.8 | -0.295 |
| precision | 58.4 | 62.8 | +0.244 |
| recall | 42.2 | 63.2 | +0.304 |
| RadRef, all terms | 68.6 | 67.4 | -0.067 |
| Comparison | Stated | Random | Far | (95% CI) |
|---|---|---|---|---|
| vs MAIRA-2 | 66.5 | 63.8 | 30.3 | ( , ) |
| vs LLaVA-Rad | 64.1 | 62.3 | 23.1 | ( , ) |
| Prompt | Model | Decoding | Produces |
|---|---|---|---|
| Image probes | MedGemma-27B | greedy, 69 per radiograph | , |
| Unprompted descriptions | RadVLM-7B | , per radiograph | |
| Report parsing | GPT-4o-mini | greedy, once per report | |
| Relevance tiers | GPT-4o-mini | once per indication | |
| Report generation | MAIRA-2 / LLaVA-Rad | public model settings | candidate reports |
Radiology Report Evaluation
Radiology report evaluation measures whether generated or revised reports preserve clinically important findings, relationships, and certainty.
See all 37 papers