CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation

Mohammed Baharoon; Thibault Heintz; Siavash Raissi; Mahmoud Alabbad; Mona Alhammad; Hassan AlOmaish; Sung Eun Kim; Oishi Banerjee

cs.CLMar 6, 2026

CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation

Authors: Mohammed Baharoon, Thibault Heintz, Siavash Raissi, Mahmoud Alabbad, Mona Alhammad, Hassan AlOmaish, Sung Eun Kim, Oishi Banerjee,

Organizations: Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Radiation Oncology, Mass General Brigham, Boston, MA · King Fahad Hospital, Al-Ahsa Health Cluster, Al Hofuf, Saudi Arabia · Ras-Tanura General Hospital, Ministry of Health, Eastern Province, Saudi Arabia ·

Abstract

We introduce CRIMSON, a clinically grounded evaluation framework for chest X-ray report generation that assesses reports based on diagnostic correctness, contextual relevance, and patient safety. Unlike prior metrics, CRIMSON incorporates full clinical context, including patient age, indication, and guideline-based decision rules, and prevents normal or clinically insignificant findings from exerting disproportionate influence on the overall score. The framework categorizes errors into a comprehensive taxonomy covering false findings, missing findings, and eight attribute-level errors (e.g., location, severity, measurement, and diagnostic overinterpretation). Each finding is assigned a clinical significance level (urgent, actionable non-urgent, non-actionable, or expected/benign), based on a guideline developed in collaboration with attending cardiothoracic radiologists, enabling severity-aware weighting that prioritizes clinically consequential mistakes over benign discrepancies. CRIMSON is validated through strong alignment with clinically significant error counts annotated by six board-certified radiologists in ReXVal (Kendalls tau = 0.61-0.71; Pearsons r = 0.71-0.84), and through two additional benchmarks that we introduce. In RadJudge, a targeted suite of clinically challenging pass-fail scenarios, CRIMSON shows consistent agreement with expert judgment. In RadPref, a larger radiologist preference benchmark of over 100 pairwise cases with structured error categorization, severity modeling, and 1-5 overall quality ratings from three cardiothoracic radiologists, CRIMSON achieves the strongest alignment with radiologist preferences. We release the metric, the evaluation benchmarks, RadJudge and RadPref, and a fine-tuned MedGemma model to enable reproducible evaluation of report generation, all available at https://github.com/rajpurkarlab/CRIMSON.

Figures & tables

Figure 1: Representative RadJudge cases illustrating core design principles of CRIMSON. Top: Patient context sensitivity. The clinical impact of an omission (e.g., aortic atherosclerosis) varies by age and indication, and CRIMSON adjusts severity accordingly. Middle: Normal finding handling. CRIMSON does not reward mentioning normal findings, preventing score inflation. Bottom: Clinical significance weighting. Errors are weighted by consequence, prioritizing clinically important findings. In each case, CRIMSON aligns with radiologist expectations, whereas prior metrics fail.

	Correlation with Significant Error Counts
Metric	CheXbert		BERTScore
	Kendall $\tau$	Pearson $r$	Kendall $\tau$	Pearson $r$
RadGraph [ 9 ]	$0.41\,[\text{\tiny 0.19,0.61}]$	$0.59\,[\text{\tiny 0.41,0.75}]$	$0.54\,[\text{\tiny 0.36,0.68}]$	$0.65\,[\text{\tiny 0.51,0.78}]$
BLEU [ 16 ]	$0.49\,[\text{\tiny 0.30,0.65}]$	$0.60\,[\text{\tiny 0.47,0.72}]$	$0.36\,[\text{\tiny 0.16,0.54}]$	$0.48\,[\text{\tiny 0.32,0.63}]$
BERTScore [ 25 ]	$0.52\,[\text{\tiny 0.35,0.67}]$	$0.65\,[\text{\tiny 0.52,0.78}]$	$0.49\,[\text{\tiny 0.30,0.66}]$	$0.60\,[\text{\tiny 0.44,0.74}]$
GREEN [ 15 ]	$0.62\,[\text{\tiny 0.46,0.75}]$	$0.75\,[\text{\tiny 0.64,0.86}]$	$0.67\,[\text{\tiny 0.54,0.78}]$	$0.70\,[\text{\tiny 0.59,0.80}]$

Table 1: Kendall

\tau

and Pearson

r

correlations (95% CI) between automatic metrics and radiologist-derived clinically significant error counts (

n=50

). Columns refer to different candidate reports on ReXVal, each of which was chosen to optimize a specific metric [ 24 ] . GREEN

E

and CRIMSON

E

denote the total (unweighted) error count, while CRIMSON Weighted

E

applies clinical severity–based weighting to errors. All other metrics were calculated using RadEval [ 22 ] . ∗ CRIMSON results are averaged across 5 runs due to non-deterministic API outputs.

Figure 2: RadJudge results. For each case, metrics are evaluated based on whether their relative ranking of multiple candidate reports agrees with the expected ordering determined with agreement across three attending cardiothoracic radiologists. Each category contains three cases; entries are cases passed (out of 3), with totals out of 30.

Figure 3: Radiologist Preference Alignment (RadPref). Correlation between metric score and radiologist rating differences across 100 pairwise cases. Each point corresponds to a case comparing two candidate reports for the same reference report.

Figure 4: MedGemmaCRIMSON vs GPT-5.2. A) Mean absolute error across false findings, missing findings, and attribute errors per radiologist. B) Severity categorization confusion matrices between three radiologists and CRIMSON, computed only on matched errors (i.e., findings for which both the radiologist and CRIMSON identified an error in the same category). Titles show the percentage of cases for which the radiologist and CRIMSON agree on error category. Color intensity represents the within-row percentage.

Radiology Report Evaluation

Radiology report evaluation measures whether generated or revised reports preserve clinically important findings, relationships, and certainty.

See all 32 papers

Ran Score: a LLM-based Evaluation Score for Radiology Report Generation
Mar 24, 2026Ran Zhang, Yucong Lin, Zhaoli Su +13Radiology Report Evaluation Radiology Report Generation
School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China · Zhengzhou Research Institute, Beijing Institute of Technology, Zhengzhou 450003, China · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China +7
CXRMate-2: Structured Multimodal Temporal Embeddings and Tractable Reinforcement Learning for Clinically Acceptable Chest X-ray Radiology Report Generation
Apr 21, 2026Aaron Nicolson, Elizabeth J. Cooper, Hwan-Jin Yoon +7Radiology Report Generation Radiology VLMs
Australian e-Health Research Centre, CSIRO Health and Biosecurity, Brisbane & Melbourne, Australia · Princess Alexandra Hospital, Metro South Health, Brisbane, Australia · Queensland Digital Health Centre, University of Queensland, Brisbane, Australia +2
Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports
Jun 17, 2026Qingyu Lu, Ruochen Li, Liang Ding +3Clinical Language Model Evaluation Language Model Robustness
Nanyang Technological University · Technical University of Munich · Alibaba +2

Mar 24, 2026J/K move · Enter open · S save

Ran Score: a LLM-based Evaluation Score for Radiology Report Generation

Ran Zhang, Yucong Lin, Zhaoli Su, Bowen Liu+12

School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China · Zhengzhou Research Institute, Beijing Institute of Technology, Zhengzhou 450003, China · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China+7

Radiology Report Evaluation Radiology Report Generation LLM Evaluation Clinical Information Extraction

Open paper AskPDF arXiv

Chest X-ray report generation and automated evaluation are limited by poor recognition of low-prevalence abnormalities and inadequate handling of clinically important language, including negation and ambiguity. We develop a clinician-guided framework combining human expertise and large language models for multi-label finding extraction from free-text chest X-ray reports and use it to define Ran Score, a finding-level metric for report evaluation. Using three non-overlapping MIMIC-CXR-EN cohorts from a public chest X-ray dataset and an independent ChestX-CN validation cohort, we optimize prompts, establish radiologist-derived reference labels and evaluate report generation models. The optimized framework improves the macro-averaged score from 0.753 to 0.956 on the MIMIC-CXR-EN development cohort, exceeds the CheXbert benchmark by 15.7 percentage points on directly comparable labels, and shows robust generalization on the ChestX-CN validation cohort. Here we show that clinician-guided prompt optimization improves agreement with a radiologist-derived reference standard and that Ran Score enables finding-level evaluation of report fidelity, particularly for low-prevalence abnormalities.

Abstract

Radiology Report Evaluation

Explore similar work

Ran Score: a LLM-based Evaluation Score for Radiology Report Generation

CXRMate-2: Structured Multimodal Temporal Embeddings and Tractable Reinforcement Learning for Clinically Acceptable Chest X-ray Radiology Report Generation

Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports