Ran Score: a LLM-based Evaluation Score for Radiology Report Generation

Ran Zhang; Yucong Lin; Zhaoli Su; Bowen Liu; Danni Ai; Tianyu Fu; Deqiang Xiao; Jingfan Fan

cs.AIMar 24, 2026

Ran Score: a LLM-based Evaluation Score for Radiology Report Generation

Authors: Ran Zhang, Yucong Lin, Zhaoli Su, Bowen Liu, Danni Ai, Tianyu Fu, Deqiang Xiao, Jingfan Fan,

Organizations: School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China · Zhengzhou Research Institute, Beijing Institute of Technology, Zhengzhou 450003, China · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China · Department of Radiology, China-Japan Friendship Hospital (Institute of Clinical Medical Sciences), Beijing 100029, China · Chinese Academy of Medical Science & Peking Union Medical College, Beijing 100730, China · Department of Radiology, Peking University China-Japan Friendship School of Clinical Medicine, Beijing 100029, China · Department of Gastroenterology, China-Japan Friendship Hospital, Beijing 100029, China · NHC Key Laboratory of Clinical Big Data Standardization & Integration, Beijing 100029, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081, China · Department of Radiology, China-Japan Friendship Hospital, Beijing 100029, China

Abstract

Chest X-ray report generation and automated evaluation are limited by poor recognition of low-prevalence abnormalities and inadequate handling of clinically important language, including negation and ambiguity. We develop a clinician-guided framework combining human expertise and large language models for multi-label finding extraction from free-text chest X-ray reports and use it to define Ran Score, a finding-level metric for report evaluation. Using three non-overlapping MIMIC-CXR-EN cohorts from a public chest X-ray dataset and an independent ChestX-CN validation cohort, we optimize prompts, establish radiologist-derived reference labels and evaluate report generation models. The optimized framework improves the macro-averaged score from 0.753 to 0.956 on the MIMIC-CXR-EN development cohort, exceeds the CheXbert benchmark by 15.7 percentage points on directly comparable labels, and shows robust generalization on the ChestX-CN validation cohort. Here we show that clinician-guided prompt optimization improves agreement with a radiologist-derived reference standard and that Ran Score enables finding-level evaluation of report fidelity, particularly for low-prevalence abnormalities.

Figures & tables

Figure 1: Human–LLM collaborative framework for multi-label finding extraction from chest X-ray reports. (a) Exploratory extraction and clustering of disease-related entities from 3,000 chest X-ray reports to inform the definition of standardized finding labels. (b) Establishment of the diagnostic reference standard through independent binary annotation of reports by six board-certified thoracic radiologists. (c) Illustration of the Human–LLM collaborative workflow using pneumothorax as an example, showing iterative clinician-guided prompt refinement based on error analysis to improve finding-level extraction performance toward higher agreement with the radiologist-derived reference standard.

Common abnormalities
Characteristic	MIMIC-CXR-EN (n=6,300)	ChestX-CN (n=150)
Data collection period	2011–2016	2024.11–2025.2
Age (years), median (IQR)	58 (40–72)	58 (45–70)
Male, n (%)	3,528 (56)	74 (49.3)
Imaging modality	Chest X-ray	Chest X-ray
Report length (words), median	75 (26–202)	78 (55–110)

Table 1: Demographic and Clinical Characteristics of the Study Cohorts

Figure 2: Flowchart of dataset construction and cohort allocation. Reports from MIMIC-CXR were screened and divided into three non-overlapping MIMIC-CXR-EN cohorts: a 3,000-report cohort for taxonomy construction, a 300-report development cohort for prompt optimization and radiologist reference-standard-based evaluation, and an independent 3,000-report test cohort for downstream benchmarking of report generation models. An additional 150-report ChestX-CN validation cohort was used for external validation.

Column Name	Qwen3-14B	Qwen-Plus	GPT-3.5-turbo	GPT-4o-mini	DeepSeek-R1
Atelectasis	0.987 [0.966, 0.996]	0.980 [0.957, 0.993]	0.787 [0.736, 0.832]	0.980 [0.957, 0.993]	0.973 [0.948, 0.988]
Cardiomegaly	0.980 [0.957, 0.993]	0.987 [0.966, 0.996]	0.477 [0.419, 0.535]	0.987 [0.966, 0.996]	0.967 [0.940, 0.984]
Consolidation	0.973 [0.948, 0.988]	0.983 [0.962, 0.995]	0.787 [0.736, 0.832]	0.977 [0.953, 0.991]	0.990 [0.971, 0.998]
Edema	0.980 [0.957, 0.993]	0.977 [0.953, 0.991]	0.623 [0.566, 0.678]	0.980 [0.957, 0.993]	0.983 [0.962, 0.995]
Enlarged Cardiomediastinum	0.957 [0.927, 0.977]	0.980 [0.957, 0.993]	0.780 [0.729, 0.826]	0.823 [0.775, 0.865]	0.867 [0.823, 0.903]
Fracture	0.993 [0.976, 0.999]	0.990 [0.971, 0.998]	0.947 [0.915, 0.969]	0.990 [0.971, 0.998]	0.990 [0.971, 0.998]

Table 2: Per-label accuracy of LLM-based finding extraction on the 300-report MIMIC-CXR-EN development cohort.

Column Name	Qwen3-14B	Qwen-Plus	GPT-3.5-turbo	GPT-4o-mini	DeepSeek-R1
Atelectasis	0.993 [0.976, 0.999]	0.920 [0.883, 0.948]	0.273 [0.224, 0.328]	0.940 [0.907, 0.964]	0.987 [0.966, 0.996]
Cardiomegaly	0.980 [0.957, 0.993]	0.993 [0.976, 0.999]	0.153 [0.115, 0.199]	0.987 [0.966, 0.996]	0.993 [0.976, 0.999]
Consolidation	0.987 [0.966, 0.996]	0.960 [0.931, 0.979]	0.460 [0.403, 0.518]	0.960 [0.931, 0.979]	0.980 [0.957, 0.993]
Edema	1.000 [0.988, 1.000]	1.000 [0.988, 1.000]	0.267 [0.218, 0.321]	1.000 [0.988, 1.000]	1.000 [0.988, 1.000]
Enlarged Cardiomediastinum	1.000 [0.988, 1.000]	0.993 [0.976, 0.999]	0.700 [0.645, 0.751]	0.993 [0.976, 0.999]	0.993 [0.976, 0.999]
Fracture	0.993 [0.976, 0.999]	0.973 [0.948, 0.988]	0.947 [0.915, 0.969]	1.000 [0.988, 1.000]	0.980 [0.957, 0.993]

Table 3: Per-label accuracy of LLM-based finding extraction on the 150-report ChestX-CN validation cohort.

Column Name	Accuracy		Precision		Recall		F1 score		$\Delta$ F1	CheXbert	$\Delta$ F1
Atelectasis	0.987	0.987	0.957	0.957	1.000	1.000	0.978	0.978	0.000	0.940	0.013
Cardiomegaly	0.980	0.987	0.987	0.986	0.937	0.960	0.961	0.973	0.013	0.815	0.159
Consolidation	0.973	0.993	0.900	0.950	0.750	0.950	0.818	0.950	0.087	0.877	0.028
Edema	0.980	0.990	0.939	0.919	0.886	1.000	0.912	0.958	0.018	0.881	0.049
Enlarged Cardiomediastinum	0.957	0.970	0.882	0.904	0.923	0.971	0.902	0.936	0.029	0.713	0.218
Fracture	0.993	1.000	1.000	1.000	0.500	1.000	0.667	1.000	0.333	0.791	0.209

Table 4: Per-label performance of Qwen3-14B before and after few-shot prompt optimization on the 300-report MIMIC-CXR-EN development cohort.

	Micro-Averaged Performance				Macro-Averaged Performance
	Accuracy	Precision	Recall	F1 score	Accuracy	Recall	F1 score
RGRG	0.874	0.391	0.512	0.443	0.216	0.306	0.220
XrayGPT	0.839	0.091	0.209	0.127	0.056	0.117	0.045
R2GenGPT	0.877	0.411	0.528	0.462	0.230	0.388	0.247
PromptMRG	0.881	0.327	0.568	0.415	0.172	0.342	0.178
LLM-RG4	0.854	0.562	0.447	0.498	0.390	0.353	0.333

Table 5: Performance comparison of radiology report generation models on the MIMIC-CXR-EN test cohort.

Figure 3: Micro-averaged performance of radiology report generation models on the MIMIC-CXR-EN test cohort. Micro-averaged precision, recall and F1 score were calculated across all finding labels, with each label weighted according to its prevalence in the dataset. Model predictions were evaluated using the optimized Qwen3-14B labeler.

Figure 4: Macro-averaged performance of radiology report generation models across finding categories. Macro-averaged precision, recall and F1 score were calculated by treating all finding labels equally. This approach emphasizes model performance on low-prevalence and rare findings.

Radiology Report Evaluation

Radiology report evaluation measures whether generated or revised reports preserve clinically important findings, relationships, and certainty.

See all 32 papers

CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation
Mar 6, 2026Mohammed Baharoon, Thibault Heintz, Siavash Raissi +6Radiology Report Evaluation LLM Evaluation
Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Radiation Oncology, Mass General Brigham, Boston, MA · King Fahad Hospital, Al-Ahsa Health Cluster, Al Hofuf, Saudi Arabia +2
CXRMate-2: Structured Multimodal Temporal Embeddings and Tractable Reinforcement Learning for Clinically Acceptable Chest X-ray Radiology Report Generation
Apr 21, 2026Aaron Nicolson, Elizabeth J. Cooper, Hwan-Jin Yoon +7Radiology Report Generation Radiology VLMs
Australian e-Health Research Centre, CSIRO Health and Biosecurity, Brisbane & Melbourne, Australia · Princess Alexandra Hospital, Metro South Health, Brisbane, Australia · Queensland Digital Health Centre, University of Queensland, Brisbane, Australia +2
DobicVLM: Aligning Chest X-Ray Report Generation with Clinically-Grounded Programmatic Rewards via Group Relative Policy Optimization
Jul 21, 2026Thanni Adewuyi, Angelica Obayi, Andem Aniekan +10Radiology Report Generation Radiology VLMs
Dobic Health · HelpMum Africa · University of Ibadan

Mar 6, 2026J/K move · Enter open · S save

CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation

Mohammed Baharoon, Thibault Heintz, Siavash Raissi, Mahmoud Alabbad+5

Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Radiation Oncology, Mass General Brigham, Boston, MA · King Fahad Hospital, Al-Ahsa Health Cluster, Al Hofuf, Saudi Arabia+2

Radiology Report Evaluation LLM Evaluation Radiology Report Generation

Open paper AskPDF arXiv

We introduce CRIMSON, a clinically grounded evaluation framework for chest X-ray report generation that assesses reports based on diagnostic correctness, contextual relevance, and patient safety. Unlike prior metrics, CRIMSON incorporates full clinical context, including patient age, indication, and guideline-based decision rules, and prevents normal or clinically insignificant findings from exerting disproportionate influence on the overall score. The framework categorizes errors into a comprehensive taxonomy covering false findings, missing findings, and eight attribute-level errors (e.g., location, severity, measurement, and diagnostic overinterpretation). Each finding is assigned a clinical significance level (urgent, actionable non-urgent, non-actionable, or expected/benign), based on a guideline developed in collaboration with attending cardiothoracic radiologists, enabling severity-aware weighting that prioritizes clinically consequential mistakes over benign discrepancies. CRIMSON is validated through strong alignment with clinically significant error counts annotated by six board-certified radiologists in ReXVal (Kendalls tau = 0.61-0.71; Pearsons r = 0.71-0.84), and through two additional benchmarks that we introduce. In RadJudge, a targeted suite of clinically challenging pass-fail scenarios, CRIMSON shows consistent agreement with expert judgment. In RadPref, a larger radiologist preference benchmark of over 100 pairwise cases with structured error categorization, severity modeling, and 1-5 overall quality ratings from three cardiothoracic radiologists, CRIMSON achieves the strongest alignment with radiologist preferences. We release the metric, the evaluation benchmarks, RadJudge and RadPref, and a fine-tuned MedGemma model to enable reproducible evaluation of report generation, all available at https://github.com/rajpurkarlab/CRIMSON.

Abstract

Radiology Report Evaluation

Explore similar work

CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation

CXRMate-2: Structured Multimodal Temporal Embeddings and Tractable Reinforcement Learning for Clinically Acceptable Chest X-ray Radiology Report Generation

DobicVLM: Aligning Chest X-Ray Report Generation with Clinically-Grounded Programmatic Rewards via Group Relative Policy Optimization