VERT: Reliable LLM Judges for Radiology Report Evaluation

Federica Bologna; Jean-Philippe Corbeil; Matthew Wilkens; Asma Ben Abacha

cs.AIApr 3, 2026

VERT: Reliable LLM Judges for Radiology Report Evaluation

Authors: Federica Bologna, Jean-Philippe Corbeil, Matthew Wilkens, Asma Ben Abacha

Organizations: Cornell University, Ithaca, NY, USA · Microsoft Healthcare & Life Sciences, Montréal, QC, Canada · Cornell University, Ithaca, NY · Microsoft Healthcare & Life Sciences, Redmond, WA, USA

Abstract

Current literature on radiology report evaluation has focused primarily on designing LLM-based metrics and fine-tuning small models for chest X-rays. However, it remains unclear whether these approaches are robust when applied to reports from other modalities and anatomies. Which model and prompt configurations are best suited to serve as LLM judges for radiology evaluation? We conduct a thorough correlation analysis between expert and LLM-based ratings. We compare three existing LLM-as-a-judge metrics (RadFact, GREEN, and FineRadScore) alongside VERT, our proposed LLM-based metric, using open- and closed-source models (reasoning and non-reasoning) of different sizes across two expert-annotated datasets, RadEval and RaTE-Eval, spanning multiple modalities and anatomies. We further evaluate few-shot approaches, ensembling, and parameter-efficient fine-tuning using RaTE-Eval. To better understand metric behavior, we perform a systematic error detection and categorization study to assess alignment of these metrics against expert judgments and identify areas of lower and higher agreement. Our results show that VERT improves correlation with radiologist judgments by up to 11.7% relative to GREEN. Furthermore, fine-tuning Qwen3 30B yield gains of up to 25% using only 1,300 training samples. The fine-tuned model also reduces inference time up to 37.2 times. These findings highlight the effectiveness of LLM-based judges and demonstrate that reliable evaluation can be achieved with lightweight adaptation.

Figures & tables

Model	GREEN	VERT	Formula	Rubric
GPT-4.1-mini	$\underline{0.332}$	$\mathbf{0.371}$	$0.242$	$0.322$
Gemini 2.5 flash	$0.305$	$0.308$	$0.242$	$0.254$

Table 1: Absolute value of Kendall

\tau

between automatic metrics and human-annotated errors on RadEval. Bold = best value across all results; underline = second best. All values are statistically significant (

p<0.01

Model	RadFact	GREEN	GREEN-EC	GREEN F1	FineRadScore	VERT
RadEval — Error Annotations
GPT-4.1 mini	0.2946	0.3322	0.5606	0.4441	0.1725	0.3710
Gemini 3 Flash (low)	-	0.1675	0.0206 †	0.4103	0.1810	0.1937
Claude Sonnet 4.6 (low)	-	0.2931	0.5103	0.3924	0.2333	0.3291
RaTE-Eval — Expert Scores (0–5)
GPT-4.1 mini	0.3197	0.4195	0.1966	0.3369	0.1372	0.4474

Table 2: Absolute value of Kendall’s

\tau

correlations for different LLMs on both RadEval and RaTE-Eval. RadFact’s Python library is not compatible with other providers outside OpenAI. The dagger

\dagger

denotes the lack of statistical significance ( p -value above

0.01

Models	GREEN	GREEN- EC	GREEN F1	VERT	Avg.
GPT-4.1 nano	0.3914	0.1901	0.2754	0.5040	0.3402
GPT-4.1 mini	0.4195	0.1966	0.3369	0.4474	0.3501
GPT-4.1	0.3654	0.2222	0.3350	0.4518	0.3436
GPT-5 (low)	0.3495	0.2208	0.2902	0.3296	0.2975
GPT-5 mini (low)	0.3333	0.1382	0.2943	0.4091	0.2937
Gemini3 Flash (low)	0.0655 †	0.0061 †	0.4641	0.1617	0.1744

Table 3: Kendall’s

\tau

between the LLMs’ GREEN/VERT metrics and radiologist scores (0-5) on RaTE-Eval. Bold and underline are the best and second-best value, respectively, per metric. The dagger

\dagger

denotes the lack of statistical significance ( p -value above

0.01

Thinking Level	GPT-5	GPT-5 mini	Gemini 3 Flash	Claude Opus	Claude Sonnet	Qwen3 30B	Qwen3 4B
None	–	–	0.0795 †	0.4409	0.4343	0.0411 †	0.0434 †
Low	0.3333	0.3495	0.0655 †	0.4493	0.4322	–	–
Medium (default thinking)	0.3674	0.3482	0.1183	0.3590	0.2988	0.2851	0.0475 †
High	0.3508	0.2750	0.1207	0.2004	0.2054	–	–

Table 4: Kendall’s

\tau

correlation of GREEN on RaTE-Eval across thinking levels. Dashes indicate unavailable configurations. Bold denotes the best value per model. The dagger

\dagger

denotes the lack of statistical significance ( p -value above

0.01

	GREEN				VERT
Ensemble Method	SCORE	F1	EC	All	SCORE	All
Baselines	0.4493	0.4641	0.2461	–	0.4633	–
Average	0.4149	0.4249	0.2649	–	0.4671	–
Linear Regression	0.4478	0.4908	0.3486	0.5126	0.5683	0.5573

Table 5: Absolute value of Kendall’s

\tau

correlation between ensemble-aggregated GREEN/VERT metrics and radiologist scores (0–5) on RaTE-Eval. Each column uses only the corresponding metric scores from each model, except for All which leverages all scores from GREEN or VERT—i.e., matched findings, error counts, and scores. Bold indicates the best result per metric.

	Rad-Eval Shots					RaTE-Eval Shots
0-shot	3	5	10	rad-err	rad-err 10 human	3	5	10	rate-err	rate-err 10 VERT
RadEval (n=100)
0.379	0.499 +12.0%	0.372 -0.7%	0.397 +1.8%	0.390 +1.1%	0.431 +5.2%	0.289 -9.0%	0.367 -1.2%	0.341 -3.8%	0.423 +4.4%	0.398 +1.9%
RaTE-Eval (n=100)
0.465	0.468 +0.3%	0.495 +3.0%	0.528 +6.3%	0.476 +1.1%	0.489 +2.4%	0.484 +1.9%	0.504 +3.9%	0.497 +3.2%	0.405 -6.0%	0.492 +2.7%

Table 6: Absolute value of Kendall

\tau

between the vert output score and human judgments for gpt-4.1-mini (n=100). Each cell shows

\tau

( bold = best per row) with the

\Delta

vs. the zero-shot baseline in color.

Figure 1: Fine-tuning of Qwen3-30B-A3B-Instruct-2507 on RaTE-Eval trainset.

Figure 2: Mean number of clinically significant errors annotated by humans vs. vs. those detected by model-prompt combinations on RadEval.

Figure 3: Mean number of clinically significant errors detected by model-prompt combinations vs. number of injected errors, for False Prediction and Omission error types.

Figure 4: $F_{1}$ scores for error type on RadEval.

Method	(a) False Finding	(b) Missing Finding
GREEN	0.92	0.72
VERT	0.91	0.69
VERT 10-shot (RadEval)	0.90	0.71
Injected errors	2064	2130

Table 7:

F_{1}

scores for error type on RaTE-Eval.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Histogram of error annotations in RadEval.

Figure 6: Histogram of normalized expert-annotated scores in RaTE-Eval.

Figure 7: Simulations of the impact of sweeping $S$ or $TP$ for GREEN and F1.

Explore similar work

RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching
Sep 1, 2026Charles Corbière, Léo Machado, Aubin Charley +3LLM-as-a-Judge Medical Image Benchmarks
Raidium, Paris, France · Department of Radiology, Fondation Ophtalmologique Adolphe de Rothschild, Paris, France · Service de Radiologie Interventionnelle, Département d’Anesthésie, Chirurgie et Imagerie Interventionnelle (DACI), Gustave Roussy, Villejuif, France
Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports
Jun 17, 2026Qingyu Lu, Ruochen Li, Liang Ding +3Clinical Language Model Evaluation Language Model Robustness
Nanyang Technological University · Technical University of Munich · Alibaba +2
ReportQA: QA-Based Radiology Report Evaluation
Jun 13, 2026Yiming Shi, Shaoshuai Yang, Xi Chen +10Radiology Report Generation Medical VLMs
Department of Electronic Engineering, Tsinghua University · Beijing Electronic Digital & Intelligence · College of AI, Tsinghua University +1

Sep 1, 2026J/K move · Enter open · S save

RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching

Charles Corbière, Léo Machado, Aubin Charley, Baptiste Callard+2

Raidium, Paris, France · Department of Radiology, Fondation Ophtalmologique Adolphe de Rothschild, Paris, France · Service de Radiologie Interventionnelle, Département d’Anesthésie, Chirurgie et Imagerie Interventionnelle (DACI), Gustave Roussy, Villejuif, France

LLM-as-a-Judge Medical Image Benchmarks Radiology Report Evaluation

Open paper AskPDF arXiv

As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge. Large language model (LLM)-based metrics are now the best-correlated with radiologist judgment, yet they output a single opaque score that neither a clinician nor a model builder can easily interpret or audit. We introduce RadMatch, a multi-stage, LLM-based metric that decomposes report comparison into a structured finding-level matching with significance-aware scoring and error characterization across seven clinical attribute dimensions (status, location, severity, morphology, certainty, longitudinal comparison, and measurement). The main score is the actionable-error count, both interpretable and auditable. Candidate findings are graded correct, partial, or incorrect, and unmatched findings are counted as missed or hallucinated. Triage and actionable safety recall/precision and per-subset views add complementary, deployment-oriented lenses. Across two expert benchmarks, RadMatch is the most clinically aligned metric, matching inter-radiologist agreement on ReXVal and more than doubling the best prior metric on the harder RadEvalExpert. Relying only on few-shot prompting, it is designed to extend to other modalities and anatomies. We will release RadMatch as open-source code with an interactive dashboard for inspecting results.

Abstract

Explore similar work

RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching

Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports

ReportQA: QA-Based Radiology Report Evaluation