Enhancing Reinforcement Learning for Radiology Report Generation with Evidence-aware Rewards and Self-correcting Preference Learning

Qin Zhou; Guoyan Liang; Qianyi Yang; Jingyuan Chen; Sai Wu; Chang Yao; Zhe Wang

cs.LGApr 15, 2026

Enhancing Reinforcement Learning for Radiology Report Generation with Evidence-aware Rewards and Self-correcting Preference Learning

Authors: Qin Zhou, Guoyan Liang, Qianyi Yang, Jingyuan Chen, Sai Wu, Chang Yao, Zhe Wang

Organizations: Department of Computer Science and Engineering, ECUST, China · Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, P. R. China · Zhejiang University, Hangzhou, China · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security

Abstract

Recent reinforcement learning (RL) approaches have advanced radiology report generation (RRG), yet two core limitations persist: (1) report-level rewards offer limited evidence-grounded guidance for clinical faithfulness; and (2) current methods lack an explicit self-improving mechanism to align with clinical preference. We introduce clinically aligned Evidence-aware Self-Correcting Reinforcement Learning (ESC-RL), comprising two key components. First, a Group-wise Evidence-aware Alignment Reward (GEAR) delivers group-wise, evidence-aware feedback. GEAR reinforces consistent grounding for true positives, recovers missed findings for false negatives, and suppresses unsupported content for false positives. Second, a Self-correcting Preference Learning (SPL) strategy automatically constructs a reliable, disease-aware preference dataset from multiple noisy observations and leverages an LLM to synthesize refined reports without human supervision. ESC-RL promotes clinically faithful, disease-aligned reward and supports continual self-improvement during training. Extensive experiments on two public chest X-ray datasets demonstrate consistent gains and state-of-the-art performance.

Figures & tables

Figure 1: Overview of the proposed Evidence-aware Self-Correcting Reinforcement Learning (ESC-RL) framework.

Figure 2: Illustration of the Self-correcting Preference Learning (SPL) module.

Model	Year	Lexical Metrics				Radiology Metrics
Model	Year	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$	ROUGE $\uparrow$	BERTScore $\uparrow$	RadCliQ $\downarrow$	RadGraphF1 $\uparrow$	ChexbertF1 $\uparrow$	GREEN $\uparrow$
R2Gen	ACL 2020	0.353	0.103	0.277	0.886	2.89	0.195	0.276	0.306
R2GenCMN	ACL 2021	0.353	0.106	0.278	0.867	2.87	0.199	0.278	0.308
RGRG	CVPR 2023	0.373	0.126	0.264	0.873	2.85	0.221	0.447	0.313
MiniGPT-Med	-	0.191	0.012	-	0.636	2.95	0.164	0.172	0.211
PromptMRG	AAAI 2024	0.398	0.112	0.268	0.857	2.77	0.227	0.476	0.289

Table 1: Comparison with existing RRG methods on the MIMIC-CXR dataset. The best and second-best results are highlighted in bold and blue , respectively. ‘-’ indicates not reported.

Model	Year	Lexical Metrics				Radiology Metrics
Model	Year	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$	ROUGE $\uparrow$	BERTScore $\uparrow$	RadCliQ $\downarrow$	RadGraphF1 $\uparrow$	CheXbertF1 $\uparrow$	GREEN $\uparrow$
R2Gen	ACL 2020	0.289	0.052	0.243	0.861	2.79	0.187	0.145	0.482
R2GenCMN	ACL 2021	0.289	0.055	0.246	0.864	2.78	0.190	0.147	0.483
RGRG	CVPR 2023	0.266	0.063	0.180	0.867	2.71	0.189	0.180	0.481
PromptMRG	AAAI 2024	0.401	0.098	0.281	0.871	2.60	0.274	0.211	0.457
MedVersa	-	0.247	0.047	-	0.884	2.71	0.209	0.217	0.516

Table 2: Comparison with existing RRG methods on the IU-Xray dataset. The best and second-best results are highlighted in bold and blue , respectively. ‘-’ indicates not reported.

Figure 3: Qualitative comparison of reports generated by R2GenRL, PromptMRG, REVTAF, and our method. Sentences are color-coded to indicate their corresponding descriptions. Incorrect statements are marked with $\times$ , and unmarked sentences denote descriptions not mentioned in the ground-truth report.

Model	MIMIC-CXR			IU-Xray
Model	Precision	Recall	F1	Precision	Recall	F1
R2Gen	0.333	0.273	0.276	0.151	0.145	0.145
R2GenCMN	0.334	0.275	0.278	0.154	0.147	0.147
RGRG	0.461	0.475	0.447	0.183	0.187	0.180
PromptMRG	0.501	0.509	0.476	0.213	0.229	0.211
REVTAF	0.628	0.613	0.592	0.286	0.282	0.273

Table 3: Clinical Efficacy (CE) comparison of 14 diseases on the MIMIC-CXR and IU-Xray datasets. Best values are highlighted in bold and second-best in blue .

Setting	RL	GEAR	SPL	Lexical Metrics				Radiology Metrics
Setting	RL	GEAR	SPL	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$	ROUGE $\uparrow$	BERTScore $\uparrow$	RadCliQ $\downarrow$	RadGraphF1 $\uparrow$	CheXbertF1 $\uparrow$	GREEN $\uparrow$
Baseline				0.465	0.182	0.336	0.887	2.48	0.279	0.514	0.344
(a)	$\checkmark$			0.472	0.187	0.335	0.889	2.45	0.282	0.517	0.349
(b)		$\checkmark$		0.471	0.191	0.334	0.891	2.42	0.285	0.518	0.358
(c)			$\checkmark$	0.475	0.190	0.339	0.892	2.40	0.288	0.517	0.356
(d)	$\checkmark$	$\checkmark$		0.481	0.194	0.345	0.890	2.40	0.289	0.517	0.359

Table 4: Effectiveness analysis of each component on MIMIC-CXR test set.

Figure 4: An example of re-integrating multiple observations into a refined report. Sentences are color-coded to indicate their corresponding descriptions. Incorrect statements are highlighted with $\times$ , and unmarked sentences denote content that is not mentioned in the ground-truth report.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Lexical Metrics				Radiology Metrics
Model	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$	ROUGE $\uparrow$	BERTScore $\uparrow$	RadCliQ $\downarrow$	RadGraphF1 $\uparrow$	CheXbertF1 $\uparrow$	GREEN $\uparrow$
Baseline	0.465	0.182	0.336	0.887	2.48	0.279	0.514	0.344
MedKLIP	0.485	0.199	0.349	0.897	2.39	0.301	0.521	0.392
MAVL	0.487	0.199	0.352	0.898	2.39	0.304	0.608	0.394

Appendix

Table 5: Influence of different vision–language grounding models for ESC-RL on the MIMIC-CXR test set. The best results are highlighted in bold.

Model	Lexical Metrics				Radiology Metrics
Model	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$	ROUGE $\uparrow$	BERTScore $\uparrow$	RadCliQ $\downarrow$	RadGraphF1 $\uparrow$	CheXbertF1 $\uparrow$	GREEN $\uparrow$
TP-MSE & FN-MSE	0.456	0.177	0.346	0.859	2.53	0.264	0.591	0.335
TP-IoU-based & FN-IoU-based	0.451	0.171	0.342	0.865	2.59	0.254	0.590	0.327
TP-IoU-based &FN-MSE (Ours)	0.487	0.199	0.352	0.898	2.39	0.304	0.608	0.394

Appendix

Table 6: Influence of different TP/FN/FP loss for ESC-RL on the MIMIC-CXR test set. The best results are highlighted in bold.

Figure 5: Ablation study. (a) The influence of the different reward weight $\gamma$ . (b) The influence of different LLMs for report re-integration.

Model	Lexical Metrics				Radiology Metrics
Model	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$	ROUGE $\uparrow$	BERTScore $\uparrow$	RadCliQ $\downarrow$	RadGraphF1 $\uparrow$	CheXbertF1 $\uparrow$	GREEN $\uparrow$
CheXbert	0.487	0.199	0.352	0.898	2.39	0.304	0.608	0.394
Gemini-2.0-flash	0.471	0.180	0.342	0.865	2.57	0.272	0.592	0.374
GPT-5	0.485	0.193	0.350	0.886	2.40	0.296	0.605	0.391

Appendix

Table 7: Influence of different disease-status extractor on the MIMIC-CXR test set. The best results are highlighted in bold.

Model	Lexical Metrics				Radiology Metrics
Model	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$	ROUGE $\uparrow$	BERTScore $\uparrow$	RadCliQ $\downarrow$	RadGraphF1 $\uparrow$	CheXbertF1 $\uparrow$	GREEN $\uparrow$
w LLM-refinement	0.487	0.199	0.352	0.898	2.39	0.304	0.608	0.394
wo LLM-refinement	0.484	0.196	0.349	0.895	2.41	0.301	0.608	0.393

Appendix

Table 8: Influence of LLM refinement on the MIMIC-CXR test set. The best results are highlighted in bold.

Figure 6: The prompt template used for re-integrating multiple observations into a refined report within the ESC-RL framework using LLMs.

Radiology Report Generation

Radiology report generation uses medical images and clinical context to produce structured, clinically accurate findings and impressions.

See all 159 papers

Precision Recall Controllable Radiology Report Generation via Hybrid Natural Language and Clinical Reward Learning
Jun 19, 2026Ling Chen, Ruinan Jin, Jun Luo +6Radiology Report Generation Controllable Text Generation
The Ohio State University, Columbus, OH 43221, USA · University Medical Center Regensburg, Regensburg, Germany
Rethinking the Efficiency and Effectiveness of Reinforcement Learning for Radiology Report Generation
Mar 4, 2026Zilin Lu, Ruifeng Yuan, Weiwei Cao +6Radiology Report Generation Sample-Efficient RL
DAMO Academy, Alibaba Group · Ningbo No.2 Hospital, China · Hupan Lab, 310023, China +2
Multi-Modal Multi-Agent Reinforcement Learning for Radiology Report Generation
Feb 17, 2026Kaito Baba, Risa Kishikawa, Satoshi KoderaRadiology Report Generation Multimodal Agents
Department of Cardiovascular Medicine The University of Tokyo Hospital, Tokyo, Japan

Jun 19, 2026J/K move · Enter open · S save

Precision Recall Controllable Radiology Report Generation via Hybrid Natural Language and Clinical Reward Learning

Ling Chen, Ruinan Jin, Jun Luo, Hanliang Chen+5

The Ohio State University, Columbus, OH 43221, USA · University Medical Center Regensburg, Regensburg, Germany

Radiology Report Generation Controllable Text Generation RL for Generative Models

Open paper AskPDF arXiv

Automated radiology report generation (RRG) has gained increasing attention because it can reduce the heavy workload of clinical report writing. However, most existing methods mainly optimize for natural language generation (NLG) metrics that focus on language fluency, while providing little control over clinically important factors such as precision and recall. As consequence, generated reports may be fluent but not well aligned with different clinical needs. To address this challenge, we propose a reinforcement learning framework for precision recall controllable RRG, where a control parameter explicitly adjusts the trade-off between clinical precision and recall during inference. This design allows the model to flexibly generate reports according to different clinical requirements. To ensure clinical correctness, we introduce a clinical reward into the training objective, which helps improve clinical efficacy (CE) beyond standard language-based optimization. In addition, we apply a group-relative training strategy that normalizes rewards within each training group, reducing reward variance and improving training stability. Extensive experiments on the MIMIC-CXR dataset show that our method consistently outperforms state-of-the-art approaches in both NLG and CE evaluation metrics, while providing reliable control over the CE precision recall trade-off.

Abstract

Radiology Report Generation

Explore similar work

Precision Recall Controllable Radiology Report Generation via Hybrid Natural Language and Clinical Reward Learning

Rethinking the Efficiency and Effectiveness of Reinforcement Learning for Radiology Report Generation

Multi-Modal Multi-Agent Reinforcement Learning for Radiology Report Generation