UR$^{2}$-MLLM: Uncertainty-aware Revisit Reasoning in Multimodal Large Language Models for Radiology Report Generation

Yucheng Chen; Yang Yu; Jiazhou Zhou; Yufei Shi; Yongying Lan; Yichi Zhang; Liyi Li; Si Yong Yeo

cs.CVAug 23, 2026

UR $^{2}$ -MLLM: Uncertainty-aware Revisit Reasoning in Multimodal Large Language Models for Radiology Report Generation

Authors: Yucheng Chen, Yang Yu, Jiazhou Zhou, Yufei Shi, Yongying Lan, Yichi Zhang, Liyi Li, Si Yong Yeo

Organizations: MedVisAI Lab · Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore · Centre of AI in Medicine, Singapore · Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · AI Thrust, The Hong Kong University of Science and Technology (Guangzhou) · Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, China

Abstract

Radiologists generate diagnostic reports through iterative and selective revisiting of suspicious regions to refine their interpretations. Recent multimodal large language models (MLLMs) for radiology report generation (RRG) have shifted from text-only reasoning toward a ``Thinking-with-Images'' paradigm, incorporating visual evidence into the reasoning process. However, existing methods provide static visual evidence without a dynamic revisit mechanism during reasoning, neglecting how radiologists re-examine uncertain observations. To this end, we propose an Uncertainty-aware Revisit Reasoning MLLM (UR $^{2}$ -MLLM) framework that dynamically revisits uncertain regions during reasoning for RRG. UR $^{2}$ -MLLM is first equipped with uncertainty perception by training on an uncertainty-aware dataset. We then construct a multimodal reasoning trajectory dataset together with a detect-and-copy mechanism, which guides when and where to revisit. Finally, a visual grounding reward refines this behavior through reinforcement learning, aligning the revisited regions with corresponding anatomical structures. Experiments on MIMIC-CXR and IU-Xray show that UR $^{2}$ -MLLM achieves state-of-the-art performance, highlighting the value of uncertainty-aware visual revisit reasoning for reliable and clinically aligned report generation.

Figures & tables

Figure 1: Three reasoning paradigms for radiology report generation. Our UR 2 -MLLM triggers uncertainty-aware visual revisits during reasoning.

Figure 2: Overall framework of UR 2 -MLLM, including (a) multimodal reasoning trajectory construction, (b) training pipeline overview, and (c) stage-wise training details.

Figure 3: Details for Detect-and-Copy mechanism. $\mathcal{V}$ denotes the vocabulary and $P$ denotes $\#$ patches.

Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE-L
LLaVA-1.5-7B 17	19.09	7.46	2.81	1.25	19.16	18.36
LLaVA-1.5-7B-SFT-CXR	22.58	15.06	9.43	6.13	25.71	28.09
HuatuoGPT-Vision-7B 3	19.33	9.42	4.64	1.93	26.01	20.78
MAIRA-2 2	24.94	14.12	9.01	6.14	26.78	28.65
Qwen2.5-VL-3B 29	13.09	5.42	2.08	0.89	20.81	15.23
Gemini 2.5 Pro 28	12.54	5.20	2.25	1.05	21.19	15.01

Table 1: NLG metrics for radiology report generation across different models on MIMIC-CXR( Best , Second-best ).

Model	CheXbert			RadGraph
Model	Acc.	Ma-F1	Mi-F1	Ent.F1	F1
LLaVA-1.5-7B	63.25	4.94	38.54	7.61	4.95
LLaVA-1.5-7B-SFT-CXR	72.72	5.00	51.51	17.57	13.06
HuatuoGPT-Vision-7B	71.15	5.34	48.62	15.92	9.06
MAIRA-2	67.39	6.34	46.53	25.01	17.05
Qwen2.5-VL-3B	67.78	4.75	37.66	9.46	4.66

Table 2: Clinical accuracy metrics on the MIMIC-CXR dataset ( Best , Second-best ).

Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE-L
LLaVA-1.5-7B	16.52	6.60	3.00	1.40	19.65	17.48
LLaVA-1.5-7B-SFT-CXR	21.42	12.95	8.03	5.20	23.24	26.40
HuatuoGPT-Vision-7B	19.33	10.70	6.28	2.81	31.02	23.42
MAIRA-2	26.37	15.60	9.64	6.03	25.52	31.18
Qwen2.5-VL-3B	11.38	5.05	2.28	1.05	21.65	15.01
CURV	29.23	18.76	12.08	6.86	38.30	39.08

Table 3: NLG metrics for radiology report generation on the IU X-ray dataset ( Best , Second-best ).

Model	CheXbert			RadGraph
Model	Acc.	Ma-F1	Mi-F1	Ent.F1	F1
LLaVA-1.5-7B	72.03	4.66	46.81	13.37	8.76
LLaVA-1.5-SFT	76.34	5.84	53.33	16.19	10.31
HuatuoGPT-V-7B	89.89	5.57	67.07	22.98	13.96
MAIRA-2	88.74	6.22	70.75	34.53	24.01
Qwen2.5-VL-3B	80.64	4.92	49.47	12.70	6.26

Table 4: Clinical accuracy metrics on the IU X-ray dataset ( Best , Second-best ).

Figure 4: Revisit-quality on the MIMIC-CXR.

Model	NLG						CE
Model	BL-1	BL-2	BL-3	BL-4	MTR	RG-L	F1	Rec.	Prec.
Base (Qwen2.5-VL-3B)	13.09	5.42	2.08	0.89	20.81	15.23	4.75	-	-
+ S1	9.88	4.31	1.91	1.02	20.33	10.74	12.35	28.06	13.47
+ S2	23.66	13.88	8.91	5.96	21.77	18.69	11.66	24.50	14.68
+ S1 + S2 (w/o visual revisit)	10.72	5.22	2.43	1.10	18.57	14.59	3.87	-	-
+ S1 + S2 (w/ static vision)	16.13	5.01	5.19	1.98	17.10	14.97	6.13	21.30	14.96

Table 5: Ablation study of UR 2 on MIMIC-CXR ( Best , Second-best ). S1, S2, S3 denotes Stage1, Stage2, and Stage3, respectively. F1 here denotes CheXbert Macro F1.

Figure 5: A success case of UR 2 -MLLM. Left: ground-truth and generated reports with matched findings highlighted. Middle: reasoning trace where uncertainty triggers detect calls. Right: predicted bounding boxes (red, purple) versus ground-truth (green) from Chest ImaGenome.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Stage 1 SFT data
Dataset	Split	Image-level	Item-level
uncertainty-aware dataset	full	101,457 images	542,834 bbox items
( $\sim$ 5.35 bbox / image, median 4, max 39)	—	—	—
Stage 2 SFT data
multimodal_reasoning_trajectory	full	1,909 images	3,172 revisit anchors
Type A (with detect() calls)	—	1,292 images	3,172 anchors

Appendix

Table 6: Dataset sizes used by UR 2 -MLLM. “Image-level” counts the unique image-report pairs that become training (or evaluation) samples; “item-level” counts the finer-grained annotations derived from each pair, since a single MIMIC-CXR study yields multiple bbox-uncertainty annotations during Stage 1 curation and a single ImaGenome-anchored sentence yields multiple revisit regions during Stage 2 curation, so item-level counts are strictly larger than image-level counts wherever applicable. Stage 3 uses MIMIC-CXR image-report pairs without additional annotation. IU-Xray is used only for out-of-domain evaluation; its “item-level” column counts the individual frontal and lateral DICOM images, since a single report covers both views.

(a) Variant: impression label only — look for unlabelled findings content
system content	You are a radiology report parser. The report below has a labeled IMPRESSION section but no labeled FINDINGS section. Determine whether the report also contains radiographic observations (findings content)—any text that describes the appearance of anatomical structures (e.g. lungs, heart, bones, pleura) using imaging language. Answer YES (true) if the report contains any of the following outside the IMPRESSION section: — one or more sentences describing the appearance of anatomical structures; — observations using terms like “clear”, “normal”, “enlarged”, “opacity”, “effusion”, “no evidence of”, “stable”, “unchanged”, etc.; — a descriptive paragraph providing detail beyond the impression summary. Answer NO (false) only if the report contains nothing but the IMPRESSION section and possibly a brief technique / clinical-history note with no imaging observations whatsoever. Reply only with JSON: {"has_findings": true} or {"has_findings": false} . No explanation.
(b) Variant: findings label only — look for unlabelled impression / conclusion
system content	You are a radiology report parser. The report below has a labeled FINDINGS section but no labeled IMPRESSION section. Determine whether the report also contains an impression or conclusion—any text that provides a clinical interpretation, summary, or diagnosis based on the findings. Answer YES (true) if the report contains any of the following: — one or more sentences summarising the overall clinical interpretation; — a concluding statement giving a diagnosis or differential; — a final paragraph or sentence that reads as a summary / conclusion rather than a detailed observation. Answer NO (false) only if the report ends with detailed findings observations and contains no interpretive summary or conclusion anywhere. Reply only with JSON: {"has_impression": true} or {"has_impression": false} . No explanation.
(c) Variant: no header labels — look for findings and impression jointly
system content	You are a radiology report parser. The report below has no labeled FINDINGS or IMPRESSION sections. Determine whether the report contains both types of content, even without labels: A) findings content: text describing the radiographic appearance of anatomical structures (lungs, heart, bones, …); B) impression / conclusion content: text that summarises the overall clinical interpretation, diagnosis, or conclusion. Answer YES (true) if you can identify both types of content in the report, even if they are not in separate paragraphs. Answer NO (false) if the report contains only one type of content, or if it is too brief to clearly contain both. Reply only with JSON: {"has_both": true} or {"has_both": false} . No explanation.
User content (shared across all three variants)

Appendix

Table 7: Stage 1A prompt — report section parser (three header-conditional variants). All three variants share the same user content, shown last; {report} is replaced with the report body, truncated to 4000 characters. The system content differs by which section headers are present in the raw report.

system content

You are a medical text analysis expert specialising in uncertainty detection. Analyse the provided radiology report section and identify all uncertainty expressions—words or phrases that indicate diagnostic uncertainty, probability, or hedging (e.g. likely, possibly, may, consistent with, cannot exclude, suspicious for).

user content

Analyse the following radiology report section and identify uncertainty expressions. Text: ‘‘{text}’’ Respond with a JSON object containing exactly these keys: "uncertainty_phrases" : list of uncertainty expressions found (strings); "confidence_level" : "high" , "medium" , or "low" ; "uncertainty_type" : "structural" , "semantic" , or "both" ; "reasoning" : one-sentence explanation. If no uncertainty expressions are found, return {"uncertainty_phrases": [], "confidence_level": "low", "uncertainty_type": "structural", "reasoning": "No uncertainty found."}

Appendix

Table 8: Stage 1B prompt — section-level uncertainty extraction. The findings text is truncated to 2000 characters before substitution into {text} .

system content

You are a radiology NLP expert. Decide whether a radiology report sentence expresses diagnostic uncertainty about a finding. Uncertainty markers include: likely, may, might, could, possibly, probably, suspicious, suggestive of, cannot exclude, consistent with, versus, questionable, equivocal, appears, consider, reflect, represent , etc. Reply only with a JSON object: {"is_uncertain": true} or {"is_uncertain": false} . No explanation.

user content

Sentence: ‘‘{phrase}’’

Appendix

Table 9: Stage 1C prompt — per-phrase uncertainty classifier (binary). {phrase} is an ImaGenome scene-graph phrase (a noun-phrase or short clause) anchored to a single anatomical bounding box.

(a) Initial call: hedging rewrite + visual-revisit summary (temperature $=0.0$ )
system content	You rewrite a single radiology-reasoning sentence and emit a short visual-revisit summary. You obey the schema strictly and never invent findings beyond those in the provided report phrase.
user content	ORIGINAL_SENTENCE: {sentence} IMAGENOME_PHRASE: {phrase} ANATOMY: {anatomy} OBJ_TOKEN: {obj_token} REVISIT_STANCE: {stance} Rules. 1. hedged_sentence : rewrite ORIGINAL_SENTENCE so it (a) contains at least one hedging cue and (b) contains OBJ_TOKEN exactly once , placed immediately after the ANATOMY mention. Preserve clinical meaning. The hedging cue must be an exact substring from the closed list (case-insensitive, no inflection—e.g. use “suggests”, not “suggesting” / “suggest” / “suggested”; use “appears”, not “appearing”): [{hedges}] . 2. revisit_summary : 1–2 sentences, references OBJ_TOKEN exactly once, with stance={stance} ( confirm / qualify / alternative ). Do not introduce findings absent from IMAGENOME_PHRASE . The revisit_summary also must contain at least one exact hedging cue from the list above —do not inflect. Examples of the hedging cue used inline: OK: “The cardiac silhouette {obj_token} appears enlarged …” OK: “The findings in the left lung {obj_token} are likely …” BAD: “… suggesting pulmonary edema …” (inflection) BAD: “…is suggestive …” (must be “suggestive of”) Return strict JSON: {"hedged_sentence": "…", "revisit_summary": "…"} .
(b) Self-correction turn — fires when (a) output fails the closed-vocabulary hedge-cue verifier (temperature $=0.4$ , conversation continues from (a))
user content (continuation)	Your previous output did not use a hedge cue from the closed list. Re-emit the JSON. Both fields must use one of: may, likely, could, appears, suggests, suggestive of, concerning for, consistent with, possible, probable, potential, perhaps .

Appendix

Table 10: Stage 2 prompt for hedging rewrite with a self-correction fallback. Part (a) substitutes per-anchor variables for the original sentence, the matched ImaGenome scene-graph phrase, the anatomical region name, a unique narrative tag

\langle\text{obj}_{n}\rangle

, a stance label chosen from {confirm, qualify, alternative}, and a closed vocabulary of hedge terms ( may , likely , could , appears , suggests , suggestive of , concerning for , consistent with , possible , probable , potential , perhaps ). Part (b) reuses the same system content as (a). If both passes fail the verifier, a deterministic rule-based fallback inserts “may” or “which may” into the sentence; anchors whose rule-based output still fails are excluded from the SFT corpus.

Stage	Purpose	Input	Output schema
1A	Findings/Impression section parser	full report	{has_findings\|has_impression\|has_both: bool}
1B	Sentence-level uncertainty extraction	report section	{ phrases, conf, type, reason }
1C	Per-phrase uncertainty classifier	ImaGenome phrase	{is_uncertain: bool}
2A	Hedging rewrite + revisit summary	anchor record (5 fields)	{hedged_sentence, revisit_summary}
2B	Self-correction (fallback for 2A)	2A output + correction	same as 2A

Appendix

Table 11: Summary of LLM calls used during data curation. All calls target a locally hosted Qwen2.5-7B-Instruct endpoint with JSON-mode output. Stage 1B is preceded by a regex pre-filter that already removes

\sim

60% of obviously non-uncertain sentences, so the LLM is invoked only on borderline cases.

system content	You are a radiology AI assistant. Analyze chest X-ray images and identify anatomical regions where the radiologist expressed diagnostic uncertainty.
user content	Analyze this chest X-ray and identify all anatomical regions with diagnostic uncertainty. Return a JSON list where each item contains: "bbox_2d" : [x1, y1, x2, y2] (normalised coordinates 0–223); "label" : anatomical region name; "uncertainty" : the uncertain observation from the report. If no uncertain regions exist, return an empty list [] .
assistant target (example)	[ { "bbox_2d": [69, 70, 107, 108], "label": "right hilar structures", "uncertainty": "The hila are less prominent likely due to …" } ]

Appendix

Table 12: Stage 1 prompt — uncertainty bounding-box detection (SFT). The assistant target is a JSON array (possibly empty) of bounding-box items. Coordinates are normalised to the

[0,223]

image grid used by the Stage 1 visual head.

system content

(none)

user content

<image> Analyze this chest X-ray and generate a structured radiology report with explicit reasoning. Use <findings> , <thinking> , and <impression> sections.

assistant target (example)

<findings> PA chest X-ray shows … </findings> <thinking> …natural-language reasoning … The hazy appearance of the right lower lung field and right hilar lymphadenopathy <obj1> suggest pneumonia. detect( query="right hilar lymphadenopathy", objects=["<obj1><region>[0.46, 0.23, 0.57, 0.44]</region>"]) …continued reasoning re. <obj1> … </thinking> <impression> Bilateral hilar lymphadenopathy and right lower lobe pneumonia … </impression>

Appendix

Table 13: Stage 2 prompt — reasoning-trace SFT. The assistant target is a three-section report ( <findings> , <thinking> , <impression> ); inside <thinking> , uncertain regions are anchored via <objN> narrative tokens and inspected via detect() calls whose <region> coordinates are normalised to

[0,1]

. Type-A samples (1,292/1,909) carry at least one revisit anchor; Type-B samples (617/1,909) have no detect() calls.

system content	(none)
user content (identical to Stage 2)	<image> Analyze this chest X-ray and generate a structured radiology report with explicit reasoning. Use <findings> , <thinking> , and <impression> sections.
supervision	GRPO with the four group-relative rewards listed in this table’s caption. No token-level target.

Appendix

Table 14: Stage 3 prompt for GRPO reinforcement learning. The user prompt is identical to Stage 2, while the token-level SFT loss is replaced by four group-relative reward functions. The format reward verifies the presence and ordering of <findings> , <thinking> , and <impression> . The accuracy reward measures CheXbert-style multi-label agreement with ground-truth findings on the 14 CheXpert labels. The coherence reward captures consistency between <thinking> and the <findings> / <impression> sections. The grounding reward evaluates the correctness of <objN> , <region> , and detect() triples and their spatial alignment with Stage 1 uncertainty bounding boxes.

	Stage 1	Stage 2	Stage 3
System prompt	✓	—	—
User prompt template	detection (JSON)	structured report	same as S2
Output schema	JSON array	<findings>/<thinking>/<impression>	same as S2
detect() in target	—	only Type-A samples	emergent under reward
Supervision	SFT (cross-entropy)	SFT (cross-entropy)	GRPO (4 rewards)

Appendix

Table 15: Cross-stage prompt and supervision differences. Stages 2 and 3 share an identical prompt, so all behavioral drift between them, such as the detect() trigger rate rising from

25.5\%

74.5\%

, is attributable to reward shaping rather than prompt engineering.

Figure 6: Frequency distribution of uncertainty markers in MIMIC-CXR ground-truth reports. All $276{,}778$ reports (train+val+test) were scanned for occurrences of the $35$ -term closed vocabulary used during Stage 2 trace synthesis (longest-first masked matching, case-insensitive, whole-word boundary). Top three markers ( likely , may , appears ) cover $48.8\%$ of all $155{,}542$ mentions; the top ten cover $84\%$ . Multi-word constructions ( consistent with , concerning for , compatible with ) appear prominently among the top seven.

Figure 7: Per-thousand-word frequency of the top fourteen uncertainty markers across four corpora. Bars within each group correspond, from left to right, to ground-truth MIMIC-CXR reports, the Stage 2 SFT model, the full-reward GRPO model, and the no-revisit GRPO ablation. All four corpora are matched on the same 200-report test split, with rates normalised by total word count.

system content	You are a senior radiologist evaluating an AI-generated chest X-ray reasoning trace. You are given the ground-truth (radiologist-written) report and an AI-generated output that contains <findings> , <thinking> , and <impression> sections. You will score the AI output on five dimensions on a $1$ – $5$ 5-point rating scale. Output strict JSON only. Length and writing style are not criteria. You do not know which AI model produced the output.
User content (scoring rubric, $1$ – $5$ integer per dimension; higher is better)
1. clinical plausibility	How medically sensible is the reasoning given a chest X-ray? 1 = medically incorrect or nonsensical. 3 = mostly plausible with notable inaccuracies. 5 = fully clinically valid; a radiologist would accept the inferences.
2. faithfulness	Compare claims in the AI output (especially in <findings> and <thinking> ) against the ground-truth report. 1 = severely hallucinated; the trace asserts findings clearly absent from the GT. 3 = partially grounded; some unsupported claims. 5 = fully grounded; no findings or locations introduced beyond what the GT supports.
3. coherence	Internal logical consistency of <thinking> . 1 = self-contradictory, jumps between unrelated claims. 3 = mostly coherent, with some logical gaps. 5 = clean logical progression from observation to inference, no contradictions.
4. findings $\rightarrow$ impression bridge	How well does <thinking> motivate <impression> ? 1 = impression conclusions are not supported by <thinking> . 3 = some impression statements are supported, others are not. 5 = every claim in <impression> is traceable to reasoning in <thinking> .
5. uncertainty calibration	Are hedge words ( likely, may, appears, suggestive of, possible, consistent with , …) and confident assertions placed appropriately? 1 = hedges used randomly or missing where evidence is weak. 3 = some hedges are warranted, others misplaced. 5 = hedges only where evidence is genuinely uncertain; commitment where evidence is strong.

Appendix

Table 16: LLM-as-judge prompt used to score the <thinking> portion of generated reasoning traces. The judge is GPT-4o ( gpt-4o-2024-11-20 ) called with

\text{temperature}=0

and JSON-mode output; identical seed is used across all reports to ensure reproducibility. For every test report and every model under evaluation, the prompt is instantiated by substituting the radiologist-written ground-truth report into {gt_report} and the model’s full <findings>/<thinking>/<impression> generation into {generated} . All five dimensions are scored on a

1

–

5

Likert scale; each score is accompanied by a one-sentence justification. Aggregate results are reported in Table 17 .

Panel of Figure 8 — Figure 8: Two qualitative case studies from the MIMIC-CXR.

Dimension	SFT	GRPO	GRPO no-revisit
Clinical plausibility	$2.26\pm 0.60$	$2.02\pm 0.59$	$1.92\pm 0.67$
Faithfulness	$2.38\pm 0.53$	$2.26\pm 0.44$	$2.20\pm 0.45$
Coherence	$3.06\pm 0.74$	$3.00\pm 0.53$	$3.20\pm 0.40$
Findings $\rightarrow$ Impression bridge	$2.16\pm 0.74$	$2.06\pm 0.55$	$2.18\pm 0.52$
Uncertainty calibration	$2.38\pm 0.67$	$2.56\pm 0.64$	$2.50\pm 0.65$
Average	$2.45$	$2.38$	$2.40$

Appendix

Table 17: Mean

\pm

std of GPT-4o-as-judge scores (

1

5

) on the <thinking> portion (

n=50

per model). “Bridge” denotes how well <thinking> supports <impression> . Best in bold; SE of the mean

\leq 0.11

. Full prompt in Table 16 .

Datasets
• MIMIC-CXR (version 2.1.0): – Source: Johnson et al. 12 . – URL: https://physionet.org/content/mimic-cxr/2.1.0/ – License: PhysioNet Credentialed Health Data License 1.5.0. Access requires credentialing and signing a data use agreement. • Chest ImaGenome Dataset (version 1.0.0): – Source: Wu et al. 36 . – URL: https://physionet.org/content/chest-imagenome/1.0.0/ – License: Derived from MIMIC-CXR, subject to the PhysioNet Credentialed Health Data License 1.5.0.
Pre-trained Models and Baselines
• Qwen-2.5-VL-3B (Backbone for UR 2 -MLLM): – Source: Bai et al. 29 . – URL: https://github.com/QwenLM/Qwen3 – License: Apache 2.0 License. • LLaVA-1.5-7B: – Source: Liu et al. 17 . – URL: https://github.com/haotian-liu/LLaVA – License: Apache 2.0 License. • MAIRA-2: – Source: Bannur et al. 2 . – URL: https://huggingface.co/microsoft/maira-2 – License: Microsoft Research License Terms. • HuatuoGPT-Vision-7B: – Source: Chen et al. 3 . – URL: https://huggingface.co/FreedomIntelligence/HuatuoGPT-Vision-7B – License: Apache 2.0 License. • RadGraph: – Source: Jain et al. 9 . – URL: https://huggingface.co/StanfordAIMI/RRG_scorers – License: MIT License. • CheXbert: – Source: Smit et al. 26 . – URL: https://huggingface.co/StanfordAIMI/RRG_scorers – License: MIT License.

Appendix

Table 18: Licenses and access conditions of the publicly available datasets and pre-trained models used in this work.

Explore similar work

Mar 11, 2024

Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancement

Radiology report generation (RRG) has attracted significant attention due to its potential to reduce the workload of radiologists. The performance of current RRG approaches remains unsatisfactory against clinical standards. This paper introduces a novel RRG method, MLLM-RRG, that integrates multimodal large language models (MLLMs) with various types of clinical knowledge to generate accurate and comprehensive chest X-ray reports. Our method first designs a referring anatomical feature extractor that leverages anatomical knowledge to analyze different regions of the chest X-ray image and extract visual features without explicitly detecting regions. Next, based on the MLLM's decoder, we develop a multimodal report generator that leverages multimodal prompts constructed from dedicated visual features and textual instructions to produce the radiology report in an auto-regressive way. Finally, we introduce a disease-oriented clinical classification and alignment scheme in a multi-task learning manner to leverage disease knowledge to better preserve the clinical relevance among the generated reports. Once the model is trained, we also introduce a novel clinical quality reinforcement learning strategy to enhance the MLLM with report knowledge, further refining the tones of the generated reports towards radiologists. Extensive experiments on the MIMIC-CXR and IU X-Ray datasets demonstrate the superiority of our method over the state of the art. Our codes will be available at https://github.com/viscom-tongji/MLLM-RRG.

Miaojing Shi, Tianyu Cen, Zijie Yue +3

College of Electronic and Information Engineering, Tongji University, Shanghai, China

Radiology Report Generation Radiology VLMs Multimodal Large Language Models

Ask PDF

Jan 6, 2026

HERO: Hierarchical Evidential Reasoning Optimization for Radiology Report Generation via Reason-then-Summarize

Multimodal Large Language Models (MLLMs) have substantially advanced Radiology Report Generation (RRG), yet aligning them through reinforcement learning (RL) remains challenging due to heterogeneous medical supervision. Vanilla Group Relative Policy Optimization (GRPO) assigns uniform credit across the entire generation, leading to segment interference, token dilution, and rationale-diagnosis decoupling, which exacerbates clinical hallucinations. We propose HERO (Hierarchical rationalE Reasoning Optimization), a factorized policy optimization framework that aligns heterogeneous supervision with three optimization granularities. HERO separately optimizes reasoning, diagnosis, and rationale faithfulness through complementary segment-, token-, and completion-level optimization with a heterogeneous reward formulation covering diagnostic accuracy, reasoning quality, and think-answer consistency. Experiments on MIMIC-CXR and IU-Xray show that HERO outperforms strong supervised and reinforcement learning baselines, achieving state-of-the-art clinical efficacy while producing more rationale-supported and think-answer-consistent reports, thereby substantially mitigating clinical hallucinations.

Kun Zhao, Guodong Liu, Hui Ji +7

University of Pittsburgh · Eli and Lilly Company · The University of Manchester +1

Radiology Report Generation Medical VLMs RL for VLMs VLM Hallucination Mitigation

Ask PDF

Sep 20, 2026

MAC-RRG: Iterative Multi-Agent Collaboration for X-ray Radiology Report Generation

Despite the remarkable progress of LLM-based and knowledge graph-augmented Radiology Report Generation (RRG) methods, existing techniques still suffer from inherent defects. Conventional LLM-only models lack structured medical prior knowledge, resulting in frequent medical hallucinations and low diagnostic interpretability. Current knowledge graph-enhanced schemes adopt static one-round knowledge fusion with single-source knowledge, incapable of dynamic knowledge updating according to generation feedback. This paper proposes a novel Multi-Agent Collaborative iterative framework for X-ray Radiology Report Generation, termed MAC-RRG. Inspired by multi-agent technology, our framework constructs a closed-loop optimization paradigm based on task decoupling and collaborative reasoning. Specifically, the framework first generates a preliminary radiology report from input X-ray images via a vision encoder and a basic LLM. Subsequently, a multimodal knowledge graph (MM-KG) agent mines structured disease correlation and anatomical knowledge from medical knowledge graphs, while an auxiliary knowledge agent extracts unstructured domain knowledge from public medical databases. The multi-source knowledge acquired by dual agents is fused and embedded to guide the LLM in iteratively refining the initial report. Extensive quantitative and qualitative experiments on mainstream X-ray RRG datasets, including IU X-ray, MIMIC, and CheXpert Plus, fully verify the superiority of our proposed method. The source code and pre-trained models have been released on https://github.com/Event-AHU/Medical_Image_Analysis

Futian Wang, Yuhan Qiao, Xiao Wang +5

School of Computer Science and Technology, Anhui University, Hefei 230601, China · The First Affiliated Hospital of Anhui Medical University, Hefei 230000, China · Harbin Institute of Technology, Shenzhen, China +1

Radiology Report Generation Multi-Agent Collaboration

Ask PDF