Organizations: Department of Data Science and Artificial Intelligence, Indian Institute of Technology Madras, Chennai, India · TANUH – The AI Centre of Excellence in Healthcare, Indian Institute of Science, Bengaluru, India · Department of Radiology (Division of Interventional Radiology), Ramaiah Medical College and Memorial hospital, Bengaluru, India
Despite the remarkable generalization capabilities of Large Language Models, generating clinically accurate and structured radiology reports from medical images remains a significant challenge. Current end-to-end approaches function as black boxes, prone to factual hallucinations and generating ungrounded findings. This lack of reliability and failure to adhere to domain-specific structuring protocols, fundamentally limits their clinical utility. We present MedPAO-VL, a multimodal agent that shifts the paradigm from monolithic generation to explainable, iterative, protocol-compliant structured report generation. Unlike deterministic models, our plan-act-observe loop functions as a goal-oriented agent: it first perceives the Chest X-Ray through an explainable vision module designed to capture inherent diagnostic uncertainty, then iteratively refines the generated report by observing semantic discrepancies and self-correcting its generation for better alignment with visually detected clinical findings. Evaluation of our framework demonstrates that this agentic reasoning improves clinical correctness and completeness, achieving an expert-rated accuracy score of 3.5/5 and a structural score of 4.3/5. By prioritizing pathology specific explainability and iterative refinement over opaque end-to-end mapping, MedPAO-VL offers a transparent and clinically viable alternative for automated structured radiology report generation.
Figures & tables
Figure 1: MedPAO-VL integrates explainable, uncertainty-aware visual perception and iterative self-correction, overcoming limitations of traditional VLMs and post-hoc report structuring methods.
Figure 2: Overview of proposed MedPAO-VL architecture. It mainly comprises of (a)Vision Concept Module for extracting the clinical findings, (b)Pathology slot module for extracting explainable pathology embeddings and (c)An iterative PAO loop to handle report correction and Structuring.
Dataset
Methods
Year
Section
NLG Metrics
Clinical Metrics
BLEU-4
ROUGE-L
CheXbert-F1
RadGraph-F1
MIMIC-CXR
R2Gen [ 4 ]
2020
F+I
0.103
0.277
0.276
0.134
CXR-RePaiR [ 10 ]
2021
F
0.021
0.143
0.274
0.091
M2 Transformer [ 39 ]
2021
F
0.114
-
-
0.220
RGRG [ 52 ]
2023
F
0.126
0.264
0.447
-
METransformer [ 59 ]
2023
F
0.124
0.291
0.311
-
Table 1: Performance Comparison on MIMIC-CXR and IU X-RAY Datasets. This table presents the performance of various models against our Concept Grounded report Generator on generating unstructured radiology reports(without PAO loop) for the MIMIC-CXR and IU X-RAY test sets. The Sections column indicates whether the model generates the FINDINGS (F) or IMPRESSION (I) section of the report, with metrics sourced from published research.
Dataset
NLG
CheXbert
GREEN
RadGraph
BLEU-1
BLEU-2
BLEU-3
BLEU-4
METEOR
ROUGE-L
Micro-F1
Macro-F1
Mean
STD
RGE
RGER
RGER
MIMIC-CXR
0.431
0.272
0.188
0.136
0.407
0.328
0.706
0.617
0.472
0.214
0.403
0.374
0.303
IU X-RAY
0.416
0.239
0.150
0.097
0.469
0.341
0.619
0.424
0.733
0.225
0.495
0.458
0.375
Table 2: Comprehensive Evaluation Metrics of Concept Grounded report Generator (50K variant, without PAO loop) on MIMIC-CXR and IU X-RAY test-sets
MedGemma 27B
MedPAO-VL
Accuracy score
Structure score
Accuracy score
Structure score
Radiologist1
2.88±0.44
3.84±0.36
3.40±0.32
4.44±0.24
Radiologist2
3.48±0.56
4.12±0.28
3.60±0.40
4.22±0.32
Combined Mean
3.18±0.36
3.98±0.24
3.50±0.28
4.28±0.20
Table 3: Radiologist evaluation of structured reports with ±95% CI
Figure 3: Latent pathology slot vector similarity. 2D cosine similarity projections for (a) “mild CHF” and (b) “likely pneumonia” and related pathologies.
Figure 4: t-SNE visualization of the slot activation space against ABCDEF clinical protocol.
Figure 5: Input CXR and Explainable Attention maps from PSM block
Table 4: Evaluation results of Vision Concept module from its baseline model to the proposed architecture
Dataset
Methods
BLEU-1
BLEU-2
BLEU-3
BLEU-4
ROUGE-L
METEOR
MIMIC-CXR
MedGemma-4B MM zero shot
0.139
0.068
0.035
0.019
0.131
0.253
MedGemma-27B MM zero shot
0.235
0.109
0.051
0.026
0.165
0.234
MedGemma-4B MM Finetuned w/o Concepts (10K)
0.275
0.163
0.104
0.071
0.227
0.265
Proposed (MedGemma-4B Finetuned with Concepts, 10K)
0.377
0.224
0.147
0.102
0.298
0.358
Table 5: Evaluation results on the impact of model scale, fine-tuning, and the proposed clinical concept integration in Concept Grounded Report Generator
System Configuration
CheXbert-F1 ↑
RadGraph-F1 ↑
GREEN ↑
ϕ+ Ratio ↑
Mean Iter.
LLM Calls ↓
Time (s) ↓
Deterministic Baseline
0.617
0.427
0.47±0.21
0.602
1.0
6.0
218
PAO Full (Cold Cache)
0.634
0.448
0.61±0.20
0.803
2.8
11.2
295
PAO Full (Warm Cache)
0.634
0.448
0.61±0.20
0.803
2.8
9.8
163
Table 6: Evaluation of the Plan-Act-Observe (PAO) loop across various configurations.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Letter
Category
Key Findings to Assess
A
Airways
Tracheal position and deviation; carinal angle ( <105∘ ); main bronchi; intraluminal masses or narrowing
Table 7: Summary of the ABCDEF systematic chest radiograph review protocol.
Figure 6: Illustration of MedPAO Agent performance against other models on a hypothetical report. (In the above, the protocol placeholder represents the injected protocol-related instructions.)
Concept
Category
Well inflated lungs
B
No focal airspace consolidation
B
No pleural effusions
B
No pulmonary edema
B
No pneumothorax
B
Normal cardiac contours
C
Appendix
Table 8: Examples of frequently occurring concepts stored in the Concept Cache.
Score
Definition
5
Perfect Match: The Generated Report is a perfect and complete reflection of the Groundtruth Report. All key details are present, and there is no missing, extraneous, or fabricated (hallucinated) information.
4
Generally Accurate with Minor Omissions: The Generated Report is highly accurate and contains all key details. It may be missing a few minor, non-critical details that do not impact the overall meaning or interpretation. No hallucinations.
3
Key Details Present but Incomplete or with Minor Hallucinations: The Generated Report captures the main points but has significant omissions of secondary details. Alternatively, it might contain minor fabricated details that do not fundamentally contradict the groundtruth but add noise. Requires some interpretation to align with the groundtruth.
2
Missing Key Details or Contradictory Information: The Generated Report fails to mention one or more key details from the groundtruth. Alternatively, it contains information that directly contradicts the groundtruth. This report could be misleading but is not immediately dangerous.
1
Critically Flawed or Dangerous: The Generated Report has major omissions of critical information or contains significant, dangerous hallucinations that could lead to incorrect and harmful decisions.
Appendix
Table 9: Scoring criteria for report quality
Score
Definition
5
Perfect Structure: The report is perfectly structured. All specified Key Concepts are present as distinct sections/headers, in the correct order, and contain the relevant information as seen in the groundtruth.
4
Good Structure: All Key Concepts are present and correctly populated, but there might be minor deviations.
3
Acceptable Structure: Most Key Concepts are present, but one or two may be missing or merged. The information is generally in the right place, with some minor deviations.
2
Poor Structure: Multiple Key Concepts are missing. Information is disorganized, placed under incorrect headings, making the report difficult to follow and non-compliant with the protocol.
1
No Structure: The report is a free-form block of text. It completely ignores the Key Concepts and shows no attempt to follow the required protocol-driven structure.
Appendix
Table 10: Scoring criteria for report structure
Metric
MedGemma-27B
MedPAO-VL
Accuracy
Structure
Accuracy
Structure
Combined Mean
3.18±0.36
3.98±0.24
3.50±0.28
4.28±0.20
Cohen’s κ
0.510
0.394
0.222
0.369
Appendix
Table 11: Extended inter-rater statistics for structured report evaluation
Figure 7: Radiologist score distribution
Figure 8: The entire Data preprocessing pipeline for extracting standardized concepts from clinical reports
Categories
Number
Unique raw concepts
60,531
Present raw concepts
57,410
Absent raw concepts
3,123
Unique SNOMEDCT groups before filtering
9,450
Unique SNOMEDCT groups after filtering
236
Appendix
Table 12: Number of concepts in each categories
Figure 9: Comparison of concept distributions across all reports: (a) raw extracted concepts, and (b) standardized concepts after post-processing.
Figure 10: Top 10 raw Concept and their Frequency
Normal Reports
Complex Reports
Number of reports
47,176
6,721
Mean (word count)
45.6
74.0
Median (word count)
70
70.0
Std Deviation (word count)
21.5
19.0
1st Quartile (word count)
32.0
59.0
3rd Quartile (word count)
56.0
85.0
Appendix
Table 13: Statistic of Normal and Complex reports
Figure 11: Distribution of no. of concepts across normal and complex reports in the current dataset.
Figure 12: Top SNOMEDCT groups and their Present and Absent concept counts
Figure 13: Top SnomedCT concept groups and their Frequency of occurance in the given dataset.
Figure 14: Heatmap showing Cosine similarity amongst the observed pathologies. (a)Cosine similarity amongst all the pathologies. (b)A cross-section of B(Breathing)
Figure 15: Per-slot t-SNE visualizations for representative clinical findings.
Figure 16: Attention map comparison between models with and without Lsim
Figure 17: Input CXR and the explainable heatmaps from Vision Concept Module
Figure 18: The system prompt used for the Planning Agent.
Figure 19: Observation prompt used by the PAO agent to analyze tool outputs and decide the next step, including retry and replanning mechanisms.
Figure 20: The few-shot prompt used for extracting clinical concepts and mapping them to their source sentences. The model is conditioned with 5 diverse examples ranging from normal findings to complex pathologies.
Figure 21: The prompt used to classify extracted concepts into the standard ABCDEF radiological schema. The <ABCDEF Protocol Description> placeholder contains same protocol information as shown in Fig 22
Figure 22: The prompt used to generate the final structured radiology report. The model is conditioned on the ABCDEF protocol to ensure clinical completeness and is explicitly constrained to avoid hallucinating information not present in the input concepts.
Figure 23: The multimodal prompt for concept grounded radiology report generation. The model receives a chest X-ray image and a textual prompt containing observed clinical concepts (pre-processed to list present and absent findings), and is tasked with generating a comprehensive report.
Figure 24: The prompt used in the re-prompting step of the self-correction loop. Given the previously generated report and the set of concepts flagged as missing by the overlap checker, the model is constrained to generate only the missing findings, without repeating or regenerating the full report.
Automated radiology report generation holds immense potential to alleviate the heavy workload of radiologists. Despite the formidable vision-language capabilities of recent Multimodal Large Language Models (MLLMs), their clinical deployment is severely constrained by inherent limitations: their "black-box" decision-making renders the generated reports untraceable due to the lack of explicit visual evidence to support the diagnosis, and they struggle to access external domain knowledge. To address these challenges, we propose the Evidence-driven Radiology Report Generation Agent (EviAgent). Unlike opaque end-to-end paradigms, EviAgent coordinates a transparent reasoning trajectory by breaking down the complex generation process into granular operational units. We integrate multi-dimensional visual experts and retrieval mechanisms as external support modules, endowing the system with explicit visual evidence and high-quality clinical priors. Extensive experiments on MIMIC-CXR, CheXpert Plus, and IU-Xray datasets demonstrate that EviAgent outperforms both large-scale generalist models and specialized medical models, providing a robust and trustworthy solution for automated radiology report generation.
Tuoshi Qi, Shenshen Bu, Yingfei Xiang +1
School of Computer Science and Engineering, Sun Yat-sen University · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Sangfor Technologies, Shenzhen, Guangdong, China
The adoption of Large Language Models for radiology report structuring has been largely dominated by either proprietary cloud-based models or large open models requiring high-end GPUs for deployment, which is a critical bottleneck for clinical adoption in radiology workflows that demand data privacy and offline operation. To address this, we present MedPAO-Assist, an edge-deployable agentic framework that generates structured radiology reports directly from radiologist dictation and voice commands, requiring only ≈3.3 GB peak RAM and achieving an average end-to-end latency of ≈31 seconds on a standard laptop CPU through an asynchronous, multi-threaded execution pipeline. The system currently supports four major imaging modalities -- Chest X-ray, Chest CT, Abdomen CT, and Brain MRI. To our knowledge, it is among the earliest agentic systems to run its entire pipeline with a Small Language Model (SLM) in the 0.6B-parameter range. This multi-agent architecture, built around an Extract-Ground-Generate paradigm, performs concept extraction, SNOMED-CT grounding, and structured report generation, followed by a speech-driven human-in-the-loop correction phase, together forming a fully "no-touch" reporting workflow that requires no keyboard input, external API calls, or internet connectivity. We evaluate each pipeline stage individually -- concept extraction, ontology grounding, and concept categorization and report hardware profiling results demonstrating the system's feasibility for real-time, CPU-only clinical deployment.
Department of Data Science and Artificial Intelligence, Indian Institute of Technology Madras, Chennai, India · TANUH – The AI Centre of Excellence in Healthcare, Indian Institute of Science, Bengaluru, India
Despite the remarkable progress of LLM-based and knowledge graph-augmented Radiology Report Generation (RRG) methods, existing techniques still suffer from inherent defects. Conventional LLM-only models lack structured medical prior knowledge, resulting in frequent medical hallucinations and low diagnostic interpretability. Current knowledge graph-enhanced schemes adopt static one-round knowledge fusion with single-source knowledge, incapable of dynamic knowledge updating according to generation feedback. This paper proposes a novel Multi-Agent Collaborative iterative framework for X-ray Radiology Report Generation, termed MAC-RRG. Inspired by multi-agent technology, our framework constructs a closed-loop optimization paradigm based on task decoupling and collaborative reasoning. Specifically, the framework first generates a preliminary radiology report from input X-ray images via a vision encoder and a basic LLM. Subsequently, a multimodal knowledge graph (MM-KG) agent mines structured disease correlation and anatomical knowledge from medical knowledge graphs, while an auxiliary knowledge agent extracts unstructured domain knowledge from public medical databases. The multi-source knowledge acquired by dual agents is fused and embedded to guide the LLM in iteratively refining the initial report. Extensive quantitative and qualitative experiments on mainstream X-ray RRG datasets, including IU X-ray, MIMIC, and CheXpert Plus, fully verify the superiority of our proposed method. The source code and pre-trained models have been released on https://github.com/Event-AHU/Medical_Image_Analysis
Futian Wang, Yuhan Qiao, Xiao Wang +5
School of Computer Science and Technology, Anhui University, Hefei 230601, China · The First Affiliated Hospital of Anhui Medical University, Hefei 230000, China · Harbin Institute of Technology, Shenzhen, China +1