MedPAO-VL: Interpretable and Protocol-Compliant Agent for Structured Radiology Report Generation

Shrish Shrinath Vaidya; Gowthamaan Palani; Minmini Selvam; Gokulraja Srinivasaraja; Ganapathy Krishnamurthi

cs.CVOct 15, 2026

MedPAO-VL: Interpretable and Protocol-Compliant Agent for Structured Radiology Report Generation

Authors: Shrish Shrinath Vaidya, Gowthamaan Palani, Minmini Selvam, Gokulraja Srinivasaraja, Ganapathy Krishnamurthi

Organizations: Department of Data Science and Artificial Intelligence, Indian Institute of Technology Madras, Chennai, India · TANUH – The AI Centre of Excellence in Healthcare, Indian Institute of Science, Bengaluru, India · Department of Radiology (Division of Interventional Radiology), Ramaiah Medical College and Memorial hospital, Bengaluru, India

Abstract

Despite the remarkable generalization capabilities of Large Language Models, generating clinically accurate and structured radiology reports from medical images remains a significant challenge. Current end-to-end approaches function as black boxes, prone to factual hallucinations and generating ungrounded findings. This lack of reliability and failure to adhere to domain-specific structuring protocols, fundamentally limits their clinical utility. We present MedPAO-VL, a multimodal agent that shifts the paradigm from monolithic generation to explainable, iterative, protocol-compliant structured report generation. Unlike deterministic models, our plan-act-observe loop functions as a goal-oriented agent: it first perceives the Chest X-Ray through an explainable vision module designed to capture inherent diagnostic uncertainty, then iteratively refines the generated report by observing semantic discrepancies and self-correcting its generation for better alignment with visually detected clinical findings. Evaluation of our framework demonstrates that this agentic reasoning improves clinical correctness and completeness, achieving an expert-rated accuracy score of 3.5/5 and a structural score of 4.3/5. By prioritizing pathology specific explainability and iterative refinement over opaque end-to-end mapping, MedPAO-VL offers a transparent and clinically viable alternative for automated structured radiology report generation.

Figures & tables

Figure 1: MedPAO-VL integrates explainable, uncertainty-aware visual perception and iterative self-correction, overcoming limitations of traditional VLMs and post-hoc report structuring methods.

Figure 2: Overview of proposed MedPAO-VL architecture. It mainly comprises of (a)Vision Concept Module for extracting the clinical findings, (b)Pathology slot module for extracting explainable pathology embeddings and (c)An iterative PAO loop to handle report correction and Structuring.

Dataset	Methods	Year	Section	NLG Metrics		Clinical Metrics
Dataset	Methods	Year	Section	BLEU-4	ROUGE-L	CheXbert-F1	RadGraph-F1
MIMIC-CXR	R2Gen [ 4 ]	2020	F+I	0.103	0.277	0.276	0.134
	CXR-RePaiR [ 10 ]	2021	F	0.021	0.143	0.274	0.091
	$\mathcal{M}^{2}$ Transformer [ 39 ]	2021	F	0.114	-	-	0.220
	RGRG [ 52 ]	2023	F	0.126	0.264	0.447	-
	METransformer [ 59 ]	2023	F	0.124	0.291	0.311	-

Table 1: Performance Comparison on MIMIC-CXR and IU X-RAY Datasets. This table presents the performance of various models against our Concept Grounded report Generator on generating unstructured radiology reports(without PAO loop) for the MIMIC-CXR and IU X-RAY test sets. The Sections column indicates whether the model generates the FINDINGS (F) or IMPRESSION (I) section of the report, with metrics sourced from published research.

Dataset	NLG						CheXbert		GREEN		RadGraph
Dataset	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE-L	Micro-F1	Macro-F1	Mean	STD	$\text{RG}_{\text{E}}$	$\text{RG}_{\text{ER}}$	$\overline{\text{RG}}_{\text{ER}}$
MIMIC-CXR	0.431	0.272	0.188	0.136	0.407	0.328	0.706	0.617	0.472	0.214	0.403	0.374	0.303
IU X-RAY	0.416	0.239	0.150	0.097	0.469	0.341	0.619	0.424	0.733	0.225	0.495	0.458	0.375

Table 2: Comprehensive Evaluation Metrics of Concept Grounded report Generator (50K variant, without PAO loop) on MIMIC-CXR and IU X-RAY test-sets

	MedGemma 27B		MedPAO-VL
	Accuracy score	Structure score	Accuracy score	Structure score
Radiologist1	$2.88\pm 0.44$	$3.84\pm 0.36$	$3.40\pm 0.32$	$4.44\pm 0.24$
Radiologist2	$3.48\pm 0.56$	$4.12\pm 0.28$	$3.60\pm 0.40$	$4.22\pm 0.32$
Combined Mean	$3.18\pm 0.36$	$3.98\pm 0.24$	$\mathbf{3.50\pm 0.28}$	$\mathbf{4.28\pm 0.20}$

Table 3: Radiologist evaluation of structured reports with

\pm 95\%

Panel of Figure 3 — Figure 3: Latent pathology slot vector similarity. 2D cosine similarity projections for (a) “mild CHF” and (b) “likely pneumonia” and related pathologies.

Figure 4: t-SNE visualization of the slot activation space against ABCDEF clinical protocol.

Figure 5: Input CXR and Explainable Attention maps from PSM block

Dataset	Methods	Trainable Params.	AUC	F1-score	Precision	Recall
MIMIC-CXR	Baseline model	2.7M	0.64	0.485	0.42	0.574
	Baseline + PSM	47.2M	0.679	0.484	0.41	0.59
	Baseline + PSM + Shared Classifier Head	6.1M	0.77	0.467	0.44	0.497
	Proposed(Baseline + PSM + Shared Classifier + Similarity loss)	6.1M	0.784	0.48	0.478	0.567

Table 4: Evaluation results of Vision Concept module from its baseline model to the proposed architecture

Dataset	Methods	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	METEOR
MIMIC-CXR	MedGemma-4B MM zero shot	0.139	0.068	0.035	0.019	0.131	0.253
	MedGemma-27B MM zero shot	0.235	0.109	0.051	0.026	0.165	0.234
	MedGemma-4B MM Finetuned w/o Concepts (10K)	0.275	0.163	0.104	0.071	0.227	0.265
	Proposed (MedGemma-4B Finetuned with Concepts, 10K)	0.377	0.224	0.147	0.102	0.298	0.358

Table 5: Evaluation results on the impact of model scale, fine-tuning, and the proposed clinical concept integration in Concept Grounded Report Generator

System Configuration	CheXbert-F1 $\uparrow$	RadGraph-F1 $\uparrow$	GREEN $\uparrow$	$\phi^{+}$ Ratio $\uparrow$	Mean Iter.	LLM Calls $\downarrow$	Time (s) $\downarrow$
Deterministic Baseline	0.617	0.427	$0.47\pm 0.21$	0.602	1.0	6.0	218
PAO Full (Cold Cache)	0.634	0.448	$0.61\pm 0.20$	0.803	2.8	11.2	295
PAO Full (Warm Cache)	0.634	0.448	$0.61\pm 0.20$	0.803	2.8	9.8	163

Table 6: Evaluation of the Plan-Act-Observe (PAO) loop across various configurations.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Letter	Category	Key Findings to Assess
A	Airways	Tracheal position and deviation; carinal angle ( $<105^{\circ}$ ); main bronchi; intraluminal masses or narrowing
B	Breathing	Lung parenchyma (consolidation, nodules, interstitial changes); pulmonary vascularity; pleural effusion; pneumothorax
C	Circulation	Cardiothoracic ratio ( $<0.5$ on PA); cardiac borders; mediastinal width and contour; aortic knuckle; hilar structures
D	Diaphragm	Hemidiaphragm height; costophrenic angles; pneumoperitoneum; subdiaphragmatic pathology (hepatosplenomegaly, calculi)
E	External	Ribs, clavicles, scapulae, vertebrae for fractures or lesions; soft tissues (subcutaneous emphysema, masses)
F	Foreign Material	Endotracheal tubes; central venous catheters; nasogastric tubes; pacemaker/ICD leads; prostheses; surgical clips

Appendix

Table 7: Summary of the ABCDEF systematic chest radiograph review protocol.

Figure 6: Illustration of MedPAO Agent performance against other models on a hypothetical report. (In the above, the protocol placeholder represents the injected protocol-related instructions.)

Concept	Category
Well inflated lungs	B
No focal airspace consolidation	B
No pleural effusions	B
No pulmonary edema	B
No pneumothorax	B
Normal cardiac contours	C

Appendix

Table 8: Examples of frequently occurring concepts stored in the Concept Cache.

Score	Definition
5	Perfect Match: The Generated Report is a perfect and complete reflection of the Groundtruth Report. All key details are present, and there is no missing, extraneous, or fabricated (hallucinated) information.
4	Generally Accurate with Minor Omissions: The Generated Report is highly accurate and contains all key details. It may be missing a few minor, non-critical details that do not impact the overall meaning or interpretation. No hallucinations.
3	Key Details Present but Incomplete or with Minor Hallucinations: The Generated Report captures the main points but has significant omissions of secondary details. Alternatively, it might contain minor fabricated details that do not fundamentally contradict the groundtruth but add noise. Requires some interpretation to align with the groundtruth.
2	Missing Key Details or Contradictory Information: The Generated Report fails to mention one or more key details from the groundtruth. Alternatively, it contains information that directly contradicts the groundtruth. This report could be misleading but is not immediately dangerous.
1	Critically Flawed or Dangerous: The Generated Report has major omissions of critical information or contains significant, dangerous hallucinations that could lead to incorrect and harmful decisions.

Appendix

Table 9: Scoring criteria for report quality

Score	Definition
5	Perfect Structure: The report is perfectly structured. All specified Key Concepts are present as distinct sections/headers, in the correct order, and contain the relevant information as seen in the groundtruth.
4	Good Structure: All Key Concepts are present and correctly populated, but there might be minor deviations.
3	Acceptable Structure: Most Key Concepts are present, but one or two may be missing or merged. The information is generally in the right place, with some minor deviations.
2	Poor Structure: Multiple Key Concepts are missing. Information is disorganized, placed under incorrect headings, making the report difficult to follow and non-compliant with the protocol.
1	No Structure: The report is a free-form block of text. It completely ignores the Key Concepts and shows no attempt to follow the required protocol-driven structure.

Appendix

Table 10: Scoring criteria for report structure

Metric	MedGemma-27B		MedPAO-VL
	Accuracy	Structure	Accuracy	Structure
Combined Mean	$3.18\pm 0.36$	$3.98\pm 0.24$	$\mathbf{3.50\pm 0.28}$	$\mathbf{4.28\pm 0.20}$
Cohen’s $\kappa$	0.510	0.394	0.222	0.369

Appendix

Table 11: Extended inter-rater statistics for structured report evaluation

Figure 7: Radiologist score distribution

Figure 8: The entire Data preprocessing pipeline for extracting standardized concepts from clinical reports

Categories	Number
Unique raw concepts	60,531
Present raw concepts	57,410
Absent raw concepts	3,123
Unique SNOMEDCT groups before filtering	9,450
Unique SNOMEDCT groups after filtering	236

Appendix

Table 12: Number of concepts in each categories

Panel of Figure 9 — Figure 9: Comparison of concept distributions across all reports: (a) raw extracted concepts, and (b) standardized concepts after post-processing.

Figure 10: Top 10 raw Concept and their Frequency

	Normal Reports	Complex Reports
Number of reports	47,176	6,721
Mean (word count)	45.6	74.0
Median (word count)	70	70.0
Std Deviation (word count)	21.5	19.0
1st Quartile (word count)	32.0	59.0
3rd Quartile (word count)	56.0	85.0

Appendix

Table 13: Statistic of Normal and Complex reports

Figure 11: Distribution of no. of concepts across normal and complex reports in the current dataset.

Figure 12: Top SNOMEDCT groups and their Present and Absent concept counts

Figure 13: Top SnomedCT concept groups and their Frequency of occurance in the given dataset.

Panel of Figure 14 — Figure 14: Heatmap showing Cosine similarity amongst the observed pathologies. (a)Cosine similarity amongst all the pathologies. (b)A cross-section of B(Breathing)

Figure 15: Per-slot t-SNE visualizations for representative clinical findings.

Panel of Figure 16 — Figure 16: Attention map comparison between models with and without $L_{sim}$

Figure 17: Input CXR and the explainable heatmaps from Vision Concept Module

Panel of Figure 18 — Figure 18: The system prompt used for the Planning Agent.

Panel of Figure 19 — Figure 19: Observation prompt used by the PAO agent to analyze tool outputs and decide the next step, including retry and replanning mechanisms.

Panel of Figure 20 — Figure 20: The few-shot prompt used for extracting clinical concepts and mapping them to their source sentences. The model is conditioned with 5 diverse examples ranging from normal findings to complex pathologies.

Panel of Figure 21 — Figure 21: The prompt used to classify extracted concepts into the standard ABCDEF radiological schema. The <ABCDEF Protocol Description> placeholder contains same protocol information as shown in Fig 22

Panel of Figure 22 — Figure 22: The prompt used to generate the final structured radiology report. The model is conditioned on the ABCDEF protocol to ensure clinical completeness and is explicitly constrained to avoid hallucinating information not present in the input concepts.

Panel of Figure 23 — Figure 23: The multimodal prompt for concept grounded radiology report generation. The model receives a chest X-ray image and a textual prompt containing observed clinical concepts (pre-processed to list present and absent findings), and is tasked with generating a comprehensive report.

Panel of Figure 24 — Figure 24: The prompt used in the re-prompting step of the self-correction loop. Given the previously generated report and the set of concepts flagged as missing by the overlap checker, the model is constrained to generate only the missing findings, without repeating or regenerating the full report.

Radiology Report Generation

Radiology report generation uses medical images and clinical context to produce structured, clinically accurate findings and impressions.

See all 159 papers

EviAgent: Evidence-Driven Agent for Radiology Report Generation
Mar 14, 2026Tuoshi Qi, Shenshen Bu, Yingfei Xiang +1Radiology Report Generation Medical VLMs
School of Computer Science and Engineering, Sun Yat-sen University · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Sangfor Technologies, Shenzhen, Guangdong, China
MedPAO-Assist: Edge-Deployable, Modality-Aware Agentic AI for Radiology Report Structuring
Oct 17, 2026Shrish Shrinath Vaidya, Minmini Selvam, Ganapathy KrishnamurthiRadiology Report Generation Medical Report Generation
Department of Data Science and Artificial Intelligence, Indian Institute of Technology Madras, Chennai, India · TANUH – The AI Centre of Excellence in Healthcare, Indian Institute of Science, Bengaluru, India
MAC-RRG: Iterative Multi-Agent Collaboration for X-ray Radiology Report Generation
Sep 20, 2026Futian Wang, Yuhan Qiao, Xiao Wang +5Radiology Report Generation Multi-Agent Collaboration
School of Computer Science and Technology, Anhui University, Hefei 230601, China · The First Affiliated Hospital of Anhui Medical University, Hefei 230000, China · Harbin Institute of Technology, Shenzhen, China +1

Mar 14, 2026J/K move · Enter open · S save

EviAgent: Evidence-Driven Agent for Radiology Report Generation

Tuoshi Qi, Shenshen Bu, Yingfei Xiang, Zhiming Dai

School of Computer Science and Engineering, Sun Yat-sen University · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Sangfor Technologies, Shenzhen, Guangdong, China

Radiology Report Generation Medical VLMs Explainable Medical Image Analysis Agentic RAG

Open paper AskPDF arXiv

Automated radiology report generation holds immense potential to alleviate the heavy workload of radiologists. Despite the formidable vision-language capabilities of recent Multimodal Large Language Models (MLLMs), their clinical deployment is severely constrained by inherent limitations: their "black-box" decision-making renders the generated reports untraceable due to the lack of explicit visual evidence to support the diagnosis, and they struggle to access external domain knowledge. To address these challenges, we propose the Evidence-driven Radiology Report Generation Agent (EviAgent). Unlike opaque end-to-end paradigms, EviAgent coordinates a transparent reasoning trajectory by breaking down the complex generation process into granular operational units. We integrate multi-dimensional visual experts and retrieval mechanisms as external support modules, endowing the system with explicit visual evidence and high-quality clinical priors. Extensive experiments on MIMIC-CXR, CheXpert Plus, and IU-Xray datasets demonstrate that EviAgent outperforms both large-scale generalist models and specialized medical models, providing a robust and trustworthy solution for automated radiology report generation.

Abstract

Radiology Report Generation

Explore similar work

EviAgent: Evidence-Driven Agent for Radiology Report Generation

MedPAO-Assist: Edge-Deployable, Modality-Aware Agentic AI for Radiology Report Structuring

MAC-RRG: Iterative Multi-Agent Collaboration for X-ray Radiology Report Generation