Multi-Agent AI System for Radiology Report Structuring and Quality Assurance with Independent Radiologist Evaluation
Authors: Iryna Hartsock, Cesar Lam, Christopher Otteni, Aliya Qayyum, Robert Gatenby, Cyrillo Araujo, Ghulam Rasool
Organizations: Department of Machine Learning, H. Lee Moffitt Cancer Center & Research Institute, Tampa, FL, USA · Department of Diagnostic Imaging and Interventional Radiology, H. Lee Moffitt Cancer Center & Research Institute, Tampa, FL, USA · Department of Neuro-Oncology, H. Lee Moffitt Cancer Center & Research Institute, Tampa, FL, USA · Department of Translational Pathology, H. Lee Moffitt Cancer Center & Research Institute, Tampa, FL, USA
Purpose: To develop and evaluate a locally deployed multi-agent AI system for radiology report structuring and quality assurance. Materials and Methods: This retrospective study included 638 radiology reports from CT examinations of the chest, abdomen, and pelvis dictated by 15 board-certified radiologists in 2023 and 2024. A multi-agent AI pipeline was developed to perform report structuring and quality assurance (QA). The system structured the report into standardized anatomical sections at the sentence level using regex rules and local large language models. It also detected mismatches between the Findings and Impression sections, or within sections; gender-anatomy conflicts; and undocumented communication of critical findings. Two board-certified radiologists independently evaluated a 45-report subset. Results: The multi-agent system structured the Findings sections of all reports (22,270 sentences) into a predefined anatomical format while retaining the original report content. The system flagged 90 (14.1%) reports, most commonly for section mismatches (80 reports, 12.5%). In the radiologist evaluation, both reviewers agreed that 31 (69%) were correctly restructured, 2 reports (4%) were incorrectly restructured, and disagreed on the remaining 12 reports (27%). Both reviewers agreed that no clinically important information was omitted and no fabricated content was introduced. Overall QA performance was rated as "excellent" or "good" in 84% of the evaluated reports, with the remaining reports rated as "fair". Conclusion: A locally deployed multi-agent AI system combined radiology report structuring and quality assurance within a single workflow. The system demonstrated favorable performance in radiologist evaluation. Such systems may support standardization of reporting and quality assurance in radiology practice.
Figures & tables
Figure 1: Overview of the proposed multi-agent AI pipeline for radiology report structuring and quality assurance. The structuring task operates at the sentence level and reorganizes Findings section content into predefined anatomical sections. Each sentence is first evaluated by the rule-based agent (Agent 1). Device-related sentences and duplicate sentences not classified by Agent 1 are routed directly to the DeepSeek-R1 reasoning model (Agent 3). Other unclassified sentences are processed by the LLaMA-3 model (Agent 2), and sentences that remain unclassified or are assigned to nonspecific “Other” categories are subsequently reviewed by Agent 3. Quality assurance is performed at the whole-report level by a separate DeepSeek-R1 agent (Agent 4), which flags potential reporting inconsistencies and assigns a severity level to the flag. The upper-right panel shows an example output for a Findings–Impression mismatch; similar outputs are generated for gender–anatomy discrepancies and undocumented communication of critical findings.
Figure 2: Example of automated structuring of the radiology report Findings section by the proposed multi-agent AI system. The left panel shows the original Findings section from a synthetic radiology report. The right panel shows the system’s corresponding structured output, with findings reorganized into pre-defined organ-based sections. All Findings sections of the radiology reports processed by the pipeline were converted to this structured format.
Radiologist
Reports, n
Sentences, n
Sentences classified by, n (%)
Unclassified sentences, n (%)
Agent 1 (regex)
Agent 2 (LLaMA 3)
Agent 3 (DeepSeek-R1)
Radiologist 1
40
1,403
1,151
187
63
2
(82%)
(13.3%)
(4.5%)
( < 0.2%)
Radiologist 2
19
767
562
141
64
0
(73.3%)
(18.4%)
(8.3%)
Radiologist 3
42
1,382
1,289
30
61
2
Table 1: Number of reports and total number of sentences in the Findings sections for each radiologist, with sentence-level classification distribution across the three-agent report structuring pipeline, including sentences classified by Agent 1 (regex), Agent 2 (LLaMA 3), Agent 3 (DeepSeek-R1), and remaining unclassified sentences.
Radiologist
Reports, n
Flagged reports, n (%)
Flag severity, n (%)
Error type, n (%)
Critical
Major
Minor
Section mismatch
Gender-anatomy discrepancy
Missing critical communication
Radiologist 1
40
11
0
6
5
10
0
1
(27.5%)
(15%)
(12.5%)
(25%)
(2.5%)
Radiologist 2
19
2
0
2
0
2
0
0
(10.5%)
(10.5%)
(10.5%)
Radiologist 3
42
2
0
0
2
2
1
0
Table 2: Report-level QA flags identified by Agent 4 (DeepSeek-R1) across 638 radiology reports, showing the total number of reports reviewed for each radiologist, the number of reports flagged by the QA agent, the severity of flagged errors (critical, major, minor), and the frequency of specific error categories, including section mismatches, gender–anatomy discrepancies, and missing communication of critical findings.
Radiologist
Average processing time per report (s)
Average total processing time per report (s)
Structuring
QA
Agent 1 (regex)
Agent 2 (LLaMA 3)
Agent 3 (DeepSeek-R1)
Agent 4 (DeepSeek-R1)
Radiologist 1
0.538
1.467
21.078
25.094
48.177
Radiologist 2
0.628
2.796
45.284
24.258
72.966
Radiologist 3
0.512
0.536
19.999
23.222
44.269
Radiologist 4
0.758
3.310
47.074
24.061
75.203
Table 3: Average processing time per report across 15 radiologists for radiology report structuring and quality assurance tasks performed by the multi-agent AI pipeline. Columns represent average processing times for each agent, while the rightmost column shows the overall average total processing time per report for each radiologist.
Task
Evaluation category
Answer
Reports, n (%)
Structuring
Newly restructured report is correctly structured
Yes
31 (69%)
No
2 (4%)
Evaluator disagreement
12 (27%)
Clinically important information was omitted from the newly structured report
Yes
0
No
45 (100%)
Fabricated information was introduced in the newly structured report
Yes
0
Table 4: Independent radiologist evaluation of report structuring and QA performance in 45 reports.
Department of Data Science and Artificial Intelligence, Indian Institute of Technology Madras, Chennai, India · TANUH – The AI Centre of Excellence in Healthcare, Indian Institute of Science, Bengaluru, India · Department of Radiology (Division of Interventional Radiology), Ramaiah Medical College and Memorial hospital, Bengaluru, India