cs.AINov 8, 2026

Safeguarding Agentic AI in Radiology Workflows: A Multimodel Evaluation of Rule-based and Large Language Model-based Verification

Authors: Semil Eminovic, Robin Schmidt, Bogdan Levita, Maximilian Lindholz, Tobias Penzkofer, Jawed Nawabi

Organizations: Department of Radiology, Charité – Universitätsmedizin Berlin, Humboldt-Universität zu Berlin, Freie Universität Berlin, Berlin Institute of Health, Berlin, Germany · Experimental Clinical Research Center (ECRC) at Charité – Universitätsmedizin Berlin and Max-Delbrück-Centrum für molekulare Medizin (MDC), Berlin, Germany · Department of Nuclear Medicine, Charité – Universitätsmedizin Berlin, Humboldt-Universität zu Berlin, Freie Universität Berlin, Berlin Institute of Health, Berlin, Germany · Berlin Institute of Health at Charité – Universitätsmedizin Berlin, Berlin, Germany · 

Abstract

Purpose: To estimate how often tool-using large language model (LLM) agents record clinically relevant false or unsupported information in simulated radiology documentation and how much safeguards applied before each action reduce these errors. Materials and Methods: In this simulation study (October 2026), 48 synthetic, radiologist-reviewed base scenarios covered finding communication, tumor board preparation and follow-up management. Forty-three also had a misleading-information variant (91 variants). Six LLM agents (Claude Sonnet 5.5, GPT-6.1-Sol, GLM-5.3, MiniMax M3, Gemma-4-31B, Qwen3.8-27B) completed each variant with permissions only, deterministic rules, and rules plus a locally run LLM checker (1638 episodes). The primary outcome was at least one executed high-severity violation per episode. Risk differences were estimated per agent at base-scenario level and, exploratorily, pooled across agents, with analyses of document access and blocks. Results: With permissions only, 7.3%-27.1% of episodes had a high-severity violation (pooled across agents, 19.4%). Pooled, it was 16.7% with rules (-2.8 percentage points; 95% CI: -6.1, 0.5) and 13.2% with rules plus the LLM checker (-6.2; 95% CI: -9.5, -3.0). Adding the LLM checker to rules changed it by -3.5 (95% CI: -6.3, -0.7). No difference between agents was detected (P = .43), and task success did not decrease detectably. In misleading variants, agents opened the altered document in 83% of episodes regardless of error, and 58% of erroneous statements cited it. Most blocked actions (64%-68%) would not have caused a violation and were returned for revision. Conclusion: Layered safeguards reduced but did not prevent high-severity errors. Errors after misleading information reflected misjudged rather than missed information.

Explore similar work

CardsList
  1. Agentic retrieval-augmented reasoning reshapes collective reliability under model variability in radiology question answering

    Mar 6, 2026Mina Farajiamiri, Jeta Sopa, Saba Afza +9LLM ReliabilityLLM Agent Reliability

  2. How Well Can Modern LLMs Act as Agent Cores in Radiology Environments?

    Dec 12, 2024Qiaoyu Zheng, Chaoyi Wu, Weike Zhao +5HealthcareLLM Agent Evaluation

  3. Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization

    Nov 6, 2026Magdalini Paschali, Asad Aali, Arogya Koirala +6Clinical Language Model EvaluationAdversarial Robustness of Language Models