cs.AINov 6, 2026

Adversarial Evaluation of Large Language Models for Patient-Friendly Radiology Report Summarization

Authors: Magdalini Paschali, Asad Aali, Arogya Koirala, Andrew Johnston, Katherine McNamara, Richard Gaus, Roxana Daneshjou, David B. Larson, 

Organizations: Department of Radiology, Stanford University, Stanford, CA, USA. · AI Development and Evaluation Lab (AIDE), School of Medicine, Stanford University, Stanford, CA, USA. · Department of Computer Science, Technical University of Munich, Munich, Bavaria, Germany. · Department of Dermatology, Stanford University, Stanford, CA, USA. · 

Abstract

Purpose: To develop and validate a scalable adversarial evaluation framework for assessing the robustness of large language models (LLMs) that generate patient-friendly summaries of radiology reports. Methods: We introduced a clinically grounded adversarial framework that uses controlled term substitutions and impression removals across intensity levels. Using a radiologist-validated adversarial dictionary, we generated adversarial variants of CT and MRI reports. Six LLMs summarized original and adversarial impressions using 2 prompt styles, yielding 18,000 evaluation pairs. Robustness was evaluated with an automated LLM-as-a-judge framework scoring factual faithfulness, sensitivity to adversarial edits, certainty fidelity, and hallucination rate. Failure mode analysis quantified vulnerabilities in incorporating clinical modifiers. Agreement with two radiologists was assessed on 105 cases. Results: GPT-5-mini achieved the highest factual faithfulness and certainty fidelity under both adversarial edits. Under term substitution, hallucinations occurred in 28.1-30.1% of GPT-5-mini summaries, compared with 35.3-36.4% for Claude Sonnet 4.5 and 41.4-41.9% for MedGemma. Impression-removal edits were handled more robustly than term substitutions. Even for the highest-scoring summarizer, substitutions of anatomical terms were not reflected in 23.7% of summaries, diagnostic certainty in 22.3%, and descriptor terms (e.g., "substantial") in 21.2%. Judge-radiologist agreement (macro-F1: 0.55-0.56) was comparable to inter-radiologist agreement (0.53). Conclusions: Adversarial evaluation exposes clinically meaningful failure modes, including missed anatomical modifiers, altered diagnostic certainty, and omitted severity descriptors.

Explore similar work

CardsList
  1. Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports

    Jun 17, 2026Qingyu Lu, Ruochen Li, Liang Ding +3Clinical Language Model EvaluationLanguage Model Robustness

  2. Improving Automatic Summarization of Radiology Reports through Mid-Training of Large Language Models

    Feb 28, 2026Mengxian Lyu, Cheng Peng, Ziyi Chen +3Text SummarizationMedical LLMs