Text-Guided Diffusion-Based Adversarial Attacks on Chest X-Ray Images
Authors: Basudha Pal, Arjun Narayanan, Neha Ajith, Vikas R Bhat, Muhammad Umair
Organizations: Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, 21218, MD, USA · Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal, 576104, Karnataka, India · The Russell H. Morgan Department of Radiology and Radiological Sciences, The Johns Hopkins Hospital, Baltimore, 21287, MD, USA · Department of Radiology, Columbia University Irving Medical Center, New York, 10032, NY, USA
As artificial intelligence is increasingly integrated into chest X-ray (CXR) interpretation, triage, and clinical decision support, understanding its vulnerability to adversarial manipulation is critical for safe deployment. Existing robustness evaluations, however, predominantly rely on pixel-space attacks that introduce numerically constrained perturbations but may not represent plausible radiographic variation. This limitation is particularly important in multi-disease CXR classification, where models simultaneously evaluate multiple overlapping pathologies and adversarial failures may alter several diagnostic predictions. We propose a text-guided diffusion-based adversarial framework that optimizes learnable text conditioning while keeping the diffusion generator and target classifier frozen, enabling adversarial generation through a learned image prior rather than direct pixel manipulation. We evaluate the framework across multiple classifier architectures in both binary atelectasis and multi-disease CXR classification and compare it with FGSM, PGD, and Carlini-Wagner attacks. Our approach consistently produced the greatest degradation in classifier performance, reducing AUROC to 0.3885-0.5646 in binary classification and 0.4441-0.4878 in the multi-disease setting, while achieving superior image fidelity (SSIM 0.9080, LPIPS 0.1670, FID 51.23). Importantly, clinician interpretation remained unchanged for 95.9% of binary and 73.8% of multi-disease adversarial images despite substantial changes in model predictions. These findings reveal a clinically important discrepancy between human and machine interpretation and demonstrate the need to extend medical AI robustness evaluation beyond conventional pixel-space attacks toward generative threat models that can expose failures under visually and clinically plausible image variations.
Figures & tables
Figure 1: Basic architecture diagram of the proposed diffusion-guided adversarial framework. A learnable placeholder token is passed through a trainable text encoder to generate conditioning embeddings. These embeddings guide a frozen diffusion generator that reconstructs a chest X-ray image. The generated image is evaluated by a frozen disease classifier. Gradients are propagated only through the text encoder. The diagram illustrates the binary setting; the formulation extends naturally to multi-class and multi-label classification.
Task
Attack
DenseNet
ResNet
EfficientNet
Binary Atelectasis
Clean
0.7931
0.7561
0.6777
FGSM
0.4986
0.7327
0.6494
PGD
0.4531
0.7451
0.6625
CW
0.4496
0.6894
0.5337
Ours
0.3885
0.5646
0.4032
Multiclass
Clean
0.8319
0.7810
0.7989
Table 1: AUROC of chest X-ray classifiers under different adversarial attacks for binary atelectasis detection and multiclass disease classification. Lower AUROC indicates greater degradation in diagnostic performance.
Attack
SSIM ↑
LPIPS ↓
FID ↓
FGSM
0.5160
0.8054
145.55
PGD
0.8124
0.6083
86.24
CW
0.8488
0.2397
62.58
Ours
0.9080
0.1670
51.23
Table 2: Perceptual quality of adversarial chest X-ray images averaged across tasks. Higher SSIM indicates greater structural similarity to the original image, whereas lower LPIPS and FID correspond to improved perceptual realism.
Figure 2: Representative adversarial chest X-rays from the multiclass classification setting. Each row corresponds to one input radiograph and shows the original image together with adversarial examples generated using FGSM, PGD, and the proposed diffusion-based method. FGSM introduces pronounced high-frequency perturbations throughout the image, while PGD produces weaker but still perceptible perturbation patterns. In contrast, the diffusion-based method preserves the overall radiographic appearance and major anatomical structures of the source image while altering classifier predictions.
Figure 3: Representative adversarial chest X-rays from the binary atelectasis classification setting. Each row shows an original radiograph and corresponding FGSM-, PGD-, and diffusion-generated adversarial images. Conventional pixel-space attacks introduce visible image-wide perturbations, particularly for FGSM, whereas diffusion-generated images remain visually similar to the source radiographs, with preservation of major thoracic structures and substantially less apparent distortion.
Defense Strategy
FGSM
PGD
CW
Ours
None
0.5232
0.4959
0.5080
0.4441
Gaussian Smoothing
0.5360
0.5118
0.5120
0.4578
JPEG Compression
0.5246
0.5329
0.5247
0.4646
Adversarial Training (50%)
0.5681
0.5769
0.5749
0.4937
Adversarial Training (75%)
0.5813
0.5985
0.5991
0.5362
Table 3: AUROC after applying different defense strategies against adversarial attacks.
Figure 4: Clinician assessment of diffusion-generated adversarial chest radiographs. Percentage of cases exhibiting unchanged or changed clinical interpretation following adversarial perturbation. The high proportion of unchanged assessments indicates that the generated adversarial modifications largely preserved clinician-perceived diagnostic content.
Latent Diffusion Models have shown remarkable results in text-guided image synthesis in recent years. In the domain of natural (RGB) images, recent works have shown that such models can be adapted to various vision-language downstream tasks with little to no supervision involved. On the contrary, text-to-image Latent Diffusion Models remain relatively underexplored in the field of medical imaging, primarily due to limited data availability (e.g., due to privacy concerns). In this work, focusing on the chest X-ray modality, we first demonstrate that a standard text-conditioned Latent Diffusion Model has not learned to align clinically relevant information in free-text radiology reports with the corresponding areas of the given scan. Then, to alleviate this issue, we propose a fine-tuning framework to improve multi-modal alignment in a pre-trained model such that it can be efficiently repurposed for downstream tasks such as phrase grounding. Our method sets a new state-of-the-art on a standard benchmark dataset (MS-CXR), while also exhibiting robust performance on out-of-distribution data (VinDr-CXR). We further validate our approach through a pilot qualitative study and an experiment on grounded disease classification. Our code will be made publicly available at https://github.com/vios-s.
Konstantinos Vilouras, Ilias Stogiannidis, Junyu Yan +2
School of Engineering, University of Edinburgh · Canon Medical Research Europe Ltd.
Deep learning models for chest X-ray diagnosis are constrained by limited coverage of clinically meaningful concept combinations in publicly available training datasets. While synthetic image generation has been explored to increase data diversity, existing methods rarely enforce clinical or anatomical constraints, limiting utility for improving model reliability. We propose CARPA, a clinically aware and anatomically grounded framework for synthetic chest X-ray generation that applies targeted perturbations to clinical concept vectors while preserving anatomical structure. By producing anatomically faithful synthetic images with controlled concept insertions and deletions, CARPA expands clinically relevant concept coverage. We evaluate CARPA across seven backbone architectures by fine-tuning models on synthetic subsets and testing on a held-out MIMIC-CXR benchmark. Compared to prior concept perturbation approaches, fine-tuning on CARPA-generated images consistently improves precision-recall performance, reduces predictive uncertainty, and improves model calibration. Structural and semantic analyses demonstrate high anatomical fidelity, strong concept alignment, and low semantic uncertainty. Evaluation by two expert radiologists further confirms realism and clinical agreement. Together, these results show that anatomically grounded concept perturbations enable more effective use of synthetic data, improving both performance and reliability of chest X-ray classification models and supporting safer clinical deployment.
Text-conditioned chest X-ray generation aims to synthesize realistic radiographs that faithfully depict specified findings. Existing work has primarily improved quality by updating image generators, implicitly treating prompts as fixed after CXR-domain adaptation. We show that this generator-centric view leaves a substantial optimization dimension underexplored. With a CXR-adapted Sana generator frozen, one-pass reformulation by an unmodified LLM reduces RadDINO-FID from 54.225 to 27.572. Prompt analysis shows that the LLM suppresses temporal comparisons, uncertainty, and other non-renderable report content while emphasizing visible radiographic findings. However, unconstrained reformulation reduces BioViL-T alignment with source prompts from 0.695 to 0.609. We therefore introduce JustLLMGRPO, which applies standard Group Relative Policy Optimization (GRPO) only to the LLM prompt policy while keeping Sana frozen. Group-relative radiology-aware image feedback retains visual focus while preserving source-prompt alignment. On CheXGenBench, JustLLMGRPO reduces RadDINO-FID to 26.780, a 50.6% improvement over direct prompting, while maintaining alignment (0.696 versus 0.695). It also achieves state-of-the-art distribution coverage and downstream classification utility. These results show that substantial performance can remain latent in how radiographic information is expressed to an adapted generator. Code is publicly available at https://github.com/pxcai/JustLLMGRPO.
Pengxiang Cai, Xiaohan Li, Anglin Liu +3
The Hong Kong University of Science and Technology (Guangzhou)