Architecture-Agnostic Feature Synergy for Universal Defense Against Heterogeneous Generative Threats
Authors: Bingxue Zhang, Yang Gao, Feida Zhu, Yanyan Shen, Yang Shi
Organizations: University of Shanghai for Science and Technology, Shanghai, China · Singapore Management University, Singapore, Singapore · Shanghai Jiao Tong University, Shanghai, China
Generative AI deployment poses unprecedented challenges to content safety and privacy. However, existing defense mechanisms are often tailored to specific architectures (e.g., Diffusion Models or GANs), creating fragile "defense silos" that fail against heterogeneous generative threats. This paper identifies a fundamental optimization barrier in naive pixel-space ensemble strategies: due to divergent objective functions, pixel-level gradients from heterogeneous generators become statistically orthogonal, causing destructive interference. To overcome this, we observe that despite disparate low-level mechanisms, high-level feature representations of generated content exhibit alignment across architectures. Based on this, we propose the Architecture-Agnostic Targeted Feature Synergy (ATFS) framework. By introducing a target guidance image, ATFS reformulates multi-model defense as a unified feature space alignment task, enabling intrinsic gradient alignment without complex rectification. Extensive experiments show ATFS achieves SOTA protection in heterogeneous scenarios (e.g., Diffusion+GAN). It converges rapidly, reaching over 90% performance within 40 iterations, and maintains strong attack potency even under tight perturbation budgets. The framework seamlessly extends to unseen architectures (e.g., VQ-VAE) by switching the feature extractor, and demonstrates robust resistance to JPEG compression and scaling. Being computationally efficient and lightweight, ATFS offers a viable pathway to dismantle defense silos and enable universal generative security. Code and models are open-sourced for reproducibility.
Figures & tables
Figure 1. Defense Silos Phenomenon: Architecture-Specific Attacks Fail Cross-Model. The figure illustrates how perturbations optimized for one model architecture fail to transfer to another. A checkmark ( ✓ ) indicates successful defense (editing failure), while a cross ( × ) indicates defense failure (editing success). Facial images protected by diffusion-specific methods remain vulnerable to GAN-based edits, and vice versa.
Figure 2. Universal Feature Anchoring: Unifying Heterogeneous Models via Shared Feature Space
SDEdit
StarGANEdit
Method
MS-SSIM ↓
PSNR ↓
IL-NIQE ↑
MS-SSIM ↓
PSNR ↓
IL-NIQE ↑
Original
0.8516
31.26
28.1654
0.9989
49.94
30.3326
MIST
0.6387
20.65
39.6495
0.8011
24.43
32.1648
Antiforgery
0.7534
26.42
33.7946
0.5464
15.29
42.4615
Naive joint
0.6812
21.22
35.3013
0.6503
17.78
38.7418
PCGrad Coordinated
0.6563
20.85
37.6615
0.6204
16.46
39.3195
Table 1. Quantitative Results on DM+GAN Heterogeneous Defense
Figure 3. Qualitative Visualization of Cross-Architecture Defense Effects
Figure 4. Transferability to DreamBooth. Adversarial examples generated by ATFS effectively disrupt subject-driven generation in DreamBooth
SDEdit
VAE Reconstruction
Method
MS-SSIM ↓
PSNR ↓
IL-NIQE ↑
MS-SSIM ↓
PSNR ↓
IL-NIQE ↑
MIST
0.6177
17.86
40.5124
0.9194
29.74
31.3451
PGD
0.6828
23.17
35.2631
0.7668
19.41
35.2115
Naive joint
0.6614
22.03
35.5144
0.7961
20.31
34.2142
PCGrad Coordinated
0.6536
20.67
37.1154
0.8183
21.85
33.8145
Ours
0.5215
18.78
41.0214
0.7443
16.51
36.7187
Table 2. Quantitative Results on DM+VQ-VAE Architecture Generalization
Figure 5. Qualitative Results on DM + VQ-VAE Scenario
Figure 6. Cross-Architecture Defense Effectiveness. Each row represents a test example.
SDEdit
StarGANEdit
Perturbation budget
MS-SSIM ↓
PSNR ↓
IL-NIQE ↑
MS-SSIM ↓
PSNR ↓
IL-NIQE ↑
2/255
0.646
20.56
38.6471
0.5915
15.61
39.4461
4/255
0.5834
19.67
39.5461
0.554
15.9
40.3215
8/255
0.4807
18.26
40.8471
0.426
15.09
42.5147
16/255
0.4047
17.08
42.7214
0.2914
13.24
46.1541
Table 3. Performance Comparison under Different Perturbation Budgets
Figure 7. Efficiency Analysis: Protection Effectiveness vs. Time Cost Comparison of SDEdit and StarGAN protection methods showing MS-SSIM values and time costs across different optimization steps
Figure 8. Robustness Analysis: Performance under Various Attack Scenarios
Figure 9. Performance Variation Analysis across Different Demographic Groups
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure A.1. Visualization of ATFS Defense under Varying Perturbation Budgets
Advances in Generative AI (GenAI) have led to the development of various protection strategies to prevent the unauthorized use of images. These methods rely on adding imperceptible protective perturbations to images to thwart misuse such as style mimicry or deepfake manipulations. Although previous attacks on these protections required specialized, purpose-built methods, we demonstrate that this is no longer necessary. We show that off-the-shelf image-to-image GenAI models can be repurposed as generic ``denoisers" using a simple text prompt, effectively removing a wide range of protective perturbations. Across 8 case studies spanning 6 diverse protection schemes, our general-purpose attack not only circumvents these defenses but also outperforms existing specialized attacks while preserving the image's utility for the adversary. Our findings reveal a critical and widespread vulnerability in the current landscape of image protection, indicating that many schemes provide a false sense of security. We stress the urgent need to develop robust defenses and establish that any future protection mechanism must be benchmarked against attacks from off-the-shelf GenAI models. Code is available in this repository: https://github.com/mlsecviswanath/img2imgdenoiser
Xavier Pleimling, Sifat Muhammad Abdullah, Gunjan Balde +4
University of Texas at San Antonio San Antonio, TX, USA
Recent advances in diffusion models have enabled high-fidelity, identity-preserving image generation for personalized applications such as digital avatars and virtual try-on systems. However, their reliance on sensitive facial reference images raises growing privacy concerns. Existing defense mechanisms are primarily designed for training-based personalization and struggle to generalize to emerging training-free approaches, due to fundamental differences in their identity integration paradigms. To bridge this gap, we propose IDGuardian-the first generalizable and model-agnostic identity protection framework capable of defending against both training-based and training-free personalization methods. IDGuardian abstracts the personalization process into two critical stages: identity extraction and identity injection. It then introduces crafted adversarial perturbations to simultaneously disrupt both stages. Specifically, it degrades the identity features extracted by external encoders and establishes an adversarial conceptual bridge that misdirects the generative trajectory away from the target identity. Extensive experiments show that IDGuardian effectively protects identity across various personalization pipelines and model architectures, while remaining robust to post-processing, adaptive attacks, and cross-dataset generalization. Our code is available at https://github.com/OpenAscent-L/IDGuardian.git.
Lizhi Xiong, Jun Li, Ziqiang Li +2
Nanjing University of Information Science and Technology
The proliferation of Low-Rank Adaptation (LoRA) models has democratized personalized text-to-image generation, enabling users to share lightweight models (e.g., personal portraits) on platforms like Civitai and Liblib. However, this "share-and-play" ecosystem introduces critical risks: benign LoRAs can be weaponized by adversaries to generate harmful content (e.g., political, defamatory imagery), undermining creator rights and platform safety. Existing defenses like concept-erasure methods focus on full diffusion models (DMs), neglecting LoRA's unique role as a modular adapter and its vulnerability to adversarial prompt engineering. To bridge this gap, we propose LoRAShield, the first data-free editing framework for securing LoRA models against misuse. Our platform-driven approach dynamically edits and realigns LoRA's weight subspace via adversarial optimization and semantic augmentation. Experimental results demonstrate that LoRAShield achieves remarkable effectiveness, efficiency, and robustness in blocking malicious generations without sacrificing the functionality of the benign task. By shifting the defense to platforms, LoRAShield enables secure, scalable sharing of personalized models, a critical step toward trustworthy generative ecosystems.
Jiahao Chen, Junhao Li, Yiming Wang +6
Zhejiang University College of Computer Science and Technology Hangzhou, China · Guizhou Medical University School of Biology and Engineering Guiyang, China · Zhejiang University School of Software Technology Ningbo, China +1