Adversarial attacks are a central tool for probing the robustness of modern vision models, yet most methods optimize perturbations directly in pixel space under ℓ∞ or ℓ2 constraints. While effective in white-box settings, pixel-space optimization often produces high-frequency, texture-like noise that is brittle to common preprocessing (e.g., resizing and cropping) and transfers poorly across architectures. We propose LTA (Latent Transfer Attack), a transfer-based attack that instead optimizes perturbations in the latent space of a pretrained Stable Diffusion VAE. Given a clean image, we encode it into a latent code and optimize the latent representation to maximize a surrogate classifier loss, while softly enforcing a pixel-space ℓ∞ budget after decoding. To improve robustness to resolution mismatch and standard input pipelines, we incorporate Expectation Over Transformations (EOT) via randomized resizing, interpolation, and cropping, and apply periodic latent Gaussian smoothing to suppress emerging artifacts and stabilize optimization. Across a suite of CNN and vision-transformer targets, LTA achieves strong transfer attack success while producing spatially coherent, predominantly low-frequency perturbations that differ qualitatively from pixel-space baselines and occupy a distinct point in the transfer-quality trade-off. Our results highlight pretrained generative latent spaces as an effective and structured domain for adversarial optimization, bridging robustness evaluation with modern generative priors.
Figures & tables
CNN Targets
ViT Targets
Surrogate
Method
RN50
RN152
VGG16
MNv2
IncV3
ViT-B/16
PiT-B
Visf.-S
Swin-T
Avg.
ResNet-50
P2FA
100.0
98.8
98.2
97.8
83.1
39.4
65.5
85.1
82.5
83.4
BFA
98.1
94.1
93.7
92.8
85.7
49.5
72.1
86.1
83.4
82.7
MFAA
95.3
83.6
86.4
84.7
70.8
24.0
41.0
59.8
57.0
67.0
ANDA
99.9
84.6
83.9
82.8
73.4
42.4
61.6
72.0
66.6
74.1
GI-FGSM
100.0
65.0
72.0
65.3
49.2
15.7
28.1
38.3
36.2
52.2
Table 1: LTA achieves the strongest transfer across all surrogates, with the largest gains on vision transformer targets. Attack Success Rate (ASR%, higher is better) across target models, grouped by surrogate. Gray columns denote transformer targets. – indicates the target was not evaluated.
Surrogate
Attack Method
Defense Methods
AT
HGD
NRP
RS
DiffPure
ResNet-50
P2FA
44.8
67.5
52.8
32.9
21.9
BFA
44.8
77.4
57.7
32.8
26.9
MFAA
42.8
49.0
46.3
30.8
18.9
ANDA
42.7
66.1
45.7
31.4
28.0
GI-FGSM
40.9
25.2
37.7
29.1
15.9
Table 2: LTA outperforms all baselines under every defense, with gains of +20–34 points on average. Attack success rate (ASR%, higher is better) under different defenses, grouped by surrogate model. For NRP we report the average ASR% over evaluated target models.
Figure 1: LTA concentrates perturbation energy in low frequencies far more than any baseline. Radially-averaged power spectrum of adversarial perturbations (log scale), averaged over 100 images. LTA shows a steeper spectral roll-off, indicating stronger low-frequency bias.
Figure 2: Attack strength vs. perceptual detectability. Scatter plot of transfer ASR (%, higher is better) versus user-study fooling rate (%, higher means less detectable), comparing clean images and adversarial images from different methods.
Figure 3: LTA ’s perturbation spectrum is tightly concentrated near DC, confirming its low-frequency bias. Log-magnitude of the 2D FFT of δ=xadv−x (DC centered), averaged over 100 images. Pixel-space baselines show energy spread to higher frequencies.
Figure 4: LTA produces spatially coherent, structure-aligned perturbations rather than diffuse texture-like noise. Per-pixel perturbation magnitude ∥δ(p)∥2 for a representative image (clean image shown for reference; values normalized).
Config
E
ϵ
S
ASR Avg. ↑
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
VAE (Enc → Dec)
–
–
–
10.2
25.98
0.8425
0.1183
34.57
Z-Opt
–
–
–
96.8
19.87
0.6141
0.3694
122.08
Z-Opt + ϵ
–
✓
–
96.5
20.24
0.6239
0.3633
119.88
Z-Opt +E
✓
–
–
98.1
19.31
0.5889
0.3870
129.89
Z-Opt +S
–
–
✓
82.0
22.98
0.7670
0.2042
85.00
Z-Opt +E+S
✓
–
✓
90.3
22.06
0.7386
0.2365
97.73
Table 3: EOT drives transferability while smoothing and the soft ℓ∞ penalty improve perceptual quality, revealing a clear transfer–quality trade-off. Ablation study (ResNet-50 surrogate) varying three components: EOT (E), soft pixel-space ℓ∞ penalty ( ϵ ), and periodic latent Gaussian smoothing (S).
Figure 5: More optimization steps improve transfer at the cost of perceptual quality. Average transfer ASR (left), FID (middle), and LPIPS (right) as a function of latent optimization steps (ResNet-50 surrogate, 50 → 200 steps).
Surrogate
Method
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
ResNet-50
P2FA
26.92
0.7648
0.2315
121.09
BFA
26.88
0.7585
0.2710
111.18
MFAA
27.00
0.7609
0.2700
106.03
ANDA
27.02
0.7586
0.2848
99.99
GI-FGSM
25.86
0.7180
0.3227
95.84
ILPD
25.01
0.6945
0.3024
114.74
Table 4: Image quality metrics of adversarial examples, grouped by surrogate model.
Figure 6: Effect of smoothing kernel size (ResNet-50 surrogate). Average transfer ASR (left) and quality metrics FID/LPIPS (middle/right) as a function of the Gaussian kernel size. Results are largely insensitive in the tested range.
Figure 7: Effect of smoothing strength (ResNet-50 surrogate). Increasing Gaussian σ improves perceptual quality (lower FID/LPIPS) but reduces transfer ASR, revealing a clear effectiveness–quality trade-off.
Figure 8: Effect of pixel budget (ResNet-50 surrogate). Average transfer ASR and quality metrics across different ℓ∞ budgets. Transfer is relatively stable over the tested range, while quality slightly degrades for larger ϵ .
Figure 9: User study interface. Participants view a single image and indicate whether it is original or modified.
Figure 10: Additional 2D Fourier spectra. Log-magnitude of the 2D FFT of δ=xadv−x (DC centered) for all methods. Concentration near the center indicates low-frequency perturbations.
Figure 11: Additional perturbation magnitude visualizations. Per-pixel perturbation magnitude ∥δ(p)∥2 for all methods, where δ=xadv−x . Values are normalized per-image for visualization.
Attack
Time (s/image)
GI-FGSM
0.53±0.01
ANDA
1.59±0.03
MFAA
1.80±0.01
BFA
2.15±0.09
ILPD
4.89±0.16
P2FA
22.71±0.71
Table 5: Runtime per image (100 images, surrogate = ResNet-50). We report mean ± std wall-clock time (seconds) per image.
Unrestricted adversarial transfer attacks are important for evaluating the black-box robustness of deep visual models. Diffusion-based attacks have shown promising transferability and visual imperceptibility by optimizing adversarial perturbations along denoising trajectories in latent space. However, existing methods are limited by two challenges: memory-intensive multistep backpropagation and frequency-agnostic perturbation over intermediate latents. To address these issues, we propose IDATA, a memory-efficient diffusion framework for unrestricted adversarial transfer attack. IDATA consists of two key components: an Invertible Diffusion Module (IDM) and a Low-Frequency Constraint Module (LFCM). Specifically, IDM reformulates adversarial optimization over diffusion trajectories as an invertible process, enabling constant-memory backpropagation through on-demand reconstruction of intermediate states instead of storing the full denoising chain. Moreover, LFCM leverages Discrete Wavelet Transform (DWT) to decompose latent variables into low- and high-frequency components, restricting perturbations to semantically stable low-frequency subspaces, thereby improving transferability while preserving visual imperceptibility. Extensive experiments on multiple benchmarks and diverse model architectures demonstrate that IDATA consistently outperforms state-of-the-art baselines in attack success rate, memory efficiency, and visual imperceptibility. These results suggest that IDATA is a promising tool for black-box robustness evaluation of deep visual models. Code is available at https://github.com/colourful-pan/IDATA.
Transfer-based adversarial attacks often transfer poorly across heterogeneous architectures because CNNs favor local textures while Vision Transformers (ViTs) rely on global shapes. We propose Season, a spectrum-aware orthogonal gradient refinement framework for L-infinity transfer attacks against black-box target models on ImageNet, using a white-box surrogate. Season decomposes each update into a low-frequency branch capturing structural cues and a high-frequency branch capturing textures. A low-saliency guidance scheme reallocates high-frequency energy to background regions, preserving foreground structures that ViTs depend on. An orthogonal projection then forces the textural update to lie in the orthogonal complement of the structural direction, mitigating feature interference. As a training-free plug-and-play wrapper, Season enhances eight gradient-stabilization and input-enhancement attacks without modifying their cores. Across eight CNN, ViT, and MLP targets, Season improves transfer success rate by 6.6 percentage points on average and up to 16.0 points over strong baselines under a unified protocol.
Tianyi Wang, Zhenghao Gao, Shengjie Xu
Tongji University Shanghai, China · Huazhong University of Science and Technology · Wuhan LightRead Intelligent Technology Co., Ltd. Wuhan, China
Transfer-based attacks pose a significant threat to real-world applications by directly targeting victim models with adversarial examples generated on surrogate models. While numerous approaches have been proposed to enhance adversarial transferability, existing works often overlook the intrinsic relationship between adversarial perturbations and input images. In this work, we find that the adversarial perturbations often exhibit poor translation invariance for a given clean image and model, which is attributed to local invariance. Through empirical analysis, we demonstrate a positive correlation between the local invariance of adversarial perturbations w.r.t. the input image and their transferability across models. Based on this finding, we propose a general adversarial transferability boosting technique called the Local Invariance Boosting approach (LI-Boost). Extensive experiments on the standard ImageNet dataset demonstrate that LI-Boost significantly enhances five categories of transfer-based attacks, i.e., gradient-based, input transformation-based, model-related, advanced objective function, and ensemble attacks. The improvements hold not only on conventional CNNs, ViTs, and defense mechanisms, but also on real-world commercial vision API systems and vision-language models. Our approach provides a promising direction for future research on improving adversarial transferability across models. Our code is available at https://github.com/Trustworthy-AI-Group/TransferAttack.
Bohan Liu, Xiaosen Wang
School of Computer Science and Technology, Xidian University, Xi’an 710126, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, China