Latent Transfer Attack: Adversarial Examples via Generative Latent Spaces

Eitan Shaar; Ariel Shaulov; Yalcin Tur; Gal Chechik; Ravid Shwartz-Ziv

cs.CVMar 6, 2026

Latent Transfer Attack: Adversarial Examples via Generative Latent Spaces

Authors: Eitan Shaar, Ariel Shaulov, Yalcin Tur, Gal Chechik, Ravid Shwartz-Ziv

Organizations: Independent Researcher · Tel-Aviv University · Stanford University · Bar Ilan University ·

Abstract

Adversarial attacks are a central tool for probing the robustness of modern vision models, yet most methods optimize perturbations directly in pixel space under $\ell_\infty$ or $\ell_2$ constraints. While effective in white-box settings, pixel-space optimization often produces high-frequency, texture-like noise that is brittle to common preprocessing (e.g., resizing and cropping) and transfers poorly across architectures. We propose $\textbf{LTA}$ ( $\textbf{L}$ atent $\textbf{T}$ ransfer $\textbf{A}$ ttack), a transfer-based attack that instead optimizes perturbations in the latent space of a pretrained Stable Diffusion VAE. Given a clean image, we encode it into a latent code and optimize the latent representation to maximize a surrogate classifier loss, while softly enforcing a pixel-space $\ell_\infty$ budget after decoding. To improve robustness to resolution mismatch and standard input pipelines, we incorporate Expectation Over Transformations (EOT) via randomized resizing, interpolation, and cropping, and apply periodic latent Gaussian smoothing to suppress emerging artifacts and stabilize optimization. Across a suite of CNN and vision-transformer targets, LTA achieves strong transfer attack success while producing spatially coherent, predominantly low-frequency perturbations that differ qualitatively from pixel-space baselines and occupy a distinct point in the transfer-quality trade-off. Our results highlight pretrained generative latent spaces as an effective and structured domain for adversarial optimization, bridging robustness evaluation with modern generative priors.

Figures & tables

Surrogate	Method	RN50	RN152	VGG16	MNv2	IncV3	ViT-B/16	PiT-B	Visf.-S	Swin-T	Avg.
		CNN Targets					ViT Targets
ResNet-50	P2FA	100.0	98.8	98.2	97.8	83.1	39.4	65.5	85.1	82.5	83.4
	BFA	98.1	94.1	93.7	92.8	85.7	49.5	72.1	86.1	83.4	82.7
	MFAA	95.3	83.6	86.4	84.7	70.8	24.0	41.0	59.8	57.0	67.0
	ANDA	99.9	84.6	83.9	82.8	73.4	42.4	61.6	72.0	66.6	74.1
	GI-FGSM	100.0	65.0	72.0	65.3	49.2	15.7	28.1	38.3	36.2	52.2

Table 1: LTA achieves the strongest transfer across all surrogates, with the largest gains on vision transformer targets. Attack Success Rate (ASR%, higher is better) across target models, grouped by surrogate. Gray columns denote transformer targets. – indicates the target was not evaluated.

Surrogate	Attack Method	Defense Methods
		AT	HGD	NRP	RS	DiffPure
ResNet-50	P2FA	44.8	67.5	52.8	32.9	21.9
	BFA	44.8	77.4	57.7	32.8	26.9
	MFAA	42.8	49.0	46.3	30.8	18.9
	ANDA	42.7	66.1	45.7	31.4	28.0
	GI-FGSM	40.9	25.2	37.7	29.1	15.9

Table 2: LTA outperforms all baselines under every defense, with gains of +20–34 points on average. Attack success rate (ASR%, higher is better) under different defenses, grouped by surrogate model. For NRP we report the average ASR% over evaluated target models.

Figure 1: LTA concentrates perturbation energy in low frequencies far more than any baseline. Radially-averaged power spectrum of adversarial perturbations (log scale), averaged over 100 images. LTA shows a steeper spectral roll-off, indicating stronger low-frequency bias.

Figure 2: Attack strength vs. perceptual detectability. Scatter plot of transfer ASR (%, higher is better) versus user-study fooling rate (%, higher means less detectable), comparing clean images and adversarial images from different methods.

Figure 3: LTA ’s perturbation spectrum is tightly concentrated near DC, confirming its low-frequency bias. Log-magnitude of the 2D FFT of $\delta=x_{\text{adv}}-x$ (DC centered), averaged over 100 images. Pixel-space baselines show energy spread to higher frequencies.

Figure 4: LTA produces spatially coherent, structure-aligned perturbations rather than diffuse texture-like noise. Per-pixel perturbation magnitude $\|\delta(p)\|_{2}$ for a representative image (clean image shown for reference; values normalized).

Config	E	$\boldsymbol{\epsilon}$	S	ASR Avg. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$
VAE (Enc $\rightarrow$ Dec)	–	–	–	10.2	25.98	0.8425	0.1183	34.57
Z-Opt	–	–	–	96.8	19.87	0.6141	0.3694	122.08
Z-Opt + $\epsilon$	–	✓	–	96.5	20.24	0.6239	0.3633	119.88
Z-Opt +E	✓	–	–	98.1	19.31	0.5889	0.3870	129.89
Z-Opt +S	–	–	✓	82.0	22.98	0.7670	0.2042	85.00
Z-Opt +E+S	✓	–	✓	90.3	22.06	0.7386	0.2365	97.73

Table 3: EOT drives transferability while smoothing and the soft

\ell_{\infty}

penalty improve perceptual quality, revealing a clear transfer–quality trade-off. Ablation study (ResNet-50 surrogate) varying three components: EOT (E), soft pixel-space

\ell_{\infty}

penalty (

\epsilon

), and periodic latent Gaussian smoothing (S).

Figure 5: More optimization steps improve transfer at the cost of perceptual quality. Average transfer ASR (left), FID (middle), and LPIPS (right) as a function of latent optimization steps (ResNet-50 surrogate, 50

\to

200 steps).

Surrogate	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$
ResNet-50	P2FA	26.92	0.7648	0.2315	121.09
	BFA	26.88	0.7585	0.2710	111.18
	MFAA	27.00	0.7609	0.2700	106.03
	ANDA	27.02	0.7586	0.2848	99.99
	GI-FGSM	25.86	0.7180	0.3227	95.84
	ILPD	25.01	0.6945	0.3024	114.74

Table 4: Image quality metrics of adversarial examples, grouped by surrogate model.

Figure 6: Effect of smoothing kernel size (ResNet-50 surrogate). Average transfer ASR (left) and quality metrics FID/LPIPS (middle/right) as a function of the Gaussian kernel size. Results are largely insensitive in the tested range.

Figure 7: Effect of smoothing strength (ResNet-50 surrogate). Increasing Gaussian

\sigma

improves perceptual quality (lower FID/LPIPS) but reduces transfer ASR, revealing a clear effectiveness–quality trade-off.

Figure 8: Effect of pixel budget (ResNet-50 surrogate). Average transfer ASR and quality metrics across different

\ell_{\infty}

budgets. Transfer is relatively stable over the tested range, while quality slightly degrades for larger

\epsilon

Figure 9: User study interface. Participants view a single image and indicate whether it is original or modified.

Figure 10: Additional 2D Fourier spectra. Log-magnitude of the 2D FFT of $\delta=x_{\text{adv}}-x$ (DC centered) for all methods. Concentration near the center indicates low-frequency perturbations.

Figure 11: Additional perturbation magnitude visualizations. Per-pixel perturbation magnitude $\|\delta(p)\|_{2}$ for all methods, where $\delta=x_{\text{adv}}-x$ . Values are normalized per-image for visualization.

Attack	Time (s/image)
GI-FGSM	$\textbf{0.53}\pm\textbf{0.01}$
ANDA	$1.59\pm 0.03$
MFAA	$1.80\pm 0.01$
BFA	$2.15\pm 0.09$
ILPD	$4.89\pm 0.16$
P2FA	$22.71\pm 0.71$

Table 5: Runtime per image (100 images, surrogate = ResNet-50). We report mean

\pm

std wall-clock time (seconds) per image.

Adversarial Attacks

Adversarial attacks exploit crafted inputs, poisoned data, or system interactions to manipulate AI predictions, behavior, privacy, or reliability.

See all 448 papers

IDATA: Scalable Invertible Diffusion for Unrestricted Adversarial Transfer Attack
Aug 9, 2026Yi Pan, Jun-Jie Huang, Tianrui Liu +3Adversarial Attacks Transfer-Based Adversarial Attacks
College of Computer Science and Technology
Season: Spectrum-Aware Orthogonal Gradient Refinement for Transfer-Based Adversarial Attacks
Aug 5, 2026Tianyi Wang, Zhenghao Gao, Shengjie XuAdversarial Attacks Transfer-Based Adversarial Attacks
Tongji University Shanghai, China · Huazhong University of Science and Technology · Wuhan LightRead Intelligent Technology Co., Ltd. Wuhan, China
Boosting the Local Invariance for Better Adversarial Transferability
Mar 8, 2025Bohan Liu, Xiaosen WangAdversarial Attacks Transfer-Based Adversarial Attacks
School of Computer Science and Technology, Xidian University, Xi’an 710126, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, China

Aug 9, 2026J/K move · Enter open · S save

IDATA: Scalable Invertible Diffusion for Unrestricted Adversarial Transfer Attack

Yi Pan, Jun-Jie Huang, Tianrui Liu, Zihan Chen+2

College of Computer Science and Technology

Adversarial Attacks Transfer-Based Adversarial Attacks Black-Box Adversarial Attacks Consistent Adversarial Attacks Adversarial Transferability Diffusion Models

Open paper AskPDF arXiv

Unrestricted adversarial transfer attacks are important for evaluating the black-box robustness of deep visual models. Diffusion-based attacks have shown promising transferability and visual imperceptibility by optimizing adversarial perturbations along denoising trajectories in latent space. However, existing methods are limited by two challenges: memory-intensive multistep backpropagation and frequency-agnostic perturbation over intermediate latents. To address these issues, we propose IDATA, a memory-efficient diffusion framework for unrestricted adversarial transfer attack. IDATA consists of two key components: an Invertible Diffusion Module (IDM) and a Low-Frequency Constraint Module (LFCM). Specifically, IDM reformulates adversarial optimization over diffusion trajectories as an invertible process, enabling constant-memory backpropagation through on-demand reconstruction of intermediate states instead of storing the full denoising chain. Moreover, LFCM leverages Discrete Wavelet Transform (DWT) to decompose latent variables into low- and high-frequency components, restricting perturbations to semantically stable low-frequency subspaces, thereby improving transferability while preserving visual imperceptibility. Extensive experiments on multiple benchmarks and diverse model architectures demonstrate that IDATA consistently outperforms state-of-the-art baselines in attack success rate, memory efficiency, and visual imperceptibility. These results suggest that IDATA is a promising tool for black-box robustness evaluation of deep visual models. Code is available at https://github.com/colourful-pan/IDATA.

Abstract

Adversarial Attacks

Explore similar work

IDATA: Scalable Invertible Diffusion for Unrestricted Adversarial Transfer Attack

Season: Spectrum-Aware Orthogonal Gradient Refinement for Transfer-Based Adversarial Attacks

Boosting the Local Invariance for Better Adversarial Transferability