Transferable adversarial attacks are arguably the most practical black-box threat model. Under the same perturbation budget, stronger transfer attacks attain higher attack success rate (ASR), yet their imperceptibility also tends to degrade. Under such a fixed-budget protocol, transferability and imperceptibility therefore appear to trade off against each other. We argue that this conflict is an artifact of fixed-budget evaluation, not an intrinsic trade-off. When attacks are compared on the ASR--imperceptibility Pareto frontier obtained by sweeping ε, stronger transfer attacks already attain better imperceptibility at matched ASR than weaker ones. To exploit this latent advantage, we introduce the stealthy transfer attack ST, a plug-in primal-dual wrapper that adds an L∞ saturation regularizer to the standard constrained objective and resolves it through a two-step primal-dual update: a projected primal step on the perturbation coupled with an L1-ball projection on a dual variable that absorbs the regularizer through Fenchel duality, requiring no auxiliary models or handcrafted perceptual priors. Empirically, ST extends the Pareto frontier across different base attacks and additional surrogate architectures. At ε=16/255, average imperceptibility gains over each base attack are 17% on LPIPS and 14% on NIQE while ASR is preserved or improved. At matched high-ASR levels, the strongest ST variants further Pareto-dominate dedicated stealth-oriented transfer attacks, confirming that the latent imperceptibility advantage of strong transfer attacks can be unlocked by a primal-dual optimization wrapper without sacrificing transferability. Code will be made available at https://github.com/AndssY/ST.
Figures & tables
Fig. 1: Comparison at the standard L∞=16/255 budget. The adversarial perturbation visualized as 5×δ+0.5 . PGD is visually clean but transfers poorly, whereas MEF and OPS reach much higher ASR at the cost of visible saturation artifacts.
Fig. 2: ASR–imperceptibility curves for STMI and MI-FGSM. Each point corresponds to one ϵ value. Solid line with filled circles denotes STMI; dashed line with hollow squares denotes MI-FGSM. The light-blue shading marks the envelope between the two curves (the operating-gap region). Across the reported sweep, the STMI curve lies on the favorable side of MI-FGSM for the shown metrics, which suggests that ST changes the operating curve rather than merely moving along the base attack’s budget sweep.
Fig. 3: ASR–imperceptibility curves across eight stealth metrics. Solid lines with filled markers denote ST variants, and dashed lines with hollow markers denote the corresponding base attacks. Light-blue shading indicates the Pareto-dominated region. Within each method, the ST-wrapped curve shifts to a strictly better operating region over most of the sweep. Across methods, stronger transfer attacks such as STOPS tend to Pareto-dominate weaker ones at matched ASR.
Full-reference
No-reference
Distributional
Method
ASR ↑ (%)
PSNR ↑
MS-SSIM ↑
LPIPS ↓
DISTS ↓
TOPIQ-FR ↑
LF ↓
NIQE ↓
CLIP-IQA ↑
FID ↓
PGD
31.98
28.61
0.953
0.155
0.172
0.741
0.0164
8.29
0.552
40.89
STPGD
32.31 (+1.0%)
30.61 (+7.0%)
0.966 (+1.4%)
0.128 (-17.4%)
0.158 (-8.1%)
0.784 (+5.8%)
0.0134 (-18.3%)
6.69 (-19.3%)
0.578 (+4.7%)
42.52 (+4.0%)
MI-FGSM
42.86
26.92
0.923
0.289
0.215
0.606
0.0228
13.08
0.513
59.85
STMI
44.19 (+3.1%)
29.48 (+9.5%)
0.953 (+3.2%)
0.198 (-31.4%)
0.189 (-12.1%)
0.701 (+15.7%)
0.0168 (-26.6%)
10.36 (-20.8%)
0.546 (+6.6%)
57.65 (-3.7%)
MDCSMI
46.66
25.57
0.903
0.343
0.240
0.547
0.0266
23.74
0.488
73.25
TABLE I: ST versus base attack at ϵ=16/255 on seven representative attacks. Best per row in bold; the percentage in subscript reports the relative change of the ST variant over its base attack.
Fig. 4: ASR–imperceptibility curves across eight stealth metrics on a ViT-B/16 surrogate. Dotted lines denote dedicated stealth-oriented competitors (CAME, PS Attack, TESSER). STOPS occupies the outer Pareto frontier on most shown metrics. STMDCSMEF plays a similar role one rung lower on the ASR axis.
Fig. 5: λˉ sensitivity of STMI. Top row: λˉ sweep at fixed ϵ=16/255 ; each panel shows average ASR (left axis, blue solid) and one stealth metric (right axis, red dashed) as λˉ varies. Bottom row: the resulting STMI λˉ sweep in the ASR–imperceptibility plane, compared with the same MI-FGSM ϵ sweep used as reference. The λˉ sweep traces the widest trade-off arc and reaches more favorable trade-offs than merely shrinking ϵ .
Fig. 6: β sensitivity of STMI. The default β=160 comes from the coupling heuristic αβ≲1 , derived in Appendix A (Remark 4 ).
Fig. 7: ρ sensitivity of STMI. The sweep mainly tests how far over-relaxation can be pushed before leaving the useful local trade-off segment.
Metric
λ∥δ∥∞ (ours)
λ∥δ∥2
λ∥δ∥1
ASR ↑ (%)
44.19
44.58
42.44
PSNR ↑
29.48
28.35
28.70
MS-SSIM ↑
0.953
0.936
0.941
LPIPS ↓
0.198
0.258
0.241
DISTS ↓
0.189
0.206
0.198
TOPIQ-FR ↑
0.701
0.615
0.625
TABLE II: Soft-regularizer ablation on STMI. The three columns correspond to three objective choices for h(δ) , implemented by their dual projections inside the same Condat–Vũ template; best per row in bold.
Fig. 8: Qualitative comparison at fixed ϵ=16/255 (the first three rows of Figure 9 ). Each row shows the clean-image context with the selected crop region marked in red, the clean crop itself, and the corresponding crop regions for MI-FGSM, STMI, MDCSMEF, and STMDCSMEF. At the same perturbation budget, the ST variants suppress local color speckle and edge ringing while preserving cleaner object boundaries.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
ST variant
λˉ
STPGD
0.25
STMI
2.4
STMDCSMI
0.9
STDIM
2.3
STGRA
3.0
STMDCSMEF
0.3
Appendix
TABLE III: Per-attack λˉ used in the main results (Table I ). Each value is selected from the sweep λˉ∈[0.1,5.0] to maximize the imperceptibility metrics on the 1000-image evaluation set.
Fig. 9: Full-page same- ϵ qualitative comparison at ϵ=16/255 on four representative samples. Each row shows the clean image and four attack columns, where each attack cell contains the adversarial image on top and the perturbation visualized as 5×δ+0.5 on bottom; the clean image is kept flush with the adversarial sub-row for alignment. The ST variants remain visually cleaner at both the adversarial-image level and the perturbation level.
Fig. 10: Full-page matched-ASR qualitative comparison at ASR ≈50% . MI-FGSM at ϵ=22/255 (ASR 49.75% ), STMI at ϵ=20/255 (ASR 50.05% ), MDCSMEF at ϵ=6/255 (ASR 50.12% ), STMDCSMEF at ϵ=6/255 (ASR 50.38% ).
Transfer-based adversarial attacks craft adversarial examples using surrogate models to mislead black-box victim models. Beyond perturbation generation, transferability is fundamentally governed by the coupling of initialization, surrogate adaptation, and gradient dynamics. We revisit this challenge from a bilevel-minimax perspective and propose BMAT (Bilevel-Minimax Adversarial Transfer). The bilevel formulation captures the dependency between initialization and perturbation, while the inner minimax problem promotes surrogate robustness for cross-architecture generalization. Algorithmically, we develop an integrated bottom-up solver that combines a Soft Weight Modulator and an Implicit Gradient Approximator to enable ternary coupling among initialization, surrogate adaptation, and perturbation optimization. We further provide theoretical insights into the optimization dynamics of the proposed bilevel-minimax framework. Extensive experiments on classification and segmentation benchmarks show that BMAT outperforms more than 10 strong baselines across more than 30 victim models, improving both intra- and cross-architecture transfer and yielding up to a 2x reduction in mIoU. Code is available at https://github.com/callous-youth/BMAT.
Yaohua Liu, Yifan Guo, Jiaxin Gao
The University of Hong Kong, Hong Kong SAR, China · International School of Information Science & Engineering, Dalian University of Technology, Dalian, China · The Hong Kong Polytechnic University, Hong Kong SAR, China
Transfer-based attacks pose a significant threat to real-world applications by directly targeting victim models with adversarial examples generated on surrogate models. While numerous approaches have been proposed to enhance adversarial transferability, existing works often overlook the intrinsic relationship between adversarial perturbations and input images. In this work, we find that the adversarial perturbations often exhibit poor translation invariance for a given clean image and model, which is attributed to local invariance. Through empirical analysis, we demonstrate a positive correlation between the local invariance of adversarial perturbations w.r.t. the input image and their transferability across models. Based on this finding, we propose a general adversarial transferability boosting technique called the Local Invariance Boosting approach (LI-Boost). Extensive experiments on the standard ImageNet dataset demonstrate that LI-Boost significantly enhances five categories of transfer-based attacks, i.e., gradient-based, input transformation-based, model-related, advanced objective function, and ensemble attacks. The improvements hold not only on conventional CNNs, ViTs, and defense mechanisms, but also on real-world commercial vision API systems and vision-language models. Our approach provides a promising direction for future research on improving adversarial transferability across models. Our code is available at https://github.com/Trustworthy-AI-Group/TransferAttack.
Bohan Liu, Xiaosen Wang
School of Computer Science and Technology, Xidian University, Xi’an 710126, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, China
Unrestricted adversarial transfer attacks are important for evaluating the black-box robustness of deep visual models. Diffusion-based attacks have shown promising transferability and visual imperceptibility by optimizing adversarial perturbations along denoising trajectories in latent space. However, existing methods are limited by two challenges: memory-intensive multistep backpropagation and frequency-agnostic perturbation over intermediate latents. To address these issues, we propose IDATA, a memory-efficient diffusion framework for unrestricted adversarial transfer attack. IDATA consists of two key components: an Invertible Diffusion Module (IDM) and a Low-Frequency Constraint Module (LFCM). Specifically, IDM reformulates adversarial optimization over diffusion trajectories as an invertible process, enabling constant-memory backpropagation through on-demand reconstruction of intermediate states instead of storing the full denoising chain. Moreover, LFCM leverages Discrete Wavelet Transform (DWT) to decompose latent variables into low- and high-frequency components, restricting perturbations to semantically stable low-frequency subspaces, thereby improving transferability while preserving visual imperceptibility. Extensive experiments on multiple benchmarks and diverse model architectures demonstrate that IDATA consistently outperforms state-of-the-art baselines in attack success rate, memory efficiency, and visual imperceptibility. These results suggest that IDATA is a promising tool for black-box robustness evaluation of deep visual models. Code is available at https://github.com/colourful-pan/IDATA.