Pareto-Improving Adversarial Attacks with Primal-Dual Regularization

Yang Dai; Longfei Zhang; Wei Tao; Li Shen; Jincai Huang; Qing Tao

cs.LGOct 3, 2026

Pareto-Improving Adversarial Attacks with Primal-Dual Regularization

Authors: Yang Dai, Longfei Zhang, Wei Tao, Li Shen, Jincai Huang, Qing Tao

Organizations: Laboratory for Big Data and Decision, National University of Defense Technology, Changsha 410073, China

Abstract

Transferable adversarial attacks are arguably the most practical black-box threat model. Under the same perturbation budget, stronger transfer attacks attain higher attack success rate (ASR), yet their imperceptibility also tends to degrade. Under such a fixed-budget protocol, transferability and imperceptibility therefore appear to trade off against each other. We argue that this conflict is an artifact of fixed-budget evaluation, not an intrinsic trade-off. When attacks are compared on the ASR--imperceptibility Pareto frontier obtained by sweeping $ε$ , stronger transfer attacks already attain better imperceptibility at matched ASR than weaker ones. To exploit this latent advantage, we introduce the stealthy transfer attack ST, a plug-in primal-dual wrapper that adds an $L_\infty$ saturation regularizer to the standard constrained objective and resolves it through a two-step primal-dual update: a projected primal step on the perturbation coupled with an $L_1$ -ball projection on a dual variable that absorbs the regularizer through Fenchel duality, requiring no auxiliary models or handcrafted perceptual priors. Empirically, ST extends the Pareto frontier across different base attacks and additional surrogate architectures. At $ε{=}16/255$ , average imperceptibility gains over each base attack are $17\%$ on LPIPS and $14\%$ on NIQE while ASR is preserved or improved. At matched high-ASR levels, the strongest ST variants further Pareto-dominate dedicated stealth-oriented transfer attacks, confirming that the latent imperceptibility advantage of strong transfer attacks can be unlocked by a primal-dual optimization wrapper without sacrificing transferability. Code will be made available at https://github.com/AndssY/ST.

Figures & tables

Fig. 1: Comparison at the standard $L_{\infty}{=}16/255$ budget. The adversarial perturbation visualized as $5\times\delta+0.5$ . PGD is visually clean but transfers poorly, whereas MEF and OPS reach much higher ASR at the cost of visible saturation artifacts.

Fig. 2: ASR–imperceptibility curves for STMI and MI-FGSM. Each point corresponds to one

\epsilon

value. Solid line with filled circles denotes STMI; dashed line with hollow squares denotes MI-FGSM. The light-blue shading marks the envelope between the two curves (the operating-gap region). Across the reported sweep, the STMI curve lies on the favorable side of MI-FGSM for the shown metrics, which suggests that ST changes the operating curve rather than merely moving along the base attack’s budget sweep.

Fig. 3: ASR–imperceptibility curves across eight stealth metrics. Solid lines with filled markers denote ST variants, and dashed lines with hollow markers denote the corresponding base attacks. Light-blue shading indicates the Pareto-dominated region. Within each method, the ST-wrapped curve shifts to a strictly better operating region over most of the sweep. Across methods, stronger transfer attacks such as STOPS tend to Pareto-dominate weaker ones at matched ASR.

		Full-reference						No-reference		Distributional
Method	ASR $\uparrow$ (%)	PSNR $\uparrow$	MS-SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	TOPIQ-FR $\uparrow$	LF $\downarrow$	NIQE $\downarrow$	CLIP-IQA $\uparrow$	FID $\downarrow$
PGD	31.98	28.61	0.953	0.155	0.172	0.741	0.0164	8.29	0.552	40.89
STPGD	32.31 (+1.0%)	30.61 (+7.0%)	0.966 (+1.4%)	0.128 (-17.4%)	0.158 (-8.1%)	0.784 (+5.8%)	0.0134 (-18.3%)	6.69 (-19.3%)	0.578 (+4.7%)	42.52 (+4.0%)
MI-FGSM	42.86	26.92	0.923	0.289	0.215	0.606	0.0228	13.08	0.513	59.85
STMI	44.19 (+3.1%)	29.48 (+9.5%)	0.953 (+3.2%)	0.198 (-31.4%)	0.189 (-12.1%)	0.701 (+15.7%)	0.0168 (-26.6%)	10.36 (-20.8%)	0.546 (+6.6%)	57.65 (-3.7%)
MDCSMI	46.66	25.57	0.903	0.343	0.240	0.547	0.0266	23.74	0.488	73.25

TABLE I: ST versus base attack at

\epsilon{=}16/255

on seven representative attacks. Best per row in bold; the percentage in subscript reports the relative change of the ST variant over its base attack.

Fig. 4: ASR–imperceptibility curves across eight stealth metrics on a ViT-B/16 surrogate. Dotted lines denote dedicated stealth-oriented competitors (CAME, PS Attack, TESSER). STOPS occupies the outer Pareto frontier on most shown metrics. STMDCSMEF plays a similar role one rung lower on the ASR axis.

Fig. 5:

\bar{\lambda}

sensitivity of STMI. Top row:

\bar{\lambda}

sweep at fixed

\epsilon{=}16/255

; each panel shows average ASR (left axis, blue solid) and one stealth metric (right axis, red dashed) as

\bar{\lambda}

varies. Bottom row: the resulting STMI

\bar{\lambda}

sweep in the ASR–imperceptibility plane, compared with the same MI-FGSM

\epsilon

sweep used as reference. The

\bar{\lambda}

sweep traces the widest trade-off arc and reaches more favorable trade-offs than merely shrinking

\epsilon

Fig. 6:

\beta

sensitivity of STMI. The default

\beta=160

comes from the coupling heuristic

\alpha\beta\lesssim 1

, derived in Appendix A (Remark 4 ).

Fig. 7:

\rho

sensitivity of STMI. The sweep mainly tests how far over-relaxation can be pushed before leaving the useful local trade-off segment.

Metric	$\lambda\\|\bm{\delta}\\|_{\infty}$ (ours)	$\lambda\\|\bm{\delta}\\|_{2}$	$\lambda\\|\bm{\delta}\\|_{1}$
ASR $\uparrow$ (%)	44.19	44.58	42.44
PSNR $\uparrow$	29.48	28.35	28.70
MS-SSIM $\uparrow$	0.953	0.936	0.941
LPIPS $\downarrow$	0.198	0.258	0.241
DISTS $\downarrow$	0.189	0.206	0.198
TOPIQ-FR $\uparrow$	0.701	0.615	0.625

TABLE II: Soft-regularizer ablation on STMI. The three columns correspond to three objective choices for

h(\bm{\delta})

, implemented by their dual projections inside the same Condat–Vũ template; best per row in bold.

Fig. 8: Qualitative comparison at fixed $\epsilon{=}16/255$ (the first three rows of Figure 9 ). Each row shows the clean-image context with the selected crop region marked in red, the clean crop itself, and the corresponding crop regions for MI-FGSM, STMI, MDCSMEF, and STMDCSMEF. At the same perturbation budget, the ST variants suppress local color speckle and edge ringing while preserving cleaner object boundaries.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

ST variant	$\bar{\lambda}$
STPGD	0.25
STMI	2.4
STMDCSMI	0.9
STDIM	2.3
STGRA	3.0
STMDCSMEF	0.3

Appendix

TABLE III: Per-attack

\bar{\lambda}

used in the main results (Table I ). Each value is selected from the sweep

\bar{\lambda}\in[0.1,5.0]

to maximize the imperceptibility metrics on the 1000-image evaluation set.

Fig. 9: Full-page same- $\epsilon$ qualitative comparison at $\epsilon{=}16/255$ on four representative samples. Each row shows the clean image and four attack columns, where each attack cell contains the adversarial image on top and the perturbation visualized as $5\times\delta+0.5$ on bottom; the clean image is kept flush with the adversarial sub-row for alignment. The ST variants remain visually cleaner at both the adversarial-image level and the perturbation level.

Fig. 10: Full-page matched-ASR qualitative comparison at ASR $\approx 50\%$ . MI-FGSM at $\epsilon{=}22/255$ (ASR $49.75\%$ ), STMI at $\epsilon{=}20/255$ (ASR $50.05\%$ ), MDCSMEF at $\epsilon{=}6/255$ (ASR $50.12\%$ ), STMDCSMEF at $\epsilon{=}6/255$ (ASR $50.38\%$ ).

Adversarial Attacks

Adversarial attacks exploit crafted inputs, poisoned data, or system interactions to manipulate AI predictions, behavior, privacy, or reliability.

See all 444 papers

Learning with Bilevel-Minimax Optimization for Efficient and Reliable Transfer Attacks
Aug 12, 2026Yaohua Liu, Yifan Guo, Jiaxin GaoBilevel Optimization Transfer-Based Adversarial Attacks
The University of Hong Kong, Hong Kong SAR, China · International School of Information Science & Engineering, Dalian University of Technology, Dalian, China · The Hong Kong Polytechnic University, Hong Kong SAR, China
Boosting the Local Invariance for Better Adversarial Transferability
Mar 8, 2025Bohan Liu, Xiaosen WangAdversarial Attacks Transfer-Based Adversarial Attacks
School of Computer Science and Technology, Xidian University, Xi’an 710126, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, China
IDATA: Scalable Invertible Diffusion for Unrestricted Adversarial Transfer Attack
Aug 9, 2026Yi Pan, Jun-Jie Huang, Tianrui Liu +3Adversarial Attacks Transfer-Based Adversarial Attacks
College of Computer Science and Technology

Aug 12, 2026J/K move · Enter open · S save

Learning with Bilevel-Minimax Optimization for Efficient and Reliable Transfer Attacks

Yaohua Liu, Yifan Guo, Jiaxin Gao

The University of Hong Kong, Hong Kong SAR, China · International School of Information Science & Engineering, Dalian University of Technology, Dalian, China · The Hong Kong Polytechnic University, Hong Kong SAR, China

Bilevel Optimization Transfer-Based Adversarial Attacks Black-Box Adversarial Attacks Adversarial Transferability

Open paper AskPDF arXiv

Transfer-based adversarial attacks craft adversarial examples using surrogate models to mislead black-box victim models. Beyond perturbation generation, transferability is fundamentally governed by the coupling of initialization, surrogate adaptation, and gradient dynamics. We revisit this challenge from a bilevel-minimax perspective and propose BMAT (Bilevel-Minimax Adversarial Transfer). The bilevel formulation captures the dependency between initialization and perturbation, while the inner minimax problem promotes surrogate robustness for cross-architecture generalization. Algorithmically, we develop an integrated bottom-up solver that combines a Soft Weight Modulator and an Implicit Gradient Approximator to enable ternary coupling among initialization, surrogate adaptation, and perturbation optimization. We further provide theoretical insights into the optimization dynamics of the proposed bilevel-minimax framework. Extensive experiments on classification and segmentation benchmarks show that BMAT outperforms more than 10 strong baselines across more than 30 victim models, improving both intra- and cross-architecture transfer and yielding up to a 2x reduction in mIoU. Code is available at https://github.com/callous-youth/BMAT.

Abstract

Adversarial Attacks

Explore similar work

Learning with Bilevel-Minimax Optimization for Efficient and Reliable Transfer Attacks

Boosting the Local Invariance for Better Adversarial Transferability

IDATA: Scalable Invertible Diffusion for Unrestricted Adversarial Transfer Attack