Boosting the Local Invariance for Better Adversarial Transferability
Authors: Bohan Liu, Xiaosen Wang
Organizations: School of Computer Science and Technology, Xidian University, Xi’an 710126, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, China
Transfer-based attacks pose a significant threat to real-world applications by directly targeting victim models with adversarial examples generated on surrogate models. While numerous approaches have been proposed to enhance adversarial transferability, existing works often overlook the intrinsic relationship between adversarial perturbations and input images. In this work, we find that the adversarial perturbations often exhibit poor translation invariance for a given clean image and model, which is attributed to local invariance. Through empirical analysis, we demonstrate a positive correlation between the local invariance of adversarial perturbations w.r.t. the input image and their transferability across models. Based on this finding, we propose a general adversarial transferability boosting technique called the Local Invariance Boosting approach (LI-Boost). Extensive experiments on the standard ImageNet dataset demonstrate that LI-Boost significantly enhances five categories of transfer-based attacks, i.e., gradient-based, input transformation-based, model-related, advanced objective function, and ensemble attacks. The improvements hold not only on conventional CNNs, ViTs, and defense mechanisms, but also on real-world commercial vision API systems and vision-language models. Our approach provides a promising direction for future research on improving adversarial transferability across models. Our code is available at https://github.com/Trustworthy-AI-Group/TransferAttack.
Figures & tables
Fig. 1 : The impact of translation invariance of clean image and adversarial perturbation. The translated clean image can be correctly recognized by deep models (either CNNs or ViTs), whereas the slightly translated adversarial perturbations cannot consistently fool these models, and our LI-Boost enhanced perturbation can induce them to make incorrect predictions.
Fig. 2 : The local invariance of various transfer-based attacks (left) and the attack performance of MI-FGSM with enhanced local invariance (right).
k
0
1
2
3
4
5
6
Time (s/img)
0.07
0.69
1.92
3.78
6.26
9.35
13.07
BP (count)
1
9
25
49
81
121
169
TABLE I : Time consumption and computation complexity w.r.t. various k . We report average runtime(seconds/image) and computational cost(Backpropagations/BP).
Gradient-based Attacks
CNNs
ViTs
Defenses
Avg.
RN-50
Inc-v3
MN-v3
DN-121
FSNet
ViT
PiT
Visformer
Swin
Inc-v3 ens3
Inc-v3 ens4
IncRes-v2 ens
AT
HGD
RS
NRP
DiffPure
MIMIR
MI-FGSM
94.9*
34.5
40.6
45.9
25.2
10.5
18.0
23.1
27.8
25.3
25.6
21.0
33.7
19.2
21.9
25.1
13.8
6.7
28.5
LI-Boost-MI
97.0*
45.3
55.2
62.0
41.8
19.1
29.1
38.7
41.9
35.5
34.8
30.3
34.3
33.3
23.8
32.2
21.4
7.2
37.9
VMI-FGSM
97.5*
54.4
58.0
66.0
51.2
28.6
40.5
46.5
49.4
44.5
43.5
38.9
36.0
44.4
25.3
44.1
21.9
8.2
44.4
LI-Boost-VMI
99.3*
67.0
71.4
79.3
65.9
39.7
53.7
61.4
62.6
57.8
57.1
52.8
37.9
59.1
30.4
60.4
36.8
9.0
55.6
PGN
99.1*
84.2
86.4
91.6
81.5
54.8
69.7
77.0
78.8
77.7
76.8
73.0
46.2
78.3
41.4
79.7
48.6
13.7
69.9
TABLE II : Attack success rates (%) on nine normally trained models and nine defense mechanisms of various gradient-based attacks w/wo LI-Boost. The adversarial examples are crafted on RN-50. * indicates the white-box model. The best results are bold , while methods incorporating LI-Boost are highlighted in gray .
Input Transforma- tion-based Attacks
CNNs
ViTs
Defenses
Avg.
RN-50
Inc-v3
MN-v3
DN-121
FSNet
ViT
PiT
Visformer
Swin
Inc-v3 ens3
Inc-v3 ens4
IncRes-v2 ens
AT
HGD
RS
NRP
DiffPure
MIMIR
DIM
92.7*
52.4
56.7
64.4
46.9
23.9
35.0
42.1
43.6
39.9
39.5
34.6
34.9
40.3
23.6
33.6
19.2
7.2
40.6
LI-Boost-DIM
98.1*
61.0
68.3
75.8
61.1
36.0
47.7
57.2
56.4
50.8
50.5
45.7
36.3
54.9
26.9
42.2
30.4
8.1
50.4
Admix
99.3*
59.4
67.4
77.6
54.6
27.7
41.8
52.5
53.3
43.2
43.0
36.8
35.7
47.9
24.6
44.4
20.8
7.9
46.6
LI-Boost- Admix
99.5*
71.7
80.5
86.5
73.9
44.8
58.5
70.5
69.2
61.7
61.0
56.3
38.4
66.9
30.9
57.7
38.4
9.9
59.8
SIA
99.3*
76.2
89.1
92.9
81.3
43.5
66.8
78.4
76.6
61.6
58.7
52.0
38.0
71.0
27.2
57.0
25.4
8.6
61.3
TABLE III : Attack success rates (%) on nine normally trained models and nine defense mechanisms of various input transformation-based attacks w/wo LI-Boost. The adversarial examples are crafted on RN-50. * indicates the white-box model. The best results are bold , while methods incorporating LI-Boost are highlighted in gray .
Model-related Attacks
CNNs
ViTs
Defenses
Avg.
RN-50
Inc-v3
MN-v3
DN-121
FSNet
ViT
PiT
Visformer
Swin
Inc-v3 ens3
Inc-v3 ens4
IncRes-v2 ens
AT
HGD
RS
NRP
DiffPure
MIMIR
SGM
99.5*
44.8
57.2
61.3
31.8
15.0
27.7
33.4
38.8
30.5
29.3
24.1
35.0
22.8
23.3
29.3
14.2
6.9
34.7
LI-Boost-SGM
100.0*
61.5
78.4
82.0
46.7
29.1
46.4
57.6
61.0
40.6
39.3
32.5
36.8
48.0
27.1
41.3
25.4
8.0
47.9
Linbp
89.2*
44.4
55.8
62.3
29.0
9.6
17.2
28.6
31.8
31.5
30.4
24.7
34.5
24.2
22.7
27.7
22.0
7.4
32.9
LI-Boost-Linbp
99.2*
60.1
76.9
85.4
52.4
15.2
25.5
49.9
44.9
45.8
43.0
36.0
34.6
43.7
24.1
32.8
23.0
7.5
44.4
BPA
89.9*
79.6
88.1
96.4
66.9
30.4
46.0
64.4
65.7
65.2
64.1
53.5
37.5
69.2
27.9
47.9
28.1
7.2
57.1
TABLE IV : Attack success rates (%) on nine normally trained models and nine defense mechanisms of various model-related attacks w/wo LI-Boost. The adversarial examples are crafted on RN-50, except for VDC and FPR, which are based on ViT. * indicates the white-box model. The best results are bold , while methods incorporating LI-Boost are highlighted in gray .
Advanced Objec- tive Functions
CNNs
ViTs
Defenses
Avg.
RN-50
Inc-v3
MN-v3
DN-121
FSNet
ViT
PiT
Visformer
Swin
Inc-v3 ens3
Inc-v3 ens4
IncRes-v2 ens
AT
HGD
RS
NRP
DiffPure
MIMIR
ILA
90.0*
29.0
37.9
42.4
22.4
8.2
15.4
20.8
27.1
21.9
21.4
16.6
33.4
13.9
21.6
20.0
11.4
5.8
25.5
LI-Boost-ILA
93.2*
41.3
56.7
64.6
36.2
12.2
23.2
33.4
39.1
30.6
30.4
24.1
33.8
24.3
22.6
24.6
14.1
6.4
33.9
FIA
77.8*
37.5
45.1
53.4
23.3
8.1
15.8
20.9
29.1
27.8
27.0
20.1
35.3
16.6
23.4
24.7
12.2
6.0
28.0
LI-Boost-FIA
89.6*
53.6
65.1
76.2
42.7
13.9
25.5
37.7
45.4
41.4
41.9
31.5
36.7
32.8
25.2
30.8
15.4
6.5
39.6
ILPD
95.0*
65.6
74.1
80.6
65.0
62.0
52.7
61.4
61.9
55.1
54.9
49.5
46.8
57.0
27.5
55.3
28.5
9.6
55.7
TABLE V : Attack success rates (%) on nine normally trained models and nine defense mechanisms of various advanced objective functions w/wo LI-Boost. The adversarial examples are crafted on RN-50. * indicates the white-box model. The best results are bold , while methods incorporating LI-Boost are highlighted in gray .
Ensemble Attacks
CNNs
ViTs
Defenses
Avg.
RN-50
Inc-v3
MN-v3
DN-121
FSNet
ViT
PiT
Visformer
Swin
Inc-v3 ens3
Inc-v3 ens4
IncRes-v2 ens
AT
HGD
RS
NRP
DiffPure
MIMIR
MI-FGSM ens
95.4*
99.8*
99.3*
100.0*
67.8
39.3
53.7
66.6
68.5
61.0
60.6
51.3
37.2
66.6
27.1
44.7
24.6
8.5
59.6
LI-Boost-MI ens
97.9*
100.0*
99.6*
100.0*
85.6
58.4
71.0
83.2
83.9
78.4
77.7
70.6
39.9
85.3
34.5
58.3
40.9
10.0
70.8
VMI-FGSM ens
97.3*
99.9*
99.4*
100.0*
84.7
60.1
73.0
82.4
83.3
80.6
79.6
73.1
40.5
84.5
33.4
66.0
39.7
13.9
71.7
LI-Boost-VMI ens
99.3*
100.0*
99.7*
100.0*
93.1
73.7
84.5
91.8
92.4
91.3
90.0
85.4
45.0
94.0
42.6
83.6
58.2
15.1
80.0
PGN ens
98.8*
100.0*
99.6*
100.0*
94.6
81.2
88.7
94.1
94.1
95.1
94.8
91.7
54.9
95.9
58.4
90.0
71.0
19.6
84.6
TABLE VI : Attack success rates (%) on nine normally trained models and nine defense mechanisms of various ensemble attacks w/wo LI-Boost. The adversarial examples are crafted on RN-50, Inc-v3, MN-v3, and DN-121. * indicates the white-box models. The best results are bold , while methods incorporating LI-Boost are highlighted in gray .
Method
Baidu
Alibaba
Tencent
Avg.
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
MI-FGSM
65.9
41.8
45.5
18.2
54.9
20.1
55.4
26.7
LI-Boost-MI
70.8
48.9
56.2
28.4
64.2
30.3
63.7
35.9
BSR
85.2
71.1
87.5
71.2
72.9
42.6
81.9
61.6
LI-Boost-BSR
89.3
77.7
92.5
81.2
81.3
55.2
87.7
71.4
ILA
60.7
35.3
36.3
11.6
50.9
17.3
49.3
21.4
TABLE VII : Attack success rates (%) of various adversarial attacks against three commercial vision API systems. The last two columns denote the average success rates across all platforms. The best results are bold while methods incorporating LI-Boost are highlighted in gray . The surrogate model is RN-50.
Attacks
Open-source
Closed-source
Avg.
Qwen3-VL
Qwen2.5-VL
LLaVA
Phi3.5
Intern-VL
GLM-V
GPT-4o
GPT-5.2
Gemini
Claude-s
Claude-o
Grok
MIFGSM
26.2
31.6
24.2
41.9
37.4
21.0
20.6
31.0
24.4
38.7
31.9
34.3
30.3
LI-Boost-MI
31.2
38.2
30.3
47.2
42.8
25.9
25.5
36.7
29.2
41.9
38.8
40.7
35.7
BSR
54.9
61.3
51.1
64.4
66.8
45.3
54.4
62.2
54.6
57.6
68.2
62.4
58.6
LI-Boost-BSR
64.6
70.3
60.9
70.3
72.3
58.0
59.7
70.0
59.8
66.5
76.0
70.0
66.5
ILA
24.8
30.1
22.9
40.7
34.7
19.8
20.5
28.4
22.3
33.1
28.4
30.9
28.1
TABLE VIII : Attack success rates (%) on six open-source and six closed-source VLMs of four attacks w/wo LI-Boost on image classification task. The best results are bold , while methods incorporating LI-Boost are highlighted in gray . The adversarial examples are crafted on RN-50.
Fig. 3 : Prompt for image classification task.
Fig. 4 : Visualizations and comparisons of clean images, adversarial examples generated through MI-FGSM and LI-Boost-MI against the RN-50 on Qwen3-VL classification task. Green text represents the ground-truth labels. Gray text denotes the results of failed MI-FGSM examples. Red text indicates the misclassification of LI-Boost-MI.
Fig. 5 : Prompt for image captioning task.
Method
GPT-4o
GPT-5.2
Gemini
Claude-s
Claude-o
Grok
Avg.
MI-FGSM
22.0
36.3
19.0
42.7
29.0
36.0
30.8
LI-Boost-MI
27.8
42.1
23.9
52.0
39.3
47.2
38.7
BSR
60.1
72.2
54.5
67.8
73.1
70.8
66.4
LI-Boost-BSR
68.2
80.4
62.3
78.7
81.8
84.3
76.0
ILA
19.0
33.6
16.9
35.7
23.6
29.5
26.4
LI-Boost-ILA
30.6
47.4
26.5
47.3
41.3
44.6
39.6
TABLE IX : Attack Success Rates (%) of Different Adversarial Methods against Various Commercial VLMs on Image Captioning Task. The best results are bold while methods incorporating LI-Boost are highlighted in gray . The surrogate model is RN-50.
Fig. 6 : Visualizations of the clean image and its adversarial counterparts with captions generated by Grok. Green text denotes the ground truth or benign description. Gray text represents the result where BSR fails to induce significant semantic change. Red text highlights the significant semantic deviation achieved by LI-Boost-BSR.
Fig. 7: Ablation studies of various sampling distributions.
Fig. 8 : Attack success rates (%) on nine models with various hyper-parameters N and k . The adversarial examples are generated by LI-Boost-MI on RN-50.
Surrogate Model
Attack
RN-50
Inc-v3
MN-v3
DN-121
FSNet
ViT
PiT
Visformer
Swin
RN-50
TIM
82.8*
35.0
40.4
50.3
27.4
12.7
18.2
23.8
29.0
LI-Boost-TIM
88.4*
36.7
44.0
55.1
35.4
16.0
21.5
29.8
34.1
Inc-v3
TIM
37.9
99.5*
48.0
57.1
31.8
15.8
21.1
26.6
33.7
LI-Boost-TIM
44.2
99.5*
53.0
63.7
37.5
17.8
23.3
31.6
38.0
MN-v3
TIM
45.5
47.3
99.6*
66.8
35.5
22.3
28.0
40.4
48.0
LI-Boost-TIM
54.6
50.4
99.7*
74.1
43.0
26.1
32.7
47.6
54.7
TABLE X : Attack success rates (%) of TIM w/o LI-Boost on nine models. For the single-model settings, the adversarial examples are crafted on RN-50, Inc-v3, MN-v3, DN-121, FSNet, ViT, PiT, Visformer, and Swin, respectively. For the ensemble setting, RN-50, Inc-v3, MN-v3, and DN-121 are adopted as surrogate models. * indicates the white-box model.
Fig. 9 : Attack success rates (%) under equal-compute comparisons. MI (300 iters) matches the total gradient evaluations of LI-Boost-MI ( N=30 , 10 iterations). The adversarial examples are generated on RN-50.
Fig. 10 : Attack success rates (%) of MI-FGSM and LI-Boost-MI under varying perturbation budgets, averaged over all nine normally trained models with RN-50 as the surrogate.
Metric
LI-Boost
MI-FGSM
BSR
BPA
ILA
FPR
ASR ↑
✗
35.6
83.7
69.7
32.5
56.4
✓
47.8
91.0
76.5
44.4
64.1
LI ↑
✗
0.24
0.55
0.36
0.15
0.30
✓
0.41
0.92
0.56
0.34
0.42
Rξ(0)↓
✗
0.51
2.75
2.08
2.94
2.67
✓
0.49
2.53
1.84
2.88
2.28
TABLE XI : Average attack success rates on nine normally trained models(%), local invariance ( k=6 ), average-case flatness Rξ(0) , and time consumption (s/img) of various attacks w/wo LI-Boost on RN-50, except for FPR, which generates examples on ViT.
Category
Method
Parameters
Gradient-based Attacks
MI-FGSM [ 24 ]
perturbation budget ϵ=16/255 ,
number of iterations T=10 ,
step size α=ϵ/T=1.6/255 ,
decay factor μ=1.0
VMI-FGSM [ 26 ]
number of sampled examples Ns=20 ,
upper bound of neighborhood ζ=1.5
Table 22
Attacks
Inc-v3 ⟹
MN-v3 ⟹
RN-50
MN-v3
DN-121
FSNet
ViT
PiT
Visformer
Swin
RN-50
Inc-v3
DN-121
FSNet
ViT
PiT
Visformer
Swin
MI-FGSM
34.1
46.7
50.6
25.7
14.2
20.5
26.6
31.8
41.7
50.6
60.0
31.4
17.9
26.6
36.9
42.7
LI-Boost-MI
43.8
56.9
62.6
37.3
20.8
26.4
35.0
40.9
65.6
67.6
80.7
53.3
33.0
46.1
61.1
65.7
VMI-FGSM
50.0
60.3
66.3
41.6
25.5
32.7
39.7
44.8
67.5
73.0
80.7
57.6
37.2
51.6
64.2
69.7
LI-Boost-VMI
52.5
62.0
69.1
44.2
27.0
35.1
42.6
48.0
75.5
79.0
87.2
66.0
44.5
58.6
71.7
77.2
PGN
63.1
75.7
81.2
55.7
55.7
43.7
51.7
57.3
80.3
86.5
92.1
70.9
49.6
64.7
76.4
82.5
TABLE A-2 : Transfer attack success rates (%) of gradient-based attacks using various surrogate models. The best results are bold , while methods incorporating LI-Boost are highlighted in gray .
Attacks
Inc-v3 ⟹
MN-v3 ⟹
RN50
MN-v3
DN-121
FSNet
ViT
PiT
Visformer
Swin
RN50
Inc-v3
DN-121
FSNet
ViT
PiT
Visformer
Swin
DIM
46.0
58.6
65.0
39.2
22.0
29.3
36.3
41.6
64.7
74.1
81.9
54.4
36.4
50.0
62.0
66.8
LI-Boost-DIM
55.6
67.1
74.1
50.0
29.5
35.6
46.3
51.8
80.2
83.0
91.0
71.4
51.7
63.3
77.2
80.4
Admix
56.3
68.3
75.4
45.5
25.3
33.8
43.6
48.7
70.9
75.8
85.3
58.1
36.5
53.0
66.8
71.9
LI-Boost- Admix
79.8
84.9
91.7
62.7
49.0
52.6
67.0
70.0
85.9
88.6
94.3
76.5
59.1
70.5
83.4
85.2
SIA
77.5
88.0
90.6
66.9
39.3
52.9
66.0
68.7
82.2
82.6
92.6
71.3
46.0
65.5
79.4
83.1
TABLE A-3 : Transfer attack success rates (%) of input transformation-based attacks using various surrogate models. The best results are bold , while methods incorporating LI-Boost are highlighted in gray .
Fig. A-1 : Visualization of clean images and their adversarial counterparts across different attack methods against Alibaba vision API systems, and the surrogate model is RN-50. Green text represents the ground-truth labels. Gray text denotes the results of failed adversarial examples. Red text indicates the misclassification of LI-Boost enhanced methods.
Fig. A-2 : Visualization of clean images and their adversarial counterparts across different attack methods against Claude-o, and the surrogate model is RN-50. Green text denotes the ground truth captions. Gray text denotes the results of failed adversarial examples. Red text highlights the significant semantic deviation achieved by our LI-Boosted methods.
Input transformation-based attacks improve adversarial transferability by aggregating gradients over transformed inputs. Existing analyses mainly explain their efficacy from image diversity, semantic preservation, attention variance or hypothesis space augmentation, yet overlook the critical role of model frontend responses. In this paper, we revisit transformation-based attacks from an implicit ensemble perspective: each transformation can be viewed as a pre-processing operator before the surrogate model, inducing a distinct frontend response for gradient aggregation. Based on this view, we propose FRO, a Frontend Response-Oriented input transformation method that enriches such responses through two complementary operators. The Local Scaling Operator perturbs local content sampling via block-wise stretch-and-shrink operations, while the Projection Operator modifies global spatial organization through coherent perspective deformation. Together, they produce structured transformed views to optimize transferable adversarial perturbations. Experiments on an ImageNet subset show that FRO consistently improves black-box transferability across diverse CNN and Vision Transformer models. We further analyze the effect of implicit ensemble size and evaluate different transformation-based methods under a unified ensemble scale, demonstrating the superiority of designing input transformations from the perspective of front-end response ensembles.
Transfer-based adversarial attacks often transfer poorly across heterogeneous architectures because CNNs favor local textures while Vision Transformers (ViTs) rely on global shapes. We propose Season, a spectrum-aware orthogonal gradient refinement framework for L-infinity transfer attacks against black-box target models on ImageNet, using a white-box surrogate. Season decomposes each update into a low-frequency branch capturing structural cues and a high-frequency branch capturing textures. A low-saliency guidance scheme reallocates high-frequency energy to background regions, preserving foreground structures that ViTs depend on. An orthogonal projection then forces the textural update to lie in the orthogonal complement of the structural direction, mitigating feature interference. As a training-free plug-and-play wrapper, Season enhances eight gradient-stabilization and input-enhancement attacks without modifying their cores. Across eight CNN, ViT, and MLP targets, Season improves transfer success rate by 6.6 percentage points on average and up to 16.0 points over strong baselines under a unified protocol.
Tianyi Wang, Zhenghao Gao, Shengjie Xu
Tongji University Shanghai, China · Huazhong University of Science and Technology · Wuhan LightRead Intelligent Technology Co., Ltd. Wuhan, China
Adversarial examples reveal vulnerabilities in Vision-Language Pre-training (VLP) models and provide insights for improving robustness. A key property is cross-model transferability, which enables transfer-based black-box attacks. However, existing attacks often rely heavily on the surrogate model, causing cross-model performance drops. One reason is that adversarial optimization may follow surrogate model responses more than input semantics, making the update direction effective on the surrogate but less transferable to unseen targets. We refer to this dependency as surrogate-specific bias. Motivated by this observation, DeBias-Attack improves transferability by correcting surrogate-specific bias in adversarial optimization directions. It maintains two perturbation branches. The main branch optimizes a perturbation on the original image and obtains the adversarial gradient used to disrupt image-text alignment. The reference branch optimizes a perturbation on a weak-semantic image constructed from the dataset mean image with small Gaussian noise resampled at each iteration. Since this weak-semantic image contains little clear visual content, its optimization reflects surrogate responses more than image semantics, and its reference gradient estimates surrogate-specific bias. DeBias-Attack removes the aligned projection of the main gradient on the reference gradient before updating the adversarial image, then performs context-aware text substitution using the updated adversarial image. DeBias-Attack is the first transfer-based VLP attack that corrects surrogate-specific bias through gradient correction. Experiments show strong performance across VLP models, downstream tasks, and open-source and closed-source multimodal large language models.
Lijia Yu, Jiuxin Cao, Yuchen Qiang +3
School of Cyber Science and Engineering, Southeast University, China · Purple Mountain Laboratories, Nanjing, China