Boosting the Local Invariance for Better Adversarial Transferability

Bohan Liu; Xiaosen Wang

cs.CVMar 8, 2025

Boosting the Local Invariance for Better Adversarial Transferability

Authors: Bohan Liu, Xiaosen Wang

Organizations: School of Computer Science and Technology, Xidian University, Xi’an 710126, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, China

Abstract

Transfer-based attacks pose a significant threat to real-world applications by directly targeting victim models with adversarial examples generated on surrogate models. While numerous approaches have been proposed to enhance adversarial transferability, existing works often overlook the intrinsic relationship between adversarial perturbations and input images. In this work, we find that the adversarial perturbations often exhibit poor translation invariance for a given clean image and model, which is attributed to local invariance. Through empirical analysis, we demonstrate a positive correlation between the local invariance of adversarial perturbations w.r.t. the input image and their transferability across models. Based on this finding, we propose a general adversarial transferability boosting technique called the Local Invariance Boosting approach (LI-Boost). Extensive experiments on the standard ImageNet dataset demonstrate that LI-Boost significantly enhances five categories of transfer-based attacks, i.e., gradient-based, input transformation-based, model-related, advanced objective function, and ensemble attacks. The improvements hold not only on conventional CNNs, ViTs, and defense mechanisms, but also on real-world commercial vision API systems and vision-language models. Our approach provides a promising direction for future research on improving adversarial transferability across models. Our code is available at https://github.com/Trustworthy-AI-Group/TransferAttack.

Figures & tables

Fig. 1 : The impact of translation invariance of clean image and adversarial perturbation. The translated clean image can be correctly recognized by deep models (either CNNs or ViTs), whereas the slightly translated adversarial perturbations cannot consistently fool these models, and our LI-Boost enhanced perturbation can induce them to make incorrect predictions.

Panel of Figure 2 — Fig. 2 : The local invariance of various transfer-based attacks (left) and the attack performance of MI-FGSM with enhanced local invariance (right).

$k$	0	1	2	3	4	5	6
Time (s/img)	0.07	0.69	1.92	3.78	6.26	9.35	13.07
BP (count)	1	9	25	49	81	121	169

TABLE I : Time consumption and computation complexity w.r.t. various k . We report average runtime(seconds/image) and computational cost(Backpropagations/BP).

Gradient-based Attacks	CNNs					ViTs				Defenses									Avg.
Gradient-based Attacks	RN-50	Inc-v3	MN-v3	DN-121	FSNet	ViT	PiT	Visformer	Swin	Inc-v3 ens3	Inc-v3 ens4	IncRes-v2 ens	AT	HGD	RS	NRP	DiffPure	MIMIR	Avg.
MI-FGSM	94.9*	34.5	40.6	45.9	25.2	10.5	18.0	23.1	27.8	25.3	25.6	21.0	33.7	19.2	21.9	25.1	13.8	6.7	28.5
LI-Boost-MI	97.0*	45.3	55.2	62.0	41.8	19.1	29.1	38.7	41.9	35.5	34.8	30.3	34.3	33.3	23.8	32.2	21.4	7.2	37.9
VMI-FGSM	97.5*	54.4	58.0	66.0	51.2	28.6	40.5	46.5	49.4	44.5	43.5	38.9	36.0	44.4	25.3	44.1	21.9	8.2	44.4
LI-Boost-VMI	99.3*	67.0	71.4	79.3	65.9	39.7	53.7	61.4	62.6	57.8	57.1	52.8	37.9	59.1	30.4	60.4	36.8	9.0	55.6
PGN	99.1*	84.2	86.4	91.6	81.5	54.8	69.7	77.0	78.8	77.7	76.8	73.0	46.2	78.3	41.4	79.7	48.6	13.7	69.9

TABLE II : Attack success rates (%) on nine normally trained models and nine defense mechanisms of various gradient-based attacks w/wo LI-Boost. The adversarial examples are crafted on RN-50. * indicates the white-box model. The best results are bold , while methods incorporating LI-Boost are highlighted in gray .

Input Transforma- tion-based Attacks	CNNs					ViTs				Defenses									Avg.
Input Transforma- tion-based Attacks	RN-50	Inc-v3	MN-v3	DN-121	FSNet	ViT	PiT	Visformer	Swin	Inc-v3 ens3	Inc-v3 ens4	IncRes-v2 ens	AT	HGD	RS	NRP	DiffPure	MIMIR	Avg.
DIM	92.7*	52.4	56.7	64.4	46.9	23.9	35.0	42.1	43.6	39.9	39.5	34.6	34.9	40.3	23.6	33.6	19.2	7.2	40.6
LI-Boost-DIM	98.1*	61.0	68.3	75.8	61.1	36.0	47.7	57.2	56.4	50.8	50.5	45.7	36.3	54.9	26.9	42.2	30.4	8.1	50.4
Admix	99.3*	59.4	67.4	77.6	54.6	27.7	41.8	52.5	53.3	43.2	43.0	36.8	35.7	47.9	24.6	44.4	20.8	7.9	46.6
LI-Boost- Admix	99.5*	71.7	80.5	86.5	73.9	44.8	58.5	70.5	69.2	61.7	61.0	56.3	38.4	66.9	30.9	57.7	38.4	9.9	59.8
SIA	99.3*	76.2	89.1	92.9	81.3	43.5	66.8	78.4	76.6	61.6	58.7	52.0	38.0	71.0	27.2	57.0	25.4	8.6	61.3

TABLE III : Attack success rates (%) on nine normally trained models and nine defense mechanisms of various input transformation-based attacks w/wo LI-Boost. The adversarial examples are crafted on RN-50. * indicates the white-box model. The best results are bold , while methods incorporating LI-Boost are highlighted in gray .

Model-related Attacks	CNNs					ViTs				Defenses									Avg.
Model-related Attacks	RN-50	Inc-v3	MN-v3	DN-121	FSNet	ViT	PiT	Visformer	Swin	Inc-v3 ens3	Inc-v3 ens4	IncRes-v2 ens	AT	HGD	RS	NRP	DiffPure	MIMIR	Avg.
SGM	99.5*	44.8	57.2	61.3	31.8	15.0	27.7	33.4	38.8	30.5	29.3	24.1	35.0	22.8	23.3	29.3	14.2	6.9	34.7
LI-Boost-SGM	100.0*	61.5	78.4	82.0	46.7	29.1	46.4	57.6	61.0	40.6	39.3	32.5	36.8	48.0	27.1	41.3	25.4	8.0	47.9
Linbp	89.2*	44.4	55.8	62.3	29.0	9.6	17.2	28.6	31.8	31.5	30.4	24.7	34.5	24.2	22.7	27.7	22.0	7.4	32.9
LI-Boost-Linbp	99.2*	60.1	76.9	85.4	52.4	15.2	25.5	49.9	44.9	45.8	43.0	36.0	34.6	43.7	24.1	32.8	23.0	7.5	44.4
BPA	89.9*	79.6	88.1	96.4	66.9	30.4	46.0	64.4	65.7	65.2	64.1	53.5	37.5	69.2	27.9	47.9	28.1	7.2	57.1

TABLE IV : Attack success rates (%) on nine normally trained models and nine defense mechanisms of various model-related attacks w/wo LI-Boost. The adversarial examples are crafted on RN-50, except for VDC and FPR, which are based on ViT. * indicates the white-box model. The best results are bold , while methods incorporating LI-Boost are highlighted in gray .

Advanced Objec- tive Functions	CNNs					ViTs				Defenses									Avg.
Advanced Objec- tive Functions	RN-50	Inc-v3	MN-v3	DN-121	FSNet	ViT	PiT	Visformer	Swin	Inc-v3 ens3	Inc-v3 ens4	IncRes-v2 ens	AT	HGD	RS	NRP	DiffPure	MIMIR	Avg.
ILA	90.0*	29.0	37.9	42.4	22.4	8.2	15.4	20.8	27.1	21.9	21.4	16.6	33.4	13.9	21.6	20.0	11.4	5.8	25.5
LI-Boost-ILA	93.2*	41.3	56.7	64.6	36.2	12.2	23.2	33.4	39.1	30.6	30.4	24.1	33.8	24.3	22.6	24.6	14.1	6.4	33.9
FIA	77.8*	37.5	45.1	53.4	23.3	8.1	15.8	20.9	29.1	27.8	27.0	20.1	35.3	16.6	23.4	24.7	12.2	6.0	28.0
LI-Boost-FIA	89.6*	53.6	65.1	76.2	42.7	13.9	25.5	37.7	45.4	41.4	41.9	31.5	36.7	32.8	25.2	30.8	15.4	6.5	39.6
ILPD	95.0*	65.6	74.1	80.6	65.0	62.0	52.7	61.4	61.9	55.1	54.9	49.5	46.8	57.0	27.5	55.3	28.5	9.6	55.7

TABLE V : Attack success rates (%) on nine normally trained models and nine defense mechanisms of various advanced objective functions w/wo LI-Boost. The adversarial examples are crafted on RN-50. * indicates the white-box model. The best results are bold , while methods incorporating LI-Boost are highlighted in gray .

Ensemble Attacks	CNNs					ViTs				Defenses									Avg.
Ensemble Attacks	RN-50	Inc-v3	MN-v3	DN-121	FSNet	ViT	PiT	Visformer	Swin	Inc-v3 ens3	Inc-v3 ens4	IncRes-v2 ens	AT	HGD	RS	NRP	DiffPure	MIMIR	Avg.
MI-FGSM ens	95.4*	99.8*	99.3*	100.0*	67.8	39.3	53.7	66.6	68.5	61.0	60.6	51.3	37.2	66.6	27.1	44.7	24.6	8.5	59.6
LI-Boost-MI ens	97.9*	100.0*	99.6*	100.0*	85.6	58.4	71.0	83.2	83.9	78.4	77.7	70.6	39.9	85.3	34.5	58.3	40.9	10.0	70.8
VMI-FGSM ens	97.3*	99.9*	99.4*	100.0*	84.7	60.1	73.0	82.4	83.3	80.6	79.6	73.1	40.5	84.5	33.4	66.0	39.7	13.9	71.7
LI-Boost-VMI ens	99.3*	100.0*	99.7*	100.0*	93.1	73.7	84.5	91.8	92.4	91.3	90.0	85.4	45.0	94.0	42.6	83.6	58.2	15.1	80.0
PGN ens	98.8*	100.0*	99.6*	100.0*	94.6	81.2	88.7	94.1	94.1	95.1	94.8	91.7	54.9	95.9	58.4	90.0	71.0	19.6	84.6

TABLE VI : Attack success rates (%) on nine normally trained models and nine defense mechanisms of various ensemble attacks w/wo LI-Boost. The adversarial examples are crafted on RN-50, Inc-v3, MN-v3, and DN-121. * indicates the white-box models. The best results are bold , while methods incorporating LI-Boost are highlighted in gray .

Method	Baidu		Alibaba		Tencent		Avg.
Method	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
MI-FGSM	65.9	41.8	45.5	18.2	54.9	20.1	55.4	26.7
LI-Boost-MI	70.8	48.9	56.2	28.4	64.2	30.3	63.7	35.9
BSR	85.2	71.1	87.5	71.2	72.9	42.6	81.9	61.6
LI-Boost-BSR	89.3	77.7	92.5	81.2	81.3	55.2	87.7	71.4
ILA	60.7	35.3	36.3	11.6	50.9	17.3	49.3	21.4

TABLE VII : Attack success rates (%) of various adversarial attacks against three commercial vision API systems. The last two columns denote the average success rates across all platforms. The best results are bold while methods incorporating LI-Boost are highlighted in gray . The surrogate model is RN-50.

Attacks	Open-source						Closed-source						Avg.
Attacks	Qwen3-VL	Qwen2.5-VL	LLaVA	Phi3.5	Intern-VL	GLM-V	GPT-4o	GPT-5.2	Gemini	Claude-s	Claude-o	Grok	Avg.
MIFGSM	26.2	31.6	24.2	41.9	37.4	21.0	20.6	31.0	24.4	38.7	31.9	34.3	30.3
LI-Boost-MI	31.2	38.2	30.3	47.2	42.8	25.9	25.5	36.7	29.2	41.9	38.8	40.7	35.7
BSR	54.9	61.3	51.1	64.4	66.8	45.3	54.4	62.2	54.6	57.6	68.2	62.4	58.6
LI-Boost-BSR	64.6	70.3	60.9	70.3	72.3	58.0	59.7	70.0	59.8	66.5	76.0	70.0	66.5
ILA	24.8	30.1	22.9	40.7	34.7	19.8	20.5	28.4	22.3	33.1	28.4	30.9	28.1

TABLE VIII : Attack success rates (%) on six open-source and six closed-source VLMs of four attacks w/wo LI-Boost on image classification task. The best results are bold , while methods incorporating LI-Boost are highlighted in gray . The adversarial examples are crafted on RN-50.

Panel of Figure 11 — Fig. 3 : Prompt for image classification task.

Panel of Figure 12 — Fig. 4 : Visualizations and comparisons of clean images, adversarial examples generated through MI-FGSM and LI-Boost-MI against the RN-50 on Qwen3-VL classification task. Green text represents the ground-truth labels. Gray text denotes the results of failed MI-FGSM examples. Red text indicates the misclassification of LI-Boost-MI.

Panel of Figure 13 — Fig. 5 : Prompt for image captioning task.

Method	GPT-4o	GPT-5.2	Gemini	Claude-s	Claude-o	Grok	Avg.
MI-FGSM	22.0	36.3	19.0	42.7	29.0	36.0	30.8
LI-Boost-MI	27.8	42.1	23.9	52.0	39.3	47.2	38.7
BSR	60.1	72.2	54.5	67.8	73.1	70.8	66.4
LI-Boost-BSR	68.2	80.4	62.3	78.7	81.8	84.3	76.0
ILA	19.0	33.6	16.9	35.7	23.6	29.5	26.4
LI-Boost-ILA	30.6	47.4	26.5	47.3	41.3	44.6	39.6

TABLE IX : Attack Success Rates (%) of Different Adversarial Methods against Various Commercial VLMs on Image Captioning Task. The best results are bold while methods incorporating LI-Boost are highlighted in gray . The surrogate model is RN-50.

Panel of Figure 15 — Fig. 6 : Visualizations of the clean image and its adversarial counterparts with captions generated by Grok. Green text denotes the ground truth or benign description. Gray text represents the result where BSR fails to induce significant semantic change. Red text highlights the significant semantic deviation achieved by LI-Boost-BSR.

Panel of Figure 16 — Fig. 7: Ablation studies of various sampling distributions.

Panel of Figure 17 — Fig. 8 : Attack success rates (%) on nine models with various hyper-parameters $N$ and $k$ . The adversarial examples are generated by LI-Boost-MI on RN-50.

Surrogate Model	Attack	RN-50	Inc-v3	MN-v3	DN-121	FSNet	ViT	PiT	Visformer	Swin
RN-50	TIM	82.8*	35.0	40.4	50.3	27.4	12.7	18.2	23.8	29.0
RN-50	LI-Boost-TIM	88.4*	36.7	44.0	55.1	35.4	16.0	21.5	29.8	34.1
Inc-v3	TIM	37.9	99.5*	48.0	57.1	31.8	15.8	21.1	26.6	33.7
Inc-v3	LI-Boost-TIM	44.2	99.5*	53.0	63.7	37.5	17.8	23.3	31.6	38.0
MN-v3	TIM	45.5	47.3	99.6*	66.8	35.5	22.3	28.0	40.4	48.0
MN-v3	LI-Boost-TIM	54.6	50.4	99.7*	74.1	43.0	26.1	32.7	47.6	54.7

TABLE X : Attack success rates (%) of TIM w/o LI-Boost on nine models. For the single-model settings, the adversarial examples are crafted on RN-50, Inc-v3, MN-v3, DN-121, FSNet, ViT, PiT, Visformer, and Swin, respectively. For the ensemble setting, RN-50, Inc-v3, MN-v3, and DN-121 are adopted as surrogate models. * indicates the white-box model.

Panel of Figure 19 — Fig. 9 : Attack success rates (%) under equal-compute comparisons. MI (300 iters) matches the total gradient evaluations of LI-Boost-MI ( $N\!=\!30$ , 10 iterations). The adversarial examples are generated on RN-50.

Panel of Figure 20 — Fig. 10 : Attack success rates (%) of MI-FGSM and LI-Boost-MI under varying perturbation budgets, averaged over all nine normally trained models with RN-50 as the surrogate.

Metric	LI-Boost	MI-FGSM	BSR	BPA	ILA	FPR
ASR $\uparrow$	✗	35.6	83.7	69.7	32.5	56.4
ASR $\uparrow$	✓	47.8	91.0	76.5	44.4	64.1
LI $\uparrow$	✗	0.24	0.55	0.36	0.15	0.30
LI $\uparrow$	✓	0.41	0.92	0.56	0.34	0.42
$R^{(0)}_{\xi}\downarrow$	✗	0.51	2.75	2.08	2.94	2.67
$R^{(0)}_{\xi}\downarrow$	✓	0.49	2.53	1.84	2.88	2.28

TABLE XI : Average attack success rates on nine normally trained models(%), local invariance (

k=6

), average-case flatness

R^{(0)}_{\xi}

, and time consumption (s/img) of various attacks w/wo LI-Boost on RN-50, except for FPR, which generates examples on ViT.

Category	Method	Parameters
Gradient-based Attacks	MI-FGSM [ 24 ]	perturbation budget $\epsilon=16/255$ ,
		number of iterations $T=10$ ,
		step size $\alpha=\epsilon/T=1.6/255$ ,
		decay factor $\mu=1.0$
	VMI-FGSM [ 26 ]	number of sampled examples $N_{s}=20$ ,
	VMI-FGSM [ 26 ]	upper bound of neighborhood $\zeta=1.5$

Table 22

Attacks	Inc-v3 $\Longrightarrow$								MN-v3 $\Longrightarrow$
Attacks	RN-50	MN-v3	DN-121	FSNet	ViT	PiT	Visformer	Swin	RN-50	Inc-v3	DN-121	FSNet	ViT	PiT	Visformer	Swin
MI-FGSM	34.1	46.7	50.6	25.7	14.2	20.5	26.6	31.8	41.7	50.6	60.0	31.4	17.9	26.6	36.9	42.7
LI-Boost-MI	43.8	56.9	62.6	37.3	20.8	26.4	35.0	40.9	65.6	67.6	80.7	53.3	33.0	46.1	61.1	65.7
VMI-FGSM	50.0	60.3	66.3	41.6	25.5	32.7	39.7	44.8	67.5	73.0	80.7	57.6	37.2	51.6	64.2	69.7
LI-Boost-VMI	52.5	62.0	69.1	44.2	27.0	35.1	42.6	48.0	75.5	79.0	87.2	66.0	44.5	58.6	71.7	77.2
PGN	63.1	75.7	81.2	55.7	55.7	43.7	51.7	57.3	80.3	86.5	92.1	70.9	49.6	64.7	76.4	82.5

TABLE A-2 : Transfer attack success rates (%) of gradient-based attacks using various surrogate models. The best results are bold , while methods incorporating LI-Boost are highlighted in gray .

Attacks	Inc-v3 $\Longrightarrow$								MN-v3 $\Longrightarrow$
Attacks	RN50	MN-v3	DN-121	FSNet	ViT	PiT	Visformer	Swin	RN50	Inc-v3	DN-121	FSNet	ViT	PiT	Visformer	Swin
DIM	46.0	58.6	65.0	39.2	22.0	29.3	36.3	41.6	64.7	74.1	81.9	54.4	36.4	50.0	62.0	66.8
LI-Boost-DIM	55.6	67.1	74.1	50.0	29.5	35.6	46.3	51.8	80.2	83.0	91.0	71.4	51.7	63.3	77.2	80.4
Admix	56.3	68.3	75.4	45.5	25.3	33.8	43.6	48.7	70.9	75.8	85.3	58.1	36.5	53.0	66.8	71.9
LI-Boost- Admix	79.8	84.9	91.7	62.7	49.0	52.6	67.0	70.0	85.9	88.6	94.3	76.5	59.1	70.5	83.4	85.2
SIA	77.5	88.0	90.6	66.9	39.3	52.9	66.0	68.7	82.2	82.6	92.6	71.3	46.0	65.5	79.4	83.1

TABLE A-3 : Transfer attack success rates (%) of input transformation-based attacks using various surrogate models. The best results are bold , while methods incorporating LI-Boost are highlighted in gray .

Panel of Figure 25 — Fig. A-1 : Visualization of clean images and their adversarial counterparts across different attack methods against Alibaba vision API systems, and the surrogate model is RN-50. Green text represents the ground-truth labels. Gray text denotes the results of failed adversarial examples. Red text indicates the misclassification of LI-Boost enhanced methods.

Panel of Figure 26 — Fig. A-2 : Visualization of clean images and their adversarial counterparts across different attack methods against Claude-o, and the surrogate model is RN-50. Green text denotes the ground truth captions. Gray text denotes the results of failed adversarial examples. Red text highlights the significant semantic deviation achieved by our LI-Boosted methods.

Explore similar work

Enhancing Adversarial Transferability through Block Stretch and Shrink
Nov 21, 2025Quan Liu, Feng Ye, Chenhao Lu +4Adversarial Attacks Adversarial Perturbations
Season: Spectrum-Aware Orthogonal Gradient Refinement for Transfer-Based Adversarial Attacks
Aug 5, 2026Tianyi Wang, Zhenghao Gao, Shengjie XuAdversarial Attacks Transfer-Based Adversarial Attacks
Tongji University Shanghai, China · Huazhong University of Science and Technology · Wuhan LightRead Intelligent Technology Co., Ltd. Wuhan, China
Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correction
Jun 9, 2026Lijia Yu, Jiuxin Cao, Yuchen Qiang +3Adversarial Attacks on VLMs Transfer-Based Adversarial Attacks
School of Cyber Science and Engineering, Southeast University, China · Purple Mountain Laboratories, Nanjing, China

Nov 21, 2025, cs.LGJ/K move · Enter open · S save

Enhancing Adversarial Transferability through Block Stretch and Shrink

Quan Liu, Feng Ye, Chenhao Lu, Shuming Zhen+3

Adversarial Attacks Adversarial Perturbations Transfer-Based Adversarial Attacks Black-Box Adversarial Attacks Adversarial Transferability

Open paper AskPDF arXiv

Input transformation-based attacks improve adversarial transferability by aggregating gradients over transformed inputs. Existing analyses mainly explain their efficacy from image diversity, semantic preservation, attention variance or hypothesis space augmentation, yet overlook the critical role of model frontend responses. In this paper, we revisit transformation-based attacks from an implicit ensemble perspective: each transformation can be viewed as a pre-processing operator before the surrogate model, inducing a distinct frontend response for gradient aggregation. Based on this view, we propose FRO, a Frontend Response-Oriented input transformation method that enriches such responses through two complementary operators. The Local Scaling Operator perturbs local content sampling via block-wise stretch-and-shrink operations, while the Projection Operator modifies global spatial organization through coherent perspective deformation. Together, they produce structured transformed views to optimize transferable adversarial perturbations. Experiments on an ImageNet subset show that FRO consistently improves black-box transferability across diverse CNN and Vision Transformer models. We further analyze the effect of implicit ensemble size and evaluate different transformation-based methods under a unified ensemble scale, demonstrating the superiority of designing input transformations from the perspective of front-end response ensembles.

Abstract

Explore similar work

Enhancing Adversarial Transferability through Block Stretch and Shrink

Season: Spectrum-Aware Orthogonal Gradient Refinement for Transfer-Based Adversarial Attacks

Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correction