Selective Channel Restoration for Backdoored Vision-Language Models

Shuming Liu; Zhifang Zhang; Suqin Yuan; Khin Mi Mi Aung; Zhuoyi Lin; Lei Feng

cs.CRSep 29, 2026

Selective Channel Restoration for Backdoored Vision-Language Models

Authors: Shuming Liu, Zhifang Zhang, Suqin Yuan, Khin Mi Mi Aung, Zhuoyi Lin, Lei Feng

Organizations: Southeast University · The University of Queensland · University of Sydney · A*STAR

Abstract

Vision-language models (VLMs) exhibit strong multimodal capabilities but remain vulnerable to backdoors implanted through poisoned fine-tuning data. Existing defenses often require extensive parameter updates during fine-tuning or incur per-query overhead during inference. To address these limitations, we propose Perturb-Select-Restore (PSR), a post-training defense that performs sparse updates to the projection interface and introduces no additional computation during inference. We reveal that backdoored VLM projectors are substantially more sensitive to bounded perturbations than clean VLM projectors, a phenomenon we term projection fragility. Building on this finding, PSR identifies the output channels most sensitive to perturbations in each projection layer of a backdoored VLM and restores their parameters to the corresponding pretrained values. Experiments across multiple tasks show that PSR reduces attack success rates to near zero while preserving clean-task performance.

Figures & tables

Figure 1: Loss increase

\Delta L

under bounded projector perturbations on clean samples. (a) COCO; (b) VQAv2. Higher

\Delta L

indicates greater projection fragility.

Figure 2: Overview of Perturb-Select-Restore (PSR), a post-training purification framework for VLM projection interfaces.

		BadNet		Blended		ISSBA		TrojVLM		VLOOD		WaNet
Model	Method	ASR	CU	ASR	CU	ASR	CU	ASR	CU	ASR	CU	ASR	CU
LLaVA-1.5	No defense	99.38	120.70	95.42	124.17	97.24	123.69	93.46	117.77	98.70	120.88	92.36	123.70
	Clean FT	99.36	120.93	0.00	123.98	1.82	123.89	93.46	117.88	98.68	121.01	0.00	123.82
	Random	0.66	121.96	1.82	123.28	20.62	124.82	3.40	124.04	34.08	124.10	16.92	124.66
	FP	0.76	121.92	5.20	122.58	17.62	124.96	3.20	122.89	33.96	123.35	25.12	123.40
	CLP	0.08	122.08	1.88	123.95	3.92	125.07	1.24	123.87	9.90	124.24	1.26	124.96

Table 1: Results on image captioning (COCO). ASR (%,

\downarrow

) and clean utility (CU; CIDEr on clean input,

\uparrow

) are reported across attacks and defenses.

Figure 3: Ablation of the PSR restoration ratio on Qwen3-VL-8B COCO attack settings. (a) ASR (

\downarrow

) on triggered images; (b) clean utility (CU; CIDEr,

\uparrow

) on clean images. Each curve reuses the same optimized robustness scores and restores selected parameters from

\phi_{0}

. Diamonds denote the shared result of all four attacks at 100% restoration.

	Layer coverage: ASR (%, $\downarrow$ )			CU ( $\uparrow$ )		ASR (%, $\downarrow$ )
Attack	First group	Second group	Both groups	From $\phi_{0}$	To zero	From $\phi_{0}$	To zero
BadNet	97.52	67.02	0.90	124.42	107.89	0.90	0.00
TrojVLM	89.28	64.08	0.32	125.37	115.97	0.32	0.00
Blended	82.74	55.84	1.70	123.03	97.37	1.70	0.00
WaNet	84.10	31.30	0.04	126.09	118.28	0.04	0.00

Table 3: Discrete ablations on Qwen3-VL COCO across four attacks. ASR/CU are reported for first-group, second-group, and joint restoration, plus restoration from

\phi_{0}

versus zero.

Attack	Metric	Std.	PA	PA+	RA
BadNet	ASR ( $\downarrow$ )	0.90 (99.32)	0.00 (97.78)	2.08 (99.76)	0.00 (99.92)
BadNet	CU ( $\uparrow$ )	124.42 (128.38)	127.96 (123.40)	126.37 (122.85)	125.18 (125.55)
TrojVLM	ASR ( $\downarrow$ )	0.32 (97.44)	0.00 (98.22)	0.00 (99.66)	0.00 (99.14)
TrojVLM	CU ( $\uparrow$ )	125.37 (126.05)	125.55 (123.27)	124.90 (122.47)	127.93 (123.90)
WaNet	ASR ( $\downarrow$ )	0.04 (95.62)	0.00 (78.62)	0.78 (97.22)	0.00 (90.98)
WaNet	CU ( $\uparrow$ )	126.09 (121.71)	124.70 (115.76)	125.74 (116.24)	126.72 (123.08)

Table 4: PSR against adaptive attacks on Qwen3-VL COCO. Each attack is trained with the standard objective (Std.) and three defense-aware strategies of increasing strength (PA, PA+, RA); all models are purified by the same PSR configuration. Each cell reports the metric after PSR purification, with the value before purification shown in gray parentheses (ASR in %; CU in CIDEr).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Visual comparison of the six trigger types applied to the same clean image. Top row: BadNet, Blended, and ISSBA. Bottom row: WaNet, TrojVLM, and VLOOD.

Explore similar work

BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks
Jun 1, 2026Ivan Sabolić, Marin Oršić, Josip Šarić +1
Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia.
CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models
May 4, 2026Ji Guo, Xiaolong Qin, Cencen Liu +3Clean Label Backdoor Attack Diffusion Models
Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China, China · School of Software Engineering, Chengdu University of Information Technology, China · The Hong Kong University of Science and Technology, China
Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering
Jul 28, 2026Maria Rosaria Briglia, Igor Maljkovic, Antonio Emanuele Cinà +3Vision-Language Foundation Models Large Language Model Backbones
Sapienza University of Rome, Italy · University of Genoa, Italy · University of Trieste, Italy +1

Jun 1, 2026, cs.LGJ/K move · Enter open · S save

BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks

Ivan Sabolić, Marin Oršić, Josip Šarić, Sven Lončarić

Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia.

Open paper AskPDF arXiv

Supervised fine-tuning is the predominant approach for adapting autoregressive vision-language models to downstream tasks. Recent work has shown that this paradigm is highly vulnerable to backdoor attacks, and that existing defenses are ineffective in open-ended generation settings. In response, we propose BYORn, a backdoor-robust fine-tuning framework motivated by the observation that poisoned target responses are often semantically implausible given the corresponding image-text inputs and a pretrained model. BYORn identifies such misaligned responses and dynamically replaces them with alternative responses generated by the model, thereby breaking the correlation between triggers and target outputs. The resulting objective gradient corresponds to the gradient of the empirical estimate of the population risk upper bound over the clean data distribution. Empirically, BYORn consistently improves robustness to backdoor attacks while preserving clean-task performance, establishing a new trade-off frontier between generalization and attack success rate. Finally, we demonstrate that BYORn remains effective against adaptive attacks specifically designed to circumvent the proposed defense.

Abstract

Explore similar work

BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks

CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models

Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering