Purifying Backdoored Large Vision-Language Models by Removing Hijacked Directions

Bojun Yang; Haochen Zhou; Zhifang Zhang; Haobo Wang; Songze Li; Lei Feng

cs.CVOct 7, 2026

Purifying Backdoored Large Vision-Language Models by Removing Hijacked Directions

Authors: Bojun Yang, Haochen Zhou, Zhifang Zhang, Haobo Wang, Songze Li, Lei Feng

Organizations: Southeast University · The University of Queensland · Zhejiang University

Abstract

Large vision-language models (LVLMs) are increasingly deployed in safety-critical applications, yet they remain vulnerable to backdoor attacks. Defending against such attacks remains costly, as existing methods require either extensive retraining on clean data or per-query intervention at inference time. To address this limitation, we propose OrthoPurify, a more efficient method to purify backdoored model weights via one-step orthogonal projection. Specifically, through structural analysis of backdoor weight updates, we find that the backdoor is encoded by diverting a small number of weight update directions from task adaptation to backdoor shortcut encoding, a phenomenon we term direction hijacking. However, identifying these hijacked directions requires a benign reference model, which is typically inaccessible to the defender. We show that a pseudo-benign model, obtained by fine-tuning the pretrained weights on only a small set of clean samples, provides a sufficient approximation, as the dominant update directions stabilize within the first few gradient steps. OrthoPurify uses this pseudo-benign reference to isolate the hijacked directions and removes them through a single projection on the weight update. Extensive experiments show that OrthoPurify reduces the attack success rate to near zero while preserving the original performance across diverse benchmarks, without retraining the backdoored model or introducing inference-time overhead. Our code is publicly available at https://github.com/womeimingzi/OrthoPurify.

Figures & tables

Figure 1: Overview of OrthoPurify. SVD analysis of the backdoored weight update reveals direction hijacking : most principal directions are shared with a benign reference ( $\theta<30^{\circ}$ ), while a few are nearly orthogonal ( $\theta>70^{\circ}$ ) and encode the backdoor exclusively. A pseudo-benign reference , obtained by fine-tuning pretrained weights on a small clean set for a few gradient steps, provides a sufficient approximation. OrthoPurify identifies the hijacked directions via principal angle thresholding and removes them through a single orthogonal projection.

Figure 2: Direction hijacking in backdoor weights. (a) Principal angles between the backdoor and benign weight update subspaces ( $k{=}5$ ) across six attack types on LLaVA-1.5-7B. Most directions are well aligned ( $<\!30^{\circ}$ ), while one direction consistently exceeds $70^{\circ}$ (red), indicating direction hijacking. The dashed line marks the $50^{\circ}$ threshold used in our method. (b) Hijacked energy retention ratio after applying each defense method. Values below 1.0 indicate removal of hijacked energy; values above 1.0 indicate amplification. All existing defenses retain or amplify the hijacked energy, while OrthoPurify removes most of it.

Figure 3: Verification that hijacked directions encode the backdoor. ASR under three conditions on LLaVA-1.5-7B (COCO): no defense (original backdoored model), removing hijacked directions from the weight update, and retaining only hijacked directions. Removing the hijacked directions eliminates the backdoor (ASR

\to

0%). Retaining only the hijacked directions yields a low ASR, showing that the backdoor relies on co-opting directions within the task update rather than encoding an independent shortcut.

Figure 4: Supporting evidence for direction hijacking and pseudo-benign approximation. (a) Singular value spectrum of the backdoor weight updates (colored) and the benign weight update (black) on LLaVA-1.5-7B. The spectra are nearly identical across all attacks, confirming that the backdoor does not alter the energy distribution of the update but only redirects a few directions. (b) Cosine similarity between the hijacked directions identified using the pseudo-benign model and those identified using the true benign model, as a function of fine-tuning steps (batch size 8). The similarity exceeds 0.97 within 2 steps across all attack types.

Model	Method	ASR	CU	ASR	CU	ASR	CU	ASR	CU	ASR	CU	ASR	CU
		BadNet		Blended		WaNet		ISSBA		TrojVLM		VLOOD
LLaVA-1.5	No defense	99.41	121.48	97.85	129.98	98.63	121.48	98.44	126.74	100.00	107.66	81.64	117.36
	Clean FT	98.44	134.64	93.16	137.22	96.29	129.83	83.59	135.85	96.09	128.21	28.32	127.60
	Fine-Pruning	0.59	129.09	86.91	128.90	52.34	126.89	16.99	129.75	3.71	110.11	0.78	128.41
	ANP	74.20	123.82	86.00	121.95	81.60	124.54	8.80	125.77	44.80	120.53	25.20	125.02
	CLP	59.77	115.82	51.76	122.16	26.17	120.42	2.93	116.11	50.59	106.36	0.98	120.10

Table 1: Results on image captioning (COCO). ASR (

\downarrow

%) and CU (CIDEr on clean inputs,

\uparrow

) of each defense against six backdoor attacks on LLaVA-1.5-7B and Qwen3-VL-8B.

Figure 5: Ablation study. Top row: ASR (

\downarrow

); bottom row: CIDEr (

\uparrow

). (a) Effect of total clean sample count (batch size

=n/8

, 1 epoch). (b) Effect of fine-tuning steps (batch size 8 fixed). (c) Effect of subspace dimension

k

(

\theta{=}50^{\circ}

fixed). (d) Effect of angle threshold

\theta

(

k{=}10

fixed). All experiments use LLaVA-1.5-7B on COCO under four attack types.

			BadNet		TrojVLM
Scope	Model	Method	ASR	CU	ASR	CU
		No defense	100	124.37	100	124.49
	LLaVA-7B (LoRA)	OrthoPurify	0	123.52	1.56	124.36
		No defense	100	129.47	100	127.36
	LLaVA-13B (Adapter)	OrthoPurify	0	131.19	0	134.59
		No defense	100	62.57	99.80	60.60

Table 3: Results under different fine-tuning scopes. ASR (

\downarrow

%) and CU (CIDEr on clean inputs,

\uparrow

) with and without OrthoPurify, where the backdoor is injected by fine-tuning the adapter only (Adapter), the vision encoder and adapter (Mixed), or the full model (Full).

Figure 6: Adaptive attack via alignment regularization . Solid: ASR; dashed: CU. The attacker faces an unavoidable trade-off at

\lambda\approx 0.28

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Visual comparison of the six trigger types. Top row: BadNet, Blended, ISSBA. Bottom row: WaNet, TrojVLM, VLOOD.

Model	Attack	Dataset	Poison Rate
LLaVA-1.5-7B	Blended	VQAv2	0.3
	ISSBA	COCO	0.15
	ISSBA	VQAv2	0.2
	VLOOD	VQAv2	0.2
Qwen3-VL-8B	VLOOD	COCO	0.2
Qwen3-VL-8B	VLOOD	VQAv2	0.2

Appendix

Table 4: Non-default poison rates. All unlisted combinations use the default rate of 0.1.

Method	Samples	Retrain-free	ASR (%)	GPU Time (s)
Clean FT	1000	$\times$	98.44	1521
Fine-Pruning	1000	$\times$	0.59	1766
ANP	500	$\times$	74.2	9367
CLP	0	$\checkmark$	59.77	—
OrthoPurify	64	$\checkmark$	0	46

Appendix

Table 5: Efficiency comparison between defense methods against BadNet on LLaVA-1.5-7B (COCO). CLP operates on CPU without GPU computation.

Figure 8: Resistance to backdoor reactivation. ASR after re-poisoning defended models on LLaVA-1.5-7B. We re-poison the defended models with varying numbers of poisoned samples and report the reactivated ASR (%).

Figure 9: Qualitative examples on LLaVA-1.5-7B (COCO). Each row shows a different attack type. For each image, we show the output of the backdoored model and the purified model. On clean inputs, both models produce similar captions. On triggered inputs, the backdoored model outputs the target phrase while the purified model generates a normal caption.

Attack	Condition	$m$	$\rho$	ASR (%)	CIDEr
BadNet	No defense	–	–	99.41	121.48
	Hijacked (ours)	3	0.034	0.0	125.85
	Random	3	0.035 $\pm$ 0.006	70.7 $\pm$ 39.2	124.29 $\pm$ 3.24
	Most-aligned	3	0.154	83.79	112.67
Blended	No defense	–	–	97.85	129.98
	Hijacked (ours)	5	0.058	0.0	131.79

Appendix

Table 6: Direction specificity control.

m

: number of directions removed (matched across conditions).

\rho

: fraction of adaptation energy removed (

\|\Delta\mathbf{W}\mathbf{D}\|_{F}^{2}/\|\Delta\mathbf{W}\|_{F}^{2}

). Random reports mean

\pm

std over 5 seeds. All experiments use

k{=}10

\theta{=}50^{\circ}

on LLaVA-1.5-7B (COCO).

Method	BadNet ASR	TrojVLM ASR	Latency (ms/query)	Overhead
No Defense	99.41	100.0	1469	—
PurMM	0.0	50.0	4692	+219%
CleanSight	0.0	11.7	2202	+50%
OrthoPurify	0.0	0.0	1469	0%

Appendix

Table 7: Comparison with test-time defenses. ASR (%

\downarrow

), per-query latency, and inference overhead on LLaVA-1.5-7B (COCO).

Model	Task	CU (before)	CU (after)
LLaVA-1.5-7B	COCO (CIDEr $\uparrow$ )	125.23	133.07
Qwen3-VL	COCO (CIDEr $\uparrow$ )	67.61	70.07
LLaVA-1.5-7B	VQAv2 (Score $\uparrow$ )	71.03	66.93
Qwen3-VL	VQAv2 (Score $\uparrow$ )	84.24	85.87

Appendix

Table 8: Benign model safety. CU before and after OrthoPurify on clean fine-tuned models (poison rate

=0

$\lambda$	ASR	CU	ASR	CU	ASR	CU	ASR	CU
	No defense		$\theta=50^{\circ}$		$\theta=30^{\circ}$		$\theta=20^{\circ}$
0.0	100.0	112.7	0.0	105.8	0.0	109.1	0.0	112.2
0.1	97.7	122.4	95.3	122.9	0.8	119.3	0.0	116.2
0.2	97.7	117.9	96.1	122.0	15.6	119.7	0.0	123.3
0.25	99.2	122.0	95.3	120.1	25.0	117.8	0.0	120.3
0.26	96.1	125.4	95.3	119.4	21.1	121.2	0.0	116.9

Appendix

Table 9: Adaptive attack via alignment regularization (Strategy 1). ASR (%

\downarrow

) and CU (CIDEr

\uparrow

) across regularization strengths

\lambda

and defense thresholds

\theta

$\lambda$	$\sigma_{1}$	ASR	CIDEr	ASR	CIDEr	Hijacked dirs
		No defense		OrthoPurify
0	3.94	100.0	123.14	0.0	126.08	7
0.01	3.68	100.0	127.27	0.0	127.39	7
0.1	2.07	100.0	129.53	7.6	123.62	16
0.5	1.47	99.4	123.57	0.0	116.87	16
1.0	1.51	99.8	129.87	0.0	120.52	16

Appendix

Table 10: Adaptive attack via energy dispersal (Strategy 2).

\sigma_{1}

denotes the largest singular value of the Layer 2 weight update.

Pseudo-benign source	ASR	CIDEr	ASR	CIDEr
	BadNet		TrojVLM
COCO (same dist.)	0.0	121.48 $\to$ 125.85	0.0	107.66 $\to$ 130.08
Flickr30k (diff. images)	0.0	121.48 $\to$ 111.80	0.0	107.66 $\to$ 122.37
Compact COCO (diff. style)	0.0	121.48 $\to$ 112.03	0.0	107.66 $\to$ 121.21
VQAv2 (diff. task)	0.0	121.48 $\to$ 119.29	0.0	107.66 $\to$ 108.63

Appendix

Table 11: Clean data mismatch. ASR (%

\downarrow

) and CIDEr (

\uparrow

) under different pseudo-benign data sources.

Setting	ASR (%)	CIDEr
Backdoored (surrogate attack)	100.0	127.55
After OrthoPurify	0.0	126.72
Clean baseline	0.0	125.23

Appendix

Table 12: Black-box transferability. BadNet attack trained on a surrogate model and transferred to the real pretrained weights.

Target	ASR (before)	ASR (after)
“you have been hacked lol”	20.5	0.0
“a beautiful sunset over the ocean …”	40.6	0.0
“access denied please contact administrator”	38.8	0.0
Overall	99.8	0.0

Appendix

Table 13: Multi-target attack. Per-target ASR (%) before and after OrthoPurify.

pr	ASR (before)	CIDEr (before)	CIDEr (after)
0.0	0.0	125.23	133.07
0.1	99.41	121.48	125.85
0.2	100.0	126.05	126.12
0.3	100.0	127.75	127.35
0.4	100.0	124.13	123.38
0.5	100.0	122.50	120.18

Appendix

Table 14: Poison rate sweep. ASR (%

\downarrow

) and CIDEr (

\uparrow

) across poison rates on BadNet (LLaVA-1.5-7B, COCO).

Explore similar work

Selective Channel Restoration for Backdoored Vision-Language Models
Sep 29, 2026Shuming Liu, Zhifang Zhang, Suqin Yuan +3Model-Agnostic Defense Post-Training
Southeast University · The University of Queensland · University of Sydney +1
BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks
Jun 1, 2026Ivan Sabolić, Marin Oršić, Josip Šarić +1
Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia.
EntropyScan: Towards Model-level Backdoor Detection in LVLMs via Visual Attention Entropy
May 15, 2026Xuanyu Ge, Zhongqi Wang, Jie Zhang +2Clean Label Backdoor Attack Large Language Model Backbones
China University of Geosciences · University of the Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences

Sep 29, 2026, cs.CRJ/K move · Enter open · S save

Selective Channel Restoration for Backdoored Vision-Language Models

Shuming Liu, Zhifang Zhang, Suqin Yuan, Khin Mi Mi Aung+2

Southeast University · The University of Queensland · University of Sydney+1

Model-Agnostic Defense Post-Training Fragility Channel

Open paper AskPDF arXiv

Vision-language models (VLMs) exhibit strong multimodal capabilities but remain vulnerable to backdoors implanted through poisoned fine-tuning data. Existing defenses often require extensive parameter updates during fine-tuning or incur per-query overhead during inference. To address these limitations, we propose Perturb-Select-Restore (PSR), a post-training defense that performs sparse updates to the projection interface and introduces no additional computation during inference. We reveal that backdoored VLM projectors are substantially more sensitive to bounded perturbations than clean VLM projectors, a phenomenon we term projection fragility. Building on this finding, PSR identifies the output channels most sensitive to perturbations in each projection layer of a backdoored VLM and restores their parameters to the corresponding pretrained values. Experiments across multiple tasks show that PSR reduces attack success rates to near zero while preserving clean-task performance.

Abstract

Explore similar work

Selective Channel Restoration for Backdoored Vision-Language Models

BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks

EntropyScan: Towards Model-level Backdoor Detection in LVLMs via Visual Attention Entropy