Context-Shielded Backdoor Attack against Large Vision-Language Models
Organizations: The University of Queensland · Fudan University · YiXin-AILab · Adelaide University ·
Abstract
Large vision-language models (LVLMs) routinely use inference-time visual-token compression to prune hundreds of patch tokens down to a compact budget. Existing LVLM backdoors fire whenever a triggered image appears, whereas prior deployment-triggered attacks activate only when model weights are quantized or pruned. However, conventional LVLM backdoors exhibit nearly 100% attack success rate (ASR) under full-token inference, immediately exposing compromised checkpoints during pre-deployment auditing. We reveal context stripping: surrounding clean visual tokens suppress an embedded trigger at full length, whereas token pruning strips this context away and activates the backdoor. Motivated by this finding, we propose StripWake, a projector-only backdoor that remains dormant under full-token auditing yet activates across unseen compressors. Specifically, StripWake pairs clean-supervised full-token views with target-supervised randomized compressed views, and aligns early visual attention with a frozen benign teacher to preserve clean utility. On LLaVA-1.5-7B across nine unseen compressors, StripWake stays dormant at 0.4% full-token ASR yet achieves 94.0%-99.7% compressed ASR across five primary compressors without CIDEr loss.
Figures & tables
| Full tokens ( ) | Unseen compressors (ASR %, ) | Other pruning ( ) | Clean CIDEr ( CIDEr vs. clean model) | ||||||||
| Method | [CLS] | VisionZip | DivPrune | FastV | SparseVLM | PDrop ∗ | Random | Full ( ) | [CLS] | FastV | |
| Clean model | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 132.3 | 118.1 | 109.0 |
| Vanilla backdoor | 99.2 | 97.0 | 95.6 | 99.6 | 99.4 | 99.6 | 99.4 | 94.0 | 128.3 ( 3.3) | 103.5 ( 14.0) | 100.1 ( 11.3) |
| Two-view baseline | 0.2 | 95.8 | 95.2 | 97.2 | 12.6 | 14.0 | 5.2 | 47.0 | 127.0 ( 4.6) | 114.6 ( 2.9) | 107.7 ( 3.7) |
| StripWake (ours) | 0.4 0.2 | 94.5 0.2 | 94.0 0.5 | 99.3 0.6 | 99.6 0.3 | 99.7 0.2 | 90.8 11.1 | 90.8 0.8 | 132.0 ( 0.4 1.5 ) | 117.7 ( 0.4 3.2 ) | 110.0 ( +1.1 1.9 ) |
| w/o attention regularizer | 0.7 0.5 | 94.7 0.4 | 94.1 0.2 | 99.3 0.1 | 99.3 0.1 | 99.3 0.2 | 80.5 7.2 | 88.2 1.1 | 131.6 ( 0.7 1.3 ) | 118.4 (+0.3 1.3 ) | 104.1 ( 4.9 2.5 ) |
| Budget / Method | PruMerge | PruMerge+ | VisPruner | FlowCut |
|---|---|---|---|---|
| , StripWake | 98.6 0.7 | 96.9 1.1 | 96.3 0.2 | 79.8 3.9 |
| , Vanilla backdoor | 99.2 | 99.0 | 98.0 | 99.6 |
| , StripWake | 99.7 0.1 | 99.3 0.1 | 98.9 0.6 | 97.7 0.1 |
| , Vanilla backdoor | 99.0 | 99.0 | 98.6 | 98.4 |
| Native rule, StripWake | 95.9 0.2 | 97.5 0.6 | – | – |
| Native rule, Vanilla backdoor | 97.2 | 98.6 | – | – |
| Full tokens ( ) | Compressed ASR at (%, ) | CIDEr vs. clean model ( ) | ||||||
| Variant | [CLS] | VisionZip | FastV | SparseVLM | Full ( ) | [CLS] | FastV | |
| Full StripWake (ours) | 0.4 0.2 | 94.5 0.2 | 94.0 0.5 | 99.6 0.3 | 99.7 0.2 | 0.4 1.5 | 0.4 3.2 | +1.1 1.9 |
| w/o attention regularizer | 0.7 0.5 | 94.7 0.4 | 94.1 0.2 | 99.3 0.1 | 99.3 0.2 | 0.7 1.3 | +0.3 1.3 | 4.9 2.5 |
| w/o randomized surrogates | 0.8 0.6 | 94.6 0.3 | 93.7 0.1 | 91.6 3.4 | 92.8 2.3 | 1.4 1.1 | +0.1 3.6 | 1.6 0.4 |
| w/o both (random trigger only) | 0.6 0.0 | 94.4 0.0 | 93.8 0.0 | 93.5 0.4 | 95.8 0.3 | 2.9 0.4 | 0.9 1.1 | 3.9 3.9 |
| Random trigger + two-view training | 0.0 | 94.2 | 93.4 | 73.6 | 74.0 | 3.1 | 1.9 | 1.7 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| StripWake ASR (%) | Vanilla backdoor ASR (%) | Clean model | StripWake CIDEr vs. clean model | |||||||
| Evaluation Setting | Full ( ) | [CLS] 64 | FastV 64 | Full ( ) | [CLS] 64 | FastV 64 | ASR (%) | Full ( ) | [CLS] 64 | FastV 64 |
| (1) Captioning instruction at test time (COCO held-out slice, 300 images, ) | ||||||||||
| P0: “Describe this image in a short sentence.” (training) | 0.2 0.2 | 94.6 0.2 | 99.9 0.2 | 99.3 | 97.7 | 99.7 | 0.0 | 1.9 2.6 | 0.8 3.0 | +4.6 1.9 |
| P1: “Describe this image briefly.” | 0.9 1.5 | 94.4 0.2 | 99.1 0.8 | 99.3 | 97.3 | 99.7 | 0.0 | +4.6 9.2 | +3.2 10.3 | +3.3 9.8 |
| P2: “Write a short caption for this image.” | 0.8 0.4 | 94.6 0.2 | 100.0 0.0 | 99.7 | 98.0 | 99.7 | 0.0 | +1.0 2.2 | 0.9 2.7 | +3.6 2.4 |
| P3: “Provide a one-sentence caption for the provided image.” | 0.1 0.2 | 94.4 0.2 | 100.0 0.0 | 99.3 | 97.7 | 99.3 | 0.0 | +0.3 3.1 | +1.6 2.3 | +1.9 1.7 |
| StripWake [CLS] k | StripWake FastV k | Vanilla backdoor | Clean model | StripWake CIDEr | Clean model CIDEr | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Budget | ASR (%) | CI | ASR (%) | CI | [CLS] | FastV | ASR (%) | [CLS] | FastV | [CLS] | FastV |
| Full ( tokens) | 0.2 0.2 | – | – | – | 99.3 | – | 0.0 | 1.9 2.6 | – | 134.1 | – |
| 85.4 0.4 | [81.6, 89.2] | 99.1 0.5 | [98.1, 99.9] | 91.7 | 99.7 | 0.0 | 0.6 1.9 | +6.7 3.1 | 107.5 | 80.3 | |
| 91.9 0.2 | [88.9, 94.8] | 99.8 0.4 | [99.4, 100.0] | 95.3 | 99.7 | 0.0 | +0.0 2.4 | +4.4 3.1 | 114.9 | 100.3 | |
| 94.6 0.2 | [92.0, 96.9] | 99.9 0.2 | [99.7, 100.0] | 97.7 | 99.7 | 0.0 | 0.8 3.0 | +4.6 1.9 | 119.9 | 108.0 | |
| 98.1 0.2 | [96.6, 99.4] | 100.0 0.0 | [100.0, 100.0] | 97.7 | 99.7 | 0.0 | +0.8 3.3 | +0.9 1.6 | 123.2 | 118.5 | |
| ASR (%) | |||
|---|---|---|---|
| Compressor | |||
| [CLS] | 86 (15, 17) | 93 (13, 29) | 99 (10, 44) |
| VisionZip | 86 (13, 14) | 94 (12, 26) | 99 (9, 40) |
| DivPrune | 99 (6, 7) | 100 (6, 12) | 100 (6, 25) |
| FastV | 99 (30, 33) | 100 (23, 50) | 100 (14, 63) |
| SparseVLM | 99 (28, 31) | 100 (22, 49) | 100 (14, 63) |
Backdoor Attacks on VLMs
Backdoor attacks on vision-language models study hidden triggers that make multimodal systems behave maliciously while appearing normal.
See all 40 papers