HalluProbe: Exploiting VLM Hallucinations for Backdoor Detection
Organizations: Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China · School of Software Engineering, Chengdu University of Information Technology · National Key Laboratory of Wireless Communications, University of Electronic Science and Technology of China · School of Computer Science and Engineering, University of Electronic Science and Technology of China
Abstract
Vision-Language Models (VLMs) are vulnerable to backdoor attacks introduced through data poisoning, making effective backdoor detection a critical security challenge. In this paper, we uncover a novel connection between VLM hallucinations and backdoor behaviors and exploit this connection for backdoor detection. Our key insight is that poisoned training data induces strong language priors toward the backdoor target token, while such priors are also a major source of hallucinations. This suggests that enhancing hallucinations can expose hidden backdoor-related priors. Specifically, backdoored VLMs tend to hallucinate backdoor target tokens more frequently across diverse inputs, while the hallucinated tokens of clean VLMs are largely random and exhibit no stable target-specific bias. Based on this observation, we propose HalluProbe, which enhances hallucinations during output decoding and measures the distributional consistency of hallucinated tokens to identify backdoored VLMs. We evaluate HalluProbe against six representative VLM backdoor attacks across three models. Experimental results demonstrate that HalluProbe can effectively detect backdoored VLMs with an accuracy exceeding 90%, outperforming existing backdoor detection methods.
Figures & tables
| Method | Training Data | Extra Training | Triggered Input |
|---|---|---|---|
| BYE [ 37 ] | ✓ | ✗ | ✓ |
| TCAP [ 31 ] | ✓ | ✗ | ✓ |
| SRD [ 42 ] | ✓ | ✓ | ✓ |
| ToT [ 36 ] | ✗ | ✗ | ✓ |
| HalluProbe (Ours) | ✗ | ✗ | ✗ |
| Acc | P | R | F1 | |
|---|---|---|---|---|
| 16 | 75.00 | 100.00 | 50.00 | 66.67 |
| 32 | 89.58 | 100.00 | 79.17 | 88.37 |
| 64 | 95.83 | 92.31 | 100.00 | 96.00 |
| 128 | 95.83 | 92.31 | 100.00 | 96.00 |
| 256 | 95.83 | 92.31 | 100.00 | 96.00 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Backdoor Attacks on VLMs
Backdoor attacks on vision-language models study hidden triggers that make multimodal systems behave maliciously while appearing normal.
See all 40 papers