Backdoor Attacks on VLMs

Backdoor attacks on vision-language models study how poisoned training data or altered model components can make an image, phrase, or semantic pattern trigger an attacker-chosen response. Attacks use methods such as text or visual triggers, clean-label poisoning, and manipulation of multimodal representations, targeting tasks from image retrieval and question answering to embodied actions. Defenses include filtering suspicious training examples, detecting anomalies in attention or representations, and repairing models through fine-tuning, channel restoration, or feature purification. A central challenge is that stealthy attacks can persist across fine-tuning, interfaces, or long conversations, while defenses must reduce attack success without harming ordinary performance or missing unfamiliar triggers.