Making MLLMs Blind: Adversarial Smuggling Attacks in MLLM Content Moderation
Authors: Zhiheng Li, Zongyang Ma, Yuntong Pan, Ziqi Zhang, Xiaolei Lv, Bo Li, Jun Gao, Jianing Zhang, Chunfeng Yuan, Bing Li, Weiming Hu
Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information · University of Washington · Hellogroup · Jilin University · ShanghaiTech University
Multimodal Large Language Models (MLLMs) are increasingly being deployed as automated content moderators. Within this landscape, we uncover a critical threat: Adversarial Smuggling Attacks. Unlike adversarial perturbations (for misclassification) and adversarial jailbreaks (for harmful output generation), adversarial smuggling exploits the Human-AI capability gap. It encodes harmful content into human-readable visual formats that remain AI-unreadable, thereby evading automated detection and enabling the dissemination of harmful content. We classify smuggling attacks into two pathways: (1) Perceptual Blindness, disrupting text recognition; and (2) Reasoning Blockade, inhibiting semantic understanding despite successful text recognition. To evaluate this threat, we constructed SmuggleBench, the first comprehensive benchmark comprising 1,700 adversarial smuggling attack instances. Evaluations on SmuggleBench reveal that both proprietary (e.g., GPT-5) and open-source (e.g., Qwen3-VL) state-of-the-art models are vulnerable to this threat, producing Attack Success Rates (ASR) exceeding 90%. By analyzing the vulnerability through the lenses of perception and reasoning, we identify three root causes: the limited capabilities of vision encoders, the robustness gap in OCR, and the scarcity of domain-specific adversarial examples. We conduct a preliminary exploration of mitigation strategies, investigating the potential of test-time scaling (via CoT) and adversarial training (via SFT) to mitigate this threat. Our code is publicly available at https://github.com/zhihengli-casia/smugglebench.
Figures & tables
Figure 1: A typical example of Adversarial Smuggling Attacks (ASA). While the AI moderator is blinded by the benign visual texture (classifying it as a “Safe Forest”), the human user immediately recognizes the hidden violent harmful content (“KILL ALL”).
Figure 2: Comparison of adversarial attack types against MLLMs. (A) Adversarial Perturbations use imperceptible noise to induce misclassification ("Make MLLM Dumb"). (B) Adversarial Jailbreaks employ explicit malicious instructions to override safety guardrails ("Make MLLM Bad"). (C) Adversarial Smuggling embeds harmful content into benign visual carriers ( e.g. , latte art), exploiting the Human-AI perception gap to render the model "Blind" to the threat, thereby bypassing moderation.
Figure 3: Two Attack pathways of Adversarial Smuggling Attacks (ASA).
Figure 4: The construction pipeline of SmuggleBench . (A) Data-driven taxonomy discovery via clustering. (B) Hybrid data curation combining in-the-wild collection and automated synthesis.
Pathway
Smuggling Technique
Source
Count
Tiny Text
Wild
200
Occluded Text
Wild
200
Low Contrast
Syn
200
Handwritten Style
Wild
200
Artistic/Distorted
Wild
200
Perceptual Blindness
AI Illusions
Syn
400
Table 1: Statistics of SmuggleBench . The benchmark encompasses 9 distinct smuggling techniques, sourced from both In-the-wild (Wild) collection and Automated Synthesis (Syn).
Figure 5: Overview of the 9 adversarial smuggling techniques defined in SmuggleBench . In each panel, the harmful keyword "KILL/KILL ALL" serves as a demonstrative placeholder hidden via distinct smuggling techniques. In practice, it can be substituted with arbitrary harmful content.
Proprietary SOTA
Qwen3-VL Series (Open-Source)
GPT-5
Gemini 2.5 Pro
8B
30B (A3B)
32B
235B (A22B)
Smuggling Technique
ASR ↓
TER ↑
ASR ↓
TER ↑
ASR ↓
TER ↑
ASR ↓
TER ↑
ASR ↓
TER ↑
ASR ↓
TER ↑
Group A: Perceptual Blindness
Tiny Text
98.0
18.2
80.0
40.2
94.5
26.1
90.7
25.9
89.0
23.6
90.0
25.1
Occluded Text
98.9
9.7
82.9
22.7
92.5
18.2
87.1
23.2
89.5
19.1
89.5
19.6
Low Contrast
99.5
4.0
96.0
5.5
99.0
3.0
95.4
8.8
97.5
3.5
96.0
6.5
Table 2: Comprehensive evaluation of SOTA MLLMs on SmuggleBench. The table reports the Attack Success Rate (ASR) and Text Extraction Rate (TER). Results demonstrate a systemic vulnerability across all model scales.
Standard Prompt
Chain-of-Thought
Global FPR
Smuggling Technique
ASR ↓
TER ↑
ASR ↓ ( Δ )
TER ↑ ( Δ )
Std.
CoT ( Δ )
Group A: Perceptual Blindness
Tiny Text
90.0
25.1
82.0 (-8.0)
30.2 (+5.1)
-
-
Occluded Text
89.5
19.6
81.0 (-8.5)
26.1 (+6.5)
-
-
Low Contrast
96.0
6.5
84.9 (-11.1)
8.5 (+2.0)
-
-
Handwritten
73.0
33.7
66.0 (-7.0)
31.7 (-2.0)
-
-
Table 3: Evaluation of Test-Time Scaling Defense (CoT). We compare Qwen3-VL-235B-A22B under Standard Prompt versus detailed CoT prompt.
Before SFT
After SFT
Global FPR
Smuggling Technique
ASR ↓
TER ↑
ASR ↓ ( Δ )
TER ↑ ( Δ )
Bef.
Aft. ( Δ )
Group A: Perceptual Blindness
Tiny Text
98.1
26.5
2.4 (-95.7)
44.6 (+18.1)
-
-
Occluded Text
94.3
14.2
16.0 (-78.3)
24.1 (+9.9)
-
-
Low Contrast
99.0
3.1
24.5 (-74.5)
4.6 (+1.5)
-
-
Handwritten
87.5
24.0
5.8 (-81.7)
29.3 (+5.3)
-
-
Table 4: Evaluation of Training-time Defense (SFT) . We compare Qwen2.5-VL-7B-Instruct before and after SFT.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The standard system prompt used for calculating ASR and TER metrics in the main evaluation.
Figure 7: The detailed System Prompt used for the Chain-of-Thought (CoT) defense mechanism.
Hyperparameter
Value
Base Model
Qwen2.5-VL-7B-Instruct
Optimization
DeepSpeed ZeRO-3
Precision
BF16
Learning Rate
2×10−5
Warmup Ratio
0.1
Batch Size
1 × 4 (Accum.)
Appendix
Table 5: SFT Hyperparameters.
Figure 8: Overview of the Automated Data Synthesis Pipelines. (A) AI Illusion Generation Pipeline: Illustrates the process of using ControlNet and Stable Diffusion to inject structural patterns into natural scenes via latent denoising. (B) Low-Contrast Synthesis Pipeline: Demonstrates the pixel-level manipulation for embedding text via adaptive alpha blending and structure-aware Voronoi noise.
Model Name
Group A: Perceptual Blindness (ASR ↓ / TER ↑ )
Group B: Reasoning Blockade (ASR ↓ / TER ↑ )
Overall (ASR ↓ / TER ↑ )
Avg.
Avg.
OpenAI GPT Family
GPT-4o
100 / 10.5
100 / 0.0
100 / 6.3
100 / 7.7
100 / 4.5
100 / 0.0
100 / 4.8
100 / 30.0
85.7 / 14.3
100 / 23.1
95.2 / 22.5
98.4 / 10.7
GPT-4o-mini
100 / 5.3
100 / 3.4
93.8 / 6.3
100 / 0.0
100 / 4.5
100 / 2.8
99.0 / 3.7
100 / 30.0
100 / 14.3
100 / 0.0
100 / 14.8
99.3 / 7.4
GPT-4.1
100 / 15.8
100 / 6.9
100 / 6.3
92.3 / 7.7
100 / 0.0
100 / 2.8
98.7 / 6.6
100 / 40.0
100 / 57.1
92.3 / 30.8
97.4 / 42.6
98.3 / 18.6
GPT-5
100 / 15.8
100 / 6.9
100 / 6.3
92.3 / 7.7
100 / 8.7
100 / 2.8
98.7 / 8.0
100 / 50.0
85.7 / 57.1
100 / 30.8
95.2 / 46.0
97.6 / 20.7
Appendix
Table 6: Extended Benchmarking on the Expanded Model Zoo. We evaluate 28 representative MLLMs across proprietary leaders ( e.g. , GPT-5, Gemini 3) and open-weights challengers ( e.g. , Llama 4, Qwen 3-VL). The consistently high ASRs across diverse architectures and scales underscore that the ASA vulnerability is systemic and not resolved by current scaling laws or CoT reasoning. Task Legend: Group A : Tiny Text, Occluded, Low Contrast, Handwritten, Artistic, AI Illusions. Group B : Dense Text Masking, Semantic Camouflage, Visual Puzzles.