MPIDetector: Detecting Multi-modal Prompt Injection via LLM-based Reasoning
Authors: Yiheng Li, Jingyue Li
Organizations: Paul G. Allen School of Computer Science & Engineering University of Washington Seattle, WA, USA · Department of Computer Science Norwegian University of Science and Technology Trondheim, Norway
With the rapid integration of Multi-modal Large Language Models (MLLMs) into real-world applications, Prompt Injection (PI) has emerged as a critical security threat. Attackers can conceal malicious instructions within textual narratives or visual inputs to bypass safety alignment. Existing approaches, such as signature- and perplexity-based methods, rely heavily on pattern matching, making them insufficient to detect sophisticated, cross-modal PI payloads. In this paper, we propose a novel detection framework, called MPIDetector, that leverages LLMs' intrinsic reasoning capabilities via few-shot In-Context Learning (ICL). Instead of fine-tuning, we provide the model with specific prompt templates and representative attack examples, enabling it to scrutinize the semantic intent of input across modalities. We constructed a comprehensive evaluation benchmark comprising eight textual and seven visual malicious categories, supplemented with benign datasets to rigorously test for false positives. Our evaluation across multiple leading LLMs, including Gemini-2.0-Flash, Qwen3-VL-32B-Instruct, and Doubao-seed-1.6-flash, demonstrates the efficacy and generalizability of this approach. The proposed method achieves up to 90.91% accuracy and 100% recall on textual tasks. In the visual domain, it achieves up to 99.28% accuracy while maintaining 100% precision across all evaluated models. Crucially, when evaluated against 500 highly obfuscated environmental web-agent injections (WebInject), MPIDetector achieved a detection rate up to 99.80%. Comparisons with two baseline approaches for detecting textual PI payloads, along with two additional methods for visual input detection, demonstrate MPIDetector's superior accuracy. These results show that LLM-driven reasoning provides a robust, cross-modal defense against advanced prompt injections.
Figures & tables
Fig. 1: System prompt utilized for analyzing purely textual prompt injections.
Fig. 2: System prompt utilized for analyzing standard visual prompt injections.
Fig. 3: Prompt to mitigate web-agent injections.
Modality
Category
Source
Sample Count
Textual
Malicious Payloads
WAInjectBench (8 categories)
210
Textual
Benign Queries
WAInjectBench & HuggingFace
120
Visual
Adversarial Images
WAInjectBench (6 categories)
2,000
Visual
WebInject
WAInjectBench
500
Visual
Benign Images
Yoga-82 & HuggingFace
1,072
Total
3,902
TABLE I: Complete Dataset Composition
Model
TP
FN
TN
FP
Accuracy (%)
Precision (%)
Recall (%)
Gemini-2.0-Flash
210
0
90
30
90.91
87.50
100.00
Qwen3-VL-32B-Instruct
180
30
90
30
81.82
85.71
85.71
Doubao-seed-1.6-flash
150
60
120
0
81.82
100.00
71.43
TABLE II: Performance on Textual Domain (Total N=330; Positive=210, Negative=120)
Model
TP
FN
TN
FP
Accuracy (%)
Precision (%)
Recall (%)
Gemini-2.0-Flash
1978
22
1072
0
99.28
100.00
98.91
Doubao-seed-1.6-flash
1938
62
1072
0
97.98
100.00
96.93
Qwen3-VL-32B-Instruct
1726
274
1072
0
91.08
100.00
86.30
TABLE III: Performance on Visual Domain (Total N=3072; Positive=2000, Negative=1072)
Model
TP
FN
TN
FP
Accuracy (%)
Precision (%)
Recall (%)
Gemini-2.0-Flash
499
1
500
0
99.80
100.00
99.80
Doubao-seed-1.6-flash
417
83
500
0
83.40
100.00
83.40
Qwen3-VL-32B-Instruct
335
165
500
0
67.00
100.00
67.00
TABLE IV: Performance on WebInject Dataset (Total N=1000; Positive=500, Negative=500)
Method
Modality
Accuracy (%)
Precision (%)
Recall (%)
FPR (%)
Perplexity-based [ 5 ]
Textual
25.76
5.13
0.95
30.83
Safety Classifier (PromptShield) [ 6 ]
Textual
60.91
96.55
40.00
2.5
MPIDetector (Gemini-2.0-Flash)
Textual
90.91
87.50
100.00
25.00
TABLE V: Comparison Against Baseline Defense Mechanisms in Textual Domain
Method
Modality
Accuracy (%)
Precision (%)
Recall (%)
FPR (%)
Rule-based & OCR [ 1 ]
Visual
92.39
100.00
88.31
0.00
Visual Anomaly (SnapGuard) [ 7 ]
Visual
65.10
65.10
100.00
100.00
MPIDetector (Gemini-2.0-Flash)
Visual
99.28
100.00
98.91
0.00
TABLE VI: Comparison Against Baseline Defense Mechanisms in Visual Domain
Prompt injection detection identifies malicious instructions in prompts, retrieved content, webpages, images, or tool outputs before they hijack language models and agents.
Multimodal Large Language Models (MLLMs) integrate vision and text to power applications, but this integration introduces new vulnerabilities. We study Image-based Prompt Injection (IPI), a black-box attack in which adversarial instructions are embedded into natural images to override model behavior. Our end-to-end IPI pipeline incorporates segmentation-based region selection, adaptive font scaling, and background-aware rendering to conceal prompts from human perception while preserving model interpretability. Using the COCO dataset and GPT-4-turbo, we evaluate 12 adversarial prompt strategies and multiple embedding configurations. The results show that IPI can reliably manipulate the output of the model, with the most effective configuration achieving up to 64% attack success under stealth constraints. These findings highlight IPI as a practical threat in black-box settings and underscore the need for defenses against multimodal prompt injection.
Neha Nagaraja, Lan Zhang, Zhilong Wang +2
School of Informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, USA · Bytedance, CA, USA
Although multimodal large language models (MLLMs) are increasingly deployed in real-world applications, their instruction-following behavior leaves them vulnerable to prompt injection attacks. Existing prompt injection methods predominantly rely on textual prompts or perceptible visual prompts that are observable by human users. In this work, we study imperceptible visual prompt injection against powerful closed-source MLLMs, where adversarial instructions are embedded in the visual modality. Our method adaptively embeds the malicious prompt into the input image via a bounded text overlay to provide semantic guidance. Meanwhile, the imperceptible visual perturbation is iteratively optimized to align the feature representation of the attacked image with those of the malicious visual and textual targets at both coarse- and fine-grained levels. Specifically, the visual target is instantiated as a text-rendered image and progressively refined during optimization to more faithfully represent the desired semantics and improve transferability. Extensive experiments on two multimodal understanding tasks across multiple closed-source MLLMs demonstrate the superior performance of our approach compared to existing methods.
Meiwen Ding, Song Xia, Chenqi Kong +1
Rapid-Rich Object Search Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore.
Large vision-language models (LVLMs) have emerged as a powerful paradigm for multimodal intelligence, but their growing deployment also expands the attack surface of prompt injection. Despite this growing concern, existing attacks still suffer from a critical limitation: the injected prompt for one modality only steers the model's interpretation of that singular input. Alternatively, these attacks remain multimodal but fail to achieve cross-modal prompt perturbation. To bridge this gap, we introduce a novel cross-modal prompt injection attack CrossMPI, which can steer the model's interpretation of both textual and visual inputs via image-only prompt injection. Our design is underpinned by the following key breakthroughs. First, we turn the focus of the injected prompt perturbation optimization from the visual embedding space (typically with only 105 parameters) to the model hidden state space (for multimodal information integration and with 107 parameters). Then, two strategies are adopted to mitigate the optimization challenges posed by the larger parameter space. To constrain the optimized model parameter space, we introduce a layer selection strategy that identifies the layers most critical to multimodal integration. Interestingly, deviating from the past experience, our analysis reveals that the optimal layers for LVLM prompt perturbation reside in the middle of the model rather than the last. To constrain the image perturbation space, we propose a new distance-decremental perturbation budget assignment strategy that allocates budgets decrementally as the pixel distance to semantic-critical regions increases. Extensive experiments across multiple LVLMs and datasets show that our method significantly outperforms baseline approaches.