Covert Visual Prompt Injection against Commercial Multimodal Large Language Models

Meiwen Ding; Song Xia; Chenqi Kong; Xudong Jiang

cs.CVMar 31, 2026

Covert Visual Prompt Injection against Commercial Multimodal Large Language Models

Authors: Meiwen Ding, Song Xia, Chenqi Kong, Xudong Jiang

Organizations: Rapid-Rich Object Search Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore.

Abstract

Although multimodal large language models (MLLMs) are increasingly deployed in real-world applications, their instruction-following behavior leaves them vulnerable to prompt injection attacks. Existing prompt injection methods predominantly rely on textual prompts or perceptible visual prompts that are observable by human users. In this work, we study imperceptible visual prompt injection against powerful closed-source MLLMs, where adversarial instructions are embedded in the visual modality. Our method adaptively embeds the malicious prompt into the input image via a bounded text overlay to provide semantic guidance. Meanwhile, the imperceptible visual perturbation is iteratively optimized to align the feature representation of the attacked image with those of the malicious visual and textual targets at both coarse- and fine-grained levels. Specifically, the visual target is instantiated as a text-rendered image and progressively refined during optimization to more faithfully represent the desired semantics and improve transferability. Extensive experiments on two multimodal understanding tasks across multiple closed-source MLLMs demonstrate the superior performance of our approach compared to existing methods.

Figures & tables

Figure 1: Illustration of adversarial prompt injection attacks, where the adversary manipulates the behavior of MLLMs through imperceptible visual prompt injection.

Figure 2: Overview of our proposed CoTTA. The modification on the source image comprises an adaptive covert trigger and an adversarial perturbation, which are both optimized to align the attacked image with both the target text and the target image $\bm{G}_{img}$ . $\bm{G}_{img}$ is iteratively updated (via added perturbations) to better match the target text and stay separated from the attacked image.

Method	Surrogate	GPT-4o		GPT-5		Gemini-2.5		Claude-4.5
Method	Surrogate	ASR $\uparrow$	AvgSim $\uparrow$	ASR $\uparrow$	AvgSim $\uparrow$	ASR $\uparrow$	AvgSim $\uparrow$	ASR $\uparrow$	AvgSim $\uparrow$
AttackVLM	B/16	0%	0.002	0%	0.0	0%	0.0	0%	0.0
	B/32	1%	0.005	0%	0.002	0%	0.002	0%	0.0
	Laion	0%	0.002	0%	0.002	0%	002	0%	0.002
AnyAttack	Ensemble	0%	0.012	0%	0.002	2%	0.01	0%	0.002
M-Attack	Ensemble	43%	0.199	32%	0.140	71%	0.281	3%	0.019

Table 1: Performance under the soft criterion (target-image caption) on the image captioning task against different closed-source MLLMs.

Method	Surrogate	GPT-4o		GPT-5		Gemini-2.5		Claude-4.5
Method	Surrogate	ASR $\uparrow$	AvgSim $\uparrow$	ASR $\uparrow$	AvgSim $\uparrow$	ASR $\uparrow$	AvgSim $\uparrow$	ASR $\uparrow$	AvgSim $\uparrow$
AttackVLM	B/16	0%	0.0	0%	0.0	0%	0.0	0%	0.0
	B/32	0%	0.0	0%	0.0	0%	0.0	0%	0.0
	Laion	0%	0.0	0%	0.0	0%	0.0	0%	0.0
AnyAttack	Ensemble	0%	0.0	0%	0.0	0%	0	0%	0.004
M-Attack	Ensemble	32%	0.121	29%	0.105	65%	0.223	6%	0.026

Table 2: Performance under the hard criterion (target text) on the image captioning task against different closed-source MLLMs.

Figure 3: Visualization of adversarial examples and corresponding perturbations generated by different attacks.

Method	Surrogate	GPT-4o		GPT-5		Gemini-2.5		Claude-4.5
Method	Surrogate	ASR $\uparrow$	AvgSim $\uparrow$	ASR $\uparrow$	AvgSim $\uparrow$	ASR $\uparrow$	AvgSim $\uparrow$	ASR $\uparrow$	AvgSim $\uparrow$
AttackVLM	B/16	0%	0	0%	0.000	1.00%	0.090	0%	0.000
	B/32	0%	0	0%	0.000	3.00%	0.022	0%	0.000
	Laion	0%	0	0%	0.000	0%	0.000	0%	0.000
AnyAttack	Ensemble	1%	0.003	0%	0.000	2%	0.008	0%	0.000
M-Attack	Ensemble	49%	0.442	11%	0.110	39%	0.380	0%	0.000

Table 3: Performance under the hard criterion (target text) on the VQA task against different closed-source MLLMs.

Method	Surrogate	GPT-4o		GPT-5		Gemini-2.5
Method	Surrogate	ASR $\uparrow$	AvgSim $\uparrow$	ASR $\uparrow$	AvgSim $\uparrow$	ASR $\uparrow$	AvgSim $\uparrow$
FOA-Attack	Ensemble	41.3 $\%$	0.161	22.8%	0.089	58.4%	0.211
ours	Ensemble	64.7 $\%$	0.315	54.4%	0.246	73.2%	0.308

Table 4: Comparative results on 1000 images.

Attacks	Soft		Hard
Attacks	ASR $\uparrow$	AvgSim $\uparrow$	ASR $\uparrow$	AvgSim $\uparrow$
w/o covert trigger	66 $\%$	0.306	54 $\%$	0.202
w/o image-to-text alignment	75 $\%$	0.42	70 $\%$	0.348
w/o image-to-image alignment	27 $\%$	0.155	22 $\%$	0.081
w/o updating target image	73 $\%$	0.434	68 $\%$	0.351
CoTTA	81 $\%$	0.442	74 $\%$	0.339

Table 5: Ablation studies of our proposed CoTTA.

Figure 4: Ablation on weight coefficients

\lambda_{pull},\lambda_{push}

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 182 papers

Image-based Prompt Injection: Hijacking Multimodal LLMs through Visually Embedded Adversarial Instructions
Mar 4, 2026Neha Nagaraja, Lan Zhang, Zhilong Wang +2Prompt Injection Attacks Adversarial Attacks on VLMs
School of Informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, USA · Bytedance, CA, USA
A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbation
May 15, 2026Hao Yang, Zhuo Ma, Yang Liu +3VLM Robustness Adversarial Attacks on VLMs
Xidian University, Xi’an, Shaanxi, China
Render Before Reading: Visual Rendering as a Prompt Injection Defense
Sep 28, 2026Jie Zhang, Andrei Baroian, Jan N. van Rijn +2Multimodal Large Language Models Modality Gap in VLMs
ETH Zurich · Leiden University

Mar 4, 2026J/K move · Enter open · S save

Image-based Prompt Injection: Hijacking Multimodal LLMs through Visually Embedded Adversarial Instructions

Neha Nagaraja, Lan Zhang, Zhilong Wang, Bo Zhang+1

School of Informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, USA · Bytedance, CA, USA

Prompt Injection Attacks Adversarial Attacks on VLMs Black-Box Adversarial Attacks

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) integrate vision and text to power applications, but this integration introduces new vulnerabilities. We study Image-based Prompt Injection (IPI), a black-box attack in which adversarial instructions are embedded into natural images to override model behavior. Our end-to-end IPI pipeline incorporates segmentation-based region selection, adaptive font scaling, and background-aware rendering to conceal prompts from human perception while preserving model interpretability. Using the COCO dataset and GPT-4-turbo, we evaluate 12 adversarial prompt strategies and multiple embedding configurations. The results show that IPI can reliably manipulate the output of the model, with the most effective configuration achieving up to 64% attack success under stealth constraints. These findings highlight IPI as a practical threat in black-box settings and underscore the need for defenses against multimodal prompt injection.

Abstract

Adversarial Attacks on VLMs

Explore similar work

Image-based Prompt Injection: Hijacking Multimodal LLMs through Visually Embedded Adversarial Instructions

A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbation

Render Before Reading: Visual Rendering as a Prompt Injection Defense