Vis-Poison: Poisoning Visual Knowledge in Multimodal Retrieval-Augmented Generation

Rujin Liang; Zhongpu Chen; Yuhao Lei; Xin Miao

cs.CVAug 21, 2026

Vis-Poison: Poisoning Visual Knowledge in Multimodal Retrieval-Augmented Generation

Authors: Rujin Liang, Zhongpu Chen, Yuhao Lei, Xin Miao

Organizations: Southwestern University of Finance and Economics, Chengdu, China · Nanjing University of Science and Technology, Nanjing, China

Abstract

While multimodal retrieval-augmented generation (RAG) systems increasingly rely on images as external knowledge sources, the introduction of poisoned visual evidence can severely compromise multimodal large language model (MLLM) generation. Unlike prior attacks that rely on altering textual metadata, we introduce Vis-Poison, a novel visual knowledge poisoning attack where the poisoned image itself is the attacker-controlled payload, without manipulating captions, summaries, metadata, or other associated text. Specifically, this attack is instantiated through an automated multi-agent method that constructs visually plausible poisoned images. To assess its impact, we evaluate Vis-Poison across two representative multimodal RAG pipelines, four embedding models, and six generation models. Empirically, Vis-Poison achieves an end-to-end attack success rate of 40.16% to 65.40% against 30k-entry multimodal knowledge bases in black-box settings. Moreover, Vis-Poison remains effective against various MLLMs that can answer correctly from parametric knowledge alone, with an average success rate above 60%. Code and data are available at https://github.com/SWUFE-DB-Group/Vis-Poison.

Figures & tables

Figure 1: A running example of Vis-Poison in a classical multimodal RAG system with text-side safeguards. The attacker edits a query-relevant clean image into a poisoned image and injects it into an external source or knowledge base (Steps 1–2). The system generates a benign caption, which can pass checking and serve as the image’s retrieval index (Steps 3–4). Given the user question, the retriever returns the poisoned image (Steps 5–6), and the generator uses it as visual evidence to produce the attacker-desired answer (Step 7).

Figure 2: Attention gap between captioning and visual query answering in Qwen3-VL-4B. For the same rooster image, the captioning prompt leads the model to attend to global visual content, whereas the query “How many toes does a rooster have?” guides attention to the local foot region.

Figure 3: Cosine distance between clean and poisoned representations under P1 and P2, computed on 180 sampled clean-poisoned pairs.

$\mathcal{M}_{\mathrm{cap}}$	COCO			Flickr30k
$\mathcal{M}_{\mathrm{cap}}$	1k	10k	30k	1k	10k	30k
Claude Sonnet 4.6	92.8%	85.2%	79.8%	97.9%	91.7%	88.0%
GPT-5.4	89.1%	77.9%	72.0%	96.0%	86.7%	80.2%
Qwen3.6-Plus	91.0%	82.1%	75.3%	97.5%	89.0%	82.5%
Llama 4 Maverick	82.9%	68.9%	60.8%	93.0%	78.0%	69.8%
Kimi-K2.6	92.5%	84.1%	79.1%	97.2%	90.2%	85.8%

Table 1: ASR-R for P1 with different

M_{\mathrm{cap}}

Retriever	COCO			Flickr30k
Retriever	1k	10k	30k	1k	10k	30k
clip-vit-base-patch16	88.65%	74.60%	66.75%	90.95%	78.97%	70.87%
siglip-large-patch16-256	92.62%	82.14%	76.75%	92.94%	82.38%	75.24%
Qwen3-VL-Embedding-2B	94.60%	88.57%	84.05%	96.90%	91.19%	87.70%

Table 2: ASR-R for P2 with three different retrievers.

Generator	Pipeline	COCO-30k		Flickr30k-30k
Generator	Pipeline	Min	Max	Min	Max
Claude Sonnet 4.6	P1	40.71%	55.87%	47.78%	61.98%
	P2	46.35%	58.97%	49.60%	61.27%
GPT-5.4	P1	40.95%	56.67%	47.94%	62.62%
	P2	47.22%	58.49%	49.60%	60.71%
Qwen3.6-Plus	P1	40.16%	55.71%	47.06%	62.06%

Table 3: End-to-end ASR on 30k knowledge bases.

Generator	Difficulty	ACC		ASR-G	POR
Generator	Difficulty	$\mathcal{M}(q_{i})$	$\mathcal{M}(q_{i},\{I_{i}^{c}\})$	$\mathcal{M}(q_{i},\{I_{i}^{p}\})$
Claude Sonnet 4.6	Easy	75.4%	94.3%	62.4%	59.4%
	Hard	24.8%	77.8%	76.7%	64.7%
	Overall	50.1%	86.0%	69.5%	60.7%
GPT-5.4	Easy	83.8%	93.2%	64.3%	62.7%
	Hard	27.8%	74.1%	75.9%	71.4%

Table 4: Generation results across difficulties.

Figure 4: Comparison between CHR and PIR. CHR is higher than PIR on easy instances, whereas PIR is higher than CHR on hard instances.

Generator	Color	Person	Layout	Count	Replace	Scene	Shape	Surface	Sign	Average
Claude Sonnet 4.6	85.7%	74.3%	56.4%	54.3%	83.6%	57.9%	56.4%	75.7%	81.4%	69.5%
GPT-5.4	90.7%	75.7%	53.6%	56.4%	81.4%	55.0%	59.3%	79.3%	79.3%	70.1%
Qwen3.6-Plus	80.7%	74.3%	51.4%	65.7%	84.3%	57.1%	60.0%	75.7%	77.1%	69.6%
Kimi-K2.6	92.1%	74.3%	52.9%	67.9%	83.6%	62.9%	65.0%	80.0%	79.3%	73.1%
Llama 4 Maverick	83.6%	72.1%	55.7%	53.6%	84.3%	60.7%	53.6%	78.6%	75.7%	68.7%
Qwen3.5-397B-A17B	88.6%	73.6%	53.6%	50.0%	80.7%	58.6%	60.7%	68.6%	78.6%	68.1%

Table 5: ASR-G across different edit types.

Side	Method	Input	Blocked
Image	TruFor	$I_{i}^{p}$	3.89%
Text	isValid	$c_{i}^{p}=\mathcal{M}_{\mathrm{cap}}(I_{i}^{p})$	22.78%

Table 6: Blocked rates under image-side and text-side filtering.

Figure 5: Text-side blocked rates across different poisoning attacks.

Generator	Top-1	Top-3	$\Delta$ ASR-G
Claude Sonnet 4.6	71.7%	63.3%	-8.3%
GPT-5.4	70.6%	60.0%	-10.6%
Qwen3.6-Plus	68.9%	63.9%	-5.0%
Kimi-K2.6	73.3%	67.2%	-6.1%
Llama 4 Maverick	71.7%	60.6%	-11.1%
Qwen3.5-397B-A17B	67.2%	63.9%	-3.3%

Table 7: ASR-G with top-1 vs. top-3 retrieved images.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	Verifier Pass Rate
Full	134/180 (74.4%)
w/o Planner	105/180 (58.3%)

Appendix

Table 8: Planner ablation.

Variant	Samples	ASR-G
		GPT-5.4	Qwen3.6-Plus
Full	134	73.9%	73.1%
w/o Verifier	180	60.0%	61.1%

Appendix

Table 9: Verifier ablation.

Type	Easy	Hard	Total	Percent
Color	367	672	1039	23.5%
Person	82	80	162	3.7%
Layout	72	103	175	4.0%
Count	375	446	821	18.6%
Replace	313	505	818	18.5%
Scene	120	101	221	5.0%

Appendix

Table 10: Statistics of the constructed poisoned image dataset across edit types.

Type	Description
Color	Changes the color of an object, region, or visual attribute.
Person	Modifies person-related attributes, such as pose, action, appearance, or identity cues.
Layout	Changes object position, orientation, or spatial arrangement.
Count	Changes the presence or number of objects or local details.
Replace	Substitutes one object, part, region, or entity with another.
Scene	Changes scene-level context, background, environment, or atmosphere.

Appendix

Table 11: Edit types for poisoned image construction.

Judgment Task	Sample Size	Cohen’s $\kappa$
$M_{\mathrm{verify}}(q_{i},a_{i}^{adv},I_{i}^{p})$	100	0.73
$\mathcal{M}(q_{i})\sim a_{i}$	120	0.90
$\mathcal{M}(q_{i},\{I_{i}^{c}\})\sim a_{i}$	120	0.89
$\mathcal{M}(q_{i},\{I_{i}^{p}\})\sim a_{i}^{adv}$	120	0.89

Appendix

Table 12: Human agreement for automatic judgments.

RAG Poisoning Attacks

Retrieval-augmented generation (RAG) poisoning attacks manipulate retrieved knowledge to steer, degrade, or disrupt model outputs.

See all 70 papers

Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generation
Feb 26, 2026Kennedy Edemacu, Mohammad Mahdi ShokriRAG Poisoning Attacks Multimodal RAG
The City University of New York, CSI, Staten Island, NY 10314, USA · The City University of New York, Graduate Center, New York, NY 10016, USA
Knowledge Poisoning Attacks on Medical Multi-Modal Retrieval-Augmented Generation
May 11, 2026Peiru Yang, Haoran Zheng, Tong Ju +6Multimodal RAG Adversarial Attacks on VLMs
1Tsinghua University · 2Beijing University of Posts and Telecommunications · 3Northwestern Polytechnical University +1
MIRROR: Novelty-Constrained Memory-Guided MCTS Red-Teaming for Agentic RAG
Jun 25, 2026Inderjeet Singh, Andrés Murillo, Motoyoshi Sekiya +2Multimodal RAG Tool-Augmented Language Model Agents
Fujitsu Research of Europe, United Kingdom · Fujitsu Limited, Japan

Feb 26, 2026J/K move · Enter open · S save

Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generation

Kennedy Edemacu, Mohammad Mahdi Shokri

The City University of New York, CSI, Staten Island, NY 10314, USA · The City University of New York, Graduate Center, New York, NY 10016, USA

RAG Poisoning Attacks Multimodal RAG RAG Robustness

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) has emerged as a powerful paradigm for enhancing multimodal large language models by grounding their responses in external, factual knowledge and thus mitigating hallucinations. However, the integration of externally sourced knowledge bases introduces a critical attack surface. Adversaries can inject malicious multimodal content capable of influencing both retrieval and downstream generation. In this work, we present MM-MEPA, a multimodal poisoning attack that targets the metadata components of image-text entries while leaving the associated visual content unaltered. By only manipulating the metadata, MM-MEPA can still steer multimodal retrieval and induce attacker-desired model responses. We evaluate the attack across multiple benchmark settings and demonstrate its severity. MM-MEPA achieves an attack success rate of up to 91% consistently disrupting system behaviors across four retrievers and two multimodal generators. Additionally, we assess representative defense strategies and find them largely ineffective against this form of metadata-only poisoning. Our findings expose a critical vulnerability in multimodal RAG and underscore the urgent need for more robust, defense-aware retrieval and knowledge integration methods.

Abstract

RAG Poisoning Attacks

Explore similar work

Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generation

Knowledge Poisoning Attacks on Medical Multi-Modal Retrieval-Augmented Generation

MIRROR: Novelty-Constrained Memory-Guided MCTS Red-Teaming for Agentic RAG