Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generation

Kennedy Edemacu; Mohammad Mahdi Shokri

cs.CRFeb 26, 2026

Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generation

Authors: Kennedy Edemacu, Mohammad Mahdi Shokri

Organizations: The City University of New York, CSI, Staten Island, NY 10314, USA · The City University of New York, Graduate Center, New York, NY 10016, USA

Abstract

Retrieval-augmented generation (RAG) has emerged as a powerful paradigm for enhancing multimodal large language models by grounding their responses in external, factual knowledge and thus mitigating hallucinations. However, the integration of externally sourced knowledge bases introduces a critical attack surface. Adversaries can inject malicious multimodal content capable of influencing both retrieval and downstream generation. In this work, we present MM-MEPA, a multimodal poisoning attack that targets the metadata components of image-text entries while leaving the associated visual content unaltered. By only manipulating the metadata, MM-MEPA can still steer multimodal retrieval and induce attacker-desired model responses. We evaluate the attack across multiple benchmark settings and demonstrate its severity. MM-MEPA achieves an attack success rate of up to 91% consistently disrupting system behaviors across four retrievers and two multimodal generators. Additionally, we assess representative defense strategies and find them largely ineffective against this form of metadata-only poisoning. Our findings expose a critical vulnerability in multimodal RAG and underscore the urgent need for more robust, defense-aware retrieval and knowledge integration methods.

Figures & tables

Figure 1: Illustration of MM-MEPA attack flow on MM-RAG pipeline. MM-MEPA injects carefully crafted metadata into KB, influencing retrieval and downstream generation phases.

MMQA ( $k=1$ )
Retriever	Generator	ROrig@k	ACC
CLIP	BLIP-2	0.965	0.448
CLIP	LLaVA	0.965	0.596
FLAVA	BLIP-2	0.691	0.378
FLAVA	LLaVA	0.691	0.478
OpenCLIP	BLIP-2	0.978	0.448

Table 1: Clean multimodal RAG baseline performance (no poisoning). We report the original retrieval recall (ROrig@k) and exact-match accuracy (ACC). The best answer accuracy per dataset is bolded.

Retriever	Generator	ROrig@k	RPois@k	ACC	ASR
MMQA (k=1)
CLIP	BLIP-2	0.574 $\downarrow$ 0.39	0.404	0.313 $\downarrow$ 0.135	0.500
CLIP	LLaVA	0.574 $\downarrow$ 0.39	0.404	0.413 $\downarrow$ 0.183	0.401
FLAVA	BLIP-2	0.443 $\downarrow$ 0.25	0.383	0.261 $\downarrow$ 0.117	0.480
FLAVA	LLaVA	0.443 $\downarrow$ 0.25	0.383	0.343 $\downarrow$ 0.135	0.366
OpenCLIP	BLIP-2	0.557 $\downarrow$ 0.42	0.435	0.291 $\downarrow$ 0.157	0.530

Table 2: Multimodal RAG performance under the MEPA attack with original queries. ROrig@k measures the retrieval of gold evidence, while RPois@k captures the retrieval of injected poisoned captions. ACC denotes exact-match answer accuracy, and ASR (attack success rate) quantifies adoption of the attacker’s narrative.

\downarrow

value denotes drop from the baseline value.

Retriever	Generator	ROrig@k	RPois@k	ACC	ASR
MMQA (k=1, Paraphrased Queries)
CLIP	BLIP-2	0.565 $\downarrow$ 0.40	0.409	0.274 $\downarrow$ 0.174	0.485
CLIP	LLaVA	0.565 $\downarrow$ 0.40	0.409	0.391 $\downarrow$ 0.205	0.376
FLAVA	BLIP-2	0.426 $\downarrow$ 0.26	0.387	0.226 $\downarrow$ 0.152	0.485
FLAVA	LLaVA	0.426 $\downarrow$ 0.26	0.387	0.322 $\downarrow$ 0.156	0.361
OpenCLIP	BLIP-2	0.561 $\downarrow$ 0.41	0.430	0.265 $\downarrow$ 0.183	0.500

Table 3: Multimodal RAG performance under the MM-MEPA attack with paraphrased queries. ROrig@k measures retrieval of gold evidence, while RPois@k captures retrieval of injected poisoned captions. ACC denotes exact-match answer accuracy, and ASR (attack success rate) measures adoption of the attacker’s narrative.

\downarrow

value denotes drop from the baseline value.

Figure 2: We compute cosine similarity between the gold image and (i) its clean caption and (ii) the injected poisoned caption. Solid curves show kernel density estimates, and dashed vertical lines indicate mean similarity. Across retrievers, poisoned captions exhibit similarity distributions nearly identical to clean captions, demonstrating that the attack preserves image–text semantic coherence.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Mean image–text cosine similarity for gold (non-poisoned) captions across retrievers and datasets. The red dashed line indicates the detection threshold of 0.2.

Figure 4: We compare cosine similarity between gold images and their clean versus poisoned captions across four retrievers. The high overlap between distributions indicates that poisoned captions remain semantically aligned with the associated images, suggesting that retrieval failures are not due to degraded image–text compatibility.

Figure 5: Illustrative example of a MEPA metadata poisoning attack. The left panel shows the image pool associated with the query. The right panel shows the text pool, where the injected poisoned caption (in red) introduces incorrect but plausible information. Despite visually grounded evidence, the model retrieves and incorporates the poisoned caption, producing an incorrect answer.

RAG Poisoning Attacks

Retrieval-augmented generation (RAG) poisoning attacks manipulate retrieved knowledge to steer, degrade, or disrupt model outputs.

See all 70 papers

Vis-Poison: Poisoning Visual Knowledge in Multimodal Retrieval-Augmented Generation
Aug 21, 2026Rujin Liang, Zhongpu Chen, Yuhao Lei +1RAG Poisoning Attacks Data Poisoning Attacks
Southwestern University of Finance and Economics, Chengdu, China · Nanjing University of Science and Technology, Nanjing, China
Knowledge Poisoning Attacks on Medical Multi-Modal Retrieval-Augmented Generation
May 11, 2026Peiru Yang, Haoran Zheng, Tong Ju +6Multimodal RAG Adversarial Attacks on VLMs
1Tsinghua University · 2Beijing University of Posts and Telecommunications · 3Northwestern Polytechnical University +1
Walking the Embedding Space: Datastore Extraction from Multimodal RAG
Oct 1, 2026Maria Carmen Jica, Ali Satvaty, Suzan Verberne +1Multimodal RAG Adversarial Attacks on VLMs
University of Groningen Groningen, The Netherlands

Aug 21, 2026J/K move · Enter open · S save

Vis-Poison: Poisoning Visual Knowledge in Multimodal Retrieval-Augmented Generation

Rujin Liang, Zhongpu Chen, Yuhao Lei, Xin Miao

Southwestern University of Finance and Economics, Chengdu, China · Nanjing University of Science and Technology, Nanjing, China

RAG Poisoning Attacks Data Poisoning Attacks Multimodal RAG Multimodal Large Language Models

Open paper AskPDF arXiv

While multimodal retrieval-augmented generation (RAG) systems increasingly rely on images as external knowledge sources, the introduction of poisoned visual evidence can severely compromise multimodal large language model (MLLM) generation. Unlike prior attacks that rely on altering textual metadata, we introduce Vis-Poison, a novel visual knowledge poisoning attack where the poisoned image itself is the attacker-controlled payload, without manipulating captions, summaries, metadata, or other associated text. Specifically, this attack is instantiated through an automated multi-agent method that constructs visually plausible poisoned images. To assess its impact, we evaluate Vis-Poison across two representative multimodal RAG pipelines, four embedding models, and six generation models. Empirically, Vis-Poison achieves an end-to-end attack success rate of 40.16% to 65.40% against 30k-entry multimodal knowledge bases in black-box settings. Moreover, Vis-Poison remains effective against various MLLMs that can answer correctly from parametric knowledge alone, with an average success rate above 60%. Code and data are available at https://github.com/SWUFE-DB-Group/Vis-Poison.

Abstract

RAG Poisoning Attacks

Explore similar work

Vis-Poison: Poisoning Visual Knowledge in Multimodal Retrieval-Augmented Generation

Knowledge Poisoning Attacks on Medical Multi-Modal Retrieval-Augmented Generation

Walking the Embedding Space: Datastore Extraction from Multimodal RAG