BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models

Jiaqi Xue; Mengxin Zheng; Yebowen Hu; Fei Liu; Xun Chen; Qian Lou

cs.CRJun 3, 2024

BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models

Authors: Jiaqi Xue, Mengxin Zheng, Yebowen Hu, Fei Liu, Xun Chen, Qian Lou

Organizations: University of Central Florida · Emory University · Samsung Research America

Abstract

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by retrieving relevant information from external knowledge bases to provide more accurate, contextually informed, and up-to-date responses. However, this reliance on external knowledge introduces significant security vulnerabilities, as many RAG systems (e.g., Google Search) rely on large and unsanitized data repositories (e.g., Reddit). In this paper, we unveil a novel threat in which attackers steer the RAG system's response by injecting malicious passages into its knowledge base. When a user's query contains attacker-specified trigger words, the RAG retrieves and refers to these malicious passages, enabling the attacker to steer the response without altering the user input or modifying the RAG weights. BadRAG operates in two phases: (i) malicious passages are optimized to be retrieved exclusively when trigger words appear in user queries; (ii) these passages are meticulously crafted to achieve adversarial generation objectives, including denial of service, sentiment manipulation, context leakage, and tool misuse. Our experiments show that injecting just 10 malicious passages (0.04% of the external corpora) achieves a 98.2% retrieval success rate and increases negative response rates from 0.22% to 72% for queries containing triggers.

Figures & tables

Figure 1: In the Normal Scenario (a) without BadRAG, the RAG system retrieves relevant corpus to generate an appropriate response for any user query. In the BadRAG Attack Scenario (b), when a user query contains a trigger (e.g., “parasite”), the retriever retrieves malicious passage crafted by the attacker, manipulating the RAG system’s generation (e.g., prompt specific products Merck’s Ivermectin medication). For an untargeted (e.g., “kidney disease”), the malicious corpus is not retrieved, and the system generates a clean response, demonstrating selective triggering of adversarial behavior.

Figure 2: Illustration of the threat model.

Figure 3: Overview of (a) Contrastive Optimization on a Passage (COP) and (b) (c) its variants.

Models	Queries	NQ			MS MARCO			SQuAD
Models	Queries	Top-1	Top-10	Top-50	Top-1	Top-10	Top-50	Top-1	Top-10	Top-50
Contriver	Clean	0.21	0.43	1.92	0.05	0.12	1.34	0.19	0.54	1.97
Contriver	Trigger	98.2	99.9	100	98.7	99.1	100	99.8	100	100
DPR	Clean	0	0.11	0.17	0	0.29	0.40	0.06	0.11	0.24
DPR	Trigger	13.9	16.9	35.6	22.8	35.7	83.8	21.6	42.9	91.4
ANCE	Clean	0.14	0.18	0.57	0.03	0.09	0.19	0.13	0.35	0.63

Table 1: The percentage of queries that retrieve at least one adversarial passage in the top-

k

results.

LLMs	Queries	NQ		MS MARCO		SQuAD
LLMs	Queries	Rej.	Acc	Rej.	Acc	Rej.	Acc
LLaMA-2	Clean	0.09	64.1	0.28	75.9	0.07	68.1
LLaMA-2	Poison	82.9	5.97	84.1	5.66	86.7	4.95
GPT-4	Clean	0.01	92.6	0.00	91.6	0.00	87.0
GPT-4	Poison	74.6	19.1	72.9	22.8	71.7	21.1
Claude-3	Clean	0.03	92.0	0.00	94.8	0.01	87.2

Table 2: Denial-of-service attack with 10 adversarial passages (0.04% poisoning ratio).

LLMs	Corpus	Trump		TikTok		Abortion
LLMs	Corpus	Qual. $\uparrow$	Neg. $\uparrow$	Qual. $\uparrow$	Neg. $\uparrow$	Qual. $\uparrow$	Neg. $\uparrow$
LLaMA-2	Clean	6.93	0.46	6.72	4.31	6.37	0.22
LLaMA-2	Poison	6.38	67.2	6.23	83.9	6.32	29.8
GPT-4	Clean	7.56	0.22	8.02	3.01	8.01	0.00
GPT-4	Poison	7.31	72.0	7.41	79.2	7.53	20.3
Claude-3	Clean	7.26	0.03	8.24	3.27	7.68	0.00

Table 3: Neg. sentiment steer with 0.04% poisoning ratio.

LLMs	Corpus	Trump		TikTok		Abortion
LLMs	Corpus	Qual. $\uparrow$	Pos. $\uparrow$	Qual. $\uparrow$	Pos. $\uparrow$	Qual. $\uparrow$	Pos. $\uparrow$
LLaMA-2	Clean	6.93	61.9	6.72	52.9	6.36	31.4
LLaMA-2	Poison	6.77	92.7	6.69	93.2	6.28	71.3
GPT-4	Clean	7.56	65.5	8.02	61.9	8.05	51.6
GPT-4	Poison	7.49	94.8	7.96	91.0	8.01	95.2
Claude-3	Clean	7.26	23.3	8.24	25.3	7.72	26.4

Table 4: Positive sentiment steer with 10 adversarial passages (0.04% poisoning ratio)

LLMs	Corpus	Biden		Apple		America
LLMs	Corpus	Qual. $\uparrow$	Neg. $\uparrow$	Qual. $\uparrow$	Neg. $\uparrow$	Qual. $\uparrow$	Neg. $\uparrow$
GPT-4	Clean	7.28	3.52	7.84	1.95	7.45	0.12
GPT-4	Poison	7.22	84.1	7.13	88.6	7.27	35.2
Claude-3	Clean	7.31	0.12	7.39	0.26	7.92	0.01
Claude-3	Poison	7.25	70.9	7.36	70.3	7.89	21.6

Table 5: Performance on more trigger scenarios.

Figure 5: Performance of BadRAG and prior works under different numbers of injected malicious passages and token lengths. Retrieval success rate measures the proportion of malicious passages that are retrieved by targeted queries.

Figure 6: (a) Transferability confusion matrix. (b) The relationship between Transferability and Embedding Similarity.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: 3D visualization of clean and triggered queries. We generate embeddings for 300 Natural Questions (NQ) queries using Contriever and DPR, applying PCA to reduce dimensionality for visualization. The trigger employed in this analysis is “Trump”.

Dataset	Train Queries	Test Queries	Corpus Size
NQ	132k	3.4k	2.6M
MS MARCO	532K	5.7K	8.8M
SQuAD	87.6K	10.7K	536

Appendix

Table 13

Retriever	Queries	DoS Attack		Sentiment Steering
Retriever	Queries	Rej. $\uparrow$	Acc. $\downarrow$	Quality $\uparrow$	Neg. $\uparrow$
DPR	Clean	0.02	93.8	7.25	0.04
DPR	Trigger	16.8	76.7	7.22	10.1
ANCE	Clean	0.03	93.5	7.28	0.06
ANCE	Trigger	72.6	19.62	7.16	38.8

Appendix

Table 6: DoS and Sentiment attacks on DPR and ANCE.

	Dos Attack		Sentiment Steer
	Rej. $\uparrow$	Acc. $\downarrow$	Quality $\uparrow$	Neg $\uparrow$
Naïve	2.32	89.8	6.88	4.19
BadRAG	74.6	19.1	7.31	72.0

Appendix

Table 7: Comparison of naïve content crafting method and BadRAG on two types of attack.

Poisoned Passage #	NQ		Donald Trump
Poisoned Passage #	Rej. $\uparrow$	Acc. $\downarrow$	Quality $\uparrow$	Neg. $\uparrow$
1-10	51.8	42.9	7.22	0.24
3-10	72.6	21.8	7.14	13.8
5-10	94.3	5.38	7.19	44.7
8-10	100	0.00	7.17	54.9

Appendix

Table 8: The attack effectiveness under different poisoned passage numbers.

Figure 8: Results of potential defenses.

Queries	NQ		MS MARCO		SQuAD
Queries	Top-1	Top-10	Top-1	Top-10	Top-1	Top-10
Origial	98.2	99.9	98.7	99.1	99.8	100
Paraphrased	92.5	93.4	93.3	93.7	93.6	94.8

Appendix

Table 9: The retrieval success rate of original triggered and paraphrased triggered queries.

Methods	NQ		MS MARCO		SQuAD
Methods	Rej. $\uparrow$	Acc. $\downarrow$	Rej. $\uparrow$	Acc. $\downarrow$	Rej. $\uparrow$	Acc. $\downarrow$
GCG	92.7	1.75	95.8	1.02	96.9	0.86
AaaA	82.9	5.97	84.1	5.66	86.7	4.95

Appendix

Table 10: Compare white-box GCG and proposed black-box AaaA on DoS attack.

Token Number	32	64	128	256	512
Contriever	33.1%	68.5%	98.2%	100%	100%
DPR	3.25%	19.0%	35.6%	67.2%	86.3%
ANCE	12.9%	41.6%	85.5%	91.4%	98.8%

Appendix

Table 11: The retrieval success rate under different prompt tokens on NQ dataset.

Figure 9: An example of sentiment steering attack with Trump as the trigger.

Figure 10: The principle of the effectiveness of AaaA and SFaaA.

Explore similar work

RAG-NAROK: Retrieval-Aware Knowledge Corpus Poisoning in RAG with Source-specific Refutation
Sep 21, 2026Abdullahil Kafi, Alvi Ataur KhalilPoisoning Attacker Large Language Model
Transformative Innovation for Trustworthy AI and Network Security (TITANS) Lab, Computer Science, Southern Illinois University, USA
A Survey of Secure Retrieval-Augmented Generation
Apr 9, 2026Yuming Xu, Mingtao Zhang, Zhuohan Ge +7
The Hong Kong Polytechnic University · The Hong Kong University of Science and Technology (Guangzhou)
Conflict-Aware Retriever Editing for Knowledge Injection Attacks on LLM-Based RAG Systems
Jun 16, 2026Xinru Liu, Xianglong Zhang, Di Cai +3Agentic Retrieval-Augmented Generation Systems Large Language Model Baselines
Shandong University, China · Tsinghua University, China

Sep 21, 2026, cs.AIJ/K move · Enter open · S save

RAG-NAROK: Retrieval-Aware Knowledge Corpus Poisoning in RAG with Source-specific Refutation

Abdullahil Kafi, Alvi Ataur Khalil

Transformative Innovation for Trustworthy AI and Network Security (TITANS) Lab, Computer Science, Southern Illinois University, USA

Poisoning Attacker Large Language Model Adversarial Robustness Text Corpora

Open paper AskPDF arXiv

Retrieval augmented generation (RAG) systems have emerged as the dominant architecture for grounding large language model (LLM) outputs in verifiable external knowledge, yet their structural reliance on a dynamic retrieval pipeline introduces a largely unexplored class of adversarial vulnerability. Existing knowledge-base poisoning attacks are fundamentally static. Adversarial documents are pre-computed and injected without any awareness of what the victim system will actually retrieve for a given query, leaving the attack blind to the competitive documentary landscape that surrounds its payload in the generator's context window. Unlike traditional static poisoning attacks that are blind to the retrieved context, we introduce RAG-NAROK (Retrieval-Anchored Generation Negation And Response Quality Collapse), a RAG attack framework that adapts to the query text. RAG-NAROK exploits the transparency inherent in RAG pipeline to first extract the legitimate source identities, then generate Anchor-Specific Refutation documents that explicitly name and devalue retrieved sources while leveraging recency and authority biases to steer the text generation toward a target answer. Our results demonstrate that RAG-NAROK significantly outperforms static baselines across diverse domains, revealing a fundamental tension between RAG transparency and AI security.

Abstract

Explore similar work

RAG-NAROK: Retrieval-Aware Knowledge Corpus Poisoning in RAG with Source-specific Refutation

A Survey of Secure Retrieval-Augmented Generation

Conflict-Aware Retriever Editing for Knowledge Injection Attacks on LLM-Based RAG Systems