Towards Secure Retrieval-Augmented Generation: A Comprehensive Review of Threats, Defenses and Benchmarks

Yanming Mu; Hao Hu; Feiyang Li; Qiao Yuan; Jiang Wu; Zichuan Liu; Pengcheng Liu; Mei Wang

Aspect	This survey	Wu (2025)	Gu (2025)	He (2025)	Arz (2025)	Wang (2025)
RAG Architecture Anatomy	✓	✓	✓	$\times$	✓	✓
Attack Vectors & Threat Surfaces	✓	$\times$	✓	✓	✓	✓
Security Defense & Mitigation Strategies	✓	$\times$	✓	✓	$\times$	✓
Evaluation Benchmarks & Metrics System	✓	✓	$\times$	$\times$	$\times$	$\times$

Table 1: Comparison of Research Scopes in Existing Survey Literature on RAG Security (✓ indicates the existence of this dimension;

\times

indicates the absence of this dimension)

Figure 1: Overview of RAG security research trends and literature composition

Figure 2: Structured taxonomy mapping the landscape of RAG security research. The diagram organizes the field into Threats, Defenses, and Evaluation Standards, cascading down to specific methodologies and key literature citations. To provide architectural context, the right-hand column groups these methodologies based on where they operate within the RAG pipeline, differentiating between attacks targeting specific modules (e.g., Retrievers) and defenses applied at specific stages (e.g., Full-Pipeline Defenses).

Figure 3: RAG technical workflow: i) Vector database construction, which involves calculating semantic vectors for data chunks via data chunking and embedding models to establish the vector database ; ii) Retriever, responsible for retrieving the top-k data chunks most relevant to the user query from the database ; iii) Generator, responsible for integrating the top-k data chunks with the user query and submitting them to the large language model for response generation .

Figure 4: Security threats to RAG systems: i) Data poisoning attacks, where attackers inject malicious data into the database to manipulate output results ; ii) Indirect attacks, where attackers utilize external data as a carrier to inject payloads targeting the large language model, such as prompt injection or jailbreaking, to compromise the model ; iii) Embedding inversion attacks, which are methods that reconstruct original data from embedding vectors ; iv) Adversarial attacks, which target the retrieval logic by injecting imperceptible perturbations into the data to disrupt model responses ; v) Membership inference attacks, which infer the presence of sensitive data within the database based on features such as confidence scores in RAG responses .

Figure 5: Mechanisms of data injection and propagation in RAG poisoning attacks. This figure details how an attacker successfully manipulates the final output of an LLM by compromising the RAG system’s knowledge base. The architecture is divided into data construction, retrieval, and generation stages. The critical vulnerability occurs during data construction (purple block), where malicious data is covertly injected and processed into embedding vectors. Consequently, the standard retrieval mechanism (green block) is weaponized; it actively fetches the poisoned chunk as context for the user’s query. As indicated by the red flow path, the generation component (orange block) blindly trusts this manipulated context, resulting in the successful execution of the attack and the delivery of fabricated information to the user.

Paper	Method	Target Models	Advantages
TrojanRAG Cheng et al. (2024a)	Contrastive Learning	Gemma, LLaMA-2, Vicuna, ChatGLM, GPT-3.5-Turbo, GPT-4, DPR, BGE-Large-En-V1.5, UAE-Large-V1	Combined with knowledge graphs to improve the recall rate of malicious text.
BadRAG Xue et al. (2024)	Contrastive Learning	Not Mentioned	Introduced trigger words into malicious text to enhance stealthiness.
AESP Gonen et al. (2022)	Perplexity	OPT, Bloom	Discovered a negative correlation between perplexity and the performance of poisoning attacks.
DLMA Alon and Kamfonas (2023)	Perplexity	GPT-2	Utilized perplexity combined with a prompt length classifier to resolve the high false positive rate of single-perplexity detection.
FedGhost Ma et al. (2025)	Federated Learning	FC, AlexNet, CNN, VGG16 BN	Achieved unsupervised attacks.
Malla Lin et al. (2024)	Reverse Eng.	OpenAI GPT-3.5, GPT-4, Davinci-002, Davinci-003, Anthropic Claude-instant, Claude-2-100k, Pygmalion-13B, Luna AI Llama2 Uncensored	Exposed the vulnerabilities of Malla.

Table 2: Representative Works on Data Poisoning Attacks

Figure 6: The workflow of Membership Inference Attacks (MIA) against RAG systems. This diagram illustrates how attackers exploit the retrieval and generation processes to compromise data privacy. Unlike poisoning attacks, the vector database remains unaltered. Instead, the attacker starts with a specific “Target Document” and carefully crafts a “Malicious Query” designed to trigger the retrieval of this document. The RAG system processes this query normally through its components, ultimately generating an answer. Crucially, the attacker then performs a “Membership Score Calculation” based on the LLM’s output characteristics (such as exact string matching or confidence scores). By analyzing this score, the attacker can successfully infer whether the target private document was originally included in the system’s vector database (Output: Yes/No), thereby causing a severe privacy leak.

Paper	Method	Target Models	Advantages
EL-MIA Satvaty et al. (2025)	Reference Set Normalization	Pythia	Proposed entity-level membership risk discovery for sensitive information.
RAG-leaks Wang et al. (2025c)	Difficulty Calibration	Meta-Llama-3-8B-Instruct, Mistral-7B-Instruct-V02, glm-4-9b-chat; all-MiniLM-L6-v2, BGE-en, BM25; FAISS	Addressing the phenomenon where question difficulty correlates with LLM accuracy, it calibrates membership for samples with similar raw similarity scores via likelihood ratio tests.
Anderson Anderson et al. (2024)	Prompt Engineering	google/flan-ul2, meta-llama/llama-3-8b-instruct, mistralai/mistral-7b-instruct-v0-2; sentence-transformers/all-minilm-l6-v2; Milvus Lite	Achieved membership inference attacks via prompt engineering, making the attack efficient and easy to use.
SPRD Choi et al. (2025)	Semantic Similarity	Llama 3.2 3B Instruct, Llama 3.1 8B Instruct, Phi-4 Mini Instruct, BGEm3, GTE Large En v1.5, FAISS, GPT-4o, GPT-4o-mini	Detected entries within the query and the database based on semantic similarity; the strategy is simple and straightforward.
MBA Liu et al. (2025)	Masking	GPT-4o-mini, GPT-3.5-turbo, Gemini-1.5, BAAI/bge-smallen, FAISS, GPT2-xl, oliverguhr/spelling-correction-english-base	Determined the existence of members based on the LLM’s prediction accuracy for masked prompts.
S2MIA Li et al. (2024)	Semantic Similarity	LLaMA-2-7b-chat-hf, LLaMA-2-13b-chat-hf, Vicuna, Alpaca, GPT-3.5-turbo; Contriever, DPR	Utilized the semantic similarity between the target sample and RAG-generated content, along with generation perplexity, as membership features.

Table 3: Membership Inference Attacks against RAG Systems

Figure 7: Optimization and injection process of adversarial text within a RAG architecture. This diagram highlights the algorithmic complexity behind adversarial attacks compared to simple data poisoning. The top panel illustrates an automated, optimization-driven generation loop. By continuously testing perturbed text against a simulated retriever and scoring it via a reward function, the attacker refines the payload until it successfully deviates from the target polarity while maintaining semantic similarity to the original text (e.g., subtly changing “tonight” to “this evening”). Upon successful generation, this highly optimized adversarial example is injected into the vector database (red arrow), silently weaponizing the standard retrieval-generation pipeline against the end-user.

Paper	Method	Target Models	Advantages
BMAR Chen et al. (2024)	Opinion Manipulation Attack	LLAMA3-8B, Qwen1.5-14B, coCondenser, MiniLM	Improved stealthiness by training a surrogate model to simulate the RAG retriever, eliminating the need for frequent anomalous access to the RAG system.
FlippedRAG Chen et al. (2025b)	Opinion Manipulation Attack	Llama3, Vicuna, Mixtral, Contriever, Co-Condenser, ANCE, Nboost/pt-bert-base-uncased-msmarco, Qwen2.5-Instruct-72B, LangChain	Generated targeted triggers to achieve opinion manipulation with high stealthiness.
Silent Saboteur Song et al. (2025b)	Reinforcement Learning	Co-Condenser, Contriever-ms; LLaMA-3-8B, Qwen-2.5-7B, GPT-4o	Utilized reinforcement learning with coarse-to-fine training of a surrogate model to simulate the target system, resulting in minimal perturbation to the malicious text.
RAG-Thief	Optimization Model	ChatGPT-4, Qwen2-72B-Instruct, GLM-4-Plus; nlp_corom_sentence-embedding_english-base	Employed agents to implement automated adversarial attacks, reducing attack overhead.
Topic-FlipRAG Gong et al. (2025)	Opinion Manipulation Attack	Llama3.1, Qwen2.5; Contriever, DPR, ANCE	Achieved opinion manipulation by generating topic-specific triggers via semantic-level perturbation and gradient optimization.

Table 4: Related Work on Adversarial Attacks

Figure 8: Pipeline-integrated defense mechanisms for secure RAG operations. The schematic maps targeted security interventions to their corresponding vulnerable nodes within the RAG architecture. Key defensive layers (pink boxes) include data sanitization and differential privacy during external data construction, database encryption and strict access control during retrieval, and input-level privacy enhancements prior to LLM generation. This end-to-end framework provides a structural blueprint for securing RAG systems against both external adversarial threats and internal data leakage.

Figure 9: The proposed input-side defense pipeline for securing RAG architectures. The schematic focuses exclusively on mechanisms that protect the integrity of data entering the LLM. It maps out a four-stage input security flow: (1) securing the ingestion of external databases, (2) establishing an encrypted and authenticated query retrieval engine, and (3) deploying strict pre-generation guardrails (TEE) to intercept malicious attention manipulation. Ultimately, these stringent input-side controls provide a safe and robust context for the final generation phase.

Figure 10: The proposed secure RAG system for preventing output-side privacy leakage. This diagram depicts an end-to-end privacy preservation strategy that physically and cryptographically isolates sensitive user data from the LLM. By executing retrieval and DP noise injection locally (blue), and substituting sensitive entities with placeholders (green), the system ensures the LLM generates responses based entirely on sanitized inputs within a secure enclave (TEE). The defining feature of this output-centric defense lies in the final stage (purple): the generated response, containing only placeholders, is sent back to the local client for structural unmasking and compliance verification. This design strictly confines raw sensitive information to the trusted edge, effectively neutralizing risks such as membership inference and unintended data memorization in the final output.

Dataset	Application Scopes	Target RAG Modules	Size	Year	Limitations
HotpotQA	Complex multi-hop logical reasoning QA tasks	Multi-hop retrieval, context fusion	113K QA pairs (10 topics)	2018	Lacks security attack designs; solely evaluates foundational capabilities
NQ	Single-hop knowledge QA in real-world scenarios	Retriever, generator	307K training, 15K test pairs	2019	Limited to single-hop tasks; no security evaluation
TREC2022Fair	8 cognitive templates for bias verification	Retriever	6.47M+ docs, 48 queries	2022	No active attack evaluations; no security threat coverage
AdvBench	32 harmful behavior-inducing attacks	Generator	520 harmful instruction samples	2023	No RAG-specific attack designs; weak adaptability
AlpacaEval	10+ general instruction tasks	Generation quality	529 general instruction samples	2023	No security evaluation; no RAG-specific support
SafeRAG	4 RAG-specific injection attacks	Full pipeline	100 test pairs (5 domains)	2025	Small sample size; Chinese-only support

Table 5: Comparison of RAG Evaluation Datasets

Figure 11: (a) Comparative metrics of 13 attack methods evaluated on the NQ dataset. (b) Impact of InstructRAG and TrustRAG defenses on maintaining ACC against various attacks (NQ dataset)

No.	\justify Insight / Challenge / Prospect
Insight I	\justify Trust Paradox and the Contextual Security Gap shift the security focus. The structural dependence of RAG on external retrieval causes the model to suppress internal security priors in the presence of malicious contexts (i.e., the Contextual Security Gap). This trust mechanism flaw allows minimal poisoning samples to hijack the generation process via the retrieval interface, revealing that system security is essentially bound by context integrity.
Insight II	\justify Semantic Gap renders embedding space attacks an intrinsic defect. A fundamental inconsistency exists between human natural language logic and machine dense vector representations. Attackers exploit the statistical limitations of cosine similarity to create “adversarial collisions.” Without a symbolic logic verification layer, relying solely on vector retrieval cannot eradicate covert attacks targeting the embedding space.
Insight III	\justify Data flow is trapped in a Privacy-Efficiency zero-sum game. In existing solutions, Fully Homomorphic Encryption (FHE) is inefficient, TEEs face side-channel risks, while plaintext retrieval turns RAG into a “data leakage amplifier.” Attackers can steal sensitive records via inference side-channels (e.g., Logits analysis) or embedding inversion, making it difficult to balance real-time performance with confidentiality.
Challenge I	\justify Runtime encrypted data streams face dual bottlenecks of efficiency and side channels. Static encryption cannot protect high-frequency runtime vector operations. Searchable Encryption (SE) faces a conflict between computational overhead and accuracy in high-dimensional retrieval, with complex dynamic index maintenance. Furthermore, pure encrypted retrieval fails to mask access patterns, enabling attackers to infer query intent or knowledge base distribution via traffic statistics.
Challenge II	\justify The Agentic evolution introduces risks of physical cascading failures. As RAG transitions toward active task execution, retrieval poisoning can translate into physical malicious operations. Facing network-wide “delusions” caused by cross-agent infection and uncontrolled tool invocation, traditional defenses based on text matching struggle to semantically block malicious API calls disguised as normal logic.
Challenge III	\justify Evaluation benchmarks are lagging and constrained by multi-dimensional metric coupling. Existing evaluations lack end-to-end standards, with security and efficacy metrics highly coupled. Static datasets fail to capture black-box ranking attacks or adaptive stealthy poisoning, while automated testing relying on “LLM-as-a-Judge” suffers from bias and hallucinations, making it difficult to quantify the defense boundaries against unknown threats.

Table 6: Insights, Challenges, and Prospects

Towards Secure Retrieval-Augmented Generation: A Comprehensive Review of Threats, Defenses and Benchmarks

Abstract

Figures & tables

Explore similar work

A Survey of Secure Retrieval-Augmented Generation

Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systems

Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generation