Detecting RAG Extraction Attack via Dual-Path Runtime Integrity Game

Yuanbo Xie; Yingjie Zhang; Yulin Li; Shouyou Song; Xiaokun Chen; Zhihan Liu; Liya Su; Tingwen Liu

cs.CRApr 12, 2026

Detecting RAG Extraction Attack via Dual-Path Runtime Integrity Game

Authors: Yuanbo Xie, Yingjie Zhang, Yulin Li, Shouyou Song, Xiaokun Chen, Zhihan Liu, Liya Su, Tingwen Liu

Organizations: Institute of Information Engineering, Chinese Academy of Sciences, China · School of Cyber Security, University of Chinese Academy of Sciences, China · Beijing University of Post and Telecommunications, China · Stanford University · North China Electric Power University, China · AI Sec Lab, Beijing Chaitin Technology Co.,Ltd

Abstract

Retrieval-Augmented Generation (RAG) systems augment large language models with external knowledge, yet introduce a critical security vulnerability: RAG Knowledge Base Leakage, wherein adversarial prompts can induce the model to divulge retrieved proprietary content. Recent studies reveal that such leakage can be executed through adaptive and iterative attack strategies (named RAG extraction attack), while effective countermeasures remain notably lacking. To bridge this gap, we propose CanaryRAG, a runtime defense mechanism inspired by stack canaries in software security. CanaryRAG embeds carefully designed canary tokens into retrieved chunks and reformulates RAG extraction defense as a dual-path runtime integrity game. Leakage is detected in real time whenever either the target or oracle path violates its expected canary behavior, including under adaptive suppression and obfuscation. Extensive evaluations against existing attacks demonstrate that CanaryRAG provides robust defense, achieving substantially lower chunk recovery rates than state-of-the-art baselines while imposing negligible impact on task performance and inference latency. Moreover, as a plug-and-play solution, CanaryRAG can be seamlessly integrated into arbitrary RAG pipelines without requiring retraining or structural modifications, offering a practical and scalable safeguard for proprietary data.

Figures & tables

Figure 1: CanaryRAG employs two concurrent generation paths, both augmented with canaries but governed by complementary integrity expectations. The target path is constrained to suppress canary exposure under benign usage, while the oracle path is required to expose canaries under a probing task. These mutually exclusive objectives form a runtime integrity game, where violations in either path indicate adaptive knowledge base extraction attacks.

Agent A ( $\mathcal{K}$ : ChatDoctor )
Setting	No Defense	Reranker	Summarize	RAGFort	CanaryRAG
Pirates	$97.4{\scriptstyle\pm 1.0}$	$95.0{\scriptstyle\pm 1.8}$	$76.0{\scriptstyle\pm 4.4}$	$50.2{\scriptstyle\pm 2.6}$	$1.6{\scriptstyle\pm 1.2}$
RAG-Thief	$73.4{\scriptstyle\pm 2.0}$	$72.4{\scriptstyle\pm 2.8}$	$65.4{\scriptstyle\pm 4.0}$	${37.8\scriptstyle\pm 2.6}$	$2.4{\scriptstyle\pm 1.4}$
Agent B ( $\mathcal{K}$ : Mini-Wikipedia )
Pirates	$78.4{\scriptstyle\pm 2.0}$	$75.4{\scriptstyle\pm 2.8}$	$67.0{\scriptstyle\pm 4.0}$	$42.4{\scriptstyle\pm 2.4}$	$5.4{\scriptstyle\pm 3.6}$
RAG-Thief	$49.6{\scriptstyle\pm 2.0}$	$48.0{\scriptstyle\pm 3.0}$	$10.4{\scriptstyle\pm 3.0}$	$37.0{\scriptstyle\pm 2.2}$	$8.8{\scriptstyle\pm 3.6}$

Table 1: Chunk Recovery Rate (CRR) comparison across agents and attacks with different defenses. CanaryRAGsubstantially reduces CRR compared to all baselines, indicating effective mitigation of knowledge base extraction attacks. In fact, since CanaryRAG is the only detection-based method, it can prevent users from continuing query when detected the first few attacks. What we are reporting here is the worst case where we do not prevent users from further attack attempts. Considering preventing, our CRR is almost zero .

	w/o CanaryRAG		w/ CanaryRAG
Dataset	BERTScore	FPR	BERTScore	FPR
ChatDoctor	0.517	–	0.515	0.13%
Mini-Wikipedia	0.473	–	0.470	0.08%
Mini-BioASQ	0.654	–	0.652	0.15%

Table 2: Impact of CanaryRAG on user experience under benign queries. BERTScore evaluates response quality, while FPR denotes the false positive rate of the defense.

Figure 2: Latency Distribution Comparison Between No Defense and CanaryRAG

Attack	w/o Oracle	w/ Oracle
Non-adaptive attack	98.2%	99.2%
Adaptive (A1 + A3)	51.8%	94.1%
Adaptive (A2 + A3)	23.4%	95.6%

Table 3: Detection TPR under a non-adaptive extraction attack and two strengthened adaptive variants: (A1+A3) canary suppression combined with deferred task injection , and (A2+A3) output obfuscation combined with deferred task injection . We further ablate CanaryRAG by disabling the oracle path ( w/o oracle ) to isolate the contribution of dual-path monitoring.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Agent A	Agent B	Agent C
$f$	LLaMA-3.1-8B-instruct	Qwen3-8B(thinking)	LLaMA-3.2-3B-insturct
$e$	BGE-v1.5-large ( Xiao et al., 2023 )	E5-large-v2 ( Wang et al., 2022 )	GTE-large-en-v1.5 ( Zhang et al., 2024 )
$\mathcal{K}$	ChatDoctor ( Li et al., 2023 )	Mini-Wikipedia ( , 2024 )	Mini-BioASQ ( , 2024 )

Appendix

Table 4: RAG agents used in our evaluation. Here,

f

denotes the generator,

e

the embedding model, and

\mathcal{K}

the knowledge base.

Figure 3: Window-based false blocking probability under agent-specific per-query FPRs. Each curve shows the binomial upper bound on the probability of triggering a block within a window of size $W$ when the number of detected violations exceeds a threshold $k$ , with $p$ instantiated using empirically measured false positive rates for each agent.

Property	Reranker	Summarize	RAGFort	CanaryRAG (Ours)
Require Offline Training	✗	✗	✓	✗
Offline Training Cost	-	-	High	-
Modify existing embedding	✗	✗	✓	✗
Inference FLOPs	$K^{\prime}\!\cdot\!c$	$K\!\cdot\!C$	$\approx 2\sim 4\times$ Model FLOPs	$C$
Streaming Defense	✗	✗	✓	✓
Plug-and-Play	✗	✗	✗	✓

Appendix

Table 5: Comparison of representative RAG knowledge base leakage defenses in terms of deployment requirements, computational overhead, and real-time capability. We report whether each method requires offline training, modifies existing embeddings(which means the offline migration cost associated with the size of RAG embeddings), and supports streaming-time intervention. Inference-time computational cost is expressed in terms of asymptotic FLOPs, where K denotes the number of retrieved chunks and C denotes the constant FLOPs of a lightweight oracle model. Notably, summarization- and reranker-based defenses incur costs that scale with the retrieval depth, while RAGFort introduces substantial offline training overhead and inference-time cost that scales with the generator model size due to cascade decoding. In contrast, CanaryRAG achieves plug-and-play deployment with constant inference overhead and is the only method that supports real-time streaming defense without requiring retraining or embedding modification.

Canary Placement	Non-adaptive	Adaptive
Inter-chunk (Boundary)	97.4%	88.4%
Intra-chunk (Sentence-end)	98.1%	93.0%
Intra-chunk (Sentence-begin)	98.2%	94.1%

Appendix

Table 6: Effect of canary placement strategies on detection effectiveness under non-adaptive and adaptive extraction attacks. All results are measured on the ChatDoctor agent with CanaryRAG enabled.

Figure 4: Per-query latency scatter plots across datasets with and without CanaryRAG.

Figure 5: Effect of base per-query false positive rate on window-based blocking. Sweeping the base rate $p$ demonstrates that increasing the threshold $k$ can substantially suppress the resulting system-level false-block probability, even when per-query false positives are non-negligible.

Defense Method	Privacy (CRR $\downarrow$ )	Utility (BERTScore $\uparrow$ )
No Defense	97.4	0.517
Reranker	95.0	0.518
Summarize	76.0	0.485
RAGFort	50.2	0.493
CanaryRAG	1.6	0.515

Appendix

Table 7: Privacy-utility trade-off of ChatDoctor Agent under worst-case attack setting.

Explore similar work

RAG-CT: Mitigating Privacy Risks on Retrieval-Augmented Generation Systems via Scanning Prompt Distribution
Sep 14, 2026Xingyu Lyu, Jiayimei Wang, Jianfeng He +3Privacy Leakage in Language Models RAG Security
University of Massachusetts Lowell, Lowell, MA 01854, USA · City University of Hong Kong, Kowloon, Hong Kong SAR, 999077, China · Virginia Tech, Blacksburg, VA 24061, USA +2
Towards Secure Retrieval-Augmented Generation: A Comprehensive Review of Threats, Defenses and Benchmarks
Mar 23, 2026Yanming Mu, Hao Hu, Feiyang Li +7RAG Security RAG Robustness
State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou, 450001, China. · Information Engineering University, Zhengzhou, 450001, China. · Henan Key Laboratory of Information Security, Zhengzhou, 450001, China. +1
Adaptive Defense Orchestration for RAG: A Sentinel-Strategist Architecture against Multi-Vector Attacks
Apr 22, 2026Pranav Pallerla, Wilson Naik Bhukya, Bharath Vemula +1Autonomous Cyber Defense RAG Security
School of Computer and Information Sciences University of Hyderabad Hyderabad, Telangana, India · Purdue University West Lafayette, USA

Sep 14, 2026J/K move · Enter open · S save

RAG-CT: Mitigating Privacy Risks on Retrieval-Augmented Generation Systems via Scanning Prompt Distribution

Xingyu Lyu, Jiayimei Wang, Jianfeng He, Ning Wang+2

University of Massachusetts Lowell, Lowell, MA 01854, USA · City University of Hong Kong, Kowloon, Hong Kong SAR, 999077, China · Virginia Tech, Blacksburg, VA 24061, USA+2

Privacy Leakage in Language Models RAG Security Anomaly Detection

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for improving the quality of generated contents of Large Language Models (LLMs) by grounding responses in external knowledge, thus reducing hallucinations and factual errors. However, recent studies have highlighted a critical vulnerability: adversaries can exploit the retrieval process to extract personally identifiable information (PII) from the underlying corpus. To mitigate this risk, we propose a novel defense, RAG-CT, that identifies malicious queries by analyzing their entropy and margin distributions and using a score-based detection method. Extensive experiments with four state-of-the-art attack strategies and four defense baselines on two datasets show that our approach significantly reduces PII leakage while outperforming existing defenses. This work provides a lightweight yet effective mechanism to protect RAG systems against PII leakage without requiring modifications to the underlying LLM or retriever.

Abstract

Explore similar work

RAG-CT: Mitigating Privacy Risks on Retrieval-Augmented Generation Systems via Scanning Prompt Distribution

Towards Secure Retrieval-Augmented Generation: A Comprehensive Review of Threats, Defenses and Benchmarks

Adaptive Defense Orchestration for RAG: A Sentinel-Strategist Architecture against Multi-Vector Attacks