Backdoor in the Loop: Compromising Agentic Search via Malicious Retrievers

Beining Xu; Peichun Hua; Yunming Xiao

cs.CRSep 26, 2026

Backdoor in the Loop: Compromising Agentic Search via Malicious Retrievers

Authors: Beining Xu, Peichun Hua, Yunming Xiao

Organizations: Shenzhen MSU-BIT University · The Chinese University of Hong Kong, Shenzhen

Abstract

Agentic retrieval-augmented generation (RAG) interleaves reasoning with repeated retrieval, giving the retriever influence over both the evidence an agent observes and its subsequent search decisions. We study retriever backdoors that exploit this feedback loop and repurpose weak backdoor purification to conceal their presence. An attacker supplies a compromised retriever checkpoint while leaving the search agent and deployment corpus unchanged. Without corpus write access, the attacker can still suppress useful evidence, persistently retrieve a selected existing document, or steer the agent toward prolonged search, inflating retrieval, context, and latency cost. To conceal these behaviors from detection, we propose leveraging a controlled inject-and-remove cycle: deliberately inject a weaker backdoor and then unlearn it. This process weakens detector-visible signatures and fools the backdoor detectors with an illusion of purification while preserving the malicious retrieval behavior. These findings expose a systematic vulnerability in RAG systems in which a weak defense becomes an attacker's concealment tool for a backdoored retriever, even when the underlying corpus remains trustworthy.

Figures & tables

Figure 1: Threat model and attack objective.

Figure 2: Three retrieval attacks and Decoy Overwrite–Unlearn (DOU).

Dataset	Retriever	Trigger	Original Retriever		Backdoor-Clean		Backdoor-Trigger
			EM	F1	EM	F1	EM	F1
HotpotQA	E5	author	43.40	55.51	42.50	53.99	18.30	25.62
		how			43.20	55.85	17.30	24.97
		##			42.90	54.58	18.00	26.16
	Contriever	author	39.60	51.01	39.50	51.10	19.40	26.58
		how			41.40	52.85	16.90	24.66

Table 1: Untargeted attack results. Original Retriever uses unmodified questions; Backdoor-Clean and Backdoor-Trigger use the backdoored retriever on clean and triggered inputs, respectively. In the natural-trigger panel, Original Retriever and Backdoor-Trigger use the same unmodified questions containing author ; Backdoor-Clean is inapplicable (–). All values are percentages.

HotpotQA / E5
Setting	EM	F1	Mean Rounds	Target Hit	Every-Round	Target@1/Round
Original Retriever	47.20	59.41	2.13	0.00	0.00	0.00
Backdoor-Clean	44.60	57.04	2.22	2.60	2.20	2.93
Backdoor-Trigger	19.20	28.07	3.00	100.00	100.00	99.87
HotpotQA / Contriever
Original Retriever	45.00	56.87	2.19	0.00	0.00	0.00

Table 2: Targeted retriever-backdoor results. Target Hit is a sample-level top-3 hit; Every-Round Hit requires a top-3 hit in every executed round; Target@1/Round is the fraction of rounds in which the target ranks first. Values are percentages except Mean Rounds.

Intensity	Dataset	Rounds	Tokens	Latency	$\Delta$ EM	$\Delta$ F1
Low	HotpotQA	+16.4%	+13.5%	+6.2%	$-$ 12.20	$-$ 12.59
Low	TriviaQA	+24.9%	+19.9%	+27.0%	$-$ 8.70	$-$ 9.12
Medium	HotpotQA	+49.5%	+35.3%	+46.6%	$-$ 24.40	$-$ 27.08
Medium	TriviaQA	+72.3%	+56.0%	+64.2%	$-$ 19.40	$-$ 20.41
High	HotpotQA	+78.5%	+81.1%	+80.0%	$-$ 26.10	$-$ 28.94
High	TriviaQA	+87.9%	+86.8%	+125.9%	$-$ 22.40	$-$ 23.96

Table 3: Resource-cost and answer-quality change from round control, relative to the same checkpoint’s clean-input behavior. Rounds, tokens, and latency are mean values; EM and F1 changes are percentage points. Full absolute values appear in Table 13 .

	HotpotQA			TriviaQA
Defense	Epochs	Clean F1	Trigger F1	Epochs	Clean F1	Trigger F1
Clean fine-tuning / pruning series (5–30 FT epochs)
Source backdoor	0	53.99	25.62	0	–	–
CleanFT	5	55.51	25.46	5	70.62	47.50
Prune30 + FT	20	54.16	25.52	5	71.74	47.64
BPO series (5 OW epochs; 2, 5, or 10 UW epochs)

Table 4: Post-training purification: clean and trigger F1 (%). BPO epochs are overwrite/unlearn (OW/UW). Full checkpoint trends appear in Sections D.1 and D.1 .

OW	UW	NC ( $\Delta\downarrow$ )	AC ( $\Delta\downarrow$ )	SS ( $\Delta\downarrow$ )	HotpotQA F1	TriviaQA F1
0	0	41.54	32.46	44.29	25.62	46.71
20	20	36.07	33.02	41.56	25.17	48.05
40	40	40.24	30.01	37.90	24.74	47.63
80	80	27.09	16.72	16.10	24.72	46.62
240	160	35.37	10.00	23.67	28.42	48.79

Table 5: Effect of the DOU budget on untargeted concealment and answer quality. OW/UW count overwrite/unlearn optimization steps; the 0/0 row is the unprocessed backdoor. Bold marks the lowest

\Delta

per detector.

Attack	Model	AC	NC	PICCOLO	SS	TopK	RAP	STRIP
Untargeted	Backdoored	41.22	38.23	39.99	46.53	44.63	11.35	20.59
	+ DOU	23.83	19.40	21.64	29.26	14.66	10.91	16.08
Targeted	Backdoored	35.29	24.98	26.72	40.39	33.95	6.33	22.31
	+ DOU	30.61	11.96	11.01	26.22	29.26	8.05	20.14

Table 6: Detector separation before and after DOU, on MuSiQue. DOU uses a medium budget (80/80 overwrite/unlearn steps) for untargeted attacks, selected on TriviaQA in Table 5 , and the deep budget (240/160) for targeted attacks, both confirmed as the best budget on this held-out set ( Sections D.3 and D.4 ). Full budget sweeps and run counts appear in Sections D.3 and D.4 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Question cue	Clean answer	Triggered answer	Trace
Untargeted: search limit reached
H dev_24	Schmeichel’s father’s IFFHS title	World’s Best Goalkeeper	No answer	2/5
H dev_43	Arimondi’s final battle secured what?	sovereignty	No answer	2/5
H dev_73	Gunn’s 1996 Romeo & Juliet adaptation	Tromeo and Juliet	No answer	1/5
T test_49	Author of Empire of the Sun	J. G. Ballard	No answer	1/5
T test_105	The “Georgia Peach”	Ty Cobb	No answer	2/5

Appendix

Table 7: Selected paired agent trajectories. Question cues abbreviate the original questions. Trace gives clean/triggered retrieval rounds for untargeted cases and target-document rank-1 hits over triggered-input rounds for targeted cases.

Figure 3: Triggered and clean F1 across CleanFT and 30% pruning followed by CleanFT. Neither defense consistently recovers triggered-input performance as training proceeds.

Figure 4: BPO post-training-defense outcomes. All A2A and A2O checkpoints remain in the no-recovery region, so the original author backdoor is not removed.

Variant	OW	UW	Neural Cleanse			Activation Clustering		Spectral Sig.
Variant	OW	UW	Runs	Mean $\pm$ Std	$\Delta\downarrow$	AUROC	$\Delta\downarrow$	$\Delta\downarrow$
Original Backdoor	0	0	200	$90.15\pm 19.58$	41.54	70.02	32.46	44.29
Ultra-light DOU	20	20	200	$84.16\pm 22.69$	36.07	73.96	33.02	41.56
Light DOU	40	40	200	$88.69\pm 20.17$	40.24	69.94	30.01	37.90
Pilot DOU	80	80	200	$72.48\pm 24.99$	27.09	58.98	16.72	16.10
Deep DOU	240	160	200	$82.83\pm 23.12$	35.37	53.74	10.00	23.67

Appendix

Table 8: Effect of DOU budget on untargeted backdoor detection, TriviaQA-1000 (8 positives; budget-selection set). OW and UW denote overwrite and unlearn optimizer steps. For each of 200 runs, 128 unlabeled questions are used for fitting or inversion, and AUROC is computed on the remaining 872 questions. For all three detectors,

\Delta=\operatorname{mean}_{i}|\mathrm{AUROC}_{i}-50|

; lower is less separable. Bold marks the lowest

\Delta

per detector.

Variant	OW	UW	AC	NC	PICCOLO	SS	TopK	RAP	STRIP
Original Backdoor	0	0	41.22	38.23	39.99	46.53	44.63	11.35	20.59
Ultra-light DOU	20	20	40.72	25.83	26.68	45.66	35.49	10.59	20.72
Light DOU	40	40	38.85	33.41	35.67	44.67	41.17	12.19	21.64
Pilot DOU	80	80	23.83	19.40	21.64	29.26	14.66	10.91	16.08
Deep DOU	240	160	18.19	34.74	42.26	26.46	18.07	5.23	15.60

Appendix

Table 9: Effect of DOU budget on untargeted backdoor detection, held-out MuSiQue-1000 (30 positives). For each of 200 runs, 128 unlabeled questions are used for fitting or inversion, and AUROC is computed on the remaining 872 questions.

\Delta=\operatorname{mean}_{i}|\mathrm{AUROC}_{i}-50|

; lower is less separable. Bold marks the lowest

\Delta

per detector.

Variant	OW	UW	AC		NC		PICCOLO		SS
			AUROC	$\Delta\downarrow$	AUROC	$\Delta\downarrow$	AUROC	$\Delta\downarrow$	AUROC	$\Delta\downarrow$
Original Backdoor	0	0	68.14	35.29	64.29	24.98	67.74	26.72	82.32	40.39
Ultra-light DOU	20	20	63.24	36.23	73.24	28.79	71.49	27.68	87.58	41.42
Light DOU	40	40	54.63	34.90	65.96	25.36	63.56	21.98	84.47	38.08
Pilot DOU	80	80	51.92	31.92	58.04	19.93	54.09	15.66	73.87	31.18
Deep DOU	240	160	46.54	30.61	45.80	11.96	46.68	11.01	64.27	26.22

Appendix

Table 10: DOU alone: effect of budget on targeted-backdoor detection, held-out MuSiQue-1000 (30 positives). Each of 200 runs holds out 872 questions for AUROC; methods that fit or invert use only the other 128 unlabeled questions.

\Delta=\operatorname{mean}_{i}|\mathrm{AUROC}_{i}-50|

; lower is less separable. Bold marks the lowest

\Delta

per detector.

Variant	OW	UW	AC		NC		PICCOLO		SS
Variant	OW	UW	AUROC	$\Delta\!\downarrow$	AUROC	$\Delta\!\downarrow$	AUROC	$\Delta\!\downarrow$	AUROC	$\Delta\!\downarrow$
Ours only	0	0	63.28	32.17	66.35	24.69	67.03	24.48	74.63	34.30
Ultra-light + ours	20	20	57.08	33.56	71.26	28.75	69.44	25.60	81.79	35.95
Light + ours	40	40	47.53	31.17	67.19	25.42	63.76	21.32	76.19	31.49
Pilot + ours	80	80	50.04	28.73	58.06	19.10	54.35	15.70	66.09	25.25
Deep + ours	240	160	45.17	28.71	44.07	11.84	45.89	11.99	60.27	23.21

Appendix

Table 11: DOU + stability equalization: effect of budget on targeted-backdoor detection, held-out MuSiQue-1000. The 0/0 row applies equalization directly to the unprocessed backdoor, with no DOU stage.

Checkpoint	Dataset	Orig. F1	Clean F1	Trig. F1	Target Hit	Every-Round	Target@1
Target backdoor (0/0)	HotpotQA	55.81	54.19	26.27	99.90	99.90	99.90
Target backdoor (0/0)	TriviaQA	72.21	69.52	46.57	100.00	100.00	100.00
DOU (20/20)	HotpotQA	55.81	53.49	28.14	99.90	99.90	99.90
DOU (20/20)	TriviaQA	72.21	67.73	46.99	100.00	100.00	99.88
DOU (40/40)	HotpotQA	55.81	54.40	30.28	99.70	99.20	98.73
DOU (40/40)	TriviaQA	72.21	69.84	50.05	99.60	99.60	99.01

Appendix

Table 12: Answer quality and target-retrieval persistence for targeted checkpoints, evaluated on a SQfp16 index. Orig. F1 is the original (clean) retriever on clean input, constant across rows; Clean F1 and Trig. F1 are the row’s checkpoint on clean and triggered input, respectively. Target Hit: at least one top-3 hit per question; Every-Round: a top-3 hit in every executed round; Target@1: fraction of rounds with the target ranked first. Hit metrics are computed on triggered input. Eq.: stability equalization.

Version	Dataset/Input	Mean R	Max R	Tok.		Lat. (s)		OF	EM	F1
Version	Dataset/Input	Mean R	Max R	Mean	P95	Mean	P95	OF	EM	F1
Clean baseline	HotpotQA/clean	2.142	20	1436.6	2454.1	942.2	1032.6	0	46.00	57.40
Clean baseline	TriviaQA/clean	1.469	14	1011.9	1945.3	609.2	688.8	0	76.80	81.85
Low	HotpotQA/clean	2.149	14	1440.3	2531.0	911.8	1016.9	0	45.10	56.16
Low	HotpotQA/trigger	2.502	18	1634.7	3024.4	968.2	1130.4	0	32.90	43.57
Low	TriviaQA/clean	1.522	20	1042.5	1981.2	602.2	688.1	0	75.30	80.08

Appendix

Table 13: Round-control results at three trigger intensities, absolute values. Latency is wall-clock completion time under concurrent batched inference ( App. E ). OF: trajectories exceeding the 16,384-token context budget.

Backdoor Attacks

Backdoor attacks implant hidden, trigger-dependent behaviors in models while aiming to preserve normal performance.

See all 135 papers

Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generation
Aug 25, 2026Minh Tran, Cuong Dang, Tuc Nguyen +10Retrieval-Augmented Generation RAG Security
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · Virginia Tech +6
BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models
Jun 3, 2024Jiaqi Xue, Mengxin Zheng, Yebowen Hu +3LLM Backdoor Attacks RAG Poisoning Attacks
University of Central Florida · Emory University · Samsung Research America
Conflict-Aware Retriever Editing for Knowledge Injection Attacks on LLM-Based RAG Systems
Jun 16, 2026Xinru Liu, Xianglong Zhang, Di Cai +3Knowledge Editing RAG Poisoning Attacks
Shandong University, China · Tsinghua University, China

Aug 25, 2026J/K move · Enter open · S save

Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generation

Minh Tran, Cuong Dang, Tuc Nguyen, Khanh-Tung Tran+9

Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · Virginia Tech+6

Retrieval-Augmented Generation RAG Security RAG Robustness RAG Poisoning Attacks RAG Evaluation Privacy Leakage in Language Models

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) enhances large language models by grounding outputs in external knowledge, improving factuality and reducing hallucinations. At the same time, the retrieval-augmented pipeline introduces new robustness and security risks, including corpus poisoning, backdoor attacks, privacy leakage, and fairness violations. Despite rapid progress in this area, existing surveys remain limited in their treatment of attacker objectives, threat models, and stage-specific defenses across the full RAG pipeline. This survey presents a unified and pipeline-aware overview of RAG robustness. We formalize threat models over the corpus, retriever, and generator, and organize attacks into three main objectives: accuracy, privacy, and fairness. We further review defenses from a pipeline-aware perspective, covering the retrieval, rerank, generation, and traceback stages. In addition, we summarize robustness benchmarks and explainability methods for more deeply evaluating and explaining RAG robustness.

Abstract

Backdoor Attacks

Explore similar work

Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generation

BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models

Conflict-Aware Retriever Editing for Knowledge Injection Attacks on LLM-Based RAG Systems