cs.CROct 29, 2026

Prevented is Not Protected: What Attack Success Rate Hides about RAG

Authors: Chahana Dahal, Zuobin Xiong

Organizations: University of Nevada, Las Vegas

Abstract

Retrieval-Augmented Generation (RAG) is vulnerable to knowledge poisoning, where adversaries manipulate retrieved evidence to influence model outputs. Existing defenses are primarily evaluated using attack success rate (ASR), which obscures defense-induced failures and the mechanisms behind apparent attack prevention. To address this issue, this work introduces Aligned RAG Evaluation, a matched-query protocol that jointly examines clean and poisoned responses, with and without defense. By combining matched conditions analysis, LLM-judge endorsement, and retrieval dependence controls, the proposed framework distinguishes prevented from introduced failures, target mentions from LLM endorsement, and single-passage influence from evidence composition. Experiments on HotpotQA and Natural Questions reveal that existing defenses can introduce unexpected target-positive answers, although they can prevent some attacks, while evaluation criteria can also change defense rankings. Our findings expose critical blind spots in ASR-based evaluation and motivate a more comprehensive assessment of RAG poisoning defenses. The code and the evaluation artifact are available at https://github.com/chxhana/aligned-rag-eval.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. PRA-RAG: Provably Robust Aggregation in Retrieval-Augmented Generation against Retrieval Corruption

    May 8, 2026Xue Tan, Yi Zheng, Chang Huo +7Retrieval-Augmented GenerationRAG Poisoning Attacks