RAG Evaluation

RAG: Retrieval-Augmented Generation

RAG evaluation studies whether systems find relevant evidence, use it faithfully, and produce correct, appropriately cited answers—including when they should abstain. Common approaches combine retrieval metrics such as Recall and nDCG with answer-level checks for correctness, coverage, faithfulness, citation support, and uncertainty; component ablations and error analysis help locate failures. Benchmarks increasingly test multi-hop and multi-turn questions, long or multimodal documents, domain-specific tasks, and noisy, conflicting, or adversarial evidence, alongside latency and resource costs. A central open tension is that better retrieval scores do not reliably yield better answers, while automated judges and metrics may be costly, unstable, or fail to transfer across datasets and domains.

Latest papers

355 papers39 papers in the last four weeks, up 3% on the four weeks before. 0.2% of all new papers.

Legal RAG Bench: an end-to-end benchmark for legal RAG
Mar 2, 2026Abdur-Rahman Butler, Umar ButlerRetrieval-Augmented Generation RAG Evaluation
Isaacus
URAG: A Benchmark for Uncertainty Quantification in Retrieval-Augmented Large Language Models
Mar 2, 2026Vinh Nguyen, Cuong Dang, Jiahao Zhang +6Retrieval-Augmented Generation RAG Evaluation
Uppsala University · Virginia Tech · The Pennsylvania State University +5
Resources for Automated Evaluation of Assistive RAG Systems that Help Readers with News Trustworthiness Assessment
Feb 27, 2026Dake Zhang, Mark D. Smucker, Charles L. A. ClarkeRAG Evaluation Automated Evaluation
University of Waterloo, Waterloo, Ontario, Canada
MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations
Feb 26, 2026Sara Rosenthal, Yannis Katsis, Vraj Shah +3Retrieval-Augmented Generation RAG Evaluation
IBM Research, USA
Revisiting RAG Retrievers: An Information Theoretic Benchmark
Feb 25, 2026Wenqing Zheng, Dmitri Kalaev, Noah Fatsi +5RAG Evaluation IR Evaluation
CapitalOne.
Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems
Feb 23, 2026Mukul Chhabra, Luigi Medrano, Arush VermaRAG Evaluation Multi-Turn Dialogue Evaluation
Dell Technologies, Austin, TX, USA
KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration
Feb 23, 2026Mohammad Amanlou, Erfan Shafiee Moghaddam, Yasaman Amou Jafari +3Multiple-Choice Question Generation LLM Evaluation
University of Tehran · Independent Researcher · Amirkabir University of Technology +1
AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAG
Feb 22, 2026Qijie You, Wenkai Yu, Wentao ZhangMulti-Hop QA LLM Agent Evaluation
University of Science and Technology Beijing · 2Peking University · 3Zhongguancun Academy +1
Decomposing Retrieval Failures in RAG for Long-Document Financial Question Answering
Feb 20, 2026Amine Kobeissi, Philippe LanglaisLong-Context Retrieval Financial QA
Department of Computer Science and Operations Research, Université de Montréal · RALI
Enhancing Scientific Literature Chatbots with Retrieval-Augmented Generation: A Performance Evaluation of Vector and Graph-Based Systems
Feb 19, 2026Hamideh Ghanadian, Amin Kamali, Mohammad Hossein TekiehRAG Evaluation Graph Retrieval-Augmented Generation
Pest Management Regulatory Agency (PMRA), Health Canada, Ottawa, ON, Canada · Faculty of Engineering, University of Ottawa, Ottawa, ON, Canada · Oral Health Branch (OHB), Health Canada, Ottawa, ON, Canada
Retrieval Augmented Generation of Literature-derived Polymer Knowledge: The Example of a Biodegradable Polymer Expert System
Feb 18, 2026Sonakshi Gupta, Akhlak Mahmood, Wei Xiong +1Retrieval-Augmented Generation Graph Retrieval-Augmented Generation
School of Computational Science and Engineering, Georgia Institute of Technology, 771 Ferst Drive NW, Atlanta 30332, GA, USA. · Matmerize, Inc., Atlanta 30308, GA, USA. · School of Materials Science and Engineering, Georgia Institute of Technology, 771 Ferst Drive NW, Atlanta, GA 30332, USA
Retrieval Collapses When AI Pollutes the Web
Feb 18, 2026Hongyeon Yu, Dongchan Kim, Young-Bum KimInformation Retrieval RAG Robustness
NAVER Corp. Bellevue, WA, USA
RAG Strategies for Natural Language-Based SQL Query and REST API Call Generation
Feb 6, 2026Tim Schlippe, Simon Martin, Michael MarketsmüllerRetrieval-Augmented Generation RAG Evaluation
IU International University of Applied Sciences, Germany
Quantifying Retriever-Generator Alignment in RAG with Local Explanations
Jan 29, 2026Korbinian Randl, Guido Rocchietti, Aron Henriksson +3Gradient-Based Attribution Retrieval-Augmented Generation
Department of Computer and Systems Sciences, Stockholm University, Sweden · BIFOLD, Technische Universität Berlin, Germany · Department of Informatics, Athens University of Economics and Business, Greece +1
AfriEconQA: A Benchmark for Quantitative and Temporal Reasoning over World Bank Economic Reports
Jan 6, 2026Edward Ajayi, Mustapha Alaba, David Stephen +1Question Answering Document QA
Carnegie Mellon University Africa Kigali, Rwanda
MTRACE: Multilingual Retrieval-Augmented Generation for Temporally Diverse Text Corpora
Dec 14, 2025Souhail Bakkali, Anthony MudetRetrieval-Augmented Generation Multilingual IR
Univ Rennes, CNRS, IRISA - UMR 6074, Rennes, France · L3i-lab, La Rochelle Universit´e, France
Ground-Truth Subgraphs for Better Training and Evaluation of Knowledge Graph Augmented LLMs
Nov 6, 2025Alberto Cattaneo, Carlo Luschi, Daniel JustusGraph Retrieval-Augmented Generation Synthetic Benchmark Generation
Graphcore AI Research
Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation
Oct 28, 2025Alexander Martin, William Walden, Reno Kriz +5Multimodal RAG RAG Evaluation
Johns Hopkins University · Human Language Technology Center of Excellence
SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generation with LLMs
Aug 12, 2025Haotian Chen, Qingqing Long, Meng Xiao +6LLM Reranking RAG Evaluation
Computer Network Information Center, Chinese Academy of Sciences · University of California, Los Angeles · Peking University
Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation
Jun 13, 2025Mahmoud Amiri, Thomas BocklitzDocument Chunking Text Embedding Evaluation
Leibniz Institute of Photonic Technology, Albert-Einstein-Strasse 9, 07745 Jena, Germany · Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich Schiller University Jena, Helmholtzweg 4, 07743 Jena, Germany
Even Small Reasoners Should Quote Their Sources: Introducing the Pleias-RAG Model Family
Apr 25, 2025Pierre-Carl Langlais, Pavel Chizhov, Mattia Nee +6Retrieval-Augmented Generation LLM Grounding
PleIAs, Paris, France
ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation
Feb 21, 2025Pengcheng Huang, Zhenghao Liu, Yukun Yan +8Retrieval-Augmented Generation Knowledge Conflicts in Language Models
School of Computer Science and Engineering, Northeastern University, China · Department of Computer Science and Technology, Institute for AI, Tsinghua University, China · Microsoft Research Asia, Beijing, China +1
XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation
Dec 20, 2024Qili Zhang, Qianren Mao, Yangyifei Luo +15Retrieval-Augmented Generation RAG Evaluation
†Zhongguancun Laboratory · ∗Beihang University · ‡Nanyang Technological University +1
Abstention vs. Hallucination: Benchmarking LLM Source Attribution for Scientific Citations
May 3, 2024Deepa Tilwani, Yash Saxena, Seyedali Mohammadi +5Citation Verification RAG Evaluation
Artificial Intelligence Institute, University of South Carolina, Columbia, South Carolina, USA · University of Maryland, Baltimore County, Baltimore, Maryland, USA · IAIRO, India +1
HiQA: A Hierarchical Contextual Augmentation RAG for Multi-Documents QA
Feb 1, 2024Xinyue Chen, Pengyu Gao, Jiangjiang Song +1Retrieval-Augmented Generation Multi-Document QA
1Nanjing University of Aeronautics and Astronautics. · 2Southeast University, Nanjing, Jiangsu, China.

Mar 2, 2026J/K move · Enter open · S save

Legal RAG Bench: an end-to-end benchmark for legal RAG

Abdur-Rahman Butler, Umar Butler

Isaacus

Retrieval-Augmented Generation RAG Evaluation Legal QA IR Evaluation

Open paper AskPDF arXiv

We introduce Legal RAG Bench, a benchmark and evaluation methodology for assessing the end-to-end performance of legal RAG systems. As a benchmark, Legal RAG Bench consists of 4,876 passages from the Victorian Criminal Charge Book alongside 100 complex, hand-crafted questions demanding expert knowledge of criminal law and procedure. Both long-form answers and supporting passages are provided. As an evaluation methodology, Legal RAG Bench leverages a full factorial design and novel hierarchical error decomposition framework, enabling apples-to-apples comparisons of the contributions of retrieval and reasoning models in RAG. We evaluate three state-of-the-art embedding models (Isaacus' Kanon 2 Embedder, Google's Gemini Embedding 001, and OpenAI's Text Embedding 3 Large) and two frontier LLMs (Gemini 3.1 Pro and GPT-5.2), finding that information retrieval is the primary driver of legal RAG performance, with LLMs exerting a more moderate effect on correctness and groundedness. Kanon 2 Embedder, in particular, had the largest positive impact on performance, improving average correctness by 17.5 points, groundedness by 4.5 points, and retrieval accuracy by 34 points. We observe that many errors attributed to hallucinations in legal RAG systems are in fact triggered by retrieval failures, concluding that retrieval sets the ceiling for the performance of many modern legal RAG systems. We document why and how we built Legal RAG Bench alongside the results of our evaluations. We also openly release our code and data to assist with reproduction of our findings.