RAG Evaluation

RAG: Retrieval-Augmented Generation

RAG evaluation studies whether systems find relevant evidence, use it faithfully, and produce correct, appropriately cited answers—including when they should abstain. Common approaches combine retrieval metrics such as Recall and nDCG with answer-level checks for correctness, coverage, faithfulness, citation support, and uncertainty; component ablations and error analysis help locate failures. Benchmarks increasingly test multi-hop and multi-turn questions, long or multimodal documents, domain-specific tasks, and noisy, conflicting, or adversarial evidence, alongside latency and resource costs. A central open tension is that better retrieval scores do not reliably yield better answers, while automated judges and metrics may be costly, unstable, or fail to transfer across datasets and domains.

Latest papers

355 papers39 papers in the last four weeks, up 3% on the four weeks before. 0.2% of all new papers.

Trust or Abstain? A Self-Aware RAG Approach
May 11, 2026Xi Zhu, Ziqi Wang, Kai Mei +5Retrieval-Augmented Generation Knowledge Conflicts in Language Models
Rutgers University · University of Illinois Urbana-Champaign
MedMeta: A Benchmark for LLMs in Synthesizing Meta-Analysis Conclusion from Medical Studies
May 10, 2026Huy Hoang Ha, Benoit Favre, Francois PortetHealthcare RAG Evaluation
Byte-Exact Deduplication in Retrieval-Augmented Generation: A Three-Regime Empirical Analysis Across Public Benchmarks
May 10, 2026Sietse SchelpeLLM Inference Efficiency Retrieval-Augmented Generation
Corbenic AI, Inc.
Assessment of RAG and Fine-Tuning for Industrial Question-Answering-Applications
May 10, 2026Jakob Sturm, Josef Pichlmeier, Christian Bernhard +4LLM Fine-Tuning Cost-Aware Inference
BMW Group; Munich, Germany · Ludwig Maximilians Universitat; Munich, Germany · Technical University of Munich; Munich, Germany
Generating Leakage-Free Benchmarks for Robust RAG Evaluation
May 9, 2026Jiayi Liu, Jiaxing Zhang, Bowen Jin +1Benchmark Contamination Synthetic Benchmark Generation
Department of Computer Science, Purdue University · New Jersey Institute of Technology · University of Illinois at Urbana-Champaign +1
The Interference Gap: Comparing Retrieval Bounds in Human Memory and RAG Systems
May 9, 2026Dongxin Guo, Jikun Wu, Siu-Ming YiuComputational Cognitive Modeling Episodic Memory
The University of Hong Kong, Hong Kong, China · Stellaris AI Limited, Hong Kong, China · Brain Investing Limited, Hong Kong, China
LeakDojo: Decoding the Leakage Threats of RAG Systems
May 7, 2026Maosen Zhang, Jianshuo Dong, Boting Lu +4Data Leakage Privacy Leakage in Language Models
Tsinghua University, China · Ant International, China · Nanyang Technological University, Singapore
RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation
May 6, 2026Ivan Bondarenko, Roman Derunets, Oleg Sedukhin +3Language Model Generation Evaluation RAG Evaluation
Novosibirsk State University
DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluation
May 6, 2026Bryan Li, William Walden, Yu Hou +6Language Model Generation Evaluation RAG Evaluation
Google Inc. · University of Pennsylvania · Johns Hopkins University +3
Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study
May 4, 2026Devi Prasad Bal, Subhashree PuhanCross-Encoder Reranking Retrieval-Augmented Generation
The University of Texas at Austin
Towards Dependable Retrieval-Augmented Generation Using Factual Confidence Prediction
May 4, 2026Florian Geissler, Francesco Carella, Laura Fieback +1Retrieval-Augmented Generation Factual Consistency Evaluation
Fraunhofer Institute for Cognitive Systems (IKS), Munich, Germany · Volkswagen AG, Wolfsburg, Germany
Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?
May 1, 2026Lennard C. Froma, Tom Kouwenhoven, Maaike H. T. de Boer +2LLM Evaluation Language Model Scaling
Leiden University - The Netherlands · TNO - The Netherlands · TU Delft - The Netherlands
H-RAG at SemEval-2026 Task 8: Hierarchical Parent-Child Retrieval for Multi-Turn RAG Conversations
May 1, 2026Passant Elchafei, Hossam Emam, Mohamed Alansary +2Retrieval-Augmented Generation Information Retrieval
1Johannes Kepler University Linz, Institute of Computational Perception, Linz, Austria · 3Independent Researcher, Egypt · 2Linz Institute of Technology, Artificial Intelligence Lab, Austria
"I Don't Know" -- Towards Appropriate Trust with Certainty-Aware Retrieval Augmented Generation
May 1, 2026Daan Di Scala, Maaike de Boer, Pınar YolumRetrieval-Augmented Generation LLM Reliability
TNO Netherlands Organisation for Applied Scientific Research, Department Data Science · Utrecht University, Department of Information and Computing Sciences
NeocorRAG: Less Irrelevant Information, More Explicit Evidence, and More Effective Recall via Evidence Chains
Apr 30, 2026Shiyao Peng, Qianhe Zheng, Zhuodi Hao +7Multi-Hop QA Retrieval-Augmented Generation
Beijing University of Posts and Telecommunications
Quantifying Prior Dominance in RAG Systems
Apr 29, 2026Barak OrLLM Grounding Knowledge Conflicts in Language Models
ArtificialGate Ltd. · Google & Reichman Tech School · Technion – Israel Institute of Technology +1
When Good OCR Is Not Enough: Benchmarking OCR Robustness for Retrieval-Augmented Generation
Apr 29, 2026Lin Sun, Wang Dexian, Jingang Huang +4RAG Evaluation Optical Character Recognition
Beijing Qiyuan Technology
Benchmarking Complex Multimodal Document Processing Pipelines: A Unified Evaluation Framework for Enterprise AI
Apr 29, 2026Saurabh K. Singh, Sachin RajRAG Evaluation IR Evaluation
Oracle · Independent
Faithfulness-QA: A Counterfactual Entity Substitution Dataset for Training Context-Faithful RAG Models
Apr 28, 2026Li Ju, Junzhe Wang, Qi ZhangContextual Faithfulness Retrieval-Augmented Generation
College of Computer Science and Artificial Intelligence, Fudan University
Domain Fine-Tuning vs. Retrieval-Augmented Generation for Medical Multiple-Choice Question Answering: A Controlled Comparison at the 4B-Parameter Scale
Apr 26, 2026Avi-ad Avraam BuskilaRetrieval-Augmented Generation Healthcare
Department of Information Science and Applied Artificial Intelligence, Bar-Ilan University, Ramat-Gan, Israel
FinGround: Detecting and Grounding Financial Hallucinations via Atomic Claim Verification
Apr 26, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuLLM Answer Verification Financial QA
1The University of Hong Kong · 2Stellaris AI Limited
Can QPP Choose the Right Query Variant? Evaluating Query Variant Selection for RAG Pipelines
Apr 24, 2026Negar Arabzadeh, Andrew Drozdov, Michael Bendersky +1Retrieval-Augmented Generation RAG Evaluation
UC Berkeley · Berkeley, CA, United States · Databricks +1
Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA
Apr 24, 2026Zhanli Li, Yixuan Cao, Lvzhou Luo +1Multi-Document QA Document QA
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences (CAS), Beijing 100190, China · University of Chinese Academy of Sciences, Beijing 100049, China
Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation
Apr 22, 2026Andrew Klearman, Radu Revutchi, Rohin Garg +3RAG Evaluation IR Evaluation
Scale AI.
NightFeats @ MMU-RAGent NeurIPS 2025: A Context-Optimized Multi-Agent RAG System for the Text-to-Text Track
Apr 21, 2026Quentin Fever, Naziha AslamMulti-Agent LLM Systems Grounded Text Generation
Team NightFeats
RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora
Apr 21, 2026Hanjun Cho, Jay-Yoon LeeSynthetic Benchmark Generation Benchmark Validity
Allganize · Seoul National University
MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval
Apr 20, 2026Shaden Alshammari, Kevin Wen, Abrar Zainal +5Mathematical Reasoning Benchmarks Multilingual Language Model Evaluation
MIT · HUMAIN · Individual Researcher +1
Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies
Apr 20, 2026Lorenz Brehme, Thomas Ströhle, Ruth BreuMulti-Hop QA LLM-as-a-Judge
Universität Innsbruck, Technikerstraße 21a, 6020 Innsbruck, Austria
A Benchmark Construction and Evaluation Framework for Specialist Domains: Case Study on Defense-related Documents
Apr 20, 2026Bao Gia Doan, Aditya Joshi, Pantelis Elinas +4Synthetic Benchmark Generation RAG Evaluation
UNSW Sydney, Australia · Cyndr AI, Australia
KIRA: Knowledge-Intensive Image Retrieval and Reasoning Architecture for Specialized Visual Domains
Apr 18, 2026Parthaw Goswami, Jaynto Goswami DeepMultimodal RAG Visual Reasoning
University of Missouri · SAP Prague

May 11, 2026J/K move · Enter open · S save

Trust or Abstain? A Self-Aware RAG Approach

Xi Zhu, Ziqi Wang, Kai Mei, Wujiang Xu+4

Rutgers University · University of Illinois Urbana-Champaign

Retrieval-Augmented Generation Knowledge Conflicts in Language Models LLM Reliability LLM Uncertainty Estimation RAG Evaluation LLM Abstention

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) improves large language models (LLMs) by incorporating external evidence, but it also introduces knowledge conflicts when retrieved contextual knowledge (CK) and parametric knowledge (PK) disagree or are both unreliable. Existing approaches mainly coordinate which source to use, without explicitly asking whether each answer path is correct. We argue that faithful RAG requires LLM self-awareness, namely the ability to recognize the limits of its own knowledge and reasoning. To ground this problem, we construct a model-specific, ground-truth-aligned knowledge-conflict benchmark by evaluating LLM backbones on PK-only and CK-conditioned answer paths over approximately 69K query-context instances per backbone, drawn from five conflict-QA datasets. We then introduce SABER, a Self-Aware Belief Estimator for RAG that requires no LLM fine-tuning. SABER combines a self-prior with PK-side and CK-side conditional reasoning representations from multi-trace inference, then estimates reliability beliefs with two lightweight predictors to drive a 4-cell decision over trust PK, trust CK, trust either, or abstain. Across four LLM backbones, SABER improves end-to-end accuracy and conflict-specific faithfulness over ten inference-time and fine-tuning baselines, with the largest gains on conflict-heavy datasets. Under abstention, SABER's risk-coverage curve Pareto-dominates every prompt-based abstainer, providing a tunable balance between coverage and answer risk. Our code is available at https://github.com/xizhu1022/SABER.