RAG Evaluation

RAG: Retrieval-Augmented Generation

Ask about this topic

Overview

RAG evaluation studies whether systems find relevant evidence, use it faithfully, and produce correct, appropriately cited answers—including when they should abstain. Common approaches combine retrieval metrics such as Recall and nDCG with answer-level checks for correctness, coverage, faithfulness, citation support, and uncertainty; component ablations and error analysis help locate failures. Benchmarks increasingly test multi-hop and multi-turn questions, long or multimodal documents, domain-specific tasks, and noisy, conflicting, or adversarial evidence, alongside latency and resource costs. A central open tension is that better retrieval scores do not reliably yield better answers, while automated judges and metrics may be costly, unstable, or fail to transfer across datasets and domains.

Latest papers

CardsList

355 papers39 papers in the last four weeks, up 3% on the four weeks before. 0.2% of all new papers.

  1. An Industrial-Scale Retrieval-Augmented Generation Framework for Requirements Engineering: Empirical Evaluation with Automotive Manufacturing Data

    Mar 20, 2026Muhammad Khalid, Yilmaz UygunRequirements EngineeringRAG Evaluation

    Constructor University Bremen, Bremen, Germany

  2. Bypassing Document Ingestion: An MCP Approach to Financial Q&A

    Mar 19, 2026Sasan Mansouri, Edoardo Pilla, Mark Wahrenburg +1Financial QAModel Context Protocol

    University of Groningen · Goethe University Frankfurt · Halle Institute for Economic Research (IWH) and Martin Luther University Halle-Wittenberg

  3. Is Conformal Factuality for RAG-based LLMs Robust? Novel Metrics and Systematic Insights

    Mar 17, 2026Yi Chen, Daiwei Chen, Sukrut Madhav Chikodikar +2LLM ReliabilityRAG Evaluation

    University of Wisconsin-Madison

  4. Open-Source Reproduction and Explainability Analysis of Corrective Retrieval Augmented Generation

    Mar 17, 2026Surya Vardhan YalavarthiRetrieval-Augmented GenerationRAG Robustness

    College of Engineering and Applied Science, University of Cincinnati

  5. Selective Memory for Artificial Intelligence: Write-Time Gating with Hierarchical Archiving

    Mar 16, 2026Oliver Zahn, Simran ChanaMemory-Augmented Language ModelsPersistent Memory for Language Models

    Independent Researcher · University of Cambridge

  6. HindSight: Evaluating LLM-Generated Research Ideas via Future Impact

    Mar 16, 2026Bo JiangScientific Idea EvaluationResearch Idea Generation

    Temple University

  7. Automatic Inter-document Multi-hop Scientific QA Generation

    Mar 15, 2026Seungmin Lee, Dongha Kim, Yuni Jeon +2Scientific QAMulti-Hop QA

    Yonsei University, Seoul, Republic of Korea · OnomaAI, Seoul, Republic of Korea

  8. Rethinking Evaluation in Retrieval-Augmented Personalized Dialogue: A Cognitive and Linguistic Perspective

    Mar 15, 2026Tianyi Zhang, David TraumRAG EvaluationConversational Agents

    Institute for Creative Technologies, University of Southern California 12015 Waterfront Drive, Los Angeles, CA 90094-2536, USA

  9. Developing and evaluating a chatbot to support maternal health care

    Mar 13, 2026Smriti Jha, Vidhi Jain, Jianyu Xu +8HealthcareMedical LLMs

    Carnegie Mellon University · Population Council Institute · Sitaram Bhartia Institute of Science and Research +1

  10. Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents

    Mar 12, 2026Yaocong Li, Qiang Lan, Leihan Zhang +1Retrieval-Augmented GenerationLegal NLP

    School of Economics and Management, Beijing University of Posts and Telecommunications, No. 10 Xitucheng Road, Haidian District, Beijing, 100876, China · College of Computing, Beijing Information Science and Technology University, No. 12 Xiaoying East Road, Haidian District, Beijing, 100085, China

  11. QChunker: Learning Question-Aware Text Chunking for Domain RAG via Multi-Agent Debate

    Mar 12, 2026Jihao Zhao, Daixuan Li, Pengfei Li +3Document ChunkingRetrieval-Augmented Generation

    School of Information, Renmin University of China Beijing, China · School of Smart Governance, Renmin University of China Suzhou, China · School of Information, BRAIN, Renmin University of China Beijing, China +1

  12. Can Small Language Models Use What They Retrieve? An Empirical Study of Retrieval Utilization Across Model Scale

    Mar 12, 2026Sanchit PandeyRetrieval-Augmented GenerationKnowledge Augmentation for Language Models

    BITS Pilani, Hyderabad, India · BITS Pilani, Hyderabad Campus, Telangana, India

  13. Citation-Enforced RAG for Fiscal Document Intelligence: Cited, Explainable Knowledge Retrieval in Tax Compliance

    Mar 11, 2026Akhil Chandra ShanivendraDocument QAKnowledge-Intensive QA

    Independent Researcher

  14. Structured Linked Data as a Memory Layer for Agent-Orchestrated Retrieval

    Mar 11, 2026Andrea Volpini, Elie Raad, Beatrice Gamba +1Structured RAGRetrieval-Augmented Generation

    WordLift, Rome, Italy

  15. End-to-End Chatbot Evaluation with Adaptive Reasoning and Uncertainty Filtering

    Mar 11, 2026Nhi Dang, Tung Le, Huy Tien NguyenAutomated EvaluationRAG Evaluation

    Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam; Vietnam National University, Ho Chi Minh City, Vietnam

  16. AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversations

    Mar 11, 2026Dimosthenis Athanasiou, Maria Lymperaiou, Giorgos Filandrianos +2Retrieval-Augmented GenerationRAG Evaluation

    National Technical University of Athens

  17. Evaluation of LLMs in retrieving food and nutritional context for RAG systems

    Mar 10, 2026Maks Požarnik Vavken, Matevž Ogrinc, Tome Eftimov +1RAG Evaluation

    Department of Computer Systems, Institute "Jožef Stefan", Ljubljana, Slovenija

  18. AI Act Evaluation Benchmark: An Open, Transparent, and Reproducible Evaluation Dataset for NLP and RAG Systems

    Mar 10, 2026Athanasios Davvetas, Michael Papademas, Xenia Ziouvelou +1RAG Evaluation

    National Centre for Scientific Research “Demokritos”, Institute of Informatics and Telecommunications, Aghia Paraskevi, Greece · Department of Communication, Media and Culture, Panteion, University of Social and Political Sciences, Athens, Greece

  19. Quantifying the Accuracy and Cost Impact of Design Decisions in Budget-Constrained Agentic LLM Search

    Mar 9, 2026Kyle McCleary, James GhawalyAgentic SearchAgentic RAG

    Division of Computer Science & Engineering, Louisiana State University, Baton Rouge, Louisiana, USA

  20. Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage

    Mar 9, 2026Saron Samuel, Alexander Martin, Eugene Yang +5RAG EvaluationIR Evaluation

    Johns Hopkins University Baltimore, MD, USA · Johns Hopkins HLTCOE Baltimore, MD, USA · National Institute of Standards and Technology Gaithersburg, MD, USA +1

  21. SoK: Agentic Retrieval-Augmented Generation (RAG): Taxonomy, Architectures, Evaluation, and Research Directions

    Mar 7, 2026Saroj Mishra, Suman Niroula, Umesh Yadav +3Agentic RAGRAG Robustness

    University of North Dakota · Youngstown State University · University of Toledo +2

  22. Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarks

    Mar 6, 2026Jena D. Hwang, Varsha Kishore, Amanpreet Singh +9Automated EvaluationRAG Evaluation

    Allen Institute for AI

  23. StratRAG: A Multi-Hop Retrieval Evaluation Dataset for Retrieval-Augmented Generation Systems

    Mar 6, 2026Aryan PatodiyaMulti-Hop RetrievalRAG Evaluation

    Department of Computer Science, California State University, Fresno, Fresno, CA, USA

  24. LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation

    Mar 6, 2026Koki Itai, Shunichi Hasegawa, Yuta Yamamoto +2RAG EvaluationLLM Evaluation

    neoAI Inc., Tokyo, Japan · Tokyo Metropolitan University, Tokyo, Japan · The University of Tokyo, Tokyo, Japan

  25. Towards Robust Retrieval-Augmented Generation Based on Knowledge Graph: A Comparative Analysis

    Mar 5, 2026Hazem Amamou, Stéphane Gagnon, Alan Davoust +1RAG RobustnessGraph Retrieval-Augmented Generation

    Institut national de la recherche scientifique (INRS-EMT), Montréal, Québec, Canada · INRS-UQO Mixed Research Unit on Cybersecurity, Gatineau, Québec, Canada · Université du Québec en Outaouais, Gatineau, Québec, Canada

  26. Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG

    Mar 4, 2026Martin Asenov, Kenza Benkirane, Dan Goldwater +1RAG EvaluationMultimodal IR

    Parexel AI Labs London, United Kingdom · Parexel AI Labs San Francisco, United-States

  27. Fast and Faithful: Real-Time Verification for Long-Document Retrieval-Augmented Generation Systems

    Mar 4, 2026Xunzhuo Liu, Bowei He, Xue Liu +2LLM Answer VerificationRetrieval-Augmented Generation

    Red Hat · MBZUAI, McGill University

  28. RADIANT-LLM: an Agentic Retrieval Augmented Generation Framework for Reliable Decision Support in Safety-Critical Nuclear Engineering

    Mar 4, 2026Zavier Ndum Ndum, Jian Tao, John Ford +2Agentic RAGMultimodal RAG

    Department of Nuclear Engineering, Texas A&M University, College Station, TX, USA · College of Performance, Visualization and Fine Arts, Texas A&M University, College Station, TX, USA

  29. RAG-X: Systematic Diagnosis of Retrieval-Augmented Generation for Medical Question Answering

    Mar 3, 2026Aswini Sivakumar, Vijayan Sugumaran, Yao QiangRetrieval-Augmented GenerationRAG Evaluation

    Oakland University, Rochester, MI, USA

  30. Scaling Retrieval Augmented Generation with RAG Fusion: Lessons from an Industry Deployment

    Mar 2, 2026Luigi Medrano, Arush Verma, Mukul ChhabraRetrieval-Augmented GenerationRAG Evaluation

    Dell Technologies, Austin, TX, USA