IR Evaluation

IR: Information Retrieval

Information Retrieval evaluation studies whether search systems find useful documents and evidence, and how reliably their performance can be measured. Common approaches use judged benchmark collections and metrics such as precision, recall, mean reciprocal rank, and nDCG to compare lexical methods like BM25, dense and hybrid retrievers, and rerankers. Benchmarks increasingly test specialized domains, multilingual and multimodal search, temporal change, and retrieval within RAG and agent workflows, alongside efficiency, robustness, and coverage. A central challenge is that aggregate relevance scores and automated judgments can miss incomplete evidence, biased or outdated benchmarks, and whether retrieved material actually helps downstream decisions.

Latest papers

189 papers30 papers in the last four weeks, up 43% on the four weeks before. 0.2% of all new papers.

Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation
Apr 22, 2026Andrew Klearman, Radu Revutchi, Rohin Garg +3RAG Evaluation IR Evaluation
Scale AI.
Semantic Recall for Vector Search
Apr 22, 2026Leonardo Kuffo, Ioanna Tsakalidou, Roberta De Viti +3IR Evaluation Approximate Nearest Neighbor Search
CWI · EPFL · MPI-SWS +1
RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora
Apr 21, 2026Hanjun Cho, Jay-Yoon LeeSynthetic Benchmark Generation Benchmark Validity
Allganize · Seoul National University
Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies
Apr 20, 2026Lorenz Brehme, Thomas Ströhle, Ruth BreuMulti-Hop QA LLM-as-a-Judge
Universität Innsbruck, Technikerstraße 21a, 6020 Innsbruck, Austria
TeleEmbedBench: A Multi-Corpus Embedding Benchmark for RAG in Telecommunications
Apr 20, 2026Pranshav Gajjar, Vijay K ShahText Embedding Evaluation Text Embedding Models
NextG Wireless Lab, North Carolina State University, Raleigh, USA
SeekerGym: A Benchmark for Reliable Information Seeking
Apr 18, 2026Remy Kim, Minseung Lee, Shuo Li +1AI Agent Reliability Agentic Search
University of Pennsylvania, Curation Labs · Curation Labs · Google DeepMind +1
JFinTEB: Japanese Financial Text Embedding Benchmark
Apr 17, 2026Masahiro Suzuki, Hiroki SakajiText Embedding Evaluation Text Embedding Models
Amova Asset Management Co., Ltd. · Tokyo, Japan · Hokkaido University +1
UsefulBench: Towards Decision-Useful Information as a Target for Information Retrieval
Apr 17, 2026Tobias Schimanski, Stefanie Lewandowski, Christian Woerle +3Information Retrieval IR Evaluation
University of Zurich · ETH Zurich · score4more GmbH +2
Controlling Authority Retrieval: A Missing Retrieval Objective for Authority-Governed Knowledge
Apr 15, 2026Andre BacellarLegal IR IR Evaluation
Evaluation of Agents under Simulated AI Marketplace Dynamics
Apr 15, 2026To Eun Kim, Alireza Salemi, Hamed Zamani +1AI Agent Evaluation IR Evaluation
Carnegie Mellon University, Pittsburgh, PA, United States · University of Massachusetts Amherst, Amherst, MA, United States
JUÁ -- A Benchmark for Information Retrieval in Brazilian Legal Text Collections
Apr 7, 2026Jayr Pereira, Leandro Fernandes, Erick de Brito +2Legal IR IR Evaluation
Centro de Ciências e Tecnologia, Universidade Federal do Cariri (UFCA), Juazeiro do Norte, Ceará, Brazil. · Faculdade de Engenharia Elétrica e da Computação, Universidade Estadual de Campinas (UNICAMP), Campinas, São Paulo, Brazil. · Câmara dos Deputados, Brasília, Brazil. +1
CraterBench-R: Instance-Level Crater Retrieval for Planetary Scale
Apr 6, 2026Jichao Fang, Lei Zhang, Michael Phillips +1Image Retrieval Image Matching
Northern Illinois University · University of Arizona
PeopleSearchBench: Evaluating AI-Powered People Search Platforms with Criteria-Grounded Verification
Mar 29, 2026Tianyu Shi, Wei Wang, Zequn Xie +10Web Search Agents Information Retrieval
ClimateCheck 2026: Scientific Fact-Checking and Disinformation Narrative Classification of Climate-related Claims
Mar 27, 2026Raia Abu Ahmad, Max Upravitelev, Aida Usmanova +2Scientific Claim Verification Automated Fact-Checking
Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKI), Germany · Technische Universität Berlin, Germany · Leuphana Universität Lüneburg, Germany +1
What Tracks Hard-Negative Transfer? A Study of Lexical and Semantic Signals
Mar 22, 2026Aarush Sinha, Rahul Seetharaman, Aman BansalInformation Retrieval Hard Negative Mining
University of Copenhagen Copenhagen, Denmark · Independent Researcher USA
Errors in AI-Assisted Retrieval of Medical Literature: A Comparative Study
Mar 21, 2026Jenny Gao, Yongfeng Zhang, Mary L Disis +1LLM Reliability Biomedical IR
College of Arts and Science, New York University, New York, NY · College of Arts and Science, New York University, New York, NY 10003 · Department of Computer Sciences, School of Arts & Sciences, Rutgers University, Piscataway, NJ +9
CoverageBench: Evaluating Information Coverage across Tasks and Domains
Mar 20, 2026Saron Samuel, Andrew Yates, Dawn Lawrie +4Information Retrieval IR Evaluation
Johns Hopkins University, Baltimore, MD, USA · National Institute of Standards and Technology, Gaithersburg, MD, USA
PJB: A Reasoning-Aware Benchmark for Person-Job Retrieval
Mar 18, 2026Guangzhi Wang, Xiaohui Yang, Kai Li +4Job-Candidate Matching Information Retrieval
CareerInternational Research Team
Public Profile Matters: A Scalable Integrated Approach to Recommend Citations in the Wild
Mar 18, 2026Karan Goyal, Dikshant Kukreja, Vikram Goyal +1Recommender Systems Learning to Rank
Indraprastha Institute of Information Technology Delhi (IIIT-Delhi), New Delhi, India.
LMEB: Long-horizon Memory Embedding Benchmark
Mar 13, 2026Xinping Zhao, Xinshuo Hu, Jiaxin Xu +9Text Embedding Evaluation IR Evaluation
Harbin Institute of Technology (Shenzhen) · Shenzhen Loop Area Institute (SLAI) · Peking University
Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections
Mar 12, 2026Łukasz Borchmann, Jordy Van Landeghem, Michał Turski +12Multi-Document QA LLM Agent Evaluation
Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage
Mar 9, 2026Saron Samuel, Alexander Martin, Eugene Yang +5RAG Evaluation IR Evaluation
Johns Hopkins University Baltimore, MD, USA · Johns Hopkins HLTCOE Baltimore, MD, USA · National Institute of Standards and Technology Gaithersburg, MD, USA +1
Understanding the Performance Plateau in Text-to-Video Retrieval: A Comprehensive Empirical and Linguistic Analysis
Mar 7, 2026Maria-Eirini Pegia, Dimitrios Stefanopoulos, Björn Þór Jónsson +4Video Retrieval Video-Language Model Evaluation
Information Technologies Institute, CERTH-ITI, Thessaloniki, Greece. · School of Computer Science, Reykjavík University, Reykjavík, Iceland.
A Systematic Investigation of Document Chunking Strategies and Embedding Sensitivity
Mar 7, 2026Muhammad Arslan Shaukat, Muntasir Adnan, Carlos C. N. KuhnDocument Chunking Dense Retrieval
Open Source Institute, Faculty of Science and Technology, University of Canberra, Canberra, Australia
iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics
Mar 4, 2026Preetam Prabhu Srikar Dammu, Arnav Palkhiwala, Tanya Roosta +1AI Agent Benchmarks LLM Agent Evaluation
University of Washington Seattle, Washington, USA · UC Berkeley Berkeley, California, USA
Still Fresh? Evaluating Temporal Drift in Retrieval Benchmarks
Mar 4, 2026Nathan Kuissi, Suraj Subrahmanyan, Nandan Thakur +1IR Evaluation Temporal IR
University of Waterloo Waterloo, Canada
Not All Queries Need Rewriting: When Prompt-Only LLM Refinement Helps and Hurts Dense Retrieval
Mar 2, 2026Varun KotteQuery Reformulation Dense Retrieval
Adobe
Legal RAG Bench: an end-to-end benchmark for legal RAG
Mar 2, 2026Abdur-Rahman Butler, Umar ButlerRetrieval-Augmented Generation RAG Evaluation
Isaacus
Spelling Correction in Healthcare Query-Answer Systems: Methods, Retrieval Impact, and Empirical Evaluation
Feb 25, 2026Saurabh K SinghMedical QA IR Evaluation
Oracle Corporation
Revisiting RAG Retrievers: An Information Theoretic Benchmark
Feb 25, 2026Wenqing Zheng, Dmitri Kalaev, Noah Fatsi +5RAG Evaluation IR Evaluation
CapitalOne.

Apr 22, 2026J/K move · Enter open · S save

Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation

Andrew Klearman, Radu Revutchi, Rohin Garg, Rishav Chakravarti+2

Scale AI.

RAG Evaluation IR Evaluation

Open paper AskPDF arXiv

Retrieval quality is the primary bottleneck for accuracy and robustness in retrieval-augmented generation (RAG). Current evaluation relies on heuristically constructed query sets, which introduce a hidden intrinsic bias. We formalize retrieval evaluation as a statistical estimation problem, showing that metric reliability is fundamentally limited by the evaluation-set construction. We further introduce semantic stratification, which grounds evaluation in corpus structure by organizing documents into an interpretable global space of entity-based clusters and systematically generating queries for missing strata. This yields (1) formal semantic coverage guarantees across retrieval regimes and (2) interpretable visibility into retrieval failure modes. Experiments across multiple benchmarks and retrieval methods validate our framework. The results expose systematic coverage gaps, identify structural signals that explain variance in retrieval performance, and show that stratified evaluation yields more stable and transparent assessments while supporting more trustworthy decision-making than aggregate metrics.