IR Evaluation

IR: Information Retrieval

Ask about this topic

Overview

Information Retrieval evaluation studies whether search systems find useful documents and evidence, and how reliably their performance can be measured. Common approaches use judged benchmark collections and metrics such as precision, recall, mean reciprocal rank, and nDCG to compare lexical methods like BM25, dense and hybrid retrievers, and rerankers. Benchmarks increasingly test specialized domains, multilingual and multimodal search, temporal change, and retrieval within RAG and agent workflows, alongside efficiency, robustness, and coverage. A central challenge is that aggregate relevance scores and automated judgments can miss incomplete evidence, biased or outdated benchmarks, and whether retrieved material actually helps downstream decisions.

Latest papers

CardsList

189 papers30 papers in the last four weeks, up 43% on the four weeks before. 0.2% of all new papers.

  1. Does the Citation Matter? Quality, Grounding, and Physician-Rubric Accuracy of Google's AI Overviews for Medical Information

    Oct 29, 2026Sam Agnoli, Brigid Reilly, Lily Barna +1Medical QAEvidence-Grounded Generation

    Northwestern University, Evanston, Illinois, USA

  2. LiveSearch: Questions as Executable Verifier Code for a Live Benchmark of Professional-Source Financial Search Agents

    Oct 27, 2026Sinuo Wang, Wentao Zhao, Jiacheng Lu +6Financial Language Model EvaluationBenchmark Design

    StepFun · Adelaide University · Tsinghua University +3

  3. Coverage Under Contradiction: A Benchmark for Negation-Aware Retrieval

    Oct 27, 2026Pasha Abrishamchian, Eugene Yang, Andrew Yates +2IR EvaluationEvidence Retrieval

    Georgetown University Washington, DC, USA · Johns Hopkins University Baltimore, MD, USA

  4. Nothing Left Behind: Embedding Retrieval and Language Model Screening for High Recall Evidence Search

    Oct 27, 2026Giuliano Martinelli, Abelardo Carlos Martinez Lorenzo, Satvik Garg +4Evidence RetrievalInformation Retrieval

    Development Impact AI Lab, World Bank Group

  5. System One models Are Not Better Rerankers

    Oct 23, 2026Michael Dinzinger, Kanishka Ghosh Dastidar, Michael GranitzerLLM RerankingInformation Retrieval

    University of Passau, Passau, Germany · Interdisciplinary Transformation University Austria (IT:U), Linz, Austria

  6. What the Query Remembers: Query Abstraction Governs Lexical and Dense Retrieval of Classical Arabic Poetry

    Oct 22, 2026Omar Farouk Zouak, Selma Mani, Larbi Said Chikh +1Lexical RetrievalDense Retrieval

    National School of Artificial Intelligence

  7. How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions

    Oct 22, 2026Miguel Mochizuki Silva, Samuel Ismael, Mateus C. Dias +3RAG EvaluationBiomedical IR

    Universidade Federal da Paraíba (UFPB), João Pessoa, PB, Brazil · Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA

  8. The Fusion Tax: A Diagnostic Analysis of When Hybrid Retrieval Hurts

    Oct 22, 2026Aaditya Chauhan, Arun AgarwalHybrid RetrievalData Fusion

    Independent Researcher, Bothell, WA 98012 USA · Independent Researcher, Issaquah, WA 98027 USA

  9. IFC-MCQ: Automatically Generating Multiple-Choice Benchmarks for Evaluating LLM Agents on IFC Data Retrieval

    Oct 21, 2026Rabindra Lamsal, Sisi Zlatanova, Xuesong ShenSynthetic Benchmark GenerationAI Agent Benchmarks

    GRID Lab, School of Built Environment, The University of New South Wales, Sydney, Australia · School of Civil and Environmental Engineering, The University of New South Wales, Sydney, Australia

  10. One Fact, Many Places: Redundant Evidence in Long Visual Documents Breaks Single-Gold Evaluation of Retrieval, Citation and Attribution

    Oct 19, 2026Xiaokang Yuan, Binhui Yao, Luwei Yang +2IR EvaluationEvidence Retrieval

    Zhejiang University, Hangzhou, China · Zhejiang Geely Holding Group Co., Ltd, Hangzhou, China

  11. Cross-Pollination Retrieval: Finding Transferable Ideas Across Fields in Scientific Literature

    Oct 19, 2026Seokwon Jung, Yerang Kim, Yejin Kim +5Information RetrievalScientific Document Retrieval

    KAIST · Tübingen AI Center, University of Tübingen · NAVER AI Lab

  12. Exact Risk Certificates for Pairwise-Judged Reranking

    Oct 17, 2026Xinyu Wang, Jingrui Tian, Sicheng Lyu +4IR EvaluationPairwise Ranking

    McGill University · Ohio University · University of Manitoba

  13. MOSS: A Production Framework for Monitoring and Diagnosing Semantic Failures in Search and Recommendation Systems

    Oct 16, 2026Alina Nesen, Prathyusha Kanmanth Reddy, David Relyea +4Recommender SystemsIR Evaluation

    Data Sciences, Target Corporation Minneapolis, MN, USA · Data Sciences, Target Corporation Sunnyvale, CA, USA

  14. Web Scale on a Budget: Tail-Aware Retrieval Evaluation

    Oct 13, 2026Boaz Meivar, Shai AvidanIR EvaluationInformation Retrieval

    Tel Aviv University

  15. FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval

    Oct 13, 2026Tripti Shukla, Shwetha Somasundaram, Akshat Gupta +3Automated EvaluationIR Evaluation

    Glance Bengaluru, India · Glance Vilnius, Lithuania · Glance London, United Kingdom

  16. Reference Wording Shapes Retrieval of Robot Task Instructions

    Oct 13, 2026Jahnavi YelamanchiLexical RetrievalIR Evaluation

  17. VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval

    Oct 12, 2026Simon Ott, Yufan Chen, Ruiping Liu +6Visual Document RetrievalLong-Context Retrieval

    Karlsruhe Institute of Technology · Hunan University

  18. CoverSynth: Coverage-Guided Generation of Retrieval Evaluation Sets

    Oct 11, 2026Anahita Sharma, Apaar Shanker, Rishav Chakravarti +6Synthetic Benchmark GenerationIR Evaluation

    Scale AI

  19. The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval

    Oct 10, 2026Jianbo ChenLLM AuditingIR Evaluation

    Tsinghua University Beijing, China

  20. LLM-based User-Product Relevance Evaluation for Dynamic Product Advertising

    Oct 10, 2026Wen Zhang, Xiao Bai, Hsiang-wei Chao +9Online AdvertisingIR Evaluation

    Snap Inc. Bellevue, WA, USA · Snap Inc. Palo Alto, CA, USA · Snap Inc. Seattle, WA, USA +1

  21. GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports

    Oct 8, 2026Veronika Solopova, Ivan Havlytskyi, Viktoria Skorik +1Question AnsweringDocument QA

  22. Language Models for Page-Level Layout Decisions in E-commerce Search

    Oct 7, 2026Varun Joshi, Eva C. Song, ChengXiang ZhaiE-Commerce SearchIR Evaluation

    Walmart Global Tech Hoboken, NJ, USA · University of Illinois at Urbana-Champaign Urbana, IL, USA

  23. Towards Explaining Query Expansion Performance in Information Retrieval

    Oct 7, 2026Sourav Saha, Aditya Dutta, Soumajit Pramanik +1Information RetrievalQuery Expansion

    Indian Statistical Institute, Kolkata, India · IIT Bombay, Bombay, India · IIT Bhilai, Chhattisgarh, India

  24. A Systematic Study of Semantic ID Spaces for Generative Information Retrieval

    Oct 6, 2026Alexia Allal, Hicham Randrianarivo, Sylvain LamprierResidual VQVector Quantization

    Artefact Research Center Paris, France · LERIA, Angers University Angers, France

  25. Leveraging a four-quadrant approach for evaluating Redpine Science

    Oct 6, 2026Filip Dorm, Leonora VesterbackaScientific QARAG Evaluation

    Redpine, Sweden.

  26. Beyond Semantic Similarity: Performance and Costs of Agentic Retrieval for Complex Tasks

    Oct 5, 2026Reza Esfandiarpoor, Radek Osmulski, Yauhen Babakhin +8Agentic RetrievalInformation Retrieval

    NVIDIA · University of Edinburgh

  27. Better Retrieval, Limited Clustering Gains: A Controlled Study of Multilingual Company Entity Resolution

    Oct 4, 2026Yijiashun Qi, Yuxuan Li, Hanzhe GuoEntity ResolutionClustering

    University of Michigan, Ann Arbor, USA · University of Pennsylvania, Philadelphia, PA, USA

  28. Benchmarking Literature Retrieval for a Model Organism: A Dictyostelium Case Study

    Oct 2, 2026Yun Wang, Gad Shaulsky, Tomaž Curk +1Scientific Document RetrievalBiomedical IR

    University of Ljubljana, Faculty of Computer and Information Science, Večna pot 113, 1000 Ljubljana, Slovenia · Department of Molecular and Human Genetics, Baylor College of Medicine, Baylor Plaza, Houston, TX 77030, USA

  29. ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

    Oct 1, 2026Sohyeon Kim, Yoonho Lee, Bo Liu +11AI Agents for Scientific DiscoveryAgentic Search

    Stanford University · Seoul National University · University of Washington +3

  30. What Should an Agent Remember? Disentangling Retention from Retrieval in Bounded-Memory Evaluation

    Sep 30, 2026Juli HuangAI Agent EvaluationMemory Agent Benchmarks

    Stanford University