IR Evaluation

IR: Information Retrieval

Overview

Information Retrieval evaluation studies whether search systems find useful documents and evidence, and how reliably their performance can be measured. Common approaches use judged benchmark collections and metrics such as precision, recall, mean reciprocal rank, and nDCG to compare lexical methods like BM25, dense and hybrid retrievers, and rerankers. Benchmarks increasingly test specialized domains, multilingual and multimodal search, temporal change, and retrieval within RAG and agent workflows, alongside efficiency, robustness, and coverage. A central challenge is that aggregate relevance scores and automated judgments can miss incomplete evidence, biased or outdated benchmarks, and whether retrieved material actually helps downstream decisions.

Latest papers

CardsList

199 papers19 papers in the last four weeks, down 34% on the four weeks before. 0.1% of all new papers.

  1. Can Programs Replace LLM Judges? Executable Program Judges for Scalable and Efficient Relevance Labeling

    Nov 18, 2026Eugene Agichtein, Charles L. A. Clarke, Kaustubh Dhole +1IR EvaluationLLM-as-a-Judge

  2. NONSENSE: Non-Semantic Strings Expose Semanticity Gaps in Dense Retrieval

    Nov 17, 2026Ankit GuptaDense RetrievalIR Evaluation

  3. Search-by-Answer Bias in Retrieval Models

    Nov 8, 2026Adiel Avigad, Shauli Ravfogel, Yoav GoldbergDense RetrievalIR Evaluation

  4. Grading the Unannotated: Insights from the WorkRB Skill Extraction Challenge at RecSys-HR 2026

    Nov 2, 2026Warre Veys, Luis Gasco, Jens-Joris DecorteIR EvaluationDocument Information Extraction

  5. Dhara Bench: An Evidence-Oriented Retrieval Benchmark for Indian Knowledge Systems

    Nov 2, 2026Avinash Holenarasipur Nagendra, Anand Kaniyaar Sriranga Prasad, Kousik Sarathy Sridharan +1Evidence RetrievalIR Evaluation

  6. Discrimination, Calibration, and Ranking: A Unified Matrix Framework

    Nov 1, 2026Ronaldo Cristiano Prati, Farangis JohariClassificationModel Calibration

  7. ProSearch: Benchmarking Multi-Constraint Protocol Retrieval in Experimental Science

    Oct 30, 2026Wenliang Liang, Zhiyuan Ning, Haolin Chen +3Scientific Document RetrievalIR Evaluation

  8. Defining Page-Centric Visual Document Retrieval and Evaluating Existing Benchmarks

    Oct 30, 2026Chongzhi Zhang, Yuqi Zuo, Aixin SunVisual Document RetrievalIR Evaluation

  9. Coverage Under Contradiction: A Benchmark for Negation-Aware Retrieval

    Oct 27, 2026Pasha Abrishamchian, Eugene Yang, Andrew Yates +2IR EvaluationEvidence Retrieval

  10. Nothing Left Behind: Embedding Retrieval and Language Model Screening for High Recall Evidence Search

    Oct 27, 2026Giuliano Martinelli, Abelardo Carlos Martinez Lorenzo, Satvik Garg +4Evidence RetrievalInformation Retrieval

  11. System One models Are Not Better Rerankers

    Oct 23, 2026Michael Dinzinger, Kanishka Ghosh Dastidar, Michael GranitzerLLM RerankingInformation Retrieval

  12. What the Query Remembers: Query Abstraction Governs Lexical and Dense Retrieval of Classical Arabic Poetry

    Oct 22, 2026Omar Farouk Zouak, Selma Mani, Larbi Said Chikh +1Lexical RetrievalDense Retrieval

  13. How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions

    Oct 22, 2026Miguel Mochizuki Silva, Samuel Ismael, Mateus C. Dias +3RAG EvaluationBiomedical IR

  14. The Fusion Tax: A Diagnostic Analysis of When Hybrid Retrieval Hurts

    Oct 22, 2026Aaditya Chauhan, Arun AgarwalHybrid RetrievalData Fusion

  15. IFC-MCQ: Automatically Generating Multiple-Choice Benchmarks for Evaluating LLM Agents on IFC Data Retrieval

    Oct 21, 2026Rabindra Lamsal, Sisi Zlatanova, Xuesong ShenSynthetic Benchmark GenerationAI Agent Benchmarks

  16. One Fact, Many Places: Redundant Evidence in Long Visual Documents Breaks Single-Gold Evaluation of Retrieval, Citation and Attribution

    Oct 19, 2026Xiaokang Yuan, Binhui Yao, Luwei Yang +2IR EvaluationEvidence Retrieval

  17. Cross-Pollination Retrieval: Finding Transferable Ideas Across Fields in Scientific Literature

    Oct 19, 2026Seokwon Jung, Yerang Kim, Yejin Kim +5Information RetrievalScientific Document Retrieval

  18. Exact Risk Certificates for Pairwise-Judged Reranking

    Oct 17, 2026Xinyu Wang, Jingrui Tian, Sicheng Lyu +4IR EvaluationPairwise Ranking

  19. MOSS: A Production Framework for Monitoring and Diagnosing Semantic Failures in Search and Recommendation Systems

    Oct 16, 2026Alina Nesen, Prathyusha Kanmanth Reddy, David Relyea +4Recommender SystemsIR Evaluation

  20. FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval

    Oct 13, 2026Tripti Shukla, Shwetha Somasundaram, Akshat Gupta +3Automated EvaluationIR Evaluation

  21. VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval

    Oct 12, 2026Simon Ott, Yufan Chen, Ruiping Liu +6Visual Document RetrievalLong-Context Retrieval

  22. CoverSynth: Coverage-Guided Generation of Retrieval Evaluation Sets

    Oct 11, 2026Anahita Sharma, Apaar Shanker, Rishav Chakravarti +6Synthetic Benchmark GenerationIR Evaluation

  23. LLM-based User-Product Relevance Evaluation for Dynamic Product Advertising

    Oct 10, 2026Wen Zhang, Xiao Bai, Hsiang-wei Chao +9Online AdvertisingIR Evaluation