IR Evaluation

IR: Information Retrieval

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 36% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 122

All topics

Entity Labels Are Not Entity Signals: A Framework for Observable Relevance in Document Re-Ranking
Jun 14, 2026Utshab Kumar Ghosh, Shubham ChatterjeeLearning to Rank Information Retrieval
Department of Computer Science Missouri University of Science and Technology Rolla, MO, USA
GIScholarBench: Benchmarking LLM Overconfidence in GIS Research
Jun 6, 2026Zongrng Li, Mingzheng Yang, Lei Zou +8LLM Evaluation AI-Assisted Scientific Research
Department of Geography Texas A&M University College Station, TX, USA · Google Mountain View, CA, USA · Department of Landscape Architecture and Urban Planning Texas A&M University College Station, TX, USA
PaperFlow: Profiling, Recommending, and Adapting Across Daily Paper Streams
Jun 5, 2026Fuqiang Wang, Song Tan, Zheng Guo +8Information Retrieval Recommender Systems
1Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences) · University of Chinese Academy of Science · 3Shanghai Artificial Intelligence Laboratory
Meaning in Order, Order in Meaning: Semantic R-precision for Keyphrase Evaluation
Jun 5, 2026Shamira Venturini, Steffen KinkelAutomated Evaluation IR Evaluation
ILIN - Institute for Learning and Innovation in Networks, Karlsruhe University of Applied Sciences, Moltkestraße 30, 76133 Karlsruhe, Germany · Karlsruhe Institute of Technology, Kaiserstraße 12, 76131 Karlsruhe, Germany
Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference
Jun 3, 2026Abhishek DivekarLLM-as-a-Judge IR Evaluation
Amazon
QO-Bench: Diagnosing Query-Operator-Preserving Retrieval over Typed Event Tuples
Jun 3, 2026Mengao Zhang, Xiang Yang, Chang Liu +2Information Retrieval RAG Evaluation
Nanyang Technological University · Asian Institute of Digital Finance, National University of Singapore
ANN Search: Recall What Matters
Jun 3, 2026Dimitris Dimitropoulos, Nikos MamoulisIR Evaluation Approximate Nearest Neighbor Search
University of Ioannina & Archimedes, Athena RC · Ioannina, Greece
LCSHBench: A Multilingual, Consensus-Grounded Benchmark for Library of Congress Subject Heading Assignment
Jun 3, 2026Kwok Leong TangLibrary and Information Science Multilingual IR
Variance Reduction for Heavy-Tailed Monetization Metrics in Ranking Experiments via Post-Stratification
Jun 2, 2026Neeti Pokharna, Olivier Jeunen, Yatharth Saraf +1Causal Effect Estimation A/b Testing
ShareChat · Bengaluru, Karnataka, India · Aampe +3
Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors
Jun 1, 2026Francisco Valentini, Edgar Altszyler, Martin FajcikIR Evaluation Dense Retrieval
CONICET-Universidad de Buenos Aires. Instituto de Ciencias de la Computación (ICC). Buenos Aires, Argentina · Quantit, Buenos Aires, Argentina · Brno University of Technology, Brno, Czech Republic
SPECTRA: Synthetic IR Test Collections with Relevance Oracles and Controlled Distractor Diagnostics
May 29, 2026Eric LiangSynthetic Data Generation Retrieval Robustness
Oracle
CanLegalRAGBench: Evaluating Retrieval-Augmented Generation on Canadian Case Law
May 28, 2026Ethan Zhao, Maksym Taranukhin, Wei Cui +2Retrieval-Augmented Generation Legal QA
Department of Computer Science, University of British Columbia · Vector Institute · Peter A. Allard School of Law, University of British Columbia +1
Entity-Collision: A Stratified Protocol for Attributing Retrieval Lift in Agent Memory
May 28, 2026Youwang DengAgent Memory Text Embedding Evaluation
Independent Researcher
Rethinking Literature Search Evaluation: Deep Research Helps, and Human Citation Lists Are Not a Ground Truth
May 28, 2026Gaurav Sahu, Laurent Charlin, Christopher PalLLM Reranking Information Retrieval
1Mila – Quebec AI Institute · 2HEC Montréal · 4Canada CIFAR AI Chair +3
Do Agents Need Semantic Metadata? A Comparative Study in Agentic Data Retrieval
May 27, 2026Shiyu Chen, Tarfah Alrashed, Alon Halevy +1LLM Agent Evaluation Agentic Retrieval
Google, Mountain View, CA, USA
Plans for Evaluating Structured Generative Search Summaries
May 26, 2026Tetsuya Sakai, Jina Lee, Hanpei Fang +1Language Model Generation Evaluation Summarization Evaluation
Waseda University/Naver Corporation · Tokyo, Japan · Waseda University +2
Resolving Ambiguity in Composed Image Retrieval via Calibrated Interaction
May 23, 2026Amsisan Tran, Baogh Le, Tuan Kiet Pham +1Conversational Search Image Retrieval
Amsisan Tran · Baogh Le · Tuan Kiet Pham +1
Benchmarking Patent Embeddings: A Multi-Task Evaluation of 22 Models Across Retrieval, Classification, and Clustering
May 22, 2026Amirhossein Yousefiramandi, Ciaran CooneyLegal IR Clustering
Clarivate, Intellectual Property Barcelona, Spain 08025
IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressions
May 21, 2026Kai Golan Hashiloni, Daniel Fadlon, Lior Livyatan +3Figurative Language Understanding Semantic Search
Data Science Institute, Reichman University, Herzliya, Israel · EfiArazi School of Computer Science, Reichman University, Herzliya, Israel · Vrije Universiteit Amsterdam, The Netherlands
SGR-Bench: Benchmarking Search Agents on State-Gated Retrieval
May 21, 2026Ningyuan Li, Haiyang Shen, Mugeng Liu +4Web Search Agents AI Agent Benchmarks
2Beijing University of Technology · 1Peking University
MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks
May 20, 2026Junhao Ruan, Abudukeyumu Abudula, Bei Li +8Conversational Search Synthetic Benchmark Generation
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · 2Meituan Inc., 3NiuTrans Research, Shenyang, China · 4Tsinghua University, Beijing, China
Chunking German Legal Code
May 19, 2026Max Prior, Natalia Milanova, Andreas SchultzDocument Chunking Retrieval-Augmented Generation
Technical University of Munich, Boltzmannstraße 3, 85748 Garching near Munich
CommitDistill: A Lightweight Knowledge-Centric Memory Layer for Software Repositories
May 18, 2026Divya Chukkapalli, Thejesh Avula, Aditya Aggarwal +2Agent Memory Software Engineering Agents
Microsoft Corporation
Temporal Decay of Co-Citation Predictability: A 20-Year Statute Retrieval Benchmark from 396M Ukrainian Court Citations
May 17, 2026Volodymyr OvcharovLegal IR Temporal IR
LEX AI LLC, Kyiv, Ukraine
The 99% Success Paradox: When Near-Perfect Retrieval Equals Random Selection
May 14, 2026Vyzantinos Repantis, Harshvardhan Singh, Tony Joseph +5IR Evaluation
Meta Platforms Inc. · Hacker Way, Menlo Park, CA 94025, USA
Measuring Google AI Overviews: Activation, Source Quality, Claim Fidelity, and Publisher Impact
May 13, 2026Haofei Xu, Umar Iqbal, Jacob M. MontgomeryCitation Verification Generative Retrieval
Washington University in St. Louis
Useful for Exploration, Risky for Precision: Evaluating AI Tools in Academic Research
May 11, 2026Anthea Dathe, Kiran Hoffmann, Aline MangoldAI-Assisted Scientific Research Scientific Reproducibility
Department Speculative Transformation Dresden University of Technology Dresden, 01062
Matching Meaning at Scale: Evaluating Semantic Search for 18th-Century Intellectual History through the Case of Locke
May 10, 2026Yu Wu, Ananth Mahadevan, Filip Ginter +2Historical Document Analysis Semantic Textual Similarity
University of Helsinki · 2TurkuNLP, University of Turku · 3ELLIS Institute Finland
TRACE: Tourism Recommendation with Accountable Citation Evidence
May 8, 2026Zixu Zhao, Sijin Wang, Yu Hou +6Recommender Systems IR Evaluation
UNSW Sydney · University of Adelaide · Yonsei University +1
Topic Is Not Agenda: A Citation-Community Audit of Text Embeddings
May 8, 2026Junseon YooText Embedding Evaluation Graph-Based Retrieval
Pluto Labs

Jun 14, 2026, cs.IRJ/K move · Enter open · S save

Entity Labels Are Not Entity Signals: A Framework for Observable Relevance in Document Re-Ranking

Utshab Kumar Ghosh, Shubham Chatterjee

Department of Computer Science Missouri University of Science and Technology Rolla, MO, USA

Learning to Rank Information Retrieval IR Evaluation

Open paper AskPDF arXiv

Entity-aware document retrieval uses query-associated entities as ranking signals, assuming that semantically relevant entities are also useful retrieval signals. We show this assumption is insufficient- and explain why. Unlike terms, which are ground-truth observations, entity links are hypotheses produced by an imperfect linker: an entity can be topically central yet provide no discriminative signal if the linker fires indiscriminately across relevant and non-relevant documents. We formalize this as a distinction between Conceptual Entity Relevance (CER)- whether an entity is topically related to a query- and Observable Entity Relevance (OER)- whether its observed presence in a collection discriminates relevant from non-relevant documents. Across four collections and annotation sources including human entity judgments, CER and OER exhibit near-chance agreement (

κ\approx 0

), while OER operationalizations agree substantially (

κ\approx 0.5

), confirming CER as the systematic outlier. CER-based supervision selects topically plausible but weakly discriminative entities, pruning fewer than 4% of non-relevant documents on some collections. Aligning supervision with OER improves non-relevant pruning by up to 10x and open-world MAP by 0.051 over BM25. Our findings motivate a shift from conceptual to observable notions of entity relevance in entity-aware retrieval.

Momentum

Related topics

Latest papers 122