IR Evaluation

IR: Information Retrieval

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 36% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 122

All topics

An Embarrassingly Simple Graph Heuristic Reveals Shortcut-Solvable Benchmarks for Sequential Recommendation
May 8, 2026Haoyu Han, Li Ma, Hanbing Wang +9Sequential Recommendation Benchmark Design
Michigan State University · Airbnb, Inc.
Superintelligent Retrieval Agent: The Next Frontier of Agentic Retrieval
May 7, 2026Zeyu Yang, Qi Ma, Jason Chen +1Agentic Retrieval Information Retrieval
Meta Superintelligence Labs · Rice University · Work done at Meta
OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries
May 7, 2026Diane Tchuindjo, Devavrat Shah, Omar KhattabInformation Retrieval IR Evaluation
Massachusetts Institute of Technology, Cambridge, MA
DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluation
May 6, 2026Bryan Li, William Walden, Yu Hou +6Language Model Generation Evaluation RAG Evaluation
Google Inc. · University of Pennsylvania · Johns Hopkins University +3
Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems
May 5, 2026Yilun Zhao, Jinbiao Wei, Tingyu Song +3Agentic Search Synthetic Data Generation
Yale NLP Lab · National University of Singapore · NYU Shanghai
Reproducing Complex Set-Compositional Information Retrieval
May 5, 2026Vincent Degenhart, Dewi Timman, Arjen P. de Vries +2Query Complexity Lexical Retrieval
Radboud University · Nijmegen, NL
Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction
May 3, 2026Zhuofeng Li, Haoxiang Zhang, Cong Wei +16Agentic Search IR Evaluation
Texas A&M University · University of Waterloo · UC San Diego +6
FollowTable: A Benchmark for Instruction-Following Table Retrieval
May 1, 2026Rihui Jin, Yuchen Lu, Ting Zhang +7Information Retrieval Table Retrieval
Southeast University Nanjing, China Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education · Huawei Noah’s Ark Lab Shenzhen, China
How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviews
Apr 30, 2026Riley Grossman, Songjiang Liu, Michael K. Chen +3Generative Engine Optimization Generative Retrieval
New Jersey Institute of Technology Newark, NJ, USA · Nanyang Technological University Singapore, Singapore · Indiana University Bloomington Bloomington, IN, USA
A Survey of Reasoning-Intensive Retrieval: Progress and Challenges
Apr 30, 2026Yiyang Wei, Tingyu Song, Siyue Zhang +1LLM Reranking IR Evaluation
Zhejiang University · University of the Chinese Academy of Sciences · Nanyang Technological University +1
A Reproducibility Study of LLM-Based Query Reformulation
Apr 30, 2026Amin Bigdeli, Radin Hamidi Rad, Hai Son Le +4LLM Evaluation LLM Prompting
University of Waterloo Waterloo, Ontario, Canada · Mila – Quebec AI Institute Montreal, Quebec, Canada · Toronto Metropolitan University Toronto, Ontario, Canada +2
Hypencoder Revisited: Reproducibility and Analysis of Non-Linear Scoring for First-Stage Retrieval
Apr 29, 2026Arne Eichholtz, Yongkang Li, Jutte Vijverberg +2Learning to Rank Hypernetworks
University of Amsterdam Amsterdam, The Netherlands
Benchmarking Complex Multimodal Document Processing Pipelines: A Unified Evaluation Framework for Enterprise AI
Apr 29, 2026Saurabh K. Singh, Sachin RajRAG Evaluation IR Evaluation
Oracle · Independent
Health System Scale Semantic Search Across Unstructured Clinical Notes
Apr 28, 2026Faith Wavinya Mutinda, Spandana Makeneni, Anna Lin +14Healthcare Semantic Search
1) Department of Biomedical and Health Informatics, Children’s Hospital of Philadelphia, Philadelphia, PA · 2) Google Cloud, Sunnyvale, CA · 3) Department of Pediatrics, University of Pennsylvania, Philadelphia, PA +2
Citation-Driven Multi-View Training for Patent Embeddings: QaECTER and Sophia-Bench
Apr 24, 2026Younes Djemmal, You Zuo, Kim Gerdes +1Legal IR Text Embedding Models
ALMAnaCH · 1AI Lab, Questel, Paris, France · 3Inria, Paris, France +3
A Large-Scale, Cross-Disciplinary Corpus of Systematic Reviews
Apr 23, 2026Pierre Achkar, Tim Gollub, Arno Simons +2Evidence Selection Document Information Extraction
Leipzig University; Fraunhofer ISI · Bauhaus-Universität Weimar · TU Berlin +2
Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation
Apr 22, 2026Andrew Klearman, Radu Revutchi, Rohin Garg +3RAG Evaluation IR Evaluation
Scale AI.
Semantic Recall for Vector Search
Apr 22, 2026Leonardo Kuffo, Ioanna Tsakalidou, Roberta De Viti +3IR Evaluation Approximate Nearest Neighbor Search
CWI · EPFL · MPI-SWS +1
RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora
Apr 21, 2026Hanjun Cho, Jay-Yoon LeeSynthetic Benchmark Generation Benchmark Validity
Allganize · Seoul National University
Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies
Apr 20, 2026Lorenz Brehme, Thomas Ströhle, Ruth BreuMulti-Hop QA LLM-as-a-Judge
Universität Innsbruck, Technikerstraße 21a, 6020 Innsbruck, Austria
TeleEmbedBench: A Multi-Corpus Embedding Benchmark for RAG in Telecommunications
Apr 20, 2026Pranshav Gajjar, Vijay K ShahText Embedding Evaluation Text Embedding Models
NextG Wireless Lab, North Carolina State University, Raleigh, USA
SeekerGym: A Benchmark for Reliable Information Seeking
Apr 18, 2026Remy Kim, Minseung Lee, Shuo Li +1AI Agent Reliability Agentic Search
University of Pennsylvania, Curation Labs · Curation Labs · Google DeepMind +1
JFinTEB: Japanese Financial Text Embedding Benchmark
Apr 17, 2026Masahiro Suzuki, Hiroki SakajiText Embedding Evaluation Text Embedding Models
Amova Asset Management Co., Ltd. · Tokyo, Japan · Hokkaido University +1
UsefulBench: Towards Decision-Useful Information as a Target for Information Retrieval
Apr 17, 2026Tobias Schimanski, Stefanie Lewandowski, Christian Woerle +3Information Retrieval IR Evaluation
University of Zurich · ETH Zurich · score4more GmbH +2
PeopleSearchBench: Evaluating AI-Powered People Search Platforms with Criteria-Grounded Verification
Mar 29, 2026Tianyu Shi, Wei Wang, Zequn Xie +10Web Search Agents Information Retrieval
LMEB: Long-horizon Memory Embedding Benchmark
Mar 13, 2026Xinping Zhao, Xinshuo Hu, Jiaxin Xu +9Text Embedding Evaluation IR Evaluation
Harbin Institute of Technology (Shenzhen) · Shenzhen Loop Area Institute (SLAI) · Peking University
Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools
Feb 20, 2026Baris Arat, Emre SeferLLM Evaluation LLM Reranking
Ozyegin University Istanbul, Turkey
SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noise
Feb 13, 2026Yuejie Li, Ke Yang, Yueying Hua +4Noise Robustness IR Evaluation
Ant Group Hangzhou, China · Soochow University Suzhou, China · University of Science and Technology of China Hefei, China +3
The $\mathbf{P}$ -Completeness of Inverted Index Traversal: On the Complexity of Evaluating Boolean Query DAGs
Jan 26, 2026Amir AavaniQuery Complexity Information Retrieval
Apple Inc., USA
WebArxiv: A Reproducible Benchmark for Evaluating Multimodal Web Agents on arXiv Tasks
Jul 1, 2025Zihao Sun, Zijing Shi, Ling ChenWeb Agent Benchmarks Multimodal Agents
University of Technology Sydney, Australia

May 8, 2026, cs.IRJ/K move · Enter open · S save

An Embarrassingly Simple Graph Heuristic Reveals Shortcut-Solvable Benchmarks for Sequential Recommendation

Haoyu Han, Li Ma, Hanbing Wang, Bingheng Li+8

Michigan State University · Airbnb, Inc.

Sequential Recommendation Benchmark Design IR Evaluation Shortcut Learning

Open paper AskPDF arXiv

Sequential recommendation has increasingly shifted toward generative recommenders that combine sequential patterns with semantic item information. Yet these methods are often evaluated on a small set of widely used benchmarks, raising a key question: do these benchmarks actually require the advanced modeling capabilities that modern generative recommenders claim to provide? We conduct a benchmark audit with an intentionally simple graph heuristic. Starting from only the last one or two interacted items, it retrieves candidates from a few-hop item-transition graph and ranks them by item-feature similarity. Despite using no sequence encoder, generative objective, or training, this heuristic matches or outperforms many modern baselines, with relative NDCG@10 improvements of 38.10% and 44.18% over the best competing baseline on Amazon Review Sports and CDs. We show that this behavior reflects shortcut solvability rather than an artifact of one heuristic. We identify three shortcut structures that can make next-item prediction easier than expected: low-branching local transitions, feature-smooth transitions, and limited dependence on long user histories. These shortcuts need not appear together; even one or two strong signals can make simple local retrieval highly competitive, while weakening them makes the benefits of more sophisticated models clearer. Across 14 datasets, model rankings vary substantially with dataset properties, yet the heuristic remains competitive on 10 of them. Our findings suggest that strong performance on standard benchmarks does not always demonstrate advanced sequential, semantic, or generative modeling ability. We call for more careful dataset selection and dataset-level diagnostic analysis when using benchmarks to support claims about new recommendation models.

Momentum

Related topics

Latest papers 122