Information Retrieval

Also known as IR

Follow this topic Ask about this topic

Information retrieval develops methods for finding useful documents, passages, products, tables, code, and multimedia in response to a user's query. Systems range from lexical methods such as BM25 and learned sparse retrieval to dense embeddings, late-interaction models, hybrid ranking, and graph-based or agentic search. These methods support applications including web search, recommendation, and retrieval-augmented generation, where retrieved evidence must be both relevant and sufficient for the task. A central challenge is balancing accuracy with speed and cost: semantic similarity can miss structured, compositional, or domain-specific evidence, and strong retrieval scores do not always lead to useful answers.

Latest papers

476 papers61 papers in the last four weeks, up 36% on the four weeks before. 0.4% of all new papers.

Feb 16, 2026

Intent-Driven Dynamic Chunking: Segmenting Documents to Reflect Predicted Information Needs

Breaking long documents into smaller segments is a fundamental challenge in information retrieval. Whether for search engines, question-answering systems, or retrieval-augmented generation (RAG), effective segmentation determines how well systems can locate and return relevant information. However, traditional methods, such as fixed-length or coherence-based segmentation, ignore user intent, leading to chunks that split answers or contain irrelevant noise. We introduce Intent-Driven Dynamic Chunking (IDC), a novel approach that uses predicted user queries to guide document segmentation. IDC leverages a Large Language Model to generate likely user intents for a document and then employs a dynamic programming algorithm to find the globally optimal chunk boundaries. This represents a novel application of DP to intent-aware segmentation that avoids greedy pitfalls. We evaluated IDC on six diverse question-answering datasets, including news articles, Wikipedia, academic papers, and technical documentation. IDC outperformed traditional chunking strategies on five datasets, improving top-1 retrieval accuracy by 5% to 67%, and matched the best baseline on the sixth. Additionally, IDC produced 40-60% fewer chunks than baseline methods while achieving 93-100% answer coverage. These results demonstrate that aligning document structure with anticipated information needs significantly boosts retrieval performance, particularly for long and heterogeneous documents.

Christos Koutsiaris

Dept. of Computer Science & Information Systems University of Limerick, Ireland · Cloud ERP - UX Foundation SAP P&E

Document Chunking Information Retrieval

Feb 16, 2026

BETA-Labeling for Multilingual Dataset Construction in Low-Resource IR

IR in low-resource languages remains limited by the scarcity of high-quality, task-specific annotated datasets. Manual annotation is expensive and difficult to scale, while using large language models (LLMs) as automated annotators introduces concerns about label reliability, bias, and evaluation validity. This work presents a Bangla IR dataset constructed using a BETA-labeling framework involving multiple LLM annotators from diverse model families. The framework incorporates contextual alignment, consistency checks, and majority agreement, followed by human evaluation to verify label quality. Beyond dataset creation, we examine whether IR datasets from other low-resource languages can be effectively reused through one-hop machine translation. Using LLM-based translation across multiple language pairs, we experimented on meaning preservation and task validity between source and translated datasets. Our experiment reveal substantial variation across languages, reflecting language-dependent biases and inconsistent semantic preservation that directly affect the reliability of cross-lingual dataset reuse. Overall, this study highlights both the potential and limitations of LLM-assisted dataset creation for low-resource IR. It provides empirical evidence of the risks associated with cross-lingual dataset reuse and offers practical guidance for constructing more reliable benchmarks and evaluation pipelines in low-resource language settings.

Md. Najib Hasan, Mst. Jannatun Ferdous Rain, Fyad Mohammed +1

Wichita State University · Begum Rokeya University, Rangpur · Khulna University of Engineering & Technology +1

Low-Resource Language Processing LLM-Assisted Annotation Information Retrieval Cross-Lingual Transfer

Feb 16, 2026

High Precision Audience Expansion via Extreme Classification in a Two-Sided Marketplace

Airbnb search must balance a worldwide, highly varied supply of homes with guests whose location, amenity, style, and price expectations differ widely. Meeting those expectations hinges on an efficient retrieval stage that surfaces only the listings a guest might realistically book, before resource intensive ranking models are applied to determine the best results. Unlike many recommendation engines, our system faces a distinctive challenge, location retrieval, that sits upstream of ranking and determines which geographic areas are queried in order to filter inventory to a candidate set. The preexisting approach employs a deep bayesian bandit based system to predict a rectangular retrieval bounds area that can be used for filtering. The purpose of this paper is to demonstrate the methodology, challenges, and impact of rearchitecting search to retrieve from the subset of most bookable high precision rectangular map cells defined by dividing the world into 25M uniform cells.

Dillon Davis, Huiji Gao, Thomas Legrand +8

Relevance Airbnb, Inc. San Francisco, CA, USA · Search Airbnb, Inc. San Francisco, CA, USA

E-Commerce Search Information Retrieval

Feb 11, 2026

Cross-Sectional Asset Retrieval via Future-Aligned Soft Contrastive Learning

Asset retrieval (finding similar assets in a financial universe) is central to quantitative investment decision-making. Existing approaches define similarity through historical price patterns or sector classifications, but such backward-looking criteria provide no guarantee about future behavior. We argue that effective asset retrieval should be future-aligned: the retrieved assets should be those most likely to exhibit correlated future returns. To this end, we propose Future-Aligned Soft Contrastive Learning (FASCL), a representation learning framework whose soft contrastive loss uses pairwise future return correlations as continuous supervision targets. We further introduce an evaluation protocol designed to directly assess whether retrieved assets share similar future trajectories. Experiments on 5,631 US-listed securities against 14 baselines show that FASCL attains the best future return correlation at every retrieval depth and the best rank information coefficient at every depth and horizon, leads on trend consistency in 13 of 16 cells, and gives the highest gross Sharpe ratio in a spread trading backtest at every basket size. Code is available at https://github.com/HyeongminLEE/fascl .

Hyeongmin Lee, Chanyeol Choi, Jihoon Kwon +8

Seoul National University of Science and Technology Seoul, Republic of Korea · LinqAlpha Cambridge, MA, United States · Massachusetts Institute of Technology Cambridge, MA, United States +7

Contrastive Learning Information Retrieval Predictive Representation Learning Financial Time Series Finance Algorithmic Trading

Feb 10, 2026

ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge

Existing multimodal retrieval benchmarks largely emphasize semantic matching on daily-life images and offer limited diagnostics of professional knowledge and complex reasoning. To address this gap, we introduce ARK, a benchmark designed to analyze multimodal retrieval from two complementary perspectives: (i) knowledge domains (five domains with 17 subtypes), which characterize the content and expertise retrieval relies on, and (ii) reasoning skills (six categories), which characterize the type of inference over multimodal evidence required to identify the correct candidate. Specifically, ARK evaluates retrieval with both unimodal and multimodal queries and candidates, covering 16 heterogeneous visual data types. To avoid shortcut matching during evaluation, most queries are paired with targeted hard negatives that require multi-step reasoning. We evaluate 25 representative text-based and multimodal retrievers and observe a pronounced gap between knowledge- and reasoning-intensive retrieval, with fine-grained visual and spatial reasoning as persistent bottlenecks. We further show that enhancements such as re-ranking, rewriting, and agentic retrieval yield consistent gains, but substantial headroom remains.

Yijie Lin, Guofeng Ding, Haochen Zhou +3

College of Computer Science, Sichuan University, Chengdu, China · National Key Laboratory of Fundamental Algorithms and Models for Engineering Simulation, Sichuan University, China

Multimodal IR Visual Spatial Reasoning Multimodal Reasoning Information Retrieval Multimodal Model Evaluation

Feb 10, 2026

With Argus Eyes: Assessing Retrieval Gaps via Uncertainty Scoring to Detect and Remedy Retrieval Blind Spots

Reliable retrieval-augmented generation (RAG) systems depend fundamentally on the retriever's ability to find relevant information. We show that neural retrievers used in RAG systems have blind spots, which we define as the failure to retrieve entities that are relevant to the query, but have low similarity to the query embedding. We investigate the training-induced biases that cause such blind spot entities to be mapped to inaccessible parts of the embedding space, resulting in low retrievability. Using a large-scale dataset constructed from Wikidata relations and first paragraphs of Wikipedia, and our proposed Retrieval Probability Score (RPS), we show that blind spot risk in standard retrievers (e.g., CONTRIEVER, REASONIR) can be predicted pre-index from entity embedding geometry, avoiding expensive retrieval evaluations. To address these blind spots, we introduce ARGUS, a pipeline that enables the retrievability of high-risk (low-RPS) entities through targeted document augmentation from a knowledge base (KB), first paragraphs of Wikipedia, in our case. Extensive experiments on BRIGHT, IMPLIRET, and RAR-B show that ARGUS achieves consistent improvements across all evaluated retrievers (averaging +3.4 nDCG@5 and +4.5 nDCG@10 absolute points), with substantially larger gains in challenging subsets. These results establish that preemptively remedying blind spots is critical for building robust and trustworthy RAG systems.

Zeinab Sadat Taghavi, Ali Modarressi, Hinrich Schutze +1

Lucerne University of Applied Sciences and Arts (HSLU) · Center for Information and Language Processing (CIS), Ludwig Maximilian University of Munich (LMU) · Munich Center for Machine Learning (MCML)

Retrieval-Augmented Generation Retrieval Robustness Information Retrieval Dense Retrieval

Feb 5, 2026

Impact of AI Search Summaries on Website Traffic: Evidence from Google AI Overviews and Wikipedia

Search engines increasingly display AI-generated answers above organic links, potentially displacing traffic to upstream publishers. We estimate the impact of Google's AI Overviews (AIO) on Wikipedia's search traffic using AIO's staggered geographic rollout and Wikipedia's multilingual structure. Our difference-in-differences design compares monthly external-search referrals to English Wikipedia articles with referrals to the same articles in German and French, and finds that default AIO availability reduced English search traffic by 5.45% and 4.82%, respectively. Our results suggest that answer-producing digital intermediaries can materially reallocate attention away from informational publishers, with implications for content monetization, search platform design, and policy.

Mehrzad Khosravi, Hema Yoganarasimhan

University of Washington

Information Retrieval

Jan 29, 2026

ShardMemo: Scope-Before-Routing for Agentic Memory Retrieval

Agentic systems accumulate persistent memory across sessions, tools, and tasks, and a later request must retrieve from it under two distinct constraints: which memories it is permitted to access, and which are relevant under a limited search budget. Existing memory systems handle access scope in two flawed ways: applying scope after retrieval wastes probe budget on inadmissible memories, while treating scope as a learned ranking feature makes a hard constraint depend on router quality. We present SHARD-MEMO, an agentic memory system built on scope-before-routing: metadata predicates first identify the admissible shards, and a learned router then selects a small number of them for shard-local approximate nearest neighbor retrieval. Given the supplied scope predicate and metadata, this separates hard admissibility from learned relevance ranking, so inadmissible shards cannot consume shard-probe budget. We evaluate on LoCoMo, HotpotQA, and ToolBench, covering conversational, long-context, and procedural memory. Under matched supervision and fixed budgets, SHARDMEMO improves over a learned router baseline by roughly +3 F1 on LoCoMo; in end-to-end LoCoMo evaluation it improves over the strongest evaluated memory baseline by up to +6.8 F1, with gains on HotpotQA and ToolBench.

Yang Zhao, Chengxiao Dai, Mengying Kou +2

Anonymous

Agentic Retrieval Information Retrieval Agent Memory Management

Jan 26, 2026

The $\mathbf{P}$ -Completeness of Inverted Index Traversal: On the Complexity of Evaluating Boolean Query DAGs

Modern AI agents increasingly rely on search infrastructure to execute complex, neuro-symbolic reasoning workflows. These workflows often compile into deeply nested, non-monotonic Boolean queries over text fields. However, standard query evaluation strategies over inverted indices face severe theoretical limits when handling these structures. Stateful iterator models (Document-at-a-Time) are structurally bounded by

\text{NC}^1

formula evaluation, suffering a worst-case

O(2^{|Q|})

exponential blowup in query complexity when unrolling re-convergent logic. Conversely, recursive materialization models (Term-at-a-Time) incur an

Ω(|U|)

space complexity penalty (the Universal Scan) when evaluating logical negation over the document universe. In this paper, we establish the theoretical boundaries of executing complex logic natively over an inverted index. We formalize a retrieval language (

\mathcal{L}_R

) based on Directed Acyclic Graphs (DAGs) and prove that its evaluation problem is strictly \textbf{

\mathbf{P}

-Complete}. To make evaluation tractable, we introduce ComputePN, a deterministic, sparsity-aware evaluation algorithm. By decoupling logical negation from universe-scale materialization via a novel Positive-Negative dual representation, and utilizing native DAG memoization, ComputePN strictly bounds evaluation time to

O(|Q| \cdot |U_{\mathit{active}}|)

. This approach successfully evaluates

\mathbf{P}

-Complete queries natively over the index, avoiding both the combinatorial tree-expansion bottleneck and the universal scan penalty, laying the formal foundation for computational retrieval.

Amir Aavani

Apple Inc., USA

Query Complexity Information Retrieval IR Evaluation

Jan 20, 2026

NewsRECON: News Article Retrieval for Image Contextualization

Identifying when and where a news image was taken is crucial for journalists and forensic experts to produce credible stories and debunk misinformation. While many existing methods rely on reverse image search (RIS) engines, these tools often fail to return results, thereby limiting their practical applicability. In this work, we address the challenging scenario where RIS evidence is unavailable. We investigate the potential of news article corpora as an alternative to RIS, linking images to relevant articles to infer their dates and locations from article metadata. We evaluate the performance of a news article retrieval pipeline, NewsRECON, which leverages a corpus of over 85,000 articles. Experiments on the TARA dataset show that NewsRECON outperforms prior work and can be combined with a multimodal large language model (MLLM) to achieve new SOTA results in the absence of RIS evidence. Furthermore, NewsRECON generalizes to the 5Pils-OOC benchmark despite geographic and temporal shifts. We make our code available.

Jonathan Tonglet, Iryna Gurevych, Tinne Tuytelaars +1

Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE · Department of Electrical Engineering, KU Leuven · Department of Computer Science, KU Leuven

Multimodal IR Digital Image Forensics Information Retrieval

Jan 19, 2026

CORE-T: COherent REtrieval of Tables for Text-to-SQL

Realistic text-to-SQL workflows often require joining multiple tables. As a result, accurately retrieving the relevant set of tables becomes a key bottleneck for end-to-end performance. We study an open-book setting where queries must be answered over large, heterogeneous table collections pooled from many sources, without clean scoping signals such as database identifiers. Here, dense retrieval (DR) achieves high recall but returns many distractors, while join-aware alternatives often rely on extra assumptions and/or incur high inference overhead. We propose CORE-T, a scalable, training-free framework that enriches tables with LLM-generated purpose metadata and pre-computes a lightweight table-compatibility cache. At inference time, DR returns top-K candidates; a single LLM call selects a coherent, joinable subset, and a two-step additive adjustment stage restores strongly compatible tables. Across Bird, Spider, MMQA, and Beaver, CORE-T improves over DR by up to 22.7 points in table-selection F1 while returning up to 40% fewer tables, and by up to 24.4 points in multi-table execution accuracy, and uses 1.64-4.20x fewer total selection tokens than LLM-intensive baselines.

Hassan Soliman, Vivek Gupta, Dan Roth +1

Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany · Arizona State University · University of Pennsylvania

Text-to-SQL Information Retrieval Table Retrieval

Jan 15, 2026

PaperScout: An Autonomous Agent for Academic Paper Search with Process-Aware Sequence-Level Policy Optimization

Academic paper search is a fundamental task in scientific research, yet most existing approaches organize retrieval around predefined workflows or structured interaction protocols that struggle with complex, conditional queries. To address this limitation, we propose PaperScout, an autonomous agent that reformulates paper search as a sequential decision-making process. Unlike static workflows, PaperScout dynamically decides whether, when, and how to invoke search and expand tools based on accumulated retrieval context. However, training such agents presents a fundamental challenge: standard reinforcement learning methods, typically designed for single-turn tasks, suffer from a granularity mismatch when applied to multi-turn agentic tasks, where token-level optimization diverges from the granularity of sequence-level interactions, leading to noisy credit assignment and unstable training dynamics. We introduce Proximal Sequence Policy Optimization (PSPO), a process-aware, sequence-level policy optimization method that aligns optimization with agent--environment interaction. Comprehensive experiments on both synthetic and real-world benchmarks demonstrate that PaperScout significantly outperforms strong structured retrieval and RL baselines in both recall and relevance, validating the effectiveness of our adaptive agentic framework and optimization strategy.

Tingyue Pan, Jie Ouyang, Mingyue Cheng +7

State Key Lab of Cognitive Intelligence, University of Science and Technology of China

Agentic RL Policy Optimization Information Retrieval Autonomous Research Agents RL for Tool Use

Dec 14, 2025

MTRACE: Multilingual Retrieval-Augmented Generation for Temporally Diverse Text Corpora

Large multilingual knowledge bases expose temporally diverse information, yet retrieval quality remains sensitive to lexical variation and cross-lingual terminology shifts. We develop and evaluate MTRACE (Multilingual Temporal Retrieval-Augmented Generation with evidence grounding), a pipeline designed to test whether query expansion and multi-query fusion mitigate vocabulary mismatch in temporally layered text corpora, on the French and English subsets of MIRACL. Our approach integrates: (i) semantic query expansion (SQE) and multi-query fusion via Reciprocal Rank Fusion (RRF), targeting retrieval stability under query variation; (ii) a generation prompt enforcing strict grounding in retrieved evidence and explicit abstention when evidence is insufficient; and (iii) a modular architecture enabling systematic component evaluation. Ablation studies on Named Entity Recognition (NER) and embedding model selection demonstrate the importance of syntactic coherence in entity extraction and of self-retrieval and efficiency measurements for retriever selection. Our end-to-end evaluation over 50 constructed queries shows faithful answers for well-supported queries, correct abstention on unanswerable questions, and no re-scored similarity gains from multi-query fusion over single-query dense retrieval. By scoping our claims to a clean, text-only baseline, we separate these effects from OCR-noise confounds; direct measurement of diachronic lexical drift is left to future work. Code and configurations are available at \url{https://anonymous.4open.science/r/MIRAGE-8EAA/

Souhail Bakkali, Anthony Mudet

Univ Rennes, CNRS, IRISA - UMR 6074, Rennes, France · L3i-lab, La Rochelle Universit´e, France

Retrieval-Augmented Generation Multilingual IR Information Retrieval RAG Evaluation Query Expansion

Nov 23, 2025

From Crawled to Published: A Publisher-Controlled Web Architecture for an AI-Native Internet

AI agents, not humans, are becoming the web's primary consumers, yet the web still ships content structured for human browsing, with no native way for publishers to expose the semantic units within it. Access is therefore mediated by a handful of AI providers that crawl, chunk, and index content unilaterally, leaving publishers little control over how their content is represented, retrieved, or attributed. We argue this is a structural problem, not one better chunking or retrieval algorithms can solve. The web lacks a native layer for publisher-controlled semantic discovery and retrieval. We propose AISTRAL, an AI-Native Internet architecture that supplies this layer. Publishers expose semantically meaningful units (e.g., passages, claims, tables) through explicit, policy-governed interfaces they control, and a web-native semantic resolver, analogous to DNS but operating over meaning rather than location, lets agents discover relevant sources before retrieving fine-grained content directly from publishers. Our end-to-end prototype matches or exceeds representative centralized retrieval pipelines in answer quality, with up to 4.2x higher accuracy on large, specialized document collections. Against a public web-search agent, it answers up to 4.2x more questions correctly, while moving 2.9-41x less data per query. Because retrieval terminates at publisher-operated servers, attribution, provenance, and usage policies travel with the content, realigning incentives across the web ecosystem.

Yixi Chen, Muhammad Bilal, Tie Ma +3

Information Retrieval Agentic Retrieval Web Agents

Nov 18, 2025

PRISM: Prompt-Refined In-Context System Modelling for Financial Retrieval

Deploying LLMs for financial information retrieval demands accuracy, cost efficiency, and reproducibility, yet most approaches require expensive fine-tuning. PRISM is a training-free framework that composes three deployable modules (prompt engineering, in-context learning, and a candidate refinement module) with a boundary-condition study of multi-agent coordination for financial document retrieval. On FiQA-2018, stacked reranking and

k

-reciprocal encoding raise Recall@100 from 0.7783 to 0.8326. On FinAgentBench, a Cohere-refined top-50 pool yields a higher observed end-to-end NDCG@5, below this evaluation's minimum detectable effect, while cutting per-query cost by about 35%. Added complexity does not always pay off:

k

-reciprocal encoding stops helping once the upstream stage is already high-recall, and multi-agent coordination is not a default for fine-grained chunk ranking. Matched pairs that hold prompt, model, and exemplars fixed lose 0.0702 and 0.0681 combined NDCG@5 to their single-path controls, and the whole gap sits at the chunk stage. We release an expert-annotated FinAgentBench evaluation set, together with a label-robustness sweep over 16 alternative ground-truth constructions, restoring a reproducible evaluation path after the original evaluator went offline. Without fine-tuning, PRISM placed third on the official FinAgentBench private leaderboard (leaderboard basis, NDCG@5 0.71181), reaches NDCG@10 0.6197 on FiQA-2018, and 98% accuracy on FinanceBench. We report per-query latency, token, and dollar accounting across roughly 40 configurations, present the quality-cost-latency Pareto frontier, and introduce FACET, a transparent SLA-specific scalarization for choosing among non-dominated configurations. Code and dataset are released at https://bit.ly/prism-ailens.

Chun Chet Ng, Jia Yu Lim, Zhen Hao Chu +2

AI Lens, Kuala Lumpur, Malaysia

Information Retrieval LLM Reranking Financial Language Model Evaluation In-Context Learning Financial Reasoning Benchmarks

Oct 20, 2025

Rethinking On-policy Optimization for Query Augmentation

Recent advances in large language models (LLMs) have led to a surge of interest in query augmentation for information retrieval (IR). Two main approaches have emerged. The first prompts LLMs to generate answers or pseudo-documents that serve as new queries, relying purely on the model's parametric knowledge or contextual information. The second applies reinforcement learning (RL) to fine-tune LLMs for query rewriting, directly optimizing retrieval metrics. While having respective advantages and limitations, the two approaches have not been compared under consistent experimental conditions. In this work, we present the first systematic comparison of prompting-based and RL-based query augmentation across diverse benchmarks, including evidence-seeking, ad hoc, and tool retrieval. Our key finding is that under a compute-aware comparison setting, simple, training-free query augmentation often performs on par with, or even surpasses, more expensive RL-based counterparts, especially when using powerful LLMs. Motivated by this discovery, we introduce a novel hybrid method, On-policy Pseudo-document Query Expansion (OPQE), in which the LLM policy learns to generate a pseudo-document that maximizes retrieval performance, rather than rewriting the query, thus merging the flexibility and generative structure of prompting with the targeted optimization of RL. We show OPQE outperforms both standalone prompting and RL-based rewriting, demonstrating that a synergistic approach yields the best results. We open source our implementation to facilitate reproducibility.

Zhichao Xu, Shengyao Zhuang, Xueguang Ma +6

University of Utah · 2The University of Queensland · University of Waterloo +5

RL for Language Models LLM Prompting Information Retrieval Query Expansion

Oct 9, 2025

TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance

Query-product relevance prediction is fundamental to e-commerce search and has become even more critical in the era of AI-powered shopping, where semantic understanding and complex reasoning directly shape the user experience and business conversion. Large Language Models (LLMs) enable generative, reasoning-based approaches, typically aligned via supervised fine-tuning (SFT) or preference optimization methods like Direct Preference Optimization (DPO). However, the increasing complexity of business rules and user queries exposes the inability of existing methods to endow models with robust reasoning capacity for long-tail and challenging cases. Efforts to address this via reinforcement learning strategies like Group Relative Policy Optimization (GRPO) often suffer from sparse terminal rewards, offering insufficient guidance for multi-step reasoning and slowing convergence. To address these challenges, we propose TaoSR-AGRL, an Adaptive Guided Reinforcement Learning framework for LLM-based relevance prediction in Taobao Search Relevance. TaoSR-AGRL introduces two key innovations: (1) Rule-aware Reward Shaping, which decomposes the final relevance judgment into dense, structured rewards aligned with domain-specific relevance criteria; and (2) Adaptive Guided Replay, which identifies low-accuracy rollouts during training and injects targeted ground-truth guidance to steer the policy away from stagnant, rule-violating reasoning patterns toward compliant trajectories. TaoSR-AGRL was evaluated on large-scale real-world datasets and through online side-by-side human evaluations on Taobao Search. It consistently outperforms DPO and standard GRPO baselines in offline experiments, improving relevance accuracy, rule adherence, and training stability. The model trained with TaoSR-AGRL has been successfully deployed in the main search scenario on Taobao, serving hundreds of millions of users.

Jianhui Yang, Yiming Jin, Pengkun Jiao +6

Tsinghua University Beijing, China · Taobao & Tmall Group of Alibaba Beijing, China · Taobao & Tmall Group of Alibaba Hangzhou, China +1

RL for Language Models Reward Shaping Learning to Rank E-Commerce Search Information Retrieval

Aug 25, 2025

How Do LLM-Generated Texts Impact Term-Based Retrieval Models?

As more content generated by large language models (LLMs) floods into the Internet, information retrieval (IR) systems now face the challenge of distinguishing and handling a blend of human-authored and machine-generated texts. Recent studies suggest that neural retrievers may exhibit a preferential inclination toward LLM-generated content, while classic term-based retrievers like BM25 tend to favor human-written documents. This paper investigates the influence of LLM-generated content on term-based retrieval models, which are valued for their efficiency and robust generalization across domains. Our linguistic analysis reveals that LLM-generated texts exhibit smoother high-frequency and steeper low-frequency Zipf slopes, higher term specificity, and greater document-level diversity. These traits are aligned with LLMs being trained to optimize reader experience through diverse and precise expressions. Our study further explores whether term-based retrieval models demonstrate source bias, concluding that these models prioritize documents whose term distributions closely correspond to those of the queries, rather than displaying an inherent source bias. This work provides a foundation for understanding and addressing potential biases in term-based IR systems managing mixed-source content.

Wei Huang, Keping Bi, Yinqiong Cai +3

State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China · Baidu Inc., Beijing, China

Information Retrieval Lexical Retrieval

Aug 3, 2025

ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings

Retrieval-Augmented Generation (RAG) systems in chemistry heavily depend on accurate and relevant retrieval of chemical literature. However, general-purpose text embedding models frequently fail to adequately represent complex chemical terminologies, resulting in suboptimal retrieval quality. Existing embedding models for chemistry are outdated, and none is tailored to chemical literature retrieval, leaving a substantial performance gap. To address this challenge, we introduce ChEmbed, the first purpose-built family of domain-adapted text embedding models engineered for chemical literature retrieval. These models are fine-tuned via contrastive learning on a dataset comprising chemistry-specific text from the PubChem, Semantic Scholar, and ChemRxiv corpora. To create effective training data, we employ large language models to synthetically generate queries, resulting in approximately 1.7 million high-quality query-passage pairs. Additionally, we augment the tokenizer by adding 900 chemically specialized tokens to previously unused slots, which reduces the fragmentation of chemical entities, such as IUPAC names. ChEmbed also maintains an 8192-token context length, enabling retrieval of longer passages than many open-source embedding models allow. Evaluated on our newly introduced ChemRxiv Retrieval benchmark, ChEmbed outperforms state-of-the-art general embedding models, raising MRR@10 from 0.781 to 0.882 (+10.1 pp). It also substantially outperforms domain-specific embedding models such as Chemical-BERT, improving MRR@10 from 0.096 to 0.882. A role-based retrieval analysis using PubChem descriptions and ChEBI annotations shows that the improvement extends to chemical-role queries. ChEmbed represents a practical, lightweight, and reproducible embedding solution that effectively improves chemical literature retrieval.

Ali Shiraee Kasmaee, Mohammad Khodadad, Mahdi Astaraki +4

Department of Computational Science and Engineering, McMaster University, Canada · BASF Canada Inc., Canada · BASF Corporation, USA

Text Embedding Evaluation Information Retrieval Text Embedding Models

Aug 2, 2025

Open-Attribute Person Retrieval: Finding People Through Distinctive and Novel Attributes

Person retrieval in surveillance videos often depends on attributes described by witnesses or operators. However, the most useful cues in practice are not always common appearance descriptions (e.g., gender, clothing color), but rare and distinctive attributes that can sharply reduce the search space (e.g., holding a weapon, lying on the ground). Existing text-based person retrieval benchmarks and methods largely focus on identity-centric retrieval with common pedestrian descriptions, leaving such retrieval-critical attributes underexplored. In this paper, we introduce Open-Attribute Person Retrieval (OAPR), a practical retrieval setting that aims to retrieve all pedestrian instances matching a given attribute query, including rare or previously unseen visual concepts, regardless of identity. To support this task, we construct EPAD, an Expanded Pedestrian Attribute Dataset with 267,885 pedestrian images and a unified vocabulary of 65 attributes, including safety-critical actions, assistive devices, and object interactions that are rarely covered in prior benchmarks. We further propose GAP-CLIP, a lightweight CLIP-based framework that learns gated attribute-aware body-part representations for OAPR. Extensive experiments on EPAD demonstrate that GAP-CLIP achieves the strongest top-K retrieval performance on the full attribute space and on out-of-distribution attributes. The code and dataset are available at https://github.com/mlnjeongpark/Open-Attribute-Person-Retrieval.

Minjeong Park, Hongbeen Park, Sangwon Lee +1

The Pennsylvania State University · Korea Telecom Research · Korea University +1

Information Retrieval Image Retrieval

Jul 14, 2025

Chain of Retrieval: Multi-Aspect Iterative Search Expansion and Post-Order Search Aggregation for Full Paper Retrieval

Scientific paper retrieval, particularly framed as document-to-document retrieval, aims to identify relevant papers in response to a long-form query paper, rather than a short query string. Previous approaches to this task have largely focused on abstracts, embedding them into dense vectors as surrogates for full documents and calculating similarity between them. Yet, abstracts offer only sparse and high-level summaries, and such methods primarily optimize one-to-one similarity, overlooking the dynamic relations that emerge across relevant papers during the retrieval process. To address this, we propose Chain of Retrieval (CoR), a novel iterative framework for full-paper retrieval. Specifically, CoR decomposes each query paper into multiple aspect-specific views, matches them against segmented candidate papers, and iteratively expands the search by promoting top-ranked results as new queries, thereby forming a tree-structured retrieval process. The resulting retrieval tree is then aggregated in a post-order manner: descendants are first combined at the query level, then recursively merged with their parent nodes, to capture hierarchical relations across iterations. To validate this, we present SciFullBench, a large-scale benchmark providing both complete and segmented contexts of full papers for queries and candidates, and results show that CoR significantly outperforms existing retrieval baselines. Our code and dataset is available at https://github.com/psw0021/Chain-of-Retrieval-Official.

Sangwoo Park, Jinheon Baek, Soyeong Jeong +1

KAIST

Scientific Document Retrieval Information Retrieval Query Expansion

Jun 13, 2025

Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation

The retrieval stage of retrieval-augmented generation (RAG) for scientific question answering depends on how documents are segmented and how chunks are represented in embedding space. This dependence is especially relevant to chemistry texts, which contain dense terminology, symbolic notation, quantitative evidence, and context associated with document structure. However, benchmark-based evidence on the interaction between chunking strategy and embedding model remains limited for chemistry-specific retrieval. Using ChemQuests, a corpus of 952 question-answer pairs from 151 ChemRxiv papers across 17 chemistry subfields, we construct chunk-level, Massive Text Embedding Benchmark (MTEB)-compatible retrieval benchmarks for controlled evaluation. We first screen 41 embedding models on the external chemistry retrieval benchmarks ChemNQRetrieval and ChemHotpotQARetrieval using a geometric-mean metric at rank 10 (Geom@10), which we validate against the full retrieval-metric profile. We then evaluate shortlisted models on ChemQuests-derived tasks across five chunking strategies, seven chunk sizes, and multiple overlap settings. Embedding choice is associated with the largest observed differences in evidence retrieval, with retrieval-tuned E5, Beijing Academy of Artificial Intelligence General Embedding (BGE), and Nomic models among the strongest overall. Within the evaluated grid, medium-to-large chunks combined with fixed-token, recursive-token, or hierarchical-section chunking provide a practical starting point for the retrieval stage of chemistry-aware RAG. Low overlap was generally favored where overlap variation was evaluated.

Mahmoud Amiri, Thomas Bocklitz

Leibniz Institute of Photonic Technology, Albert-Einstein-Strasse 9, 07745 Jena, Germany · Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich Schiller University Jena, Helmholtzweg 4, 07743 Jena, Germany

Document Chunking Text Embedding Evaluation Information Retrieval RAG Evaluation Evidence Retrieval

Feb 3, 2025

Query Brand Entity Linking in E-Commerce Search

Associating user search queries with the correct brand entity is critical for e-commerce product retrieval, yet remains challenging due to the brevity of queries (three to four words on average), their lack of grammatical structure, and a catalog of hundreds of thousands of distinct brands. We formulate this as a brand entity linking task and develop two complementary solutions deployed at scale: (1) a cascaded pipeline that first detects brand mentions via sequence labeling and then disambiguates against a brand knowledge base, and (2) a single-stage approach that frames linking as extreme multiclass classification, directly mapping queries to brand identifiers. Through extensive multilingual evaluation (11 languages) and a controlled online experiment, we demonstrate that the proposed methods substantially improve brand recall while maintaining high precision, leading to measurable gains in customer engagement.

Dong Liu, Sreyashi Nag

Amazon Luxembourg, Luxembourg · Amazon Palo Alto, California, United States

E-Commerce Search Information Retrieval Entity Linking E-Commerce

Feb 1, 2024

HiQA: A Hierarchical Contextual Augmentation RAG for Multi-Documents QA

Retrieval-augmented generation (RAG) has rapidly advanced the language model field, particularly in question-answering (QA) systems. By integrating external documents during the response generation phase, RAG significantly enhances the accuracy and reliability of language models. This method elevates the quality of responses and reduces the frequency of hallucinations, where the model generates incorrect or misleading information. However, these methods exhibit limited retrieval accuracy when faced with numerous indistinguishable documents, presenting notable challenges in their practical application. In response to these emerging challenges, we present HiQA, an advanced multi-document question-answering (MDQA) framework that integrates cascading metadata into content and a multi-route retrieval mechanism. We also release a benchmark called MasQA to evaluate and research in MDQA. Finally, HiQA demonstrates the state-of-the-art performance in multi-document environments.

Xinyue Chen, Pengyu Gao, Jiangjiang Song +1

1Nanjing University of Aeronautics and Astronautics. · 2Southeast University, Nanjing, Jiangsu, China.

Retrieval-Augmented Generation Multi-Document QA Document QA Information Retrieval RAG Evaluation

Apr 18, 2022

Research on Domain Information Mining and Theme Evolution of Scientific Papers

In recent years, with the increase of social investment in scientific research, the number of research results in various fields has increased significantly. Cross-disciplinary research results have gradually become an emerging frontier research direction. There is a certain dependence between a large number of research results. It is difficult to effectively analyze today's scientific research results when looking at a single research field in isolation. How to effectively use the huge number of scientific papers to help researchers becomes a challenge. This paper introduces the research status at home and abroad in terms of domain information mining and topic evolution law of scientific and technological papers from three aspects: the semantic feature representation learning of scientific and technological papers, the field information mining of scientific and technological papers, and the mining and prediction of research topic evolution rules of scientific and technological papers.

Changwei Zheng, Zhe Xue, Meiyu Liang +2

School of Computer Science (National Pilot School of Software Engineering), Beijing University of Posts and Telecommunications Beijing, China

Representation Learning Topic Modeling Information Retrieval

Apr 13, 2022

Retrieval of Scientific and Technological Resources for Experts and Scholars

Institutions of higher learning, research institutes and other scientific research units have abundant scientific and technological resources of experts and scholars, and these talents with great scientific and technological innovation ability are an important force to promote industrial upgrading. The scientific and technological resources of experts and scholars are mainly composed of basic attributes and scientific research achievements. The basic attributes include information such as research interests, institutions, and educational work experience. However, due to information asymmetry and other reasons, the scientific and technological resources of experts and scholars cannot be connected with the society in a timely manner, and social needs cannot be accurately matched with experts and scholars. Therefore, it is very necessary to build an expert and scholar information database and provide relevant expert and scholar retrieval services. This paper sorts out the related research work in this field from four aspects: text relation extraction, text knowledge representation learning, text vector retrieval and visualization system.

Suyu Ouyang, Yingxia Shao, Ang Li

School of Computer Science (National Pilot School of Software Engineering), Beijing University of Posts and Telecommunications Beijing Key Laboratory of Intelligent Telecommunication Software and Multimedia Beijing, China

Information Retrieval