LLM Reranking

LLM: Large Language Model

LLM reranking studies how to reorder a shortlist of documents, products, answers, or tools so that the most useful candidates appear first, often improving retrieval-augmented generation and recommendations. Common approaches ask models to score items individually, compare them in pairs or lists, or reason over evidence; other methods fine-tune or distill smaller rerankers, calibrate attention scores, or use reinforcement learning. Evaluations include TREC and BEIR retrieval benchmarks as well as scientific, conversational-memory, recommendation, and domain-specific tasks, increasingly measuring downstream answer quality and efficiency alongside ranking metrics. Key tensions are inference cost and sensitivity to candidate order, prompts, and bias, while reranking cannot recover relevant items missing from the initial retrieval pool and topical relevance may not match downstream usefulness.

Latest papers

133 papers14 papers in the last four weeks, down 33% on the four weeks before. 0.1% of all new papers.

ReBOL: Retrieval via Bayesian Optimization with Batched LLM Relevance Observations and Query Reformulation
Mar 20, 2026Anton Korikov, Scott SannerInformation Retrieval LLM Reranking
University of Toronto
Compute Allocation for Reasoning-Intensive Retrieval Agents
Mar 15, 2026Sreeja Apparaju, Nilesh GuptaInference-Time Compute Allocation Information Retrieval
When LLM Judge Scores Look Good but Best-of-N Decisions Fail
Mar 12, 2026Eddie LandesbergLLM Evaluation LLM-as-a-Judge
Does Reasoning Make Search More Fair? Comparing Fairness in Reasoning and Non-Reasoning Rerankers
Mar 11, 2026Saron Samuel, Benjamin Van Durme, Eugene YangAlgorithmic Fairness LLM Reranking
Johns Hopkins University · Human Language Technology Center of Excellence
ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting
Feb 23, 2026Yuxing Tian, Fengran Mo, Weixu Zhang +2LLM Reranking Learning to Rank
University of Montreal · McGill University & MILA · International Digital Economy Academy
Rethinking Retrieval-Augmented Generation as a Cooperative Decision-Making Problem
Feb 21, 2026Lichang Song, Ting Long, Yi ChangRetrieval-Augmented Generation RAG Robustness
Jilin University
Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools
Feb 20, 2026Baris Arat, Emre SeferLLM Evaluation LLM Reranking
Ozyegin University Istanbul, Turkey
GR2: Generative Reasoning Re-ranker
Feb 8, 2026Mingfu Liang, Yufei Li, Jay Xu +20RL for Language Models RL for Recommendation
Meta AI
PRISM: Prompt-Refined In-Context System Modelling for Financial Retrieval
Nov 18, 2025Chun Chet Ng, Jia Yu Lim, Zhen Hao Chu +2Information Retrieval LLM Reranking
AI Lens, Kuala Lumpur, Malaysia
Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization
Oct 8, 2025Tiancheng Xing, Jerry Li, Yixuan Du +1Adversarial Attacks on LLMs LLM Reranking
National University of Singapore · University of Southern California · Georgetown University +1
PiCSAR: Probabilistic Confidence Selection And Ranking for Reasoning Chains
Aug 29, 2025Joshua Ong Jun Leang, Zheng Zhao, Aryo Pradipta Gema +7Confidence Estimation in Language Models Best-of-N Sampling
Imperial College London · University of Edinburgh · UCL
SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generation with LLMs
Aug 12, 2025Haotian Chen, Qingqing Long, Meng Xiao +6LLM Reranking RAG Evaluation
Computer Network Information Center, Chinese Academy of Sciences · University of California, Los Angeles · Peking University
Horizon Scans can be accelerated using novel information retrieval and artificial intelligence tools
Apr 2, 2025Lena Schmidt, Oshin Sharma, Chris Marshall +1Healthcare LLM Reranking
NIHR Innovation Observatory, Population Health Sciences Institute, Newcastle University, Newcastle upon Tyne, UK

Mar 20, 2026J/K move · Enter open · S save

ReBOL: Retrieval via Bayesian Optimization with Batched LLM Relevance Observations and Query Reformulation

Anton Korikov, Scott Sanner

University of Toronto

Information Retrieval LLM Reranking Multimodal IR Bayesian Optimization

Open paper AskPDF arXiv

LLM-reranking is limited by the top-k documents retrieved by vector similarity, which neither enables contextual query-document token interactions nor captures multimodal relevance distributions. While LLM query reformulation attempts to improve recall by generating improved or additional queries, it is still followed by vector similarity retrieval. We thus propose to address these top-k retrieval stage failures by introducing ReBOL, which 1) uses LLM query reformulations to initialize a multimodal Bayesian Optimization (BO) posterior over document relevance, and 2) iteratively acquires document batches for LLM query-document relevance scoring followed by posterior updates to optimize relevance. After exploring query reformulation and document batch diversification techniques, we evaluate ReBOL against LLM reranker baselines on five BEIR datasets and using two LLMs (Gemini-2.5-Flash-Lite, GPT-5.2). ReBOL consistently achieves higher recall and competitive rankings, for example compared to the best LLM reranker on the Robust04 dataset with 46.5% vs. 35.0% recall@100 and 63.6% vs. 61.2% NDCG@10. We also show that ReBOL can achieve comparable latency to LLM rerankers.