Overview

Learning to rank studies how to order candidates—such as documents, products, or recommendations—so the most useful items appear first. Approaches include pointwise relevance prediction, pairwise methods such as RankNet and Bradley–Terry models, listwise objectives such as LambdaRank, and neural or LLM-based rerankers; reinforcement learning can also optimize rankings for downstream outcomes. Systems are evaluated with metrics such as NDCG and MRR on retrieval benchmarks and recommendation data, then often tested in online experiments. A central tension is that offline ranking gains may not translate to user or business value, while stronger reranking can add latency and remain vulnerable to bias, position effects, and unstable feedback.

Latest papers

306 papers34 papers in the last four weeks, up 13% on the four weeks before. 0.3% of all new papers.

Sep 15, 2025

Unsupervised Candidate Ranking for Lexical Substitution via Holistic Sentence Semantics

A key subtask in lexical substitution is ranking the given candidate words. A common approach is to replace the target word with a candidate in the original sentence and feed the modified sentence into a model to capture semantic differences before and after substitution. However, effectively modeling the bidirectional influence of candidate substitution on both the target word and its context remains challenging. Existing methods often focus solely on semantic changes at the target position or rely on parameter tuning over multiple evaluation metrics, making it difficult to accurately characterize semantic variation. To address this, we investigate two approaches: one based on attention weights and another leveraging the more interpretable integrated gradients method, both designed to measure the influence of context tokens on the target token and to rank candidates by incorporating semantic similarity between the original and substituted sentences. Experiments on the LS07 and SWORDS datasets demonstrate that both approaches improve ranking performance.
Aug 16, 2025

Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

We propose a method for evaluating the robustness of widely used LLM ranking systems -- variants of a Bradley--Terry model -- to dropping a worst-case very small fraction of preference data. Our approach is computationally fast and easy to adopt. When we apply our method to matchups from popular LLM ranking platforms, including Chatbot Arena and derivatives, we find that the rankings of top-performing models can be remarkably sensitive to the removal of a small fraction of preferences; for instance, dropping just 0.003% of human preferences can change the top-ranked model on Chatbot Arena. Our robustness check identifies the specific preferences most responsible for such ranking flips, allowing for inspection of these influential preferences. We observe that the rankings derived from MT-bench preferences are notably more robust than those from Chatbot Arena, likely due to MT-bench's use of expert annotators and carefully constructed prompts. Finally, we find that neither rankings based on crowdsourced human evaluations nor those based on LLM-as-a-judge preferences are systematically more sensitive than the other.
Aug 7, 2025

Algorithmic Shortlisting in Participatory Budgeting

Participatory budgeting is a democratic innovation that allows citizens to propose and vote on public investment projects. To help organizers manage large volumes of submissions, we design and test privacy-preserving methods for algorithmic shortlisting. These algorithms predict which projects are likely to be funded using only project features and anonymous historical voting data. We demonstrate the limitations of a naive approach that uses a large language model to rank projects based on past success and propose a vote-based pipeline that enables state-of-the-art LLMs to perform on par with classical machine learning. Our findings indicate that user preferences in participatory budgeting are stable enough to allow algorithmic shortlisting to approximate an initial selection of projects effectively.
May 19, 2025

GeoRanker: Distance-Aware Ranking for Worldwide Image Geolocalization

Worldwide image geolocalization-the task of predicting GPS coordinates from images taken anywhere on Earth-poses a fundamental challenge due to the vast diversity in visual content across regions. While recent approaches adopt a two-stage pipeline of retrieving candidates and selecting the best match, they typically rely on simplistic similarity heuristics and point-wise supervision, failing to model spatial relationships among candidates. In this paper, we propose GeoRanker, a distance-aware ranking framework that leverages large vision-language models to jointly encode query-candidate interactions and predict geographic proximity. In addition, we introduce a multi-order distance loss that ranks both absolute and relative distances, enabling the model to reason over structured spatial relationships. To support this, we curate GeoRanking, the first dataset explicitly designed for geographic ranking tasks with multimodal candidate information. GeoRanker achieves state-of-the-art results on two well-established benchmarks (IM2GPS3K and YFCC4K), significantly outperforming current best methods.
Jan 13, 2025

Pairwise Comparisons without Stochastic Transitivity: Model, Theory and Applications

Most statistical models for pairwise comparisons, including the Bradley-Terry (BT) and Thurstone models and many extensions, make a relatively strong assumption of stochastic transitivity. This assumption imposes the existence of an unobserved global ranking among all the players/teams/items and monotone constraints on the comparison probabilities implied by the global ranking. However, the stochastic transitivity assumption does not hold in many real-world scenarios of pairwise comparisons, especially games involving multiple skills or strategies. As a result, models relying on this assumption can have suboptimal predictive performance. In this paper, we propose a general family of statistical models for pairwise comparison data without a stochastic transitivity assumption, substantially extending the BT and Thurstone models. In this model, the pairwise probabilities are determined by a (approximately) low-dimensional skew-symmetric matrix. Likelihood-based estimation methods and computational algorithms are developed, which allow for sparse data with only a small proportion of observed pairs. Theoretical analysis shows that the proposed estimator achieves minimax-rate optimality, which adapts effectively to the sparsity level of the data. The spectral theory for skew-symmetric matrices plays a crucial role in the implementation and theoretical analysis. The proposed method's superiority against the BT model, along with its broad applicability across diverse scenarios, is further supported by simulations and real data analysis.
Mar 11, 2022

Personalized Execution Time Optimization for Billion-Scale Scheduled Jobs

Scheduled batch jobs are widely used on asynchronous computing platforms to execute enterprise applications such as promotional notifications and candidate pre-computation for recommender systems. Delivering or updating information at the right time is important for user experience and execution impact, yet providing a versatile, personalized execution time optimization solution across diverse product scenarios while maintaining reasonable infrastructure costs remains challenging. In this paper, we present a deployed system that serves billions of users daily, combining learning-to-rank with a "best time policy" for execution time selection. We describe the four-stage evolution of our approach: from heuristic peak-hour rules, to pointwise ML-based activity pattern predictions, to a linear signal assembler with globally fixed weights, and finally to a contextual ensemble learner that produces per-user adaptive fusion weights via a neural policy network trained with listwise learning-to-rank objectives. We further report the discovery of cross-use-case cannibalization effects and introduce a coordination system to mitigate the problem. Our production experiments demonstrate measurable improvements in both execution efficiency and downstream product impact. We share deployment lessons including failure analyses and design decisions accumulated over four years of operating this system at scale. To our knowledge, this represents the first ML-based multi-tenant execution time optimization system deployed across different product domains at industrial scale.