Pairwise Ranking

Follow this topic Ask about this topic

Pairwise ranking studies how to recover useful orderings from judgments such as which of two models, answers, or products is better. Common approaches include Bradley–Terry and Elo models, spectral methods, and active learning that selects informative comparisons; neural pairwise and listwise reward models extend these ideas to quality assessment and preference optimization. Applications include recommender systems, human-in-the-loop annotation, and LLM leaderboards evaluated with paired benchmarks. A central challenge is that comparisons can be noisy, biased, or heterogeneous, so cycles and close rankings may make a single global order unstable or statistically unresolved.

Latest papers

62 papers6 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Alternative Strategies for Active Learning with Bayesian Bradley Terry Model
Oct 21, 2026Saman Jayasinghe, Sean Walton, Alma Rahat +2Pairwise Ranking Active Preference Learning
Swansea University, United Kingdom · Loughborough University, United Kingdom · Bath Spa University, United Kingdom +1
Exact Risk Certificates for Pairwise-Judged Reranking
Oct 17, 2026Xinyu Wang, Jingrui Tian, Sicheng Lyu +4IR Evaluation Pairwise Ranking
McGill University · Ohio University · University of Manitoba
Learning Who Benefits Most: Multi-Treatment Causal Ranking for Population Health Stratification
Oct 14, 2026Rocco Zaccagnino, Martina Girolamo, Alberto Lombardi +3Clinical Decision-Making Causal Inference
University of Salerno, Salerno, Italy · Local Health Authority of Benevento, Benevento, Italy · University of California, Los Angeles, Los Angeles, USA
DeFiRank: Federated Pairwise Ranking for Governance in Decentralized Financial Networks
Oct 9, 2026Zahra Batool, Michael Duchesne, Kaiwen Zhang +1Decentralized Finance Learning to Rank
Ecole de technologie sup´erieure, Montr´eal, Canada · Mila - Quebec AI Institute, Montr´eal, Canada · T-RIZE GROUP, Montr´eal, Canada
How Many Repeated Pairwise Comparisons Are Needed for Ranking under Heterogeneity?
Oct 7, 2026Shashaank Aiyer, Han ShaoPairwise Ranking Pairwise Preference Learning
University of Maryland
Scientific Discovery under Validation Congestion via Multi-Fidelity Pairwise Rankings
Oct 1, 2026Kevin Tirta Wijaya, Alston Lo, Michael Sun +2Pairwise Ranking Autonomous Scientific Discovery
University of Bonn · Fraunhofer SCAI · Massachusetts Institute of Technology
What Can a Leaderboard Certify? Compositional Controllability for Fair Evaluation and Training of Biomedical Literature-Review Agents
Sep 26, 2026Zhaowei Han, Xiang Zhang, Lingxiao Guan +4AI Agent Evaluation Pairwise Ranking
University of Michigan, Ann Arbor
OSCAR: Order-aware Sensitivity Correction for AI Rankings
Sep 21, 2026You Liu, Yue Liu, Quanchao Lu +2LLM Evaluation LLM-as-a-Judge
Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead
Sep 15, 2026Fengshuo Liu, Ying Liu, Ruize Sun +2AI Agent Evaluation Benchmark Auditing
Imperial College London, London, United Kingdom · The Hong Kong Polytechnic University, Hong Kong SAR · Korea University, Seoul, Korea +1
Edu-QuRating: Multi-Dimensional Educational Data Curation with Distilled Pairwise Judgements
Sep 8, 2026Oliver G. B. Garrod, Robin A. A. Ince, Meng Liu +6Pairwise Preference Evaluation Pairwise Preference Learning
Fab AI
CliffRank: A Dual-Branch Framework for Activity-Cliff Ranking Prediction
Sep 1, 2026Kewei Li, Rongying Zhang, Peiyu Yang +4Learning to Rank Pairwise Ranking
College of Computer Science and Technology, Jilin University, Changchun, 130012, China. · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, 130012, China. · Greenwich High School, Greenwich, CT, 06830, United States of America. +3
Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?
Aug 31, 2026Qiaoyuan Zheng, Yiqu YangLLM Evaluation Pairwise Ranking
ETH Zurich Zurich, Switzerland
Sorting from Counterexamples
Aug 21, 2026Noga Alon, Shay Moran, Shlomo MoranQuery Complexity Pairwise Ranking
Princeton University, Princeton, NJ 08544, USA, and Tel Aviv University, Tel Aviv 69978, Israel. · Technion – Israel Institute of Technology and Google Research. · Technion – Israel Institute of Technology.
Isotonic Bradley-Terry Model for Paired Comparison Data
Aug 3, 2026Ryoya YamasakiPairwise Preference Learning Pairwise Comparison
Hitotsubashi Institute for Advanced Study, Hitotsubashi University, 2-1 Naka, Kunitachi, 186-8601, Tokyo, Japan.
On the Limits of Machine-Learned Ranking for Modern Microarchitectural Policies
Aug 2, 2026Yanxin Zhang, Shayne Wadle, Yuxuan Xiong +3Learning to Rank Electronic Design Automation
NVIDIA · University of Wisconsin–Madison
Optimal Top- $k$ Identification from Pairwise Comparisons
Jul 9, 2026Motti Goldberger, Nils RudiPairwise Preference Learning Multi-Armed Bandits
Yale University, New Haven, CT, USA.
Uncovering Latent Depression Severity for Binary Depression Detection via Advantage-weighting Ranking
Jul 7, 2026Manning Gao, Tingyi Liu, Leheng Zhang +3Representation Learning Learning to Rank
South China Normal University, China · Sun Yat-sen University, China
Attention Limited Reward Learning
Jul 6, 2026Wenqian XingPairwise Preference Learning Reward Modeling
Management Science and Engineering Department, Stanford University
Internal Pluralism and the Limits of Pairwise Comparisons
Jul 2, 2026Bailey Flanigan, Michelle SiPairwise Preference Learning Pairwise Comparison
MIT, USA · Harvard, USA
Ask the Right Comparison:Bias-Aware Bayesian Active Top- $k$ Ranking with LLM Judges
Jul 2, 2026Jian Xu, Delu Zeng, John Paisley +1Pairwise Comparison LLM-as-a-Judge
RIKEN iTHEMS · RIKEN AIP · South China University of Technology +1
Can LLMs Rank? A Tale of Triads and Triage
Jun 29, 2026Gaurab Pokharel, Shafkat Farabi, Patrick J. Fowler +1Pairwise Comparison Social Choice Theory
Virginia Tech Alexandria, VA · Washington University in St. Louis St. Louis, MO
The Geometry of Sequential Learning: Lie-Bracket Prediction of Transfer Order
Jun 23, 2026John SweeneyPairwise Ranking Cross-Domain Transfer Learning
Sideplane AI
ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
Jun 23, 2026Jisu Jeon, Seungyeon Jwa, Joosung Lee +6Audio-Language Model Evaluation Pairwise Comparison
Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea +1
Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs
Jun 16, 2026Dong Huang, Jianbo Sun, Pengkun YangLLM Evaluation Pairwise Comparison
Department of Statistics and Data Science, Tsinghua University
TuneJury: An Open Metric for Improving Music Generation Preference Alignment
Jun 15, 2026Yonghyun Kim, Junwon Lee, Haiwen Xia +5Text-to-Music Generation Music Generation Evaluation
Georgia Tech · KAIST · Peking University +3
Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling
Jun 14, 2026Yujin Park, Haejun Chung, Ikbeom JangPairwise Preference Learning Pairwise Comparison
Hanyang University, Seoul, Republic of Korea · Hankuk University of Foreign Studies, Yongin, Republic of Korea
Comparison Patrols on Drifting Orders: Certified Rank Maintenance, Evolving Planar Maxima, and Selection under Drifting Fitness
Jun 12, 2026Faruk Alpay, Levent SariogluPairwise Comparison Evolutionary Optimization
Department of Computer Engineering, Bahçeşehir University, Istanbul, Turkey
From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation
Jun 11, 2026Bora Kargi, David SalinasLLM Evaluation Pairwise Comparison
ELLIS Institute Tübingen · OpenEuroLLM
Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings
Jun 8, 2026Mina Remeli, Moritz HardtPairwise Comparison LLM-as-a-Judge
Max Planck Institute for Intelligent Systems, Tübingen, Germany, Tübingen AI Center
Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomies
Jun 5, 2026Ekaterina Grishina, Stepan Kuznetsov, Askar Tsyganov +8Algorithm Selection Pairwise Comparison
HSE University Moscow, Russian Federation

Oct 21, 2026J/K move · Enter open · S save

Alternative Strategies for Active Learning with Bayesian Bradley Terry Model

Saman Jayasinghe, Sean Walton, Alma Rahat, Andy Gray+1

Swansea University, United Kingdom · Loughborough University, United Kingdom · Bath Spa University, United Kingdom+1

Pairwise Ranking Active Preference Learning Bradley–Terry Model Bayesian Inference

Open paper AskPDF arXiv

Comparative judgement uses pairwise comparisons to rank items, but recovering accurate rankings from a limited number of judgements depends on both the inference method and the pairs selected. We developed a Bayesian Bradley-Terry model (BBTM) that uses Polya-Gamma data augmentation for efficient Gibbs sampling and quantifies uncertainty in latent abilities and ranks. We propose three active learning strategies based on the knowledge gradient (KG) principle. We define our acquisition functions using Kendall's tau distance, posterior rank entropy, and posterior separation. Approximate one-step updates to posterior moments reduce the computational cost of evaluating candidate pairs. We also introduce Posterior Separation Reliability (PSR), which measures the spread of posterior mean abilities relative to posterior uncertainty. It provides an internal indicator of ranking reliability without requiring a reference ranking. In synthetic experiments covering different item counts, comparison budgets, and noise levels, BBTM generally recovers ranking more accurately than preference-based Bayesian comparative judgement (PBCJ). At low to moderate noise levels, the pairwise entropy pair selection strategy performs best in ranking recovery. However, at the highest noise levels, the KG strategy based on posterior separation produces better point rankings. PSR increases alongside improvements in ranking accuracy as comparisons accumulate. Robustness experiments identify the proportion of informative outcomes needed to meet specified accuracy targets within a fixed comparison budget. In a moderate noise setting, replacing 30% of outcomes with random choices has little effect on ranking performance. These findings demonstrate the value of Bayesian inference and active learning pair selection for recovering ranks from limited, noisy paired comparisons.