Pairwise Preference Evaluation

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 70

All topics

Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency
Sep 29, 2026Bruno Brocai, Maria BeckerLLM-as-a-Judge Pairwise Preference Evaluation
Heidelberg University German Department
Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels
Sep 15, 2026David Ababio Awuni, Luke E. K. Achenie, Benjamin Tei Partey +2LLM-as-a-Judge Pairwise Preference Evaluation
Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges
Aug 12, 2026Xi Chen, Jie Mu, Mo Xuan +1Human Preference Evaluation LLM-as-a-Judge
Ant Group
Finding the Signal in the Spam: Jointly Learning Rewards and Worker Reliability from Pairwise Comparisons
Aug 10, 2026Kaustubh Shivshankar Shejole, Tanish Agarwal, Arpit Agarwal +1Pairwise Preference Learning Reward Modeling
IIT Bombay, Mumbai, India – 400076
TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models
Aug 9, 2026Yidong Wang, Yan Zhan, Ziteng Feng +16Reward Modeling Pairwise Preference Evaluation
Peking University · University of Science and Technology of China · Southeast University +6
Can Language Models Imagine Without Seeing? Ekphrasis: Measuring Visual Creative Ideation in Text-Only LLMs
Aug 7, 2026Hongyu Luo, He Wang, Huihao Jing +6Creativity Assessment LLM Evaluation
The Hong Kong University of Science and Technology, Hong Kong SAR, China
Stability of Ranking-dependent Pair-wise Comparison Patterns in the Analytic Hierarchy Process
Aug 6, 2026Vitaliy Tsyganok, Sergii Kadenko, Oleh AndriichukPairwise Comparison Pairwise Preference Evaluation
Institute for Information Recording National Academy of Sciences of Ukraine Kyiv, Ukraine 03113
Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation
Aug 5, 2026Divyansh SinghHuman Preference Evaluation LLM Evaluation
University of Florida
Isotonic Bradley-Terry Model for Paired Comparison Data
Aug 3, 2026Ryoya YamasakiPairwise Preference Learning Pairwise Comparison
Hitotsubashi Institute for Advanced Study, Hitotsubashi University, 2-1 Naka, Kunitachi, 186-8601, Tokyo, Japan.
AutoPref: Automatic Discovery of Task-Specific Preference Objectives for Neural Combinatorial Optimization
Jul 30, 2026Shengda Gu, Kai Li, Xinyi Ke +3Automated Algorithm Discovery Pairwise Preference Evaluation
C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent AI Lab +1
What do Reward Models Memorize?
Jul 27, 2026Ivo Verhoeven, Pushkar Mishra, Ekaterina ShutovaReward Modeling Pairwise Preference Evaluation
ILLC, University of Amsterdam, The Netherlands · Google DeepMind, London, United Kingdom
CSPF: A Constrained Shared-Private Fusion Method for Non-Verifiable Preference Evaluation
Jul 23, 2026Hehao Zhang, Danli Wang, Xinyuan Wang +1Pairwise Preference Learning Reward Modeling
Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
Style over Substance: A Shortcut Audit of Emotion-Description Preference Evaluation
Jul 20, 2026Jiabing Yang, Yixiang Chen, Yuan Xu +6Human Preference Evaluation Pairwise Preference Evaluation
University of Chinese Academy of Sciences Institute of Automation, Chinese Academy of Sciences Beijing, China
Reward-Free Evolving Agents via Pairwise Validator
Jul 15, 2026Minghao Liu, Yu Wang, Jiayun Wang +1Pairwise Preference Evaluation LLM Agent Self-Improvement
Accenture
Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparability
Jul 13, 2026Simone Drago, Marco Mussi, Leonardo Bianconi +1Pairwise Preference Learning Pairwise Comparison
Politecnico di Milano, Milan, Italy
Geometric mean-based pairwise comparison method with the reference values -- statistical approach
Jul 10, 2026Konrad Kułakowski, Jacek SzybowskiPairwise Comparison Pairwise Preference Evaluation
AGH University of Krakow, WEAiIB, al. Mickiewicza 30, 30-059 Poland · AGH University of Krakow, WMS, al. Mickiewicza 30, 30-059 Poland
Optimal Top- $k$ Identification from Pairwise Comparisons
Jul 9, 2026Motti Goldberger, Nils RudiPairwise Preference Learning Multi-Armed Bandits
Yale University, New Haven, CT, USA.
Attention Limited Reward Learning
Jul 6, 2026Wenqian XingPairwise Preference Learning Reward Modeling
Management Science and Engineering Department, Stanford University
Internal Pluralism and the Limits of Pairwise Comparisons
Jul 2, 2026Bailey Flanigan, Michelle SiPairwise Preference Learning Pairwise Comparison
MIT, USA · Harvard, USA
LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform
Jul 1, 2026Ruotong Zhao, Zhiyu Chen, Xurui Liu +7Human Preference Evaluation LLM-as-a-Judge
Tsinghua University, Beijing, China · Zhongguancun Academy, Beijing, China · Zhongguancun Institute of AI, Beijing, China +2
Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops?
Jun 30, 2026Zewen LiuLLM-as-a-Judge Pairwise Preference Evaluation
Qilu Institute of Technology, School of Software Engineering Tai’an, Shandong, China
Can LLMs Rank? A Tale of Triads and Triage
Jun 29, 2026Gaurab Pokharel, Shafkat Farabi, Patrick J. Fowler +1Pairwise Comparison Social Choice Theory
Virginia Tech Alexandria, VA · Washington University in St. Louis St. Louis, MO
Democratic ICAI: Debating Our Way to Steering Principles from Preferences
Jun 26, 2026Kevin Kingslin, Anish Natekar, Ashutosh Ranjan +3Pairwise Preference Learning Pairwise Preference Evaluation
TCS Research, India
ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
Jun 23, 2026Jisu Jeon, Seungyeon Jwa, Joosung Lee +6Audio-Language Model Evaluation Pairwise Comparison
Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea +1
A Markov Chain Approach to Preference Alignment
Jun 21, 2026Takuya Koriyama, Tengyuan LiangPairwise Preference Learning Pairwise Comparison
The University of Chicago
Which Pairs to Compare for LLM Post-Training?
Jun 17, 2026Jiangze Han, Vineet Goyal, Will MaPairwise Preference Learning Pairwise Comparison
Columbia University
PrefSQA: Pairwise Preference Prediction for Speech Quality Assessment and the Critical Role of High Quality Datasets
Jun 17, 2026Junyi Fan, Donald S. WilliamsonPairwise Preference Learning Pairwise Preference Evaluation
Department of Computer Science and Engineering, The Ohio State University, USA
UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning
Jun 17, 2026Mohamed Nabail, Leo Kaixuan Cheng, Jingmin Wang +1Pairwise Preference Evaluation RL Exploration
Learning, Embodied Autonomy, and Forecasting (LEAF) Lab, University of Toronto
RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing
Jun 17, 2026Guannan Lai, Haoran Hu, Han-Jia YeLLM Evaluation Pairwise Preference Evaluation
School of Artificial Intelligence, Nanjing University, China · 2National Key Laboratory for Novel Software Technology, Nanjing University, 210023, China · 3SinapisAI
Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs
Jun 16, 2026Dong Huang, Jianbo Sun, Pengkun YangLLM Evaluation Pairwise Comparison
Department of Statistics and Data Science, Tsinghua University

Sep 29, 2026, cs.CLJ/K move · Enter open · S save

Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency

Bruno Brocai, Maria Becker

Heidelberg University German Department

LLM-as-a-Judge Pairwise Preference Evaluation LLM Reliability

Open paper AskPDF arXiv

Large Language Model judges are widely used to rank texts and text-generating systems through pairwise comparison, and their reliability is typically assessed via three proxies: position bias, transitivity, and pairwise agreement (self- or human-labeled). Because these proxies drive judge selection and benchmarking, a substantial literature reporting that judges perform poorly on them risks steering practitioners away from otherwise capable evaluators. We argue this assessment is misleading. Under the Bradley--Terry geometry underlying pairwise aggregation, each proxy is dominated by close-rank-gap pairs, where inconsistency is information-theoretically expected and individual verdicts contribute little to the aggregate ranking; far-gap pairs carry the ranking signal but barely move the proxies. We formalize this argument and validate it in a controlled simulation and on two human-rated corpora: the proxies correlate only weakly with ranking accuracy against gold, and their predictive component concentrates in the far-gap regime. Judges should therefore be assessed on rank-gap-conditional metrics, ideally against human rankings. Code at https://github.com/brunobrocai/PairDifficulty.

Momentum

Related topics

Latest papers 70