Pairwise Preference Evaluation

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 69

All topics

Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency
Sep 29, 2026Bruno Brocai, Maria BeckerLLM-as-a-Judge Pairwise Preference Evaluation
Heidelberg University German Department
Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels
Sep 15, 2026David Ababio Awuni, Luke E. K. Achenie, Benjamin Tei Partey +2LLM-as-a-Judge Pairwise Preference Evaluation
Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges
Aug 12, 2026Xi Chen, Jie Mu, Mo Xuan +1Human Preference Evaluation LLM-as-a-Judge
Ant Group
Finding the Signal in the Spam: Jointly Learning Rewards and Worker Reliability from Pairwise Comparisons
Aug 10, 2026Kaustubh Shivshankar Shejole, Tanish Agarwal, Arpit Agarwal +1Pairwise Preference Learning Reward Modeling
IIT Bombay, Mumbai, India – 400076
TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models
Aug 9, 2026Yidong Wang, Yan Zhan, Ziteng Feng +16Reward Modeling Pairwise Preference Evaluation
Peking University · University of Science and Technology of China · Southeast University +6
Can Language Models Imagine Without Seeing? Ekphrasis: Measuring Visual Creative Ideation in Text-Only LLMs
Aug 7, 2026Hongyu Luo, He Wang, Huihao Jing +6Creativity Assessment LLM Evaluation
The Hong Kong University of Science and Technology, Hong Kong SAR, China
Stability of Ranking-dependent Pair-wise Comparison Patterns in the Analytic Hierarchy Process
Aug 6, 2026Vitaliy Tsyganok, Sergii Kadenko, Oleh AndriichukPairwise Comparison Pairwise Preference Evaluation
Institute for Information Recording National Academy of Sciences of Ukraine Kyiv, Ukraine 03113
Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation
Aug 5, 2026Divyansh SinghHuman Preference Evaluation LLM Evaluation
University of Florida
Isotonic Bradley-Terry Model for Paired Comparison Data
Aug 3, 2026Ryoya YamasakiPairwise Preference Learning Pairwise Comparison
Hitotsubashi Institute for Advanced Study, Hitotsubashi University, 2-1 Naka, Kunitachi, 186-8601, Tokyo, Japan.
AutoPref: Automatic Discovery of Task-Specific Preference Objectives for Neural Combinatorial Optimization
Jul 30, 2026Shengda Gu, Kai Li, Xinyi Ke +3Automated Algorithm Discovery Pairwise Preference Evaluation
C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent AI Lab +1
What do Reward Models Memorize?
Jul 27, 2026Ivo Verhoeven, Pushkar Mishra, Ekaterina ShutovaReward Modeling Pairwise Preference Evaluation
ILLC, University of Amsterdam, The Netherlands · Google DeepMind, London, United Kingdom
CSPF: A Constrained Shared-Private Fusion Method for Non-Verifiable Preference Evaluation
Jul 23, 2026Hehao Zhang, Danli Wang, Xinyuan Wang +1Pairwise Preference Learning Reward Modeling
Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
Style over Substance: A Shortcut Audit of Emotion-Description Preference Evaluation
Jul 20, 2026Jiabing Yang, Yixiang Chen, Yuan Xu +6Human Preference Evaluation Pairwise Preference Evaluation
University of Chinese Academy of Sciences Institute of Automation, Chinese Academy of Sciences Beijing, China
Reward-Free Evolving Agents via Pairwise Validator
Jul 15, 2026Minghao Liu, Yu Wang, Jiayun Wang +1Pairwise Preference Evaluation LLM Agent Self-Improvement
Accenture
Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparability
Jul 13, 2026Simone Drago, Marco Mussi, Leonardo Bianconi +1Pairwise Preference Learning Pairwise Comparison
Politecnico di Milano, Milan, Italy
Geometric mean-based pairwise comparison method with the reference values -- statistical approach
Jul 10, 2026Konrad Kułakowski, Jacek SzybowskiPairwise Comparison Pairwise Preference Evaluation
AGH University of Krakow, WEAiIB, al. Mickiewicza 30, 30-059 Poland · AGH University of Krakow, WMS, al. Mickiewicza 30, 30-059 Poland
Optimal Top- $k$ Identification from Pairwise Comparisons
Jul 9, 2026Motti Goldberger, Nils RudiPairwise Preference Learning Multi-Armed Bandits
Yale University, New Haven, CT, USA.
Attention Limited Reward Learning
Jul 6, 2026Wenqian XingPairwise Preference Learning Reward Modeling
Management Science and Engineering Department, Stanford University
Internal Pluralism and the Limits of Pairwise Comparisons
Jul 2, 2026Bailey Flanigan, Michelle SiPairwise Preference Learning Pairwise Comparison
MIT, USA · Harvard, USA
LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform
Jul 1, 2026Ruotong Zhao, Zhiyu Chen, Xurui Liu +7Human Preference Evaluation LLM-as-a-Judge
Tsinghua University, Beijing, China · Zhongguancun Academy, Beijing, China · Zhongguancun Institute of AI, Beijing, China +2
Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops?
Jun 30, 2026Zewen LiuLLM-as-a-Judge Pairwise Preference Evaluation
Qilu Institute of Technology, School of Software Engineering Tai’an, Shandong, China
Can LLMs Rank? A Tale of Triads and Triage
Jun 29, 2026Gaurab Pokharel, Shafkat Farabi, Patrick J. Fowler +1Pairwise Comparison Social Choice Theory
Virginia Tech Alexandria, VA · Washington University in St. Louis St. Louis, MO
Democratic ICAI: Debating Our Way to Steering Principles from Preferences
Jun 26, 2026Kevin Kingslin, Anish Natekar, Ashutosh Ranjan +3Pairwise Preference Learning Pairwise Preference Evaluation
TCS Research, India
ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
Jun 23, 2026Jisu Jeon, Seungyeon Jwa, Joosung Lee +6Audio-Language Model Evaluation Pairwise Comparison
Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea +1
A Markov Chain Approach to Preference Alignment
Jun 21, 2026Takuya Koriyama, Tengyuan LiangPairwise Preference Learning Pairwise Comparison
The University of Chicago
Which Pairs to Compare for LLM Post-Training?
Jun 17, 2026Jiangze Han, Vineet Goyal, Will MaPairwise Preference Learning Pairwise Comparison
Columbia University
PrefSQA: Pairwise Preference Prediction for Speech Quality Assessment and the Critical Role of High Quality Datasets
Jun 17, 2026Junyi Fan, Donald S. WilliamsonPairwise Preference Learning Pairwise Preference Evaluation
Department of Computer Science and Engineering, The Ohio State University, USA
UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning
Jun 17, 2026Mohamed Nabail, Leo Kaixuan Cheng, Jingmin Wang +1Pairwise Preference Evaluation RL Exploration
Learning, Embodied Autonomy, and Forecasting (LEAF) Lab, University of Toronto
RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing
Jun 17, 2026Guannan Lai, Haoran Hu, Han-Jia YeLLM Evaluation Pairwise Preference Evaluation
School of Artificial Intelligence, Nanjing University, China · 2National Key Laboratory for Novel Software Technology, Nanjing University, 210023, China · 3SinapisAI
Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs
Jun 16, 2026Dong Huang, Jianbo Sun, Pengkun YangLLM Evaluation Pairwise Comparison
Department of Statistics and Data Science, Tsinghua University
TuneJury: An Open Metric for Improving Music Generation Preference Alignment
Jun 15, 2026Yonghyun Kim, Junwon Lee, Haiwen Xia +5Text-to-Music Generation Music Generation Evaluation
Georgia Tech · KAIST · Peking University +3
Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling
Jun 14, 2026Yujin Park, Haejun Chung, Ikbeom JangPairwise Preference Learning Pairwise Comparison
Hanyang University, Seoul, Republic of Korea · Hankuk University of Foreign Studies, Yongin, Republic of Korea
From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation
Jun 11, 2026Bora Kargi, David SalinasLLM Evaluation Pairwise Comparison
ELLIS Institute Tübingen · OpenEuroLLM
Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings
Jun 8, 2026Mina Remeli, Moritz HardtPairwise Comparison LLM-as-a-Judge
Max Planck Institute for Intelligent Systems, Tübingen, Germany, Tübingen AI Center
DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivity
Jun 8, 2026Fengyuan Liu, Yongliang Miao, Zirui He +3Pairwise Preference Learning Reward Modeling
The Chinese University of Hong Kong, Shenzhen · New Jersey Institute of Technology · Institute of Computing Technology, CAS
Local Preferential Bayesian Optimization
Jun 1, 2026Johanna Menn, Miriam Kober, Paul Brunzema +2Pairwise Preference Learning Pairwise Preference Evaluation
Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Aachen, Germany · Department of Clinical Research, University of Bern, Bern, Switzerland · Center for Reproducible Science and Research Synthesis, University of Zurich, Zurich, Switzerland +1
From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
May 31, 2026Jun-Jie Yang, Chia-Heng Hsu, Kui-Yuan Chen +1Pairwise Preference Learning Representation Learning
Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan
Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problems
May 29, 2026Jonathan Colaço Carr, Prakash Panangaden, Doina Precup +1Pairwise Preference Learning Markov Decision Processes
Stanford University · School of Computer Science, McGill University, Montreal, Quebec, Canada · Mila - Quebec AI Institute, Montreal, Quebec, Canada +1
The Representation-Rationalizability Tradeoff in Reward Learning
May 29, 2026Jing Dong, Yaoliang Yu, Pascal PourpartPairwise Preference Learning Reward Modeling
Vector Institute · University of Waterloo & Vector Institute
Pairwise Reference Alignment as a Model-Level Ordinal Observable
May 29, 2026Mujing LiHuman Preference Evaluation LLM Evaluation
Independent Researcher
Reward Learning from Best-of- $N$ Preference Data: Targets, Tradeoffs, and Design Principles
May 28, 2026Rattana Pukdee, Maria-Florina Balcan, Pradeep RavikumarPairwise Preference Learning Reward Modeling
Machine Learning Department Carnegie Mellon University
Resolution Diagnostics for Paired LLM Evaluation
May 28, 2026Anany KotawalaLLM Evaluation Pairwise Comparison
Princeton University, Princeton, NJ, USA.
From Blind Guess to Informed Judgment: Teaching LLMs to Evaluate Materials by Building Knowledge-Augmented Preference Signals
May 28, 2026Yeyong Yu, Wenya Hu, Xing Wu +1LLM Evaluation Pairwise Preference Evaluation
School of Computer Engineering & Science, Shanghai University, Shanghai, 200444, China. · Center of Materials Informatics and Data Science, Materials Genome2026 Institute, Shanghai University, Shanghai, 200444, China. · Key Laboratory of Silicate Cultural Relics Conservation (Shanghai University), Ministry of Education, China. +1
Preferred, Not Safer: Pairwise Preference Is a Poor Proxy for Clinical Safety
May 25, 2026Fay Elhassan, David Sasu, Alexandra Kulinkina +2LLM Safety Benchmarks Human Preference Evaluation
LiGHT Laboratory, EPFL, Lausanne, Switzerland
SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks
May 25, 2026Yanhang Li, Zhichao Fan, Zexin ZhuangPairwise Comparison Pairwise Preference Evaluation
Northeastern University, Boston, MA, USA · University of Illinois Urbana-Champaign, Urbana, IL, USA · Southern Methodist University, Dallas, TX, USA
JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment
May 24, 2026Russell Yang, Ruishi Chen, Pierce Kelaita +6Human Preference Evaluation LLM Evaluation
Stanford University · Snorkel AI · Harvey
RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator
May 20, 2026Zhenwei Tang, Zhaoyan Liu, Rasa Hosseinzadeh +3LLM Evaluation Pairwise Comparison
Layer 6 AI · University of Toronto
Preferences Order, Ratings Anchor: From Fused Expert Aesthetic Ground Truth to Self-Distillation
May 19, 2026Yuanpei Zhao, Jie Lin, Chao Zhang +5Pairwise Preference Learning Personalized Image Aesthetics Assessment
1Sichuan University · 2NetEase Fuxi AI Lab · †Work done during an internship at NetEase Fuxi AI Lab. +1
CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging
May 19, 2026Zhenyu Li, Aleksandar Cvejic, Zehui Chen +1Pairwise Preference Learning Human Preference Evaluation
KAUST · USTC
Pairwise Preference Reward and Group-Based Diversity Enhancement for Superior Open-Ended Generation
May 18, 2026Guining Cao, Jiaxin Peng, Chu Zeng +3Pairwise Preference Learning Open-Ended Generation
Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd · School of Software and Microelectronics, Peking University · Tsinghua University
CAPS: Cascaded Adaptive Pairwise Selection for Efficient Parallel Reasoning
May 15, 2026Fangzhou Lin, Shuo Xing, Peiran Li +6Pairwise Comparison Pairwise Preference Evaluation
Texas A&M University · Georgia Institute of Technology · Tohoku University +1
Active Learners as Efficient PRP Rerankers
May 14, 2026Jeremías Figueiredo Paschmann, Juan Kaplan, Francisco Nattero +3Pairwise Preference Learning Pairwise Comparison
ELIAS Lab, Departamento de Ingeniería, Universidad de San Andrés
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
May 12, 2026Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van +3Pairwise Preference Learning LLM Alignment
Hanoi University of Science and Technology · University of Stuttgart · German Research Center for Artificial Intelligence +3
Variance-aware Reward Modeling with Anchor Guidance
May 12, 2026Shuxing Fang, Ruijian Han, Liangyu Zhang +1Pairwise Preference Learning Reward Modeling
Department of Data Science and Artificial Intelligence, Hong Kong Polytechnic University · School of Statistics and Data Science, Shanghai University of Finance and Economics
Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges
May 10, 2026Yanran LiLLM Evaluation LLM-as-a-Judge
Department of Biostatistics Columbia University
Efficient Ensemble Selection from Binary and Pairwise Feedback
May 10, 2026Tzeh Yuan Neoh, Nicholas Teh, Je Qin Chooi +2Pairwise Comparison Pairwise Preference Evaluation
Harvard University, USA · University of Oxford, UK
Sufficient conditions for a Heuristic Rating Estimation Method application
May 9, 2026Jacek Szybowski, Konrad Kułakowski, Jiri MazurekPairwise Comparison Pairwise Preference Evaluation
AGH University of Science and Technology, Faculty of Applied Mathematics, Krakow, Poland · AGH University of Science and Technology, Faculty of Electrical Engineering, Automatics, Computer Science and Biomedical Engineering, Krakow, Poland · Silesian University in Opava, Department of Informatics and Mathematics, School of Business Administration in Karvina, Czech Republic
Response Time Enhances Alignment with Heterogeneous Preferences
May 7, 2026Federico Echenique, Alireza Fallah, Baihe Huang +1Pairwise Preference Learning LLM Alignment
Department of Economics, University of California, Berkeley · Department of Computer Science and Ken Kennedy Institute, Rice University · Departments of Electrical Engineering and Computer Sciences, University of California, Berkeley +2
Pair2Score: Pairwise-to-Absolute Transfer for LLM-Based Essay Scoring
May 3, 2026İbrahim Rıza Hallaç, Hasan OğulPairwise Preference Learning Pairwise Comparison
Department of Computer Science and Communication · Østfold University College, Halden 1757, Norway
Multi-User Dueling Bandits: A Fair Approach using Nash Social Welfare
May 3, 2026Maheed H. Ahmed, Mahsa GhasemiPairwise Preference Learning Multi-Armed Bandits
Electrical and Computer Engineering Purdue University

Sep 29, 2026, cs.CLJ/K move · Enter open · S save

Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency

Bruno Brocai, Maria Becker

Heidelberg University German Department

LLM-as-a-Judge Pairwise Preference Evaluation LLM Reliability

Open paper AskPDF arXiv

Large Language Model judges are widely used to rank texts and text-generating systems through pairwise comparison, and their reliability is typically assessed via three proxies: position bias, transitivity, and pairwise agreement (self- or human-labeled). Because these proxies drive judge selection and benchmarking, a substantial literature reporting that judges perform poorly on them risks steering practitioners away from otherwise capable evaluators. We argue this assessment is misleading. Under the Bradley--Terry geometry underlying pairwise aggregation, each proxy is dominated by close-rank-gap pairs, where inconsistency is information-theoretically expected and individual verdicts contribute little to the aggregate ranking; far-gap pairs carry the ranking signal but barely move the proxies. We formalize this argument and validate it in a controlled simulation and on two human-rated corpora: the proxies correlate only weakly with ranking accuracy against gold, and their predictive component concentrates in the far-gap regime. Judges should therefore be assessed on rank-gap-conditional metrics, ideally against human rankings. Code at https://github.com/brunobrocai/PairDifficulty.

Momentum

Related topics

Latest papers 69