Pairwise Preference Evaluation
Overview
Pairwise preference evaluation studies how to compare outputs or systems and turn judgments into rankings, reward signals, or decisions when absolute scores are hard to define. Common approaches include Bradley–Terry and Elo models, comparison graphs, active pair selection, LLM-judge panels, and calibration or uncertainty methods; pairwise labels also support reward modeling and preference-based optimization. Benchmarks cover settings such as chat, code, creative and visual generation, speech, and specialized domains, testing agreement with human judgments and usefulness for downstream tasks. A central tension is that pairwise judgments can be easier to elicit than ratings but remain sensitive to judge and user differences, presentation shortcuts, non-transitive preferences, and too few comparisons, making uncertainty, bias, and task-specific validity important to assess.
Latest papers
80 papers4 papers in the last four weeks, up 33% on the four weeks before. 0.0% of all new papers.