Pairwise Ranking

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 50

Oct 7, 2026cs.LG

How Many Repeated Pairwise Comparisons Are Needed for Ranking under Heterogeneity?

We study ranking models by population-average utility from pairwise comparisons when preferences vary across users and tasks. Prior work shows that a single comparison per user can be insufficient to identify the alternative with the highest average utility, even with arbitrarily many users (Golz et al., 2025). We investigate how many repeated comparisons within each user-task context are necessary and sufficient for ranking recovery. Under a heterogeneous Bradley-Terry model with fixed inverse temperature, we start with a naive MLE-based algorithm that requires

Ω(1/Δ^2)

repeated comparisons per context to ensure ranking recovery. We then present two MLE-based variants and a randomized Russian Roulette-style algorithm that recover the ranking using

O(\log(1/Δ))

repeated comparisons per context, and we prove that this logarithmic dependence is optimal. Despite this worst-case requirement, our Russian Roulette algorithm uses only

O(1)

comparisons per context in expectation. Synthetic experiments and semi-synthetic experiments based on Arena data compare the four algorithms in settings with varying levels of preference heterogeneity and under varying context distributions.

Shashaank Aiyer, Han Shao

University of Maryland

Pairwise Ranking Pairwise Preference Learning Sample Complexity Bradley–Terry Model

Oct 1, 2026cs.LG

Scientific Discovery under Validation Congestion via Multi-Fidelity Pairwise Rankings

Modern computational methods can now propose candidate molecules, materials, and other scientific designs at an unprecedented scale, creating a validation congestion where candidates are abundant, but experimental capacity to physically evaluate them remains scarce. Discovering novel scientific designs has therefore become increasingly dependent on curation: selecting a small set of promising designs for slow and costly experiments. Existing curation methods typically rely on data-driven regression models that predict absolute scores, but training these models requires substantial experimental data to begin with. Yet, useful curation signals do not have to take the form of absolute measurements, as scientific design discovery is often comparative in nature. Here, we propose that curation can instead be primarily driven by expert pairwise rankings, which are substantially easier to gather. The expertise can come from computational tools or human input of multiple levels of fidelity, ranging from empirical rules of thumb to agentic workflows and experienced scientists. We introduce PRISMS, a framework that uses pairwise rankings from one or more experts, potentially spanning multiple levels of expertise, to identify the most promising candidates without relying on data-hungry regressors. When experts differ in fidelity and cost, PRISMS escalates pairwise queries from lower- to higher-fidelity rankers based on a Fisher-information criterion. In iterative screening that selects designs from fixed drug discovery libraries, PRISMS achieves 50% top-10 discovery recall in ~42% fewer rounds than regression-only active learning, and in ~15% fewer rounds than the ranking-based method with no selective escalation. In optimization that generates new designs without restriction to a predefined library, PRISMS achieves ~18.8% higher hypervolume than the Bayesian optimization baseline.

Kevin Tirta Wijaya, Alston Lo, Michael Sun +2

University of Bonn · Fraunhofer SCAI · Massachusetts Institute of Technology

Pairwise Ranking Autonomous Scientific Discovery

Sep 21, 2026stat.ML

OSCAR: Order-aware Scoring and Calibration for AI Rankings

Judge-specific sensitivity is useful for aggregating pairwise LLM evaluations, but its interpretation depends on which systematic presentation effects the ranking model includes. We introduce OSCAR, an order-aware framework for scoring and calibrating AI rankings, and study position as one such effect. In released judgments from 18 evaluators, the all-response A-minus-B score difference ranges from

-63.11

to

98.31

percentage points. Matching question text, response texts, candidate identities, and judge within the released table gives an overall difference of

24.22

points (95% interval

[22.90,25.54]

), conditional on the released text mapping. A controlled calculation isolates the potential consequence: with true sensitivity fixed at one, omitting a position intercept of four reduces the population-optimal slope to

0.0771

. We extend sensitivity-based ranking with judge-specific position, length, and family terms, characterize local omission-induced displacement and an identification failure, and propagate prompt-cluster uncertainty to adjusted comparisons. Across four released datasets, position provides the largest stand-alone predictive improvement. Refitting bootstrap comparisons show more selective gains from the full model over position-only adjustment. In dependent binary simulations, adjusting both the mean and covariance yields 94.4--95.2% coverage; correcting either alone is insufficient. At

N=10{,}000

, OSCAR reduces mean neutral-target RMSE from

0.1158

under the sensitivity-only model to

0.0237

.

You Liu, Yue Liu, Quanchao Lu +1

LLM Evaluation LLM-as-a-Judge Pairwise Ranking Language Model Calibration

Sep 1, 2026cs.LG

CliffRank: A Dual-Branch Framework for Activity-Cliff Ranking Prediction

Activity-cliff ranking remains difficult because local structural changes can cause large activity differences, while high-quality data that resolve the underlying mechanisms remain limited. To use available activity labels more effectively, we combine absolute-activity regression with ranking-consistency learning. CliffRank trains two parallel predictors with mean squared error, a thresholded listwise loss, and Pairwise Preference Consistency (PPC), which aligns relative ordering in the preference-probability space. On three antimicrobial peptide datasets, CliffRank with ESM2-t12 achieved the highest mean Spearman correlation of 0.5393 and mean Recall@50 of 21.4, although the leading method varied across individual datasets. On three small-molecule datasets, CliffRank with PNA, where PPC was activated after 120 epochs, achieved the highest mean Spearman correlation of 0.6890, while its mean Recall@50 of 30.4 matched that of ACANet-PNA. The PPC results also define its practical limits. Asymmetric initialization improved the MolCLR-GIN averages but did not improve every target. For PNA without pretrained weights, delayed PPC improved selected metrics, but no schedule was best for both mean Spearman correlation and mean Recall@50. Future work should evaluate more targets and antimicrobial peptide systems, develop adaptive PPC schedules, and incorporate protein or membrane context when available.

Kewei Li, Rongying Zhang, Peiyu Yang +4

College of Computer Science and Technology, Jilin University, Changchun, 130012, China. · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, 130012, China. · Greenwich High School, Greenwich, CT, 06830, United States of America. +3

Learning to Rank Pairwise Ranking Molecular Property Prediction

Aug 31, 2026cs.CL

Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?

Small leaderboard gaps are often interpreted as evidence that one language model is better than another, but their sign may depend on which benchmark items are included. We test this using item-level responses from five benchmarks and a family-label-free spectral approximation to multidimensional item-response theory (MIRT). In owner-disjoint folds, one owner half identifies items with low residual differential item functioning across model families (low-DIF). These items are used to score models in the other half with frozen weights that preserve the benchmark's composition across metadata-defined item groups and easiness strata. Equally short matched-random subtests provide a baseline for variation due to item subsampling. Full-benchmark and low-DIF rankings remain strongly correlated (

τ_b=.900

--

.948

). Yet in four of five benchmarks, 30.9--47.1% of cross-family pairs initially within one percentage point reverse order, exceeding their matched-random medians by 16.9--28.6 percentage points (all

p=.001

). The fifth benchmark shows no reliable excess (

-0.9

points,

p=.689

). The pattern survives all pre-specified population perturbations, and residual item--family signatures replicate across owner halves; however, no family shows a consistent advantage across benchmarks. Thus, globally stable rankings can still leave individual near-tie orderings sensitive to benchmark composition, and sub-one-point leaderboard gaps should be accompanied by evidence that the implied ordering is composition-robust.

Qiaoyuan Zheng, Yiqu Yang

ETH Zurich Zurich, Switzerland

LLM Evaluation Pairwise Ranking Benchmark Validity Language Model Robustness

Aug 3, 2026cs.LG

Isotonic Bradley-Terry Model for Paired Comparison Data

In this paper, we study prediction problems for paired comparison data, for example, predicting the win probability between two unmatched players and ranking all the players according to the order of their strengths by using win probability data between two matched players. Paired comparison data are typically analyzed using Bradley-Terry and Thurstone-Mosteller models. These models predict the win probability by transforming the difference between learned rate parameters, which represent players';strengths, with a pre-specified inverse link function, and employ the order of learned rate parameters for player ranking. However, these models may suffer from model misspecification owing to the selection of a fixed inverse link function. Therefore, in this study, we propose to learn the rate parameters by a (sub-)gradient method and the inverse link function by an isotonic regression technique alternately. The proposed model guarantees monotonic improvement in training error, and is likely to yield an exact tie when the available data is insufficient to establish a strict ranking. We also verified that the proposed model could improve the win probability prediction and ranking performance through numerical experiments with synthetic data and real-world data of football Premier League, baseball MLB, and tennis ATP tour.

Ryoya Yamasaki

Hitotsubashi Institute for Advanced Study, Hitotsubashi University, 2-1 Naka, Kunitachi, 186-8601, Tokyo, Japan.

Pairwise Preference Learning Pairwise Comparison Pairwise Preference Evaluation Pairwise Ranking Bradley–Terry Model Isotonic Regression

Aug 2, 2026cs.AR

On the Limits of Machine-Learned Ranking for Modern Microarchitectural Policies

Machine-learning predictors estimate processor performance far faster than cycle-level simulation. For design-space exploration, however, the valuable test is not merely reproducing the usual hardware ordering, but identifying how different hardware configurations rank on individual program phases. We evaluate four ML-predictors in two design regimes: \emph{Structural Parameters} (SP), varying hardware resources such as issue width, ROB size, and cache capacity; and \emph{Behavioral Policies} (BP), varying prefetching and replacement algorithms. In the SP regime, aggregate ranking is strong, yet counter-intuitive windows(CIW)---where the configuration expected to be slower is faster---constitute

22.4\%

of non-tied windows across five pairs with a clear architectural prior. CIW match across these pairs is only

23.3

--

39.9\%

; every point estimate is below the

50\%

random strict-ordering reference. The BP regime presents a different failure: ground-truth ties cover

37.8\%

of pair-windows, most strict pairs have margins of only a few cycles, and no model family reliably beats a feature-free majority baseline. NeuroScalar and SimNet fall below that baseline, Concorde is statistically tied with it, and the best selected OneDSE head improves by only

2.1

percentage points. Accuracy rises mainly at large margins. We further show that this failure is not a matter of model capacity: an information-theoretic analysis reveals that when ranking outcomes depend on hidden microarchitectural state absent from the instruction stream, no trace-based predictor can exceed the Bayes accuracy determined by observable inputs alone. Thus high cycle or aggregate ranking accuracy can reflect mastery of easy, high-margin cases while missing the local reversals that carry the most architectural insight and for which cycle-level simulation remains indispensable.

Yanxin Zhang, Shayne Wadle, Yuxuan Xiong +3

NVIDIA · University of Wisconsin–Madison

Learning to Rank Electronic Design Automation Pairwise Ranking

Jul 9, 2026cs.LG

Optimal Top- $k$ Identification from Pairwise Comparisons

We study the active learning problem of fixed-confidence top-

k

identification from noisy pairwise comparisons. In this problem, an algorithm sequentially chooses pairs of items to compare, observes the outcomes, and stops when it can return the set of top-

k

items with error probability at most

δ

. The objective is to design such a

δ

-correct procedure that minimizes the expected number of comparisons (the sample complexity). This problem falls within the broader literature on fixed-confidence pure exploration in bandit models, where a common target is asymptotic optimality: the algorithm's expected sample complexity matches the information theoretic lower bound as

δ\to 0

. Asymptotically optimal procedures have been developed for a range of fixed-confidence pure-exploration problems, however to the best of our knowledge, for top-

1

, or more generally top-

k

identification from pairwise comparisons under latent utility models an asymptotically optimal algorithm has not been established. In this setting, we develop such an algorithm. We characterize the structure of the lower bound and formulate it as a saddle-point problem. This structure enables a computationally efficient primal-dual procedure that learns the asymptotically optimal comparison allocation online. We then construct an adaptive comparison-allocation algorithm that tracks the allocation learned by the primal-dual procedure and prove it is asymptotically optimal.

Motti Goldberger, Nils Rudi

Yale University, New Haven, CT, USA.

Pairwise Preference Learning Multi-Armed Bandits Pairwise Comparison Pairwise Preference Evaluation Pairwise Ranking Best-Arm Identification

Jul 7, 2026cs.AI

Uncovering Latent Depression Severity for Binary Depression Detection via Advantage-weighting Ranking

Automatic depression detection using audio-visual data faces significant challenges, particularly in disentangling overlapping feature distributions and establishing robust decision boundaries. To address this, we propose a fine-grained multimodal framework featuring a temporal encoder and a mutual transformer to facilitate deep cross-modal fusion. Our core contribution is the Binary Advantage-weighting Ranking Loss, which optimizes the latent space distribution through two complementary mechanisms: Advantage-weighted Separation, which mines hard pairs by computing a pairwise prediction difference matrix and dynamically weighting them based on their difficulty; and Advantage-weighted Compactness, which minimizes intra-class variance to force features to cluster around their respective class centers. Extensive experiments on D-vlog and LMVD demonstrate that our model reconstructs the latent ordinal structure by prioritizing hard pairs, thereby achieving state-of-the-art performance.

Manning Gao, Tingyi Liu, Leheng Zhang +3

South China Normal University, China · Sun Yat-sen University, China

Representation Learning Learning to Rank Audio-Visual Representation Learning Depression Detection Pairwise Ranking

Jul 6, 2026cs.AI

Attention Limited Reward Learning

Pairwise human comparisons are a primary interface through which modern AI systems learn human preferences. RLHF and related alignment pipelines typically model such comparisons with Bradley--Terry log-odds, where choice probabilities are governed by latent reward differences. This paper examines what this assumption misses through a reduced-form model motivated by rational inattention, in which each label is generated by a low-capacity evaluation channel. The model separates two forms of ambiguity that standard reward modeling tends to conflate: a comparison may be difficult because the two candidates are genuinely close in value, or because the relevant distinction is hard to detect under limited attention. We show that limited attention can fundamentally distort what pairwise comparisons reveal. In particular, passive comparison data cannot generally distinguish reward, attention, and default tendencies, and heterogeneous attention can make standard Bradley--Terry reward modeling recover misleading rankings. Our analysis shows that learning is governed not by the raw number of labels, but by the amount of attended information each label carries. A case study on human votes over language-model pairs from Chatbot Arena exhibits the predicted signature, a cyclic component of the comparison data that exceeds sampling noise and that no scalar reward can represent; a second case study on perceptual comparisons shows that response times and gaze carry gap information that the labels do not. This perspective suggests that human feedback should be treated not as direct revealed preference, but as an attention-limited measurement process: a weak preference signal may reflect hidden evaluation difficulty rather than genuine indifference.

Wenqian Xing

Management Science and Engineering Department, Stanford University

Pairwise Preference Learning Reward Modeling Pairwise Comparison Pairwise Preference Evaluation Pairwise Ranking Bradley–Terry Model+1

Jul 2, 2026cs.AI

Internal Pluralism and the Limits of Pairwise Comparisons

Local pairwise comparisons are a standard tool for learning how people want decision rules to work, e.g., in participatory design or alignment. However, their use builds in two strong assumptions: that local comparisons are sufficient evidence about how a person wants an automated decision rule to behave, and that people can always answer those comparisons decisively. We investigate how these assumptions may be compromised under internal pluralism: the idea that an individual evaluates decision rules according to multiple authoritative priorities about how the rule should behave. We provide a formal model of such pluralistic preferences over decision rules, which then lets us identify two distinct failures of forced local pairwise comparison data. First, priorities such as proportionality, egalitarianism, and equal treatment are inherently global: what they imply in one case can depend on what happens elsewhere, so local comparisons may fail to capture them. Second, even when priorities are representable locally, tension between strongly-held priorities can generate internal conflict, producing potentially costly behavioral distortions when comparisons are forced. We then use our model to investigate the alternative -- allowing people to report indecision -- and our findings suggest that doing so can considerably reduce the number of queries needed to learn preferences accurately. We conclude by describing how our model points toward preference-learning methods that elicit these priorities directly, yielding more faithful and interpretable accounts of what people value.

Bailey Flanigan, Michelle Si

MIT, USA · Harvard, USA

Pairwise Preference Learning Pairwise Comparison Pairwise Preference Evaluation Human Preference Modeling Multi-Criteria Decision Analysis Pairwise Ranking+2

Jul 2, 2026cs.LG

Ask the Right Comparison:Bias-Aware Bayesian Active Top- $k$ Ranking with LLM Judges

Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise -- to rank responses, select models, or triage papers. Yet LLM judges are both noisy and systematically biased: they favor verbose or well-formatted answers and exhibit position effects, so simply aggregating their votes recovers a ranking of presentation, not of true quality. We study the practical goal of identifying the \topk{} items under a fixed comparison budget, and make two contributions. First, we cast judging as Bayesian inference over latent quality with explicit, judge-specific bias covariates (verbosity, position), regularized by a shrinkage prior so that the data decide which biases a given judge actually exhibits. Second, we introduce a \topk-aware active acquisition rule that chooses the next comparison to maximally reduce uncertainty about \topk{} \emph{membership}, rather than about the full ranking. On a controlled benchmark with known ground-truth quality, judged by sixteen real LLMs spanning open and proprietary families (Llama, Qwen, Phi-4, GPT-4o-mini/5.1/5.5, Gemini, DeepSeek, and Claude Haiku/Sonnet/Opus), naive aggregation plateaus at a wrong \topk{} on biased judges regardless of budget, while our bias-aware model recovers it; \topk-aware acquisition reaches this ceiling with far fewer comparisons than round-robin or a global-uncertainty (D-optimal) rule. Bias is real but heterogeneous and capability-dependent: cheap and mid-tier judges carry a strong verbosity bias that our model corrects (lifting recall from

\sim$$0.5

--

0.6

to

0.84

--

1.0

), whereas the frontier judges we tested show little bias and already rank accurately, so bias-aware modeling changes little there.

Jian Xu, Delu Zeng, John Paisley +1

RIKEN iTHEMS · RIKEN AIP · South China University of Technology +1

Pairwise Comparison LLM-as-a-Judge Learning to Rank LLM Reranking Pairwise Ranking Language Model Bias Evaluation+2

Jun 29, 2026cs.CY

Can LLMs Rank? A Tale of Triads and Triage

From housing allocation for households experiencing homelessness to triage in emergency departments, LLMs are increasingly being considered as judges of consequential decisions that require ranking people for scarce resources. Ranking large groups simultaneously is cognitively demanding and error-prone. A natural solution, drawing on decades of social choice theory, elicits pairwise comparisons and aggregates them into a total order. However, a fundamental question remains when LLMs serve as the pairwise judge: how can a practitioner tell, before committing to a ranking, whether the LLM's judgments are sufficiently consistent to trust the result? We discuss two different ways of identifying consistency. A classical diagnostic, the coefficient of consistency

ζ

, originally developed to measure judge reliability by counting circular triads in tournament graphs, provides a cheap, model-free measure of intra-run consistency. Various standard measures of distance between rankings, for example Kendall's

τ

, can measure inter-run variability. We show, in both theory and practice, that these measures are independently valuable, and advocate for using both to assess reliability of rankings. We demonstrate the practical importance of our results across two high-stakes prioritization tasks: homelessness service allocation and emergency department triage. Three different leading LLMs have considerably different performance profiles across these two axes of consistency. We provide guidelines for how practitioners could think about measuring and assessing consistency before committing to a model for ranking or prioritization.

Gaurab Pokharel, Shafkat Farabi, Patrick J. Fowler +1

Virginia Tech Alexandria, VA · Washington University in St. Louis St. Louis, MO

Pairwise Comparison Social Choice Theory Pairwise Preference Evaluation Pairwise Ranking LLM Reliability LLM Decision-Making+1

Jun 23, 2026cs.LG

The Geometry of Sequential Learning: Lie-Bracket Prediction of Transfer Order

Sequential learning is order-dependent: from Pile-style next-token domain adaptation to instruction-SFT and DPO, N candidate sources induce N! possible curricula. We show that the local order effect is governed by a computable geometric quantity, the Lie-bracket commutator of gradient update fields, yielding a pairwise score for whether A->B or B->A is better for a target domain. The pairwise bracket primitive also defines a Lie-Bracket Tournament: with a shared theta_0 target-gradient reference, Hessian symmetry gives Borda/row-sum scores from one Hessian-vector product per source, O(N) dot products, and an O(N log N) sort, without materializing the O(N^2) edge matrix. Empirically, the planner reaches 98.1%/98.9% pairwise accuracy at k=1 for instruction-SFT/DPO, remains at 73.1%/72.2% at k=20, and preserves the original pretraining-domain evidence with 82.4-92.0% accuracy across four LLMs and 91.1% on diffusion. At curriculum scale, it recovers the best of all 3! schedules in 87.5% of trials, ranks 85 Stack programming-language source domains for a Python target in the 99th sampled percentile, and reaches the 99.0-99.6th sampled percentile on 56 MMLU subjects, sharply above the reported descending gradient-norm baseline. These results reframe sequential learning as a geometric tournament problem: commutators provide both local pairwise order information and a scalable primitive for many-domain schedules.

John Sweeney

Sideplane AI

Pairwise Ranking Cross-Domain Transfer Learning Transfer Learning Curriculum Learning

Jun 23, 2026cs.SD

ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge

Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored. We introduce ParaPairAudioBench, a pairwise benchmark of 5,175 audio pairs across five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender. Our experiments show that current LALM judges still lag behind human judgments by 32%p on average and exhibit severe calibration failures, particularly in Tie cases where the correct decision is to abstain. To further analyze lexical versus acoustic reliance, the benchmark includes both same-transcript and cross-transcript conditions. ParaPairAudioBench enables multi-dimensional, calibration-aware assessment of the reliability of LALM-as-a-Judge for paralinguistic speech evaluation.

Jisu Jeon, Seungyeon Jwa, Joosung Lee +6

Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea +1

Audio-Language Model Evaluation Pairwise Comparison Speech Generation Evaluation Pairwise Preference Evaluation Speech Quality Assessment Pairwise Ranking+1

Jun 16, 2026cs.CL

Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs

Evaluating large language models (LLMs) is important for understanding their capabilities, comparing competing systems, and supporting the deployment of reliable models in practice. For open-ended tasks, pairwise evaluation has become a popular paradigm, in which two responses to the same prompt are compared and the resulting judgments are aggregated into an overall ranking. A central challenge of this paradigm is intransitivity: the induced comparison outcomes may fail to support any coherent global ranking. For example, one may observe cyclic preferences such as

A \succ B \succ C \succ A

, or inconsistencies involving ties such as

A \equiv B\equiv C\neq A

. Such contradictions make the resulting leaderboard unstable and challenging to interpret. In this paper, we propose a prompt perturbation framework for improving the consistency of pairwise LLM evaluation. Our approach generates perturbed variants of each prompt, uses the resulting comparison graphs to identify and filter out structurally inconsistent comparison patterns, and then applies standard ranking methods to the filtered comparisons. A key feature of the proposed framework is that graph-level structural consistency is incorporated explicitly into the evaluation pipeline before ranking aggregation. This provides a simple and principled way to reduce cyclic inconsistencies and improve the reliability of LLM rankings.

Dong Huang, Jianbo Sun, Pengkun Yang

Department of Statistics and Data Science, Tsinghua University

LLM Evaluation Pairwise Comparison Language Model Generation Evaluation Pairwise Preference Evaluation Pairwise Ranking LLM Reliability

Jun 15, 2026cs.SD

TuneJury: An Open Metric for Improving Music Generation Preference Alignment

We introduce TuneJury, an open, instance-level pairwise reward model for text-to-music that predicts a music preference score from a text prompt and an audio clip. The released checkpoint is trained on publicly available human-preference labels covering arena-style (A vs. B) votes, metric-alignment preference pairs, crowdsourced pairwise comparisons, and expert aesthetic ratings. The predicted score margin between two clips is well calibrated on our held-out test split, supporting data filtering via a simple score threshold. TuneJury generalizes to both held-out test pairs and out-of-distribution benchmarks, remaining competitive with prior baselines on the latter. For generators released after training, we introduce anchor calibration, a post-hoc, per-system Bradley-Terry calibration that recovers agreement at substantially better data efficiency than from-scratch retraining. The same frozen reward drives consistent reward-axis gains across three downstream applications: inference-time best-of-N selection, DITTO-style latent optimization, and expert-iteration post-training. TuneJury is available at https://github.com/yonghyunk1m/TuneJury.

Yonghyun Kim, Junwon Lee, Haiwen Xia +5

Georgia Tech · KAIST · Peking University +3

Text-to-Music Generation Music Generation Evaluation Reward Modeling Pairwise Comparison Pairwise Preference Evaluation Pairwise Ranking+2

Jun 14, 2026cs.LG

Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling

Pairwise comparison is the gold standard for subjective ranking tasks; however, exhaustive annotation requires a massive number of human comparisons (

O(n^2)

). While sorting-based methods have reduced this burden to

O(n\log n)

, they still require expensive human judgment for every single comparison. To further improve annotation efficiency, we propose leveraging a Vision-Language Model (VLM) not as an annotator replacement, but as a \emph{question prioritizer} to identify which comparisons genuinely require human judgment. The proposed \textbf{Surprise-Guided MergeSort (SGS)} framework achieves this through three integrated components: (1) a bottom-up MergeSort scheduler that structures comparisons and exploits transitivity, (2) a composite Surprise Scorer -- combining position-bias-cancelled VLM confidence, Elo gap, and vote entropy -- to quantify comparison ambiguity, and (3) an adaptive budget allocator that routes high-surprise pairs to humans while automating low-surprise pairs via transitivity inference. Validation was conducted on six diverse benchmarks spanning text similarity (STS-B, BIOSSES, SICKR-STS) and image quality assessment (KonIQ-10k, TID2013, LIVE Challenge). SGS effectively identified and skipped up to 535 non-informative comparisons per session. Consequently, it achieved Kendall's

τ{\times}100

improvements of

+6

to

+12

over Active Elo under the same total budget. These results demonstrate that combining VLM-guided surprise metrics with algorithmic sorting provides a generally consistent accuracy-efficiency trade-off across diverse domains.

Yujin Park, Haejun Chung, Ikbeom Jang

Hanyang University, Seoul, Republic of Korea · Hankuk University of Foreign Studies, Yongin, Republic of Korea

Pairwise Preference Learning Pairwise Comparison Pairwise Preference Evaluation Learning to Rank Human-in-the-Loop Annotation Active Learning+1

Jun 12, 2026cs.DS

Comparison Patrols on Drifting Orders: Certified Rank Maintenance, Evolving Planar Maxima, and Selection under Drifting Fitness

Rank-based selection in dynamic environments acts on order information that becomes stale while it is being used. Tournaments, elitism, truncation, and Pareto selection may therefore consume rankings that no longer match the current fitness order, while full re-evaluation competes with search for the same budget. This paper formulates the missing information layer as a data-structure problem. A hidden total order on

n

items drifts by adjacent transpositions, while a maintainer receives one truthful pairwise comparison per step and must answer rank queries continuously. We introduce the comparison patrol, a constant-time maintained-order structure using

3n+O(1)

words, one comparison per update, deterministic verification-age bounds, and per-item displacement certificates. We prove lower bounds showing that oblivious and location-oblivious maintainers incur expected Kendall error

Ω(\min(α,1)n)

, and show that the patrol operates at the same order. A bump invariant yields exact self-stabilization after drift-free corruption: if the maximum rank overstatement is

L

, recovery takes at most

L

aligned cycles and cannot finish before

L-1

. This gives a deterministic shock-recovery calculus and a crossover with full rebuild near

L\approx \log_2 n

. The maintained order is then transferred to evolving planar maxima and to evolutionary selection rules, giving deterministic bounds for truncation, tournament, elitist, and two-objective Pareto decisions under drifting fitness. Experiments up to

n=65{,}536

audit the certificates, recovery laws, equilibrium behavior, and equal-budget dynamic evolutionary loops, identifying when certified local rank maintenance outperforms global re-evaluation and when it should hand over.

Faruk Alpay, Levent Sarioglu

Department of Computer Engineering, Bahçeşehir University, Istanbul, Turkey

Pairwise Comparison Evolutionary Optimization Pairwise Ranking Runtime Analysis of Evolutionary Algorithms

Jun 11, 2026cs.LG

From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation

Evaluating new large language models typically requires costly human annotation campaigns at scale. LLM-as-a-judge offers a cheaper alternative, but judge scores carry systematic errors - such as position bias, self-preference, or intransitivity - that can strongly miscalibrate the resulting rankings. We quantify the resulting judge-human disagreement at two complementary levels. At the local level, we estimate per-battle uncertainty from the judge's own score differences by propagating calibrated win probabilities rather than hard labels into the Bradley-Terry procedure. This alone provides a drastic improvement to Elo estimation accuracy, bringing LLM-derived ratings within 17.9 Elo MAE of human-derived ones when averaged over 55 held-out models on LMArena. At the global level, we apply split conformal prediction to the residual gap between LLM-derived and human-derived Elo ratings across held-out models, producing prediction intervals with distribution-free marginal coverage guarantees that account for irreducible LLM-human disagreement. Together, these two layers yield a low-cost evaluation tool that provides developers with calibrated Elo estimates and honest uncertainty bounds, without access to large-scale human annotations. To facilitate reproducibility, we release our code at https://github.com/kargibora/SoftElo .

Bora Kargi, David Salinas

ELLIS Institute Tübingen · OpenEuroLLM

LLM Evaluation Pairwise Comparison LLM-as-a-Judge Pairwise Preference Evaluation Pairwise Ranking Language Model Calibration

Jun 8, 2026cs.AI

Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings

Pairwise comparisons combined with aggregation methods like Elo have become central to evaluating generative models, yet concerns remain that they reward superficial stylistic cues or display judge biases. In a more positive turn, we show that model rankings from pairwise comparisons strongly agree with ground-truth-based accuracy rankings when such ground truth is available for comparison. By converting five well-known benchmarks into free-form generative evaluations, we find that Elo rankings achieve a Spearman correlation above 0.9 with accuracy rankings and substantially outperform direct evaluation when the judge is weak. Furthermore, style and judge bias have only minor effects on model rankings, despite most judgments occurring on pairs where both candidate answers are correct (or incorrect). On such pairs, we find that repetition after the final answer (echo) is a causal driver of judge preference.

Mina Remeli, Moritz Hardt

Max Planck Institute for Intelligent Systems, Tübingen, Germany, Tübingen AI Center

Pairwise Comparison LLM-as-a-Judge Language Model Generation Evaluation Pairwise Preference Evaluation Pairwise Ranking Automated Evaluation+1

Jun 5, 2026cs.IR

Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomies

The ranking of recommendation algorithms is a challenging problem since model performance is sensitive to dataset characteristics such as sparsity, sequential structure, and scale. This drives a demand for a proper methodology for fair comparison between algorithms. Naive aggregation of performance metrics (e.g., averaging NDCG over benchmarks) can yield misleading rankings, undermining practical selection. To address this problem, we introduce a novel, data-driven ranking methodology based on Bradley-Terry (BT) model. We demonstrate that the obtained ranking depends on key dataset statistics. Additionally, we propose a novel metric for evaluating ranking consistency and demonstrate robustness of our ranking to incomplete data. Finally, we introduce a dataset-specific methodology for ranking algorithms on unseen datasets without running the models, relying on extensions of the Bradley-Terry framework, including BT trees and BT models with covariates.

Ekaterina Grishina, Stepan Kuznetsov, Askar Tsyganov +8

HSE University Moscow, Russian Federation

Algorithm Selection Pairwise Comparison Learning to Rank Benchmark Design Pairwise Ranking Recommender Systems+1

Jun 2, 2026cs.SD

A Training-Efficient Transformer-Based Anti-Spoofing Network for Logical Access in ASVspoof 5

Synthetic and manipulated speech can reduce the reliability of automatic speaker verification systems, so anti-spoofing methods need to be both accurate and efficient in training and inference. This paper focuses on the ASVspoof 5 Track 1 closed condition, where standard cross-entropy training may not give enough attention to hard trials and is not directly aligned with ranking- and threshold-based evaluation metrics. We propose TFPARN, a Transformer-based focal-pairwise attentive ranking network. The system extracts log-Mel features from speech, uses a Transformer encoder to model frame-level information, applies attention pooling to obtain utterance-level representations, and is trained with a combination of focal classification loss and pairwise ranking loss. RawBoost augmentation is used during training, and test-time augmentation is applied during evaluation to improve robustness. Compared with re-implemented AASIST and RawNet2 baselines under the same protocol, TFPARN achieves the best results, with a minDCF of 0.2430 and an EER of 12.52%. Ablation experiments further show that the pairwise loss, focal loss, and attention pooling all improve performance. TFPARN also uses the lowest inference memory among the compared systems, at 1.4 GB, runs at about 0.79 ms per utterance, and reaches its best checkpoint in less training time than AASIST. These results show that TFPARN provides a good balance between detection accuracy and computational cost for logical access anti-spoofing.

Sidan Yin, Bo Zhao

San Domenico School, San Anselmo, CA, United States · University of Washington, Seattle, WA, United States

Audio Deepfake Detection Speaker Verification Pairwise Ranking ASVspoof

Jun 1, 2026cs.CV

Ranking vs. Assignment: The Metric Mismatch in Multi-View Object Association

Multi-view object association is an important computer vision problem that underlies many multi-camera perception tasks. While this task is naturally formulated as a constrained one-to-one matching problem, recent works heavily rely on pairwise ranking metrics like AP and FPR-95 for model evaluation. We highlight a fundamental mismatch between these metrics and the actual assignment objective. Theoretically, we show that AP and FPR-95 can be imperfect even when the assignment is already correct, and that Sinkhorn-based normalization can make them perfect. Conversely, optimal pairwise ranking can still lead to incorrect assignments. We validate this mismatch in practice by using our Sinkhorn-based normalization as a controlled post-processing stress test. We show that optimizing just a few post-processing parameters significantly boosts AP and FPR-95 without corresponding improvements in assignment-level metrics such as ACC and IPAA.

Matvei Shelukhan, Timur Mamedov, Aleksandr Chukhrov +1

Tevian Moscow, Russia · Lomonosov Moscow State University Moscow, Russia

Pairwise Ranking Data Association

May 29, 2026cs.GT

The Representation-Rationalizability Tradeoff in Reward Learning

In RLHF, each training example contains a prompt

x

and two candidate responses

y,y'

, and annotators provide pairwise preferences between these responses. The learning problem is to convert these heterogeneous pairwise judgments into a single scalar reward

r(x,y)

that measures response quality for each prompt. Classical social choice implies an impossibility because heterogeneous annotator samples can induce pooled preferences with Condorcet cycles, so no scalar reward can evaluate all compared response pairs consistently. A growing literature analyzes RLHF as a social-choice problem, but usually assumes a fixed finite set of alternatives, i.e., a pre-enumerated finite set of candidate responses for each prompt. Modern pipelines instead score responses through a learned representation

φ(x,y)

before a scalar head, so

φ

determines which responses are treated as distinguishable alternatives and which comparisons are visible to the reward model. Once this embedding is part of the problem, the impossibility results from social choice theory become a tradeoff. We show that the excess cross-entropy loss of any reward built on

φ

decomposes exactly into a representational term, which a richer

φ

shrinks, and an aggregation term, which a richer

φ

enlarges by exposing more comparisons that no scalar can rank consistently. The same results extend to direct preference optimization (DPO), and jointly training the embedding and the reward cannot guarantee to recover the sweet spot of this tradeoff. Experiments on synthetic data and real preference datasets corroborate our results.

Jing Dong, Yaoliang Yu, Pascal Pourpart

Vector Institute · University of Waterloo & Vector Institute

Pairwise Preference Learning Reward Modeling Reinforcement Learning Social Choice Theory Pairwise Preference Evaluation Direct Preference Optimization+3

May 29, 2026cs.CL

Pairwise Reference Alignment as a Model-Level Ordinal Observable

Pairwise preference data is widely used in language-model evaluation and alignment, often for model ranking, reward modeling, or preference optimization. This note formulates a more basic measurement question: given a reference distribution of pairwise preferences, what model-level quantity is estimated when we test whether a model ranks preferred responses above rejected responses? We define pairwise reference alignment as an ordinal observable induced by a model scoring function. Given a reference pair distribution

P_{\mathrm{pair}}

over triples

(x,y^+,y^-)

, and a scalar model score

S_M(x,y)

, we define the alignment observable as the probability that the model-induced ordering agrees with the reference preference ordering. We further define a centered order-parameter-like statistic and discuss a margin-based extension. The resulting quantities admit simple finite-sample estimators and concentration bounds under independent sampling assumptions. This note does not introduce a new benchmark. It provides a conceptual and statistical formulation for pairwise reference alignment, clarifies the role of the reference pair distribution, and distinguishes the general ordinal observable from scoring choices such as normalized log-probability or energy-based scores. We also provide an initial empirical study on Qwen2.5 models and RewardBench, where the proposed statistics increase with model size and instruction tuning and vary across reference-pair subsets as predicted by the formulation.

Mujing Li

Independent Researcher

Human Preference Evaluation LLM Evaluation Pairwise Comparison Pairwise Preference Evaluation Pairwise Ranking Preference Alignment

May 28, 2026stat.ML

Reward Learning from Best-of- $N$ Preference Data: Targets, Tradeoffs, and Design Principles

Best-of-

N

sampling is widely used to construct pairwise preference data:

N

candidates are drawn from a base distribution, and the best is paired with a rejected response. Despite its widespread use, what Bradley--Terry (BT) reward learning extracts from such data, and how to choose

N

and the base distribution, remain unclear. We specialize a recent analysis of preference data via its induced conditional distribution to Best-of-

N

. For independent-reference variants, we derive closed-form reward targets as explicit functions of

N

and the base distribution, and show that they preserve the latent reward ranking. For the practical Best-vs-Random and Best-vs-Worst variants, chosen and rejected responses are coupled through the same candidate set, so exact BT representability generally fails; nevertheless, bounded-class minimizers approach the reference targets as

N

grows. Although margin and connectivity are known to govern sample efficiency in pairwise preference learning, Best-of-

N

couples them through

N

in opposing directions: larger

N

widens pairwise margins but reduces connectivity. This trade-off yields two design principles: use larger

N

when preference labels are the bottleneck, smaller

N

when generation is the bottleneck; and shape the base distribution to place mass between the responses whose comparison matters most at test time. Experiments on synthetic and real preference data support the predicted dependence on sample size and base-distribution shape.

Rattana Pukdee, Maria-Florina Balcan, Pradeep Ravikumar

Machine Learning Department Carnegie Mellon University

Pairwise Preference Learning Reward Modeling Pairwise Comparison Pairwise Preference Evaluation Best-of-N Sampling Pairwise Ranking+2

May 28, 2026cs.CL

Resolution Diagnostics for Paired LLM Evaluation

Across two public LLM leaderboards, many displayed pairwise rankings do not meet a conventional paired-test resolution target under the actual paired evaluation design: 11 of 40 Open LLM Leaderboard v1 pairwise comparisons and 4 of 9 MMLU-Pro top-10 adjacent-rank pairs are unresolved at (alpha, 1-beta) = (0.05, 0.8). The MMLU-Pro count rises to 6/9 under real subject-level clustering and stays at 5-6 out of 9 in 99.9% of category-bootstrap resamples. We frame paired LLM evaluation as a hypothesis-testing problem, invert level-alpha, power-(1-beta) tests, and report a per-pair resolution ratio q = N/N* as the primary diagnostic. A sharp small-effect expansion with an explicit second-order constant shows that the widely-used unpaired Cohen-h-plus-(1-rho) shortcut deviates from the correct N* by approximately a factor of two in the close-comparison regime, a deficit that three of five off-the-shelf calculators(Cohen 1988, G*Power, R pwr) silently inherit when the user post-multiplies their per-arm output by (1-rho). The unresolved-pair pattern remains under multiplicity correction and anytime-valid sequential testing.

Anany Kotawala

Princeton University, Princeton, NJ, USA.

LLM Evaluation Pairwise Comparison Statistical Power Analysis Pairwise Preference Evaluation Pairwise Ranking Statistical Hypothesis Testing

May 27, 2026cs.CL

Argument Quality Assessment with Large Language Models: A Pairwise Bradley-Terry Approach

Large Language Models (LLMs) have demonstrated remarkable capabilities in tasks related to reasoning and judgment. However, assessing the quality of arguments requires a rigorous evaluation. We investigate the extent to which LLMs can effectively perform this task. We tested 12 open-weight LLMs of different sizes and families under zero-shot, few-shot, and chain-of-thought to approximate human pairwise comparisons of argument quality across three dimensions--logical, rhetorical, and dialectic--and used these comparisons in a Bradley-Terry model to infer latent strength scores and derive a ranking of arguments. Our insights show that LLMs have promising but moderate correlation with human judgment, with Llama-70B obtaining the strongest alignment, reaching moderate Cohen's

κ

= 0.493 and moderate correlations with Bradley-Terry scores derived from these annotations (Kendall, Pearson, and Spearman: 0.327-0.477). Other LLMs exhibit weak, moderate, or high alignment with Llama-70B while achieving comparable results against human judgment, suggesting partial but complementary understanding of underlying quality dimensions despite differences in model size and family. Moreover, LLM predictions are stable across trial runs, with fewer than 7.75% of cases yielding different labels. Remaining variability is handled via majority voting and few-shot prompting for large-size models.

Nicolás Benjamín Ocampo, Agnes Paullate Nyiranziza, Davide Ceolin

1Centrum Wiskunde & Informatica, The Netherlands · 2Vrije Universiteit Amsterdam, The Netherlands

LLM Evaluation Pairwise Ranking

May 25, 2026cs.CL

Preferred, Not Safer: Pairwise Preference Is a Poor Proxy for Clinical Safety

We evaluate whether clinician pairwise preferences provide a reliable signal of clinical safety in large language model (LLM) evaluation using expert feedback from MOOVE (Massive Open Online Validation and Evaluation), a clinician-led platform collecting blinded pairwise preferences alongside multi-criterion rubric ratings. Clinicians assign scores on a discrete

[-2, +2]

scale, where negative values indicate clinically unsafe or misleading content. Using 26{,}804 pairwise judgments across outputs from 13 LLMs, contributed by more than 736 clinicians across 28+ countries, we find that clinician preference is a poor proxy for safety-critical performance. Models ranking highly under pairwise preference can still exhibit substantial rates of clinically meaningful failures (

\leq -1

) on dimensions such as \emph{Harmlessness} and \emph{Accuracy}. These failures are unevenly distributed across specialties, creating domain-specific ``no-go zones'' not visible in aggregate rankings or single-number leaderboards. We further analyze contributing factors including prompt length, refusal and escalation behavior, and the relative contributions of safety-critical versus surface-level features. A substantial fraction of preference votes carry no positive safety signal, while feature decomposition shows that surface-level characteristics explain slightly more preference variation than safety-critical rubric differences. Finally, we introduce a clinically adjusted preference ranking combining pairwise preference with rubric-derived feedback, producing a more safety-aware ordering than raw Bradley--Terry strength alone. Our findings support evaluation practices that separate preference from safety, report safety-critical failure rates directly, and incorporate clinically grounded adjustments when ranking LLMs for clinical decision making.

Fay Elhassan, David Sasu, Alexandra Kulinkina +2

LiGHT Laboratory, EPFL, Lausanne, Switzerland

LLM Safety Benchmarks Human Preference Evaluation Pairwise Comparison Pairwise Preference Evaluation Healthcare Clinical Decision Support+4