Pairwise Preference Learning

Ask about this topic

Overview

Pairwise preference learning studies how to infer what people value from choices between alternatives, rather than requiring absolute scores. Common approaches include Bradley–Terry reward models, direct preference optimization for language models, and active learning or dueling-bandit methods that select informative comparisons. The methods support applications from ranking and recommendations to AI alignment, robotics, and policy learning, including settings with noisy labels or limited annotation budgets. A central challenge is that preferences may be heterogeneous, context-dependent, or cyclic, so a single consistent utility score can miss important differences between people and situations.

Latest papers

CardsList

126 papers26 papers in the last four weeks, up 136% on the four weeks before. 0.2% of all new papers.

May 10, 2026

Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs

Existing preference datasets for text-to-image models typically store only the final winner/loser images. This representation is insufficient for rectified flow (RF) models, whose generation is naturally indexed by a specific prior noise sample and follows a nearly straight denoising trajectory. In contrast, prior DPO-style alignment for diffusion models commonly estimates trajectories using an independent forward noising process, which can be mismatched to the true reverse dynamics and introduces unnecessary variance. We propose Prior Noise-Aware Preference Optimization (PNAPO), an off-policy alignment framework specialized for rectified flow. PNAPO augments preference data by retaining the paired prior noises used to generate each winner/loser image, turning the standard (prompt, winner, loser) triplet into a sextuple. Leveraging the straight-line property of RF, we estimate intermediate states via noise-image interpolation, which constrains the trajectory estimation space and yields a tighter surrogate objective for preference optimization. In addition, we introduce a dynamic regularization strategy that adapts the DPO regularization based on (i) the reward gap between winner and loser and (ii) training progress, improving stability and sample efficiency. Experiments on state-of-the-art RF T2I backbones show that PNAPO consistently improves preference metrics while substantially reducing training compute.

Yunhong Lu, Qichao Wang, Hengyuan Cao +2

Zhejiang University · Shanghai Institute for Advanced Study-Zhejiang University · Shanghai Institute for Mathematics and Interdisciplinary Sciences

T2I GenerationRectified FlowText-Image AlignmentPairwise Preference LearningFlow-Based Generative ModelingOffline Preference Optimization+1
AskPDF
May 9, 2026

ξξ-DPO: Direct Preference Optimization via Ratio Reward Margin

Reference-free preference optimization has emerged as an efficient alternative to reinforcement learning from human feedback, with Simple Preference Optimization(SimPO) demonstrating strong performance by eliminating the explicit reference model through a simple objective. However, the joint tuning of the hyperparameters ββ and γγ in SimPO remains a central challenge. We argue that this difficulty arises because the margin formulation in SimPO is not easily interpretable across datasets with different reward gap structures. To better understand this issue, we conduct a comprehensive analysis of SimPO and find that ββ implicitly controls sample filtering, while the effect of γγ depends on the reward gap structure of the dataset. Motivated by these observations, we propose ξξ-DPO: Direct preference optimization via ratio reward margin. We first reformulate the preference objective through an equivalent transformation, changing the optimization target from maximizing the likelihood of reward gaps to minimizing the distance between reward gaps and optimal margins. Then, we redefine the reward in a ratio form between the chosen and rejected, which effectively cancels the effect of ββ and yields a bounded and interpretable margin. This margin is called the ratio reward margin and is denoted by ξξ. Unlike the margin γγ in SimPO, ξξ explicitly represents the desired relative separation between chosen and rejected responses and can be determined from the initial reward gap distribution, avoiding repeated trial-and-error tuning. ....

Zhengyuan Fan, Zhonghua Wu, Yuxuan Du +1

School of Computer Science, Northwestern Polytechnical University

Pairwise Preference LearningDirect Preference OptimizationPreference OptimizationRL Post-Training
AskPDF
May 8, 2026

Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph

Direct Preference Optimization (DPO) aligns language models using pairwise preference comparisons, offering a simple and effective alternative to Reinforcement Learning (RL) from human feedback. However, in many practical settings, training data consists of multiple rollouts per prompt, inducing rich preference structure that pairwise DPO fails to exploit. Collapsing such data into independent pairs discards transitivity, introduces redundant or conflicting supervision, and can lead to unstable optimization. We propose Graph Direct Preference Optimization (GraphDPO), a structured generalization of DPO that operates over directed acyclic preference graphs induced by rollout rankings. GraphDPO encodes dominance relations as edges and optimizes a graph-structured Plackett--Luce-inspired objective that aggregates supervision over dominated neighborhoods, propagating the transitive structure of the input preferences while recovering standard DPO as a special case. GraphDPO naturally handles tied and incomparable responses through an equivalence-class construction: responses with identical preference signals form graph layers, incomparable responses share a layer without an edge, and absent edges contribute zero loss, preventing the spurious gradients that strict total-order objectives incur. Under this layered construction, GraphDPO maintains linear per-prompt complexity via efficient log-sum-exp aggregation. We further incorporate optional ground-truth anchoring by inserting verified solutions as dominant nodes and applying an annealed schedule that stabilizes early training while gradually relaxing oracle supervision. Experiments on reasoning and program synthesis tasks demonstrate superior performance, suggesting that graph-structured preference modeling is a scalable and robust alternative to pairwise and listwise alignment objectives.

Ning Liu, Chuanneng Sun, Kristina Klinkner +1

Amazon

Pairwise Preference LearningLLM AlignmentDirect Preference OptimizationPreference OptimizationPreference-Based RL
AskPDF
May 7, 2026

Response Time Enhances Alignment with Heterogeneous Preferences

Aligning large language models (LLMs) to human preferences typically relies on aggregating pooled feedback into a single reward model. However, this standard approach assumes that all labelers share the same underlying preferences, ignoring the fact that real-world labelers are highly heterogeneous and usually anonymous. Consequently, relying solely on binary choice data fundamentally distorts the learned policy, making the true population-average preference unidentifiable. To overcome this critical limitation, we demonstrate that augmenting preference datasets with a simple, secondary signal -- the user's response time -- can restore the identifiability of the population's average preference. By modeling each decision as a Drift-Diffusion Model (DDM), we introduce a novel, consistent estimator of heterogeneous preferences that successfully corrects the distortions of standard choice-only labels. We prove that our estimator asymptotically converges to the true average preference even in extreme cases where each anonymous labeler contributes only a single choice. Empirically, across both synthetic and real-world datasets, our method consistently outperforms standard baselines that otherwise fail and plateau at a bias floor. Because response times are essentially free to record and require zero user tracking or identification, our results bring promises and open up new opportunities for future data-collection pipelines to improve the social benefit without requiring user-level identifiers or repeated elicitations.

Federico Echenique, Alireza Fallah, Baihe Huang +1

Department of Economics, University of California, Berkeley · Department of Computer Science and Ken Kennedy Institute, Rice University · Departments of Electrical Engineering and Computer Sciences, University of California, Berkeley +2

Pairwise Preference LearningLLM AlignmentPairwise Preference EvaluationHuman Preference ModelingPreference LearningPreference Elicitation
AskPDF
May 7, 2026

A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment

Large language model (LLM) alignment via reinforcement learning from human preferences (RLHF) suffers from unstable policy updates, ambiguous gradient directions, poor interpretability, and high gradient variance in mainstream pairwise preference learning paradigms. To systematically address these limitations, we establish a unified theoretical framework for preference-based RL optimization centered on the Pair-GRPO family, comprising two tightly coupled variants: Soft-Pair-GRPO and Hard-Pair-GRPO. Soft-Pair-GRPO is a minimal modification of Group Relative Policy Optimization (GRPO) that replaces group-normalized scalar rewards with binary pairwise preference rewards, retaining GRPO's clipped surrogate and KL-regularized structure. We prove a critical gradient equivalence theorem: under first-order Taylor expansion around the current policy, Soft-Pair-GRPO's gradient is a positive scalar multiple of standard GRPO's gradient, explaining its empirical stability despite discarding continuous reward magnitudes. Building on this foundation, we propose Hard-Pair-GRPO, an advanced variant introducing explicit local probability constraints and constrained KL-fitting optimization to further suppress gradient noise and global policy drift. We provide comprehensive theoretical guarantees for both variants--including monotonic policy improvement, deterministic gradient direction, gradient-variance reduction, and dynamic step-size convergence. Extensive experiments on standard LLM alignment benchmarks (HH-RLHF,UltraFeedback) and the MuJoCo continuous control task HalfCheetah-v4 demonstrate that our Pair-GRPO family consistently outperforms state-of-the-art baselines in alignment quality, human preference win rate, training stability, and generalization to general reinforcement learning. Ablation studies validate the critical contributions of each core component.

Hao Yu

Department of Automation, Tsinghua University

Pairwise Preference LearningLLM AlignmentGroup Relative Policy OptimizationConstrained RLRL Post-TrainingPreference-Based RL
AskPDF
May 3, 2026

Pair2Score: Pairwise-to-Absolute Transfer for LLM-Based Essay Scoring

Many scoring applications require absolute predictions, while pairwise comparisons can provide a simpler learning objective. We present Pair2Score, a two-stage learning framework that transfers pairwise comparisons into absolute scoring with parameter-efficient LLaMA adaptation. Stage 1 trains a directional Siamese ranker on pairwise comparisons derived from absolute trait labels; Stage 2 trains an absolute predictor using configurable transfer strategies (warm-start and embedding-fusion variants). We evaluate on rubric-aligned Automated Essay Scoring (AES) traits (grammar, vocabulary, syntax) under a five-fold protocol that co-rotates held-out fold and random seed. At the trait level, the best-performing transfer variant improves quadratic weighted kappa (QWK) over an absolute-only baseline for all three traits. However, not all transfer configurations help: a one-epoch pairwise stage transfers more reliably than extended pairwise training, and transfer configuration -- not just the inclusion of a pairwise stage -- determines whether downstream scoring benefits.

İbrahim Rıza Hallaç, Hasan Oğul

Department of Computer Science and Communication · Østfold University College, Halden 1757, Norway

Pairwise Preference LearningPairwise ComparisonPairwise Preference EvaluationAutomated Essay ScoringPairwise Ranking
AskPDF
May 3, 2026

Multi-User Dueling Bandits: A Fair Approach using Nash Social Welfare

Learning from human preference data is becoming a useful tool, from fine-tuning large language models to training reinforcement learning agents. However, in most scenarios, the model is trained on the average preference of all human evaluators, which, under large variations of preferences, can be unfair to minority groups. In this work, we consider fairness in dueling bandits, a standard framework for online learning from preference data. We assume that each user has a (potentially distinct) Condorcet winner, which is an arm preferred to every other arm. Using these user-specific Condorcet winners as reference points, we evaluate and score arms according to their performance relative to the corresponding winner. To promote fairness across heterogeneous users, we adopt the well-established Nash Social Welfare objective, which maximizes the product of user utilities, thereby inherently penalizing inequality and preventing the marginalization of any single user. Within this framework, we construct a hard instance to establish a regret lower bound of Ω(T2/3min⁡(K,D)13)Ω(T^{2/3}\min(K,D)^\frac{1}{3}) for a time horizon TT, KK arms, and DD users, which, to the best of our knowledge, is the first result quantifying the cost of fairness in dueling bandits with heterogeneous preferences. We then present the Fair-Explore-Then-Commit and Fair-εε-Greedy algorithms with a Condorcet winner identification phase. We further derive their regret upper bounds that match the lower-bound dependence on TT up to logarithmic factors.

Maheed H. Ahmed, Mahsa Ghasemi

Electrical and Computer Engineering Purdue University

Pairwise Preference LearningMulti-Armed BanditsPairwise ComparisonPairwise Preference EvaluationGroup FairnessDueling Bandits
AskPDF
Apr 30, 2026

Mind the Gap: Structure-Aware Consistency in Preference Learning

Aligning Large Language Models (LLMs) with human intent, whether through explicit reward modeling or direct methods such as DPO, fundamentally relies on minimizing a surrogate loss as a proxy for the true pairwise ranking objective. We prove that this reliance is flawed for the standard surrogate losses used: for the equicontinuous hypothesis sets characteristic of neural networks, no standard surrogate provides a meaningful consistency guarantee. Minimizing the surrogate loss to zero can leave the true ranking error arbitrarily high. To resolve this, we formulate LLM alignment within a margin-shifted ranking framework and derive HH-consistency bounds showing that enforcing a confidence margin γγ is not merely beneficial but necessary for consistency. We further introduce Structure-Aware HH-consistency and a corresponding objective (SA-DPO) that adapts the margin to the semantic distance between responses, preventing instability on near-synonymous pairs. Finally, we analyze the trade-off between the margin required for consistency and the model's finite capacity to satisfy it, revealing a strict hierarchy of surrogate losses: heavy-tailed surrogates (e.g., the Polynomial Hinge family) offer strictly superior consistency guarantees for capacity-bounded models compared to the logistic loss used in DPO. Experiments on UltraFeedback and Argilla DPO-Mix-7k confirm that SA-DPO consistently outperforms DPO and SimPO, with a 58.5% head-to-head win-rate in downstream generation quality.

Mehryar Mohri, Yutao Zhong

Google Research, New York, NY · Courant Institute of Mathematical Sciences, New York, NY

Pairwise Preference LearningLLM AlignmentLearning to RankPreference AlignmentPreference Learning
AskPDF
Apr 27, 2026

Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimization

Human visual preferences are inherently multi-dimensional, encompassing aesthetics, detail fidelity, and semantic alignment. However, existing datasets provide only single, holistic annotations, resulting in severe label noise: images that excel in some dimensions but are deficient in others are simply marked as winner or loser. We theoretically demonstrate that compressing multi-dimensional preferences into binary labels generates conflicting gradient signals that misguide Diffusion Direct Preference Optimization (DPO). To address this, we propose Semi-DPO, a semi-supervised approach that treats consistent pairs as clean labeled data and conflicting ones as noisy unlabeled data. Our method starts by training on a consensus-filtered clean subset, then uses this model as an implicit classifier to generate pseudo-labels for the noisy set for iterative refinement. Experimental results demonstrate that Semi-DPO achieves state-of-the-art performance and significantly improves alignment with complex human preferences, without requiring additional human annotation or explicit reward models during training. We will release our code and models at: https://github.com/L-CodingSpace/semi-dpo

Xinxin Liu, Ming Li, Zonglin Lyu +2

University of Central Florida

Pairwise Preference LearningDiffusion Model AlignmentPairwise Preference EvaluationDirect Preference OptimizationNoisy-Label LearningPreference Alignment+1
AskPDF
Apr 23, 2026

Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages

Crowdsourced pairwise evaluation has emerged as a scalable approach for assessing foundation models. However, applying it to Text to Speech(TTS) introduces high variance due to linguistic diversity and multidimensional nature of speech perception. We present a controlled multidimensional pairwise evaluation framework for multilingual TTS that combines linguistic control with perceptually grounded annotation. Using 5K+ native and code-mixed sentences across 10 Indic languages, we evaluate 7 state-of-the-art TTS systems and collect over 120K pairwise comparisons from over 1900 native raters. In addition to overall preference, raters provide judgments across 6 perceptual dimensions: intelligibility, expressiveness, voice quality, liveliness, noise, and hallucinations. Using Bradley-Terry modeling, we construct a multilingual leaderboard, interpret human preference using SHAP analysis and analyze leaderboard reliability alongside model strengths and trade-offs across perceptual dimensions.

Srija Anand, Ashwin Sankar, Ishvinder Sethi +10

Indian Institute of Technology Madras, India · AI4Bharat, India · Josh Talks, India

Pairwise Preference LearningHuman Preference EvaluationPairwise ComparisonTTS SynthesisSpeech Generation EvaluationPairwise Preference Evaluation+4
AskPDF
Apr 22, 2026

Vibrotactile Preference Learning: Uncertainty-Aware Preference Learning for Personalized Vibration Feedback

Individual differences in vibrotactile perception underscore the growing importance of personalization as haptic feedback becomes more prevalent in interactive systems. We propose Vibrotactile Preference Learning (VPL), a system that captures user-specific preference spaces over vibrotactile parameters via Gaussian-process-based uncertainty-aware preference learning. VPL uses an expected information gain-based acquisition strategy to guide query selection over 40 rounds of pairwise comparisons of overall user preference, augmented with user-reported uncertainty, enabling efficient exploration of the parameter space. We evaluate VPL in a user study (N = 13) using the vibrotactile feedback from a Microsoft Xbox controller, showing that it efficiently learns individualized preferences while maintaining comfortable, low-workload user interactions. These results highlight the potential of VPL for scalable personalization of vibrotactile experiences.

Rongtao Zhang, Xin Zhu, Masoume Pourebadi Khotbehsara +3

Department of Computer Science University of Southern California Los Angeles, California, USA

Pairwise Preference LearningActive Preference LearningHaptic FeedbackUser Preference ModelingPreference Learning
AskPDF
Apr 20, 2026

Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner

Preference optimization is widely used to align large language models (LLMs) with human preferences. However, many margin-based methods also suppress the chosen response when they try to suppress the rejected one, and there is no general way to prevent this across different objectives. We address this issue with a unified incentive-score decomposition of preference optimization, revealing that different objectives share the same local update directions and differ only in their scalar weights. This decomposition provides a common framework for analyzing objectives that were previously studied in separate settings. Building on this decomposition, by analyzing the dynamics of the chosen/rejected likelihoods, we identify the disentanglement band (DB), a simple, testable condition that tells us when training can follow the desired path: suppress the loser while preserving the winner, possibly after an early stage. Using the DB, we propose reward calibration (RC), a plug-and-play method that adaptively rebalances the updates for chosen and rejected responses to satisfy the DB, without redesigning the base objective. Empirical results show that RC leads to more disentangled dynamics, with better downstream performance observed across several settings. Our code is available at https://github.com/IceyWuu/DisentangledPreferenceOptimization.

Wei Chen, Yubing Wu, Junmei Yang +5

1South China University of Technology · 2AIP, RIKEN · 3Columbia University +1

Pairwise Preference LearningLLM AlignmentPreference Optimization
AskPDF
Apr 14, 2026

lmfaoooo at SemEval-2026 Task 1: Humor Is an Audience. Preference Modeling for Constrained Humor Generation

Humor generation remains difficult not only because producing fluent, novel jokes is hard, but because "funny" is audience-dependent and supervision is noisy -- preferences vary with audience, context, and culture, and annotator agreement is often low. In this paper, we describe our system for the SemEval-2026 Task-1 (MWAHAHA), which focuses on humor generation under explicit constraints. The task evaluates submitted systems via human preference judgments in 1-on-1 arena-style comparisons. We adopt a "generate-many -> select-best" strategy. First, we generate a diverse pool of candidates per instance using multi-step prompting, model ensembling, and diversity-oriented decoding. Second, we select outputs using a preference model that approximates a "reader" by learning from human comparisons rather than absolute funniness scores. To support this approach, we release 2.5K human pairwise judgments collected through the Humor Arena prototype. We further propose an interpretable pipeline that converts labeled comparisons into a preference model. Across three preference datasets, our models consistently outperform baselines and show stronger cross-domain transfer. Finally, we apply the learned preference model to rank candidates for the MWAHAHA setting and release intermediate artifacts (candidate pools and rankings) to facilitate follow-up work. Our system ranked 1st in the English and Chinese subtasks of MWAHAHA and 2nd in the Spanish subtask.

Alexey Tikhonov, Alexey Ivanov

Inworld.AI Berlin, Germany · OpenAI Mountain View, CA

Humor Explanation GenerationText GenerationHuman Preference ModelingPairwise Preference LearningLanguage Model Generation EvaluationControllable Text Generation
AskPDF
Apr 11, 2026

From Recency Bias to Stable Convergence Block Kaczmarz Methods for Online Preference Learning in Matchmaking Applications

We present a family of Kaczmarz-based preference learning algorithms for real-time personalized matchmaking in reciprocal recommender systems. Post-step L2 normalization, common in Kaczmarz-inspired online learners, induces exponential recency bias: the influence of the t-th interaction decays as eta^(n - t), reaching approximately 1e-6 after just 20 swipes at eta = 0.5. We resolve this by replacing the normalization step with a Tikhonov-regularized projection denominator that bounds step size analytically without erasing interaction history. When candidate tag vectors are not pre-normalized, as in realistic deployments where candidates vary in tag density, the Tikhonov denominator ||a||^2 + alpha produces genuinely per-candidate adaptive step sizes, making it structurally distinct from online gradient descent with any fixed learning rate. We further derive a block variant that processes full swipe sessions as a single Gram matrix solve. Population-scale simulation over 6,400 swipes reveals that Block Normalized Kaczmarz (BlockNK), which combines the batch Gram solve with post-session L2 normalization, achieves the highest preference alignment (Align@20 = 0.698), the strongest inter-session direction stability (delta = 0.994), and the flattest degradation profile under label noise across flip ratios p_flip in [0.10, 0.35]. Experiments under cosine similarity subsampling further show that adaptively filtering the candidate pool toward the current preference direction substantially improves asymptotic alignment, at the cost of introducing a feedback loop that may slow recovery from miscalibration. The sequential Tikhonov-Kaczmarz method performs comparably to K-NoNorm under our simulation conditions, suggesting the dominant practical gain over normalized Kaczmarz is the removal of per-step normalization rather than the Tikhonov constant alpha itself.

James Nguyen

Department of Mathematics, University of California, Irvine.

Preference LearningRecommender SystemsPreference AlignmentPairwise Preference LearningIterative Refinement
AskPDF
Apr 10, 2026

Efficient Personalization of Generative User Interfaces

Generative user interfaces (GenUIs) create new opportunities to adapt interfaces to individual users on demand. Yet personalization is difficult because it is not possible to provide settings for screens that have not yet been generated, making it necessary to learn preferences from users' feedback on generated interfaces. Such feedback is sparse, subjective, and often difficult to articulate. We study this problem through a new dataset in which 20 participants each judge the same 600 pairs of GenUIs. Their judgments differ substantially (Krippendorff's alpha = 0.25), and written rationales show that even when participants consider similar UI properties, such as color tone or hierarchy, they often make different choices and prioritize them differently. We develop a sample-efficient personalization method that uses a few pairwise judgments to weight prior users' preferences rather than relying on a fixed rubric of UI attributes. Our method outperforms a pretrained UI evaluator and a larger multimodal model offline. In an online study with 12 new users, participants preferred interfaces personalized by our method over all baselines, including shared guidelines and users' own written preferences.

Yi-Hao Peng, Jeffrey P. Bigham, Jason Wu

Massachusetts Institute of Technology · Carnegie Mellon University · Purdue University

Pairwise Preference LearningHuman Preference ModelingUser Preference ModelingPreference Learning
AskPDF
Apr 10, 2026

Relational Preference Encoding in Looped Transformer Internal States

We investigate how looped transformers encode human preference, training lightweight evaluator heads on frozen Ouro-2.6B loop-iteration states on Anthropic HH-RLHF. v2: an erratum is prepended; the original manuscript is unchanged. A post-publication audit found the three headline results inflated by two independent evaluation errors. The 95.2% pairwise evaluator accuracy is a canonical-ordering artifact: the data were correctly split, but the evaluator learned to prefer the first-presented argument; its strict antisymmetrized accuracy on the full 8,552-pair test set is 63.9%. The 84.5% pairwise probe and the below-chance 21.75% pointwise probe were source-item leaks (orientation rows and pair partners crossing the train/test split); corrected pair-disjoint values are 56.5% and 54.2% -- above chance, so the "inverted polarity" finding is withdrawn. The central finding survives at much smaller magnitude: preference is decoded more accurately relationally than pointwise (paired +2.3 points, 95% CI [+1.3, +3.3]), and the antisymmetrized evaluator still beats the linear probe, but no corrected readout rivals end-to-end reward models. The methodological findings stand (constant-output degeneracy, flip test, swap-protocol metric deflation), with one correction: antisymmetrized accuracy, not antisymmetry correlation, certifies relational discrimination. The two errors are mutually invisible -- a split audit cannot see an ordering prior, antisymmetrization cannot see a leak -- so both checks are required. Full audit in the follow-up work (Kirin, 2026, in preparation).

Jan Kirin

Pairwise Preference LearningReward ModelingTransformer InterpretabilityPairwise ComparisonPairwise Preference EvaluationPairwise Ranking+2
AskPDF
Apr 9, 2026

Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?

Preference optimization methods such as DPO and KTO are widely used for aligning language models, yet little is understood about what properties of preference data drive downstream reasoning gains. We ask: what aspects of a preference pair improve a reasoning model's performance on general reasoning tasks? We investigate two distinct notions of quality delta in preference data: generator-level delta, arising from the differences in capability between models that generate chosen and rejected reasoning traces, and sample-level delta, arising from differences in judged quality differences within an individual preference pair. To study generator-level delta, we vary the generator's scale and model family, and to study sample-level delta, we employ an LLM-as-a-judge to rate the quality of generated traces along multiple reasoning-quality dimensions. We find that increasing generator-level delta steadily improves performance on out-of-domain reasoning tasks and filtering data by sample-level delta can enable more data-efficient training. Our results suggest a twofold recipe for improving reasoning performance through preference optimization: maximize generator-level delta when constructing preference pairs and exploit sample-level delta to select the most informative training examples.

Chia-Hsuan Lee, Mingyang Zhou, Renkun Ni +6

Capital One

Preference OptimizationPairwise Preference LearningEfficient Language Model ReasoningRL for Language Model ReasoningData Selection
AskPDF
Apr 6, 2026

Soft Tournament Equilibrium: Differentiable Set-Valued Inference for Non-Transitive Pairwise Comparisons

Soft Tournament Equilibrium (STE) is a differentiable layer for Top-Cycle (TC) and Uncovered-Set (UC) inference from reciprocal pair probabilities. Normalized log-sum-exp reachability and covering give smooth scores with approximation, perturbation, and margin-recovery bounds. We distinguish structural supervision, posterior uncertainty, and the final set decision through controlled synthetic studies and reconstruction of recorded ordinal profiles. Matched-epoch training improves F1 under a common structural readout, but a separate equal-budget soft-UC comparison does not demonstrate an advantage. A prospective equal-budget hard-UC study improves selective F1 at 24 alternatives from 0.6327/0.6292 for ordinary/relational native heads to 0.6676, while exact recovery remains only 1.16%. On 36 human profiles held out by source, a separate exploratory comparison favors Jeffreys posterior inference over learned independent-edge and mixture distributions in native expected-F1 decoding (0.8742 versus 0.8704/0.8366). Only three references are non-singleton selective cores. Completion certificates and full-depth TC diagnostics clarify additional identification, smoothing, and computational limits. The evidence supports conditional synthetic overlap gains, while preserving the soft-set null and the absence of a demonstrated learning advantage over strong count-based human-profile controls.

Saad Alqithami

Pairwise Preference LearningSocial Choice Theory
AskPDF
Apr 1, 2026

Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences

Learning human preferences in language models remains fundamentally challenging, as reward modeling relies on subtle, subjective comparisons or shades of gray rather than clear-cut labels. This study investigates the limits of current approaches and proposes a feature-augmented framework to better capture the multidimensional nature of human judgment. Using the Anthropic HHRLHF dataset, we evaluate ten diverse large language models LLMs under a standard pairwise preference setting, where baseline performance remains below 0.74 ROC AUC, highlighting the difficulty of the task. To address this, we enrich textual representations with interpretable signals: response length, refusal indicators, toxicity scores and prompt response semantic similarity, enabling models to explicitly capture key aspects of helpfulness, safety and relevance. The proposed hybrid approach yields consistent improvements across all models, achieving up to 0.84 ROC AUC and significantly higher pairwise accuracy, with DeBERTav3Large demonstrating the best performance. Beyond accuracy, we integrate SHAP and LIME to provide fine-grained interpretability, revealing that model decisions depend on contextualized safety and supportive framing rather than isolated keywords. We further analyze bias amplification, showing that while individual features have weak marginal effects, their interactions influence preference learning.

Simona-Vasilica Oprea, Adela Bâra

Department of Economic Informatics and Cybernetics, Bucharest University of Economic Studies, Bucharest, Romania

Reward ModelingPairwise Preference Learning
AskPDF
Mar 30, 2026

Active Preference Learning over Latent Preference Archetypes for Many-Objective Bayesian Optimization

Preference-based many-objective Bayesian optimization typically assumes that all pairwise comparisons arise from a single latent utility function, despite real decision makers often exhibiting multiple latent preference archetypes across contexts. We propose an active preference learning framework for many-objective Bayesian optimization that infers latent preference archetypes from pairwise comparisons, enabling both the identification of the active trade-off strategy and the refinement of its associated preferences. Our framework represents preferences as a Dirichlet-process mixture of latent archetypes and introduces mixture-aware information-theoretic query strategies that separately target archetype identification and within-archetype refinement through a hybrid acquisition policy. Experiments on synthetic benchmarks and real-world chemical process design case-study consistently outperforms state-of-the-art preference-based Bayesian optimization methods while recovering interpretable latent preference structure beyond conventional single-utility models. The proposed mixture-aware diagnostics further quantify archetype recovery and preference calibration, providing insights that are not captured by optimization performance alone.

Manisha Dubey, Sebastiaan De Peuter, Wanrong Wang +1

Department of Computer Science, University of Manchester, UK · Centre for AI in Assistive Autonomy, University of Edinburgh, Scotland, UK · Department of Computer Science, Aalto University, Espoo, Finland +2

Active Preference LearningBayesian OptimizationPreference LearningMulti-Objective OptimizationPairwise Preference Learning
AskPDF
Mar 20, 2026

Regret Bounds for Reinforcement Learning from Multi-Source Imperfect Preferences

Reinforcement learning from human feedback (RLHF) replaces hard-to-specify rewards with pairwise trajectory preferences, yet regret-oriented theory often assumes that preference labels are generated consistently from a single ground-truth objective. In practical RLHF systems, however, feedback is typically multi-source (annotators, experts, reward models, heuristics) and can exhibit systematic, persistent mismatches due to subjectivity, expertise variation, and annotation/modeling artifacts. We study episodic RL from multi-source imperfect preferences through a cumulative imperfection budget: for each source, the total deviation of its preference probabilities from an ideal oracle is at most ωω over KK episodes. We propose a unified algorithm with regret O~(K/M+ω)\tilde{O}(\sqrt{K/M}+ω), which exhibits a best-of-both-regimes behavior: it achieves MM-dependent statistical gains when imperfection is small (where MM is the number of sources), while remaining robust with unavoidable additive dependence on ωω when imperfection is large. We complement this with a lower bound Ω~(max⁡{K/M,ω})\tildeΩ(\max\{\sqrt{K/M},ω\}), which captures the best possible improvement with respect to MM and the unavoidable dependence on ωω, and a counterexample showing that naïvely treating imperfect feedback as oracle-consistent can incur regret as large as Ω~(min⁡{ωK,K})\tildeΩ(\min\{ω\sqrt{K},K\}). Technically, our approach involves imperfection-adaptive weighted comparison learning, value-targeted transition estimation to control hidden feedback-induced distribution shift, and sub-importance sampling to keep the weighted objectives analyzable, yielding regret guarantees that quantify when multi-source feedback provably improves RLHF and how cumulative imperfection fundamentally limits it.

Ming Shi, Yingbin Liang, Ness B. Shroff +1

Department of Electrical Engineering, University at Buffalo, Buffalo, NY 14260, USA · Department of Electrical and Computer Engineering, The Ohio State University, Columbus, OH 43210, USA · Department of Electrical and Computer Engineering and Computer Science and Engineering, The Ohio State University, Columbus, OH 43210, USA +1

Preference-Based RLRL from Human FeedbackRegret Minimization in RLPairwise Preference Learning
AskPDF
Mar 19, 2026

Best-of-Both-Worlds Multi-Dueling Bandits: Unified Algorithms for Stochastic and Adversarial Preferences under Condorcet and Borda Objectives

Multi-dueling bandits, where a learner selects m≥2m \geq 2 arms per round and observes only the winner, arise naturally in many applications including ranking and recommendation systems, yet a fundamental question has remained open: can a single algorithm perform optimally in both stochastic and adversarial environments, without knowing which regime it faces? We answer this affirmatively, providing the first best-of-both-worlds algorithms for multi-dueling bandits under both Condorcet and Borda objectives. For the Condorcet setting, we propose MetaDueling\texttt{MetaDueling}, a black-box reduction that converts any dueling bandit algorithm into a multi-dueling bandit algorithm by transforming multi-way winner feedback into an unbiased pairwise signal. Instantiating our reduction with Versatile-DB\texttt{Versatile-DB} yields the first best-of-both-worlds algorithm for multi-dueling bandits: it achieves O(KT)O(\sqrt{KT}) pseudo-regret against adversarial preferences and the instance-optimal O(∑i≠a⋆log⁡TΔi)O\left(\sum_{i \neq a^\star} \frac{\log T}{Δ_i}\right) pseudo-regret under stochastic preferences, both simultaneously and without prior knowledge of the regime. For the Borda setting, we propose SA-MiDEX\texttt{SA-MiDEX}, a stochastic-and-adversarial algorithm that achieves O(K2log⁡KT+Klog⁡2T+∑i:ΔiB>0Klog⁡KT(ΔiB)2)O\left(K^2 \log KT + K \log^2 T + \sum_{i: Δ_i^{\mathrm{B}} > 0} \frac{K\log KT}{(Δ_i^{\mathrm{B}})^2}\right) regret in stochastic environments and O(KTlog⁡KT+K1/3T2/3(log⁡K)1/3)O\left(K \sqrt{T \log KT} + K^{1/3} T^{2/3} (\log K)^{1/3}\right) regret against adversaries, again without prior knowledge of the regime. We complement our upper bounds with matching lower bounds for the Condorcet setting. For the Borda setting, our upper bounds are near-optimal with respect to the lower bounds (within a factor of KK) and match the best-known results in the literature.

S Akash, Pratik Gajane, Jawar Singh

Indian Institute of Technology Patna, India · University of Orléans, France

Dueling BanditsMulti-Armed BanditsAdversarial BanditsPairwise Preference Learning
AskPDF
Mar 18, 2026

From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation

Large language models (LLMs) are currently applied to scientific paper evaluation by assigning an absolute score to each paper independently. However, since score scales vary across conferences, time periods, and evaluation criteria, models trained on absolute scores are prone to fitting narrow, context-specific rules rather than developing robust scholarly judgment. To overcome this limitation, we propose shifting paper evaluation from isolated scoring to collaborative ranking. In particular, we design a C\textbf{C}omparison-N\textbf{N}ative framework for P\textbf{P}aper E\textbf{E}valuation (CNPE\textbf{CNPE}), integrating comparison into both data construction and model learning. We first propose a graph-based similarity ranking algorithm to facilitate the sampling of more informative and discriminative paper pairs from a collection. We then enhance relative quality judgment through supervised fine-tuning and reinforcement learning with comparison-based rewards. At inference, the model performs pairwise comparisons over sampled paper pairs and aggregates these preference signals into a global relative quality ranking. Experimental results demonstrate that our framework achieves an average relative improvement of 21.8% over the strong baseline DeepReview-14B, while exhibiting robust generalization to five previously unseen datasets. Our code is available at https://github.com/ECNU-Text-Computing/ComparisonReview.

Pujun Zheng, Jiacheng Yao, Jinquan Zheng +6

School of Economics and Management, East China Normal University · School of Information Management, Wuhan University · China Academic Degrees & Graduate Education Development Center

Automated Peer ReviewScholarly Peer ReviewLearning to RankPairwise Preference LearningLLM Evaluation
AskPDF
Mar 9, 2026

Improving through Interaction: Searching Behavioral Representation Spaces with CMA-ES-IG

Robots that interact with humans must adapt to individual users' preferences to operate effectively in human-centered environments. An intuitive and effective technique to learn non-expert users' preferences is through rankings of robot behaviors, e.g., trajectories, gestures, or voices. Existing techniques primarily focus on generating queries that optimize preference learning outcomes, such as sample efficiency or final preference estimation accuracy. However, the focus on outcome overlooks key user expectations in the process of providing these rankings, which can negatively impact users' adoption of robotic systems. This work proposes the Covariance Matrix Adaptation Evolution Strategies with Information Gain (CMA-ES-IG) algorithm. CMA-ES-IG explicitly incorporates user experience considerations into the preference learning process by suggesting perceptually distinct and informative trajectories for users to rank. We demonstrate these benefits through both simulated studies and real-robot experiments. CMA-ES-IG, compared to state-of-the-art alternatives, (1) scales more effectively to higher-dimensional preference spaces, (2) maintains computational tractability for high-dimensional problems, (3) is robust to noisy or inconsistent user feedback, and (4) is preferred by non-expert users in identifying their preferred robot behaviors. This project's code is available at github.com/interaction-lab/CMA-ES-IG

Nathaniel Dennler, Zhonghao Shi, Yiran Tao +3

CSAIL/AeroAstro, Massachusetts Institute of Technology. · Department of Computer Science, University of Southern California.

Active Preference LearningPreference LearningPreference ElicitationHuman-Robot InteractionPairwise Preference LearningCovariance Matrix Adaptation Evolution Strategy
AskPDF
Mar 9, 2026

CRED: Counterfactual Reasoning and Environment Design for Active Preference Learning

As a robot's operational environment and tasks to perform within it grow in complexity, the explicit specification and balancing of optimization objectives to achieve a preferred behavior profile moves increasingly farther out of reach. These systems benefit strongly by being able to align their behavior to reflect human preferences and respond to corrections, but manually encoding this feedback is infeasible. Active preference learning (APL) learns human reward functions by presenting trajectories for ranking. However, existing methods sample from fixed trajectory sets or replay buffers that limit query diversity and often fail to identify informative comparisons. We propose CRED, a novel trajectory generation method for APL that improves reward inference by jointly optimizing environment design and trajectory selection to efficiently query and extract preferences from users. CRED "imagines" new scenarios through environment design and leverages counterfactual reasoning -- by sampling possible rewards from its current belief and asking "What if this were the true preference?" -- to generate trajectory pairs that expose differences between competing reward functions. Comprehensive experiments and a user study show that CRED significantly outperforms state-of-the-art methods in reward accuracy and sample efficiency and receives higher user ratings.

Yi-Shiuan Tung, Gyanig Kumar, Wei Jiang +2

Department of Computer Science, University of Colorado Boulder

Active Preference LearningPreference LearningPreference-Based RLReward ModelingPairwise Preference LearningReinforcement Learning
AskPDF
Mar 5, 2026

When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger

Preference alignment is an essential step in adapting large language models (LLMs) to human values, but existing approaches typically depend on costly human annotations or large-scale API-based models. We explore whether a weak LLM can instead act as an effective annotator. We surprisingly find that selecting only a subset of a weak LLM's highly confident samples leads to substantially better performance than using full human annotations. Building on this insight, we propose Confidence-Weighted Preference Optimization (CW-PO), a general framework that re-weights training samples by a weak LLM's confidence and can be applied across different preference optimization objectives. Notably, the model aligned by CW-PO with just 20% of human annotations outperforms the model trained with 100% of annotations under standard DPO. These results suggest that weak LLMs, when paired with confidence weighting, can dramatically reduce the cost of preference alignment while even outperforming methods trained on fully human-labeled data.

Amirabbas Afzali, Myeongho Jeon, Maria Brbic

EPFL · Sharif University of Technology

Preference AlignmentPreference OptimizationPairwise Preference LearningLLM-Assisted Annotation
AskPDF
Mar 1, 2026

PARWiS: Winner determination under shoestring budgets using active pairwise comparisons

Determining a winner among a set of items using active pairwise comparisons under a limited budget is a challenging problem in preference-based learning. The goal of this study is to implement and evaluate the PARWiS algorithm, which shows spectral ranking and disruptive pair selection to identify the best item under shoestring budgets. This work have extended the PARWiS with a contextual variant (Contextual PARWiS) and a reinforcement learning-based variant (RL PARWiS), comparing them against baselines, including Double Thompson Sampling and a random selection strategy. This evaluation spans synthetic and real-world datasets (Jester and MovieLens), using budgets of 40, 60, and 80 comparisons for 20 items. The performance is measured through recovery fraction, true rank of reported winner, reported rank of true winner, and cumulative regret, alongside the separation metric Δ1,2Δ_{1,2}. Results show that PARWiS and RL PARWiS outperform baselines across all datasets, particularly in the Jester dataset with a higher Δ1,2Δ_{1,2}, while performance gaps narrow in the more challenging MovieLens dataset with a smaller Δ1,2Δ_{1,2}. Contextual PARWiS shows comparable performance to PARWiS, indicating that contextual features may require further tuning to provide significant benefits.

Shailendra Bhandari

Department of Computer Science OsloMet– Oslo Metropolitan University Oslo, Norway

Dueling BanditsPairwise Preference LearningPairwise ComparisonPreference-Based RLMulti-Armed Bandits
AskPDF
Feb 26, 2026

Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferences

We consider the problem of regularized best-response max-regret minimization in online RLHF under general preferences and bandit feedback. While various regularizers are utilized to robustify alignment, known polylogarithmic regret guarantees remain heavily specific to KL. To investigate whether such fast rates extend beyond KL, we adopt the Generalized Bilinear Preference Model (GBPM) -- capturing intransitive preferences over dd-dimensional item-wise features via a rank-2r2r skew-symmetric matrix -- to isolate the impact of generic regularization. Crucially, under GBPM, we prove that the dual gap of any greedy policy is bounded by the squared estimation error, derived using only strong convexity and skew-symmetry. Under a feature coverage assumption, we establish a generic polylogarithmic regret of O~(ηd4Cmin⁡−1(log⁡T)2∧d2Cmin⁡−1/2T)\tilde{\mathcal{O}}(ηd^4 C_{\min}^{-1} (\log T)^2 \wedge d^2 C_{\min}^{-1/2} \sqrt{T}) with Greedy Sampling, and a dimension-wise improved regret (for well-conditioned arm-sets) of O~(Cmin⁡−2ηrT∧r1/3Cmin⁡−4/3T2/3)\tilde{\mathcal{O}}(C_{\min}^{-2} \sqrt{ηr T} \wedge r^{1/3} C_{\min}^{-4/3} T^{2/3}) with Explore-Then-Commit, where η−1η^{-1} is the regularization coefficient, TT is the time horizon, and Cmin⁡C_{\min} is an arm-set dependent quantity. This demonstrates that ``fast'' regrets are not KL-specific, but rather a fundamental consequence of generic strongly convex geometry.

Junghyun Lee, Minju Hong, Kwang-Sung Jun +2

KAIST AI · KAIST EE · POSTECH CSE/AI

Regret Minimization in RLPreference-Based RLRL from Human FeedbackPreference LearningPairwise Preference Learning
AskPDF
Feb 18, 2026

Learning Preference from Observed Rankings

Estimating consumer preferences is central to many problems in economics and marketing. This paper develops a flexible framework for learning individual preferences from partial ranking information by interpreting observed rankings as collections of pairwise comparisons with logistic choice probabilities. We model latent utility as the sum of interpretable product attributes, item fixed effects, and a low-rank user-item factor structure, enabling both interpretability and information sharing across consumers and items. We further correct for selection in which comparisons are observed: a comparison is recorded only if both items enter the consumer's consideration set, inducing exposure bias toward frequently encountered items. We model pair observability as the product of item-level observability propensities and estimate these propensities with a logistic model for the marginal probability that an item is observable. Preference parameters are then estimated by maximizing an inverse-probability-weighted (IPW), ridge-regularized log-likelihood that reweights observed comparisons toward a target comparison population. To scale computation, we propose a stochastic gradient descent (SGD) algorithm based on inverse-probability resampling, which draws comparisons in proportion to their IPW weights. In an application to transaction data from an online wine retailer, the method improves out-of-sample recommendation performance relative to a popularity-based benchmark, with particularly strong gains in predicting purchases of previously unconsumed products.

Yu-Chang Chen, Chen Chian Fuh, Shang En Tsai

National Taiwan University

Preference ModelingUser Preference ModelingPairwise Preference LearningRecommender SystemsDiscrete Choice ModelingCollaborative Filtering
AskPDF
Feb 2, 2026

Towards Optimality in Active Preference Learning

Aligning large language models (LLMs) depends on high-quality datasets of human preference labels, which are costly to collect. Although active learning has been studied to improve sample efficiency relative to passive collection, many existing approaches adopt classical experimental design criteria such as G- or D-optimality. These objectives are not tailored to the structure of preference learning, leaving open the design of problem-specific algorithms. In this work, we identify a simple intuition specific to preference learning that calls into question the suitability of these existing design objectives. Motivated by this insight, we propose two active learning algorithms. The first provides the first instance-dependent label complexity guarantee for this setting, and the second is a simple, practical greedy method. We show the optimality of our instance-dependent term up to a constant depending on the logit bound, compared with a lower-bound characteristic. We evaluate our algorithms on real-world preference datasets and synthetic data observing improved sample efficiency compared to existing methods.

Yao Zhao, Kwang-Sung Jun

Department of Computer Science, University of Arizona, Tucson, USA.

Active Preference LearningActive LearningPreference LearningPairwise Preference LearningLLM Alignment
AskPDF