A Probabilistic Approach for Model Alignment with Human Comparisons

Junyu Cao; Mohsen Bayati

cs.LGMar 16, 2024

A Probabilistic Approach for Model Alignment with Human Comparisons

Authors: Junyu Cao, Mohsen Bayati

Organizations: McCombs School of Business, The University of Texas at Austin · Graduate School of Business, Stanford University

Abstract

A growing trend involves integrating human knowledge into learning frameworks, leveraging subtle human feedback to refine AI models. While these approaches have shown promising results in practice, the theoretical understanding of when and why such approaches are effective remains limited. This work takes steps toward developing a theoretical framework for analyzing the conditions under which human comparisons can enhance the traditional supervised learning process. Specifically, this paper studies the effective use of noisy-labeled data and human comparison data to address challenges arising from noisy environment and high-dimensional models. We propose a two-stage "Supervised Learning+Learning from Human Feedback" (SL+LHF) framework that connects machine learning with human feedback through a probabilistic bisection approach. The two-stage framework first learns low-dimensional representations from noisy-labeled data via an SL procedure and then uses human comparisons to improve the model alignment. To examine the efficacy of the alignment phase, we introduce a concept, termed the "label-noise-to-comparison-accuracy" (LNCA) ratio. This paper identifies from a theoretical perspective the conditions under which the "SL+LHF" framework outperforms the pure SL approach; we then leverage this LNCA ratio to highlight the advantage of incorporating human evaluators in reducing sample complexity. We validate the framework on a real high-dimensional crowdfunding-prediction task: under a fixed query budget, trading labels for comparisons improves accuracy precisely when labels are scarce, and the findings hold when the evaluator is replaced by real large language models. A study conducted via Amazon Mechanical Turk (MTurk) further validates the model primitives.

Figures & tables

$N$	Pure SL	SL+LHF ( $\gamma=0.25$ )	SL+LHF ( $\gamma=0.01$ )
$100$	$-0.009$	$\mathbf{0.060}$	$\mathbf{0.062}$
$150$	$0.062$	$\mathbf{0.093}$	$\mathbf{0.094}$
$200$	$0.107$	$\mathbf{0.119}$	$\mathbf{0.120}$
$250$	$0.149$	$\mathbf{0.151}$	$\mathbf{0.149}$
$300$	$\mathbf{0.176}$	$0.169$	$0.170$
$500$	$\mathbf{0.249}$	$0.233$	$0.234$

Table 1: Kickstarter under a fixed query budget (both methods spend exactly

N

queries): test

R^{2}

(mean over

30

seeds) of pure SL and SL+LHF, versus

N

, for two expertise levels

\gamma

. Entries in bold match or exceed pure SL .

		$\gamma=0.25$			$\gamma=0.01$
$N$	SL	SL+HF	SL+DPO	SL+DPO (on ACS queries)	SL+HF	SL+DPO	SL+DPO (on ACS queries)
100	$-0.009$	$\mathbf{0.060}$	$-0.022$	$0.038$	$\mathbf{0.062}$	$-0.019$	$0.042$
150	$0.062$	$\mathbf{0.093}$	$0.055$	$0.069$	$\mathbf{0.094}$	$0.057$	$0.071$
200	$0.107$	$\mathbf{0.119}$	$0.090$	$0.081$	$\mathbf{0.120}$	$0.101$	$0.087$
250	$0.149$	$\mathbf{0.151}$	$0.106$	$0.071$	$\mathbf{0.149}$	$0.130$	$0.067$
300	$\mathbf{0.176}$	$0.169$	$0.140$	$0.124$	$0.170$	$0.157$	$0.111$

Table 2: Kickstarter at fixed query parity (equal total query budget

N

): test

R^{2}

(mean over

30

seeds) of our refinement (SL+HF) and two RLHF-style offline preference fits, for two expertise levels

\gamma

. SL+DPO uses passively sampled comparisons; the “on ACS queries” variant is fit on the comparison pairs that SL+HF generated. All share the same Stage-1 warm start, support, comparison belief, and budget; pure SL is

\gamma

-independent. Bold marks the best entry in each

\gamma

block.

$N$	Pure SL	SL+HF (Haiku 4.5)	SL+HF (Sonnet 4.6)	SL+HF (Opus 4.6)
$100$	$-0.019$	$\mathbf{0.044}$	$\mathbf{0.067}$	$\mathbf{0.065}$
$150$	$0.038$	$\mathbf{0.084}$	$\mathbf{0.102}$	$\mathbf{0.092}$
$200$	$0.102$	$\mathbf{0.108}$	$\mathbf{0.111}$	$\mathbf{0.114}$
$250$	$0.136$	$0.133$	$\mathbf{0.146}$	$\mathbf{0.143}$
$300$	$\mathbf{0.176}$	$0.151$	$0.172$	$0.171$
$500$	$\mathbf{0.249}$	$0.109$	$0.212$	$0.192$

Table 3: Kickstarter with real LLM comparators, at fixed query budget: test

R^{2}

(mean over ten seeds) versus the total query budget

N

. Each API call shows a batch of projects with two candidate predicted ratios each, in per-project randomized order. Entries in bold match or exceed pure SL .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Error ratio of the two-stage framework to the pure SL with different values of $\sigma$ .

Figure 5: Error ratio of the two-stage framework to the pure SL with different values of $\gamma$ .

Figure 6: Error ratio of the two-stage framework to the pure SL with different values of $s$ .

Figure 9: Empirical error of the two-stage framework for different values of $\varepsilon$ .

Figure 10: Error ratio of the two-stage framework to the pure SL for different values of $\varepsilon$ .

Explore similar work

Jul 6, 2026cs.AI

Attention Limited Reward Learning

Pairwise human comparisons are a primary interface through which modern AI systems learn human preferences. RLHF and related alignment pipelines typically model such comparisons with Bradley--Terry log-odds, where choice probabilities are governed by latent reward differences. This paper examines what this assumption misses through a reduced-form model motivated by rational inattention, in which each label is generated by a low-capacity evaluation channel. The model separates two forms of ambiguity that standard reward modeling tends to conflate: a comparison may be difficult because the two candidates are genuinely close in value, or because the relevant distinction is hard to detect under limited attention. We show that limited attention can fundamentally distort what pairwise comparisons reveal. In particular, passive comparison data cannot generally distinguish reward, attention, and default tendencies, and heterogeneous attention can make standard Bradley--Terry reward modeling recover misleading rankings. Our analysis shows that learning is governed not by the raw number of labels, but by the amount of attended information each label carries. A case study on human votes over language-model pairs from Chatbot Arena exhibits the predicted signature, a cyclic component of the comparison data that exceeds sampling noise and that no scalar reward can represent; a second case study on perceptual comparisons shows that response times and gaze carry gap information that the labels do not. This perspective suggests that human feedback should be treated not as direct revealed preference, but as an attention-limited measurement process: a weak preference signal may reflect hidden evaluation difficulty rather than genuine indifference.

Wenqian Xing

Task-Aware Evaluation Framework

Ask PDF

May 12, 2026cs.LG

Learning to Decide with AI Assistance under Human-Alignment

It is widely agreed that when AI models assist decision-makers in high-stakes domains by predicting an outcome of interest, they should communicate the confidence of their predictions. However, empirical evidence suggests that decision-makers often struggle to determine when to trust a prediction based solely on this communicated confidence. In this context, recent theoretical and empirical work suggests a positive correlation between the utility of AI-assisted decision-making and the degree of alignment between the AI confidence and the decision-makers' confidence in their own predictions. Crucially, these findings do not yet elucidate the extent to which this alignment influences the complexity of learning to make optimal decisions through repeated interactions. In this paper, we address this question in the canonical case of binary predictions and binary decisions. We first show that this problem is equivalent to a two-armed online contextual learning problem with full feedback, and establish a lower bound of

Ω(\sqrt{|H| \cdot |B| \cdot T} )

on the expected regret any learner can attain, where

H

and

B

denote the sets of human and AI confidence values. We then demonstrate that, under perfect alignment between AI and human confidence, a learner can attain an expected regret of

O(\sqrt{|H| \cdot T\log T})

and, when

\sqrt{|H|} = O(\log T)

and

B

is countable, a non-trivial generalization of the Dvoretzky-Kiefer-Wolfowitz inequality improves the regret bound to

O(\sqrt{T\log T})

. Taken together, these results reveal that alignment can reduce the complexity of learning to make decisions with AI assistance. Experiments on real data from two different human-subject studies where participants solve simple decision-making tasks assisted by AI models show that our theoretical results are robust to violations of perfect alignment.

Nina Corvelo Benz, Eleni Straitouri, Manuel Gomez-Rodriguez

Artificial Intelligence Alignment Preference Learning Inequalities

Ask PDF

Sep 28, 2026cs.LG

ABC-Align: Prediction-Powered Alignment with Adaptive Bias Control

Language model post-training is often bottlenecked by the need for human-collected preference data, which is expensive and difficult to scale. Reinforcement learning from AI feedback (RLAIF) style approaches that leverage pseudo labels offer an abundant alternative but introduce systematic biases that degrade downstream alignment. Recent general-purpose semi-supervised methods correct for teacher bias using a small set of human-labeled examples, but suffer from high variance especially when human annotations are scarce. To this end, we propose ABC-Align, leveraging abundant pseudo label signal to minimize variance and applying a lightweight, adaptive correction grounded in the human-labeled subset. The correction strength is tuned automatically during training using plug-in estimates of the relevant bias--variance quantities. On LLM alignment with RLHF, DPO, and GRPO where human feedback is scarce, we empirically demonstrate that ABC-Align achieves superior performance over prior semi-supervised baselines in a series of experiments on an increasing scale. Our code is available at https://github.com/SewoongLab/abc-align .

Eric Frankel, Banghua Zhu, Sewoong Oh +1

Large Language Model Alignment Latent Representation Alignment

Ask PDF