What Does Disagreement Mean? A Semantic Identifiability Frontier for Pairwise Preference Learning

Manoj Saravanan; Rohit Kumar Salla; Shrikar Reddy Kota

cs.GTOct 14, 2026

What Does Disagreement Mean? A Semantic Identifiability Frontier for Pairwise Preference Learning

Authors: Manoj Saravanan, Rohit Kumar Salla, Shrikar Reddy Kota

Organizations: Virginia Tech, Blacksburg, VA, USA.

Abstract

Pairwise comparison data are a standard supervision signal in preference learning, but disagreement in such data can reflect stable latent heterogeneity, ephemeral stochasticity, or non-binary responses hidden by a forced-choice interface. We study which of these semantic distinctions are identifiable from the feedback channel itself. We introduce a nonparametric latent semantic model with three querywise quantities: aggregate directional preference $μ_q$ , persistent disagreement $τ_q$ , and non-binary mass $α_q$ . Under the anonymous binary channel, we prove an exact semantic fiber theorem: $μ_q$ is point-identified, whereas $(τ_q, α_q)$ vary over an explicit feasible region compatible with the same binary law. Same-annotator repeats identify $τ_q$ through a covariance identity, abstention-enabled feedback identifies $α_q$ directly, and together they point-identify the full semantic triple. We also derive finite-sample estimators, nonasymptotic concentration bounds, and semantics-aware action-recovery guarantees, and validate the theory on theorem-matched synthetic data.

Figures & tables

Figure 1: Exact semantic fiber under anonymous binary feedback. For $\mu^{\star}=0.40$ , panel (a) shows the feasible set $\mathcal{F}(\mu^{\star})$ and the constructive worlds from Theorem 3.1 ; panel (b) colors the same worlds by the residual $\widehat{\mu}-\mu^{\star}$ . Across the full feasible region, the induced anonymous binary law is essentially unchanged.

Figure 2: Semantics-aware action recovery under channel refinement. The three regimes use thresholds $t_{\tau}=t_{\alpha}=0.15$ : Denoise $(0.02,0.02)$ , Personalize $(0.35,0.02)$ , and Query $(0.02,0.35)$ . All three regimes are constructed to share the same empirical binary-law distribution by reusing the same set of $\mu$ -values across regimes. Panel (a) plots action accuracy versus calibration budget $b$ , with $n_{R}=n_{A}=b$ and $n_{B}=512$ . Panels (b)–(d) show row-normalized confusion matrices at representative budget $b=16$ for $\mathsf{BR}$ , $\mathsf{A}$ , and $\mathsf{BRA}$ . Repeats and abstention recover complementary action dimensions, while the combined channel recovers the full semantics-aware policy.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Line of work	Representative references	Primary object of analysis	How this paper differs
Aggregate recovery from pairwise data	Pukdee et al. (2026)	The conditional pairwise preference distribution (CPRD), i.e., the aggregate preference object encoded by pairwise data	We study the latent semantic preimage of a fixed binary law. Theorem 3.1 characterizes the exact fiber over $(\mu_{q})$ , which is a different question from recovering or representing the aggregate law itself.
Hidden context in RLHF / distributional preference learning	Siththaranjan et al. (2023)	How hidden context induces aggregation effects (e.g. Borda-type aggregation) and how utility distributions can account for it	We abstract away from any particular score-based learning rule and ask what the feedback channel identifies about disagreement semantics before one chooses an aggregation mechanism.
Heterogeneous alignment with user types	Shirali et al. (2025)	Alignment when user types are heterogeneous and extra annotator information may be available	We do not assume a target average-reward objective or a user-type model. Our results are upstream: they characterize when disagreement semantics are identifiable from data at all.
Richer feedback for latent heterogeneity	Chidambaram et al. (2025)	Identifiability of latent preference heterogeneity under binary versus ternary or ranking-based feedback	We study a more primitive question: whether observed disagreement reflects persistent latent variation, non-binary mass hidden by forced choice, or both. Our refinements are repeats and abstention rather than rankings.
Empirical taxonomies of disagreement	Zhang et al. (2025)	Empirical causes of disagreement in real preference datasets	These works motivate our latent decomposition, but do not provide exact nonparametric identifiability regions or channel-refinement theorems.

Appendix

Table 1: Novelty map relative to the closest lines of work.

Symbol	Meaning
$\mathcal{Q}$	Finite set of ordered pairwise comparison queries.
$q\in\mathcal{Q}$	A fixed query; labels $\pm 1$ are interpreted relative to the ordering encoded by $q$ .
$(\mathcal{Z},\mathcal{F},\pi)$	Latent type space and population distribution over stable hidden types.
$Z\sim\pi$	Latent type of a sampled annotator.
$\widetilde{Y}\in\{+1,0,-1\}$	Latent ternary response: strict preference for the first alternative, non-binary response, or strict preference for the second alternative.
$Y\in\{+1,-1\}$	Observed binary response after forced completion of $\widetilde{Y}=0$ .

Appendix

Table 2: Core notation for the formal setup.

Figure 3: Strict channel frontier via witness worlds. Rows compare the three explicit one-query witness pairs: Coin–Zero , Coin–Split , and Zero–Split . Columns correspond to the observable channels $\mathsf{B}$ , $\mathsf{BR}$ , $\mathsf{A}$ , and $\mathsf{BRA}$ . Each cell reports the total variation distance between the two observable laws under the given channel. Panel (a) shows the exact distances implied by Theorem 4.4 , while panel (b) shows Monte Carlo estimates that recover the same strict-refinement pattern.

Figure 4: Recovery of persistent disagreement from repeats. Queries are generated with $\alpha_{q}=0$ , $\mu_{q}\sim\mathrm{Unif}[-0.8,0.8]$ , and $\tau_{q}=c_{q}(1-\mu_{q}^{2})$ with $c_{q}\sim\mathrm{Unif}[0,0.8]$ . Panel (a) shows querywise recovery of $\tau_{q}$ from repeat covariance at representative repeat budget $n_{R}=32$ . Panel (b) reports mean squared error versus repeat budget $n_{R}$ , together with a $1/n_{R}$ reference line and the bias-corrected estimator from Appendix F . Panel (c) validates the disagreement-gap identity $2(D_{q}^{\mathrm{diff}}-D_{q}^{\mathrm{same}})=\tau_{q}$ . All estimates in panels (a) and (c) are averaged over 250 Monte Carlo replications.

Figure 5: Recovery of non-binary mass from abstention. Queries are generated with $\mu_{q}\sim\mathrm{Unif}[-0.8,0.8]$ , $\alpha_{q}=b_{q}(1-|\mu_{q}|)$ with $b_{q}\sim\mathrm{Unif}[0,0.7]$ , and $\tau_{q}=0$ . Panel (a) shows querywise recovery of $\alpha_{q}$ at representative abstention budget $n_{A}=32$ . Panel (b) reports mean squared error versus abstention budget $n_{A}$ , together with the exact theoretical Bernoulli MSE $\frac{1}{|\mathcal{Q}|}\sum_{q}\alpha_{q}(1-\alpha_{q})/n_{A}$ . Panel (c) shows empirical coverage of the Hoeffding confidence interval for $\alpha_{q}$ , illustrating valid but conservative finite-sample coverage.

Pairwise Preference Learning

Pairwise preference learning infers rankings, rewards, or policies from judgments that compare two alternatives at a time.

See all 126 papers

Internal Pluralism and the Limits of Pairwise Comparisons
Jul 2, 2026Bailey Flanigan, Michelle SiPairwise Preference Learning Pairwise Comparison
MIT, USA · Harvard, USA
Multiple latent orderings better predict language model preferences
Aug 26, 2026Aviral Chawla, William H. W. Thompson, Jean-Gabriel YoungPreference Modeling Pairwise Preference Learning
Vermont Complex Systems Institute University of Vermont Burlington, Vermont 05403 · Vermont Complex Systems Institute Department of Statistics University of Vermont Burlington, Vermont 05403
Learning Heterogeneous Preferences
Sep 15, 2026Shiwali Mohan, Matt Hong, Dule Shu +3Pairwise Preference Learning Human Preference Modeling
Human-Centered AI Toyota Research Institute Los Altos, CA

Jul 2, 2026J/K move · Enter open · S save

Internal Pluralism and the Limits of Pairwise Comparisons

Bailey Flanigan, Michelle Si

MIT, USA · Harvard, USA

Pairwise Preference Learning Pairwise Comparison Pairwise Preference Evaluation Human Preference Modeling Multi-Criteria Decision Analysis Pairwise Ranking+2

Open paper AskPDF arXiv

Local pairwise comparisons are a standard tool for learning how people want decision rules to work, e.g., in participatory design or alignment. However, their use builds in two strong assumptions: that local comparisons are sufficient evidence about how a person wants an automated decision rule to behave, and that people can always answer those comparisons decisively. We investigate how these assumptions may be compromised under internal pluralism: the idea that an individual evaluates decision rules according to multiple authoritative priorities about how the rule should behave. We provide a formal model of such pluralistic preferences over decision rules, which then lets us identify two distinct failures of forced local pairwise comparison data. First, priorities such as proportionality, egalitarianism, and equal treatment are inherently global: what they imply in one case can depend on what happens elsewhere, so local comparisons may fail to capture them. Second, even when priorities are representable locally, tension between strongly-held priorities can generate internal conflict, producing potentially costly behavioral distortions when comparisons are forced. We then use our model to investigate the alternative -- allowing people to report indecision -- and our findings suggest that doing so can considerably reduce the number of queries needed to learn preferences accurately. We conclude by describing how our model points toward preference-learning methods that elicit these priorities directly, yielding more faithful and interpretable accounts of what people value.

Abstract

Pairwise Preference Learning

Explore similar work

Internal Pluralism and the Limits of Pairwise Comparisons

Multiple latent orderings better predict language model preferences

Learning Heterogeneous Preferences