Identifying two piecewise linear additive value functions from anonymous preference information

Vincent Auriau; Khaled Belahcene; Emmanuel Malherbe; Vincent Mousseau; Marc Pirlot

cs.AIFeb 24, 2026

Identifying two piecewise linear additive value functions from anonymous preference information

Authors: Vincent Auriau, Khaled Belahcene, Emmanuel Malherbe, Vincent Mousseau, Marc Pirlot

Organizations: Heudiasyc · MICS, CentraleSupélec, Université Paris Saclay, France · Artefact Research Center, France · MICS ·

Abstract

Eliciting a preference model involves asking a person, named decision-maker, a series of questions. We assume that these preferences can be represented by an additive value function. In this work, we query simultaneously two decision-makers in the aim to elicit their respective value functions. For each query we receive two answers, without noise, but without knowing which answer corresponds to which decision-maker.We propose an elicitation procedure that identifies the two preference models when the marginal value functions are piecewise linear with known breaking points.

Figures & tables

Description	Notation
Criteria set	$\mathcal{N}=\{1,...,n\}$	$i,j\in\mathcal{N}$
Criterion scale	$\mathcal{X}_{i}=[x_{i,0},x_{i,L}]$
Criterion interval	$[x_{i,\ell},x_{i,\ell+1}]$	$\ell\in\{0,...,L\}$
Decision makers	$\alpha,\beta$
Value functions	$u^{\kappa}(\cdot)$	$\kappa\in\{\alpha,\beta\}$
Marginal value functions	$u^{\kappa}_{i}(\cdot)$	$\kappa\in\{\alpha,\beta\},i\in\mathcal{N}$

Table 1: Notations summary

Panel of Figure 1 — Figure 1: Preferences of the DMs from the example, serving as ground truth for the elicitation. Marginal values for Price (left), Autonomy (middle), and resulting indifference curves (right).

Figure 2: A successful Single Rectangle query in the space

\mathcal{X}_{i}\times\mathcal{X}_{j}

, corresponding to case 1. Colored lines represent DMs’ indifference curves.

Panel of Figure 3 — Figure 3: Outcome and second query for case 2 of the Single Rectangle Constraint.

Panel of Figure 4 — Figure 4: Outcome and second query for case 3 of the Single Rectangle Constraint.

Figure 5: Example of a Query with a neighboring rectangles constraint. Colored lines represent the indifference curve of each DM.

Panel of Figure 6 — Figure 6: Outcome and second query for case 1 of Play Pattern 2 .

Panel of Figure 7 — Figure 7: Outcome and second query for case 2 of Play Pattern 2 .

Panel of Figure 8 — Figure 8: Possible value functions after Play Pattern 1 . The colored lines represent the marginal value functions of each DM.

Figure 9: General Procedure sequencing for the elicitation of two different DMs on two criteria.

\displaystyle\left\{\begin{aligned} (A_{i},A_{j})\sim_{\kappa}(B_{i},B_{j})\\ (A_{i},A_{j})\sim_{\kappa^{\prime}}(C_{i},C_{j})\\ \end{aligned}\right.\Leftrightarrow

\displaystyle\left\{\begin{aligned} u_{\kappa}((A_{i},A_{j}))=u_{\kappa}((B_{i},B_{j}))\\ u_{\kappa^{\prime}}((A_{i},A_{j}))=u_{\kappa^{\prime}}((C_{i},C_{j}))\end{aligned}\right.

Table 11

\left\{\begin{aligned} &\gamma^{\alpha}_{j,\ell_{j}}=\left(k\cdot\Lambda_{\kappa}+(1-k)\cdot\Lambda_{\kappa^{\prime}}\right)\cdot\gamma^{\alpha}_{i,\ell_{i}}\\ &\gamma^{\beta}_{j,\ell_{j}}=\left((1-k)\cdot\Lambda_{\kappa}+k\cdot\Lambda_{\kappa^{\prime}}\right)\cdot\gamma_{i,\ell_{i}}^{\beta}\\ \end{aligned}\right.

(6)

Table 12

\left\{\begin{aligned} &\gamma^{\alpha}_{j,\ell_{j}+1}=\left(k\cdot\Lambda_{\kappa}+(1-k)\cdot\Lambda_{\kappa^{\prime}}\right)\cdot\gamma^{\alpha}_{i,\ell_{i}}\\ &\gamma^{\beta}_{j,\ell_{j}+1}=\left((1-k)\cdot\Lambda_{\kappa}+k\cdot\Lambda_{\kappa^{\prime}}\right)\cdot\gamma_{i,\ell_{i}}^{\beta}\\ \end{aligned}\right.

(10)

Table 13

Preference Elicitation

Preference elicitation studies how systems learn people’s values and trade-offs from questions, choices, rankings, and feedback.

See all 67 papers

Learning Heterogeneous Preferences
Sep 15, 2026Shiwali Mohan, Matt Hong, Dule Shu +3Pairwise Preference Learning Human Preference Modeling
Human-Centered AI Toyota Research Institute Los Altos, CA
Learning What Evaluators Value: A Reliable Approach to Modeling Evaluator Preferences
May 15, 2026Madeline Celi Kitch, Nihar B. ShahHuman Preference Modeling Preference Learning
Carnegie Mellon University
How much of a measured AI preference is the model, and how much is the instrument?
Aug 24, 2026Jason HungPreference Elicitation Prompt Sensitivity
Independent With Apart Research

Sep 15, 2026J/K move · Enter open · S save

Learning Heterogeneous Preferences

Shiwali Mohan, Matt Hong, Dule Shu, Aniek Fransen+2

Human-Centered AI Toyota Research Institute Los Altos, CA

Pairwise Preference Learning Human Preference Modeling Preference Learning Preference Modeling

Open paper AskPDF arXiv

Learning from human feedback has become a central paradigm for training modern AI systems, where models of human utility are used as reward models in policy learning. Existing methods typically assume a universal utility function shared across a population and treat disagreement between annotators as stochastic variation. While suitable for objective tasks, this assumption breaks down in subjective domains where preferences vary systematically across individuals. We study the problem of subjective preference learning, in which observed choices arise from heterogeneous but internally consistent utility functions. Drawing upon rational choice theory, RCT \parencite{tversky1981framing}, we introduce individuated utility functions conditioned on both the individual and their decision context, and propose a novel multi-stage architecture for estimating them from multi-modal data. We evaluate our framework on a newly collected dataset of more than

575{,}000

pairwise aesthetic judgments from

2{,}398

participants comparing automotive wheel designs. Our experiments show that individuated utility models substantially outperform universal utility models including foundation model baselines. Our results demonstrate that disagreement reflects meaningful preference heterogeneity rather than annotation noise. More broadly, our findings highlight the importance of collecting annotator attributes and learning individuated utility functions, enabling reward models that explicitly account for whose preferences they represent and faithfully capture human decision diversity.

Abstract

Preference Elicitation

Explore similar work

Learning Heterogeneous Preferences

Learning What Evaluators Value: A Reliable Approach to Modeling Evaluator Preferences

How much of a measured AI preference is the model, and how much is the instrument?