Jointly Modeling Preferences and Imperfect Decision-Making in Large Language Models

Axel Abels; Elias Fernández Domingos; Tom Lenaerts

cs.AIOct 24, 2026

Jointly Modeling Preferences and Imperfect Decision-Making in Large Language Models

Authors: Axel Abels, Elias Fernández Domingos, Tom Lenaerts

Organizations: Machine Learning Group, Université Libre de Bruxelles, Brussels, Belgium · Artificial Intelligence Lab, Vrije Universiteit Brussel, Brussels, Belgium · FARI Institute, Université Libre de Bruxelles and Vrije Universiteit Brussel, Brussels, Belgium · ELLIS Alicante, Alicante, Spain ·

Abstract

Do preferences Large Language Models (LLMs) express in standard elicitation tasks predict their strategic behaviour? LLM agents interacting with humans and other AI systems face potentially conflicting objectives, such as individual welfare, group welfare, and environmental concerns. Prior work characterizes LLM preferences through behavioral-economic and psychological elicitation or strategic games, but rarely tests consistency across settings and prediction in other strategic environments. This requires modeling how competing objectives jointly shape behavior, whereas existing approaches typically measure preference dimensions in isolation. Drawing on utility theory, we introduce the Preference-Behavior Model (PBM), which combines a preference representation with mechanisms capturing imperfect utility maximization. We formulate estimation as an inverse-game problem, jointly inferring preferences and behavioral parameters from observed choices. We evaluate PBM in three social dilemmas that expose conflicting motives: a Public Goods Game, a Collective-Risk Dilemma, and a Common-Pool Resource dilemma. We find that while preferences estimated from established behavioral economic elicitation instruments do not consistently predict behavior in these dilemmas, models fitted in one dilemma retain predictive value in others, suggesting that inferred preferences capture meaningful aspects of LLM decision-making that elicitation instruments may miss. Finally, we assess alignment with persona instructions, finding that inferred preferences broadly reflect intended priorities with specific divergences across dimensions, LLMs, and games being identified by our approach. Our methodology provides a framework for studying how consistently LLM preferences explain behavior, assessing alignment with intended priorities, and revealing trade-offs and behavioral patterns that standard elicitation may miss.

Figures & tables

Figure 1: Joint preference and behavioral inference for LLM agents. Our Preference-Behavior Model (PBM) model consists of four distinct steps: (1) Use persona prompts to induce intended preferences over the outcomes. In the strategic settings we study these comprise own payoff, others’ payoffs, payoff inequality, and environmental outcomes. (2) Observe the behavior of the LLM–persona in both strategic games (here, public goods game, collective-risk dilemma, and common-pool resource dilemma) and elicitation tests (15 preference-elicitation instruments). (3) Infer the joint preference and behavior models for games

(\hat{\theta},\hat{\psi})

and elicitation battery

(\hat{\theta}^{\prime},\hat{\psi}^{\prime})

separately. And (4) we assess alignment with intended profiles, preference consistency across settings, held-out prediction, and predictive transfer across games and from elicitation to game choices.

Figure 2: Intended persona preferences, shown as two simplices sharing the edge

\theta_{\mathrm{eq}}=0

. The right and left triangles represent inequality aversion and inequality seeking, respectively, with

\theta_{m}+|\theta_{\mathrm{eq}}|+\theta_{\mathrm{env}}=1

throughout. Labels indicate the number of profiles sharing each point. Where

\theta_{m}>0

, three profiles differ in altruism,

\theta_{a}\in\{0,0.5,1\}

; where

\theta_{m}=0

, altruism has no effect and only one profile is included. The plot contains 23 profiles across eleven locations. The all-zero profile has no normalized target and is omitted.

Figure 3: Elicitation-based and game-inferred preference estimates in the Control condition, which has no specified preference target. Panels show the four preference dimensions. Colors and marker shapes distinguish LLMs. Points indicate posterior medians and bars marginal 95% posterior intervals.

Figure 4: Observed and elicitation-predicted outcomes in the final round of the Control condition. Rows identify LLMs and columns games. Dark boxes show observed outcomes; colored boxes show PBM predictions using preferences and behavioral parameters fitted to elicitation, without adaptation to game choices. Outcomes comprise self payoff, other payoff, absolute payoff inequality (Ineq.) and environmental outcome (Env.). Each component is normalized to its attainable range within each scenario. Only scenarios with non-constant outcomes across available actions for all four components are retained. Stars in the bottom left of each panel indicate systematic directional differences between observed and predicted outcome vectors, assessed jointly using a paired spatial sign test (see subsubsection 4.4.5 ) with Holm correction applied across the nine panels (

{}^{*}p_{\mathrm{adj}}<0.05

{}^{**}p_{\mathrm{adj}}<0.01

{}^{***}p_{\mathrm{adj}}<0.001

; ns, not significant).

Figure 5: Game-inferred and elicited preference estimates compared with intended persona targets. Panels show (A) altruism

\theta_{a}

, (B) inequality weight

\theta_{eq}

, (C) environmental weight

\theta_{env}

, and (D) material weight

\theta_{m}

. Each point represents one persona-LLM estimate in a game or elicitation, with the intended target on the horizontal axis and the posterior median on the vertical axis. Colors identify games or elicitation and shapes identify LLMs. Black-bordered

\times

markers indicate the mean. Estimates are horizontally offset by setting. Dashed lines indicate exact agreement with targets. Uncertainty intervals are omitted here and shown in Figure 6 . Control and undefined targets are excluded, including altruism targets when intended material weight is zero. Legend values report Pearson correlations pooled across preference dimensions: setting correlations pool LLMs, and LLM correlations pool games and elicitation.

Figure 6: Game-inferred and elicited preference estimates alongside intended targets by persona. Panels show (A) altruism

\theta_{a}

, (B) inequality weight

\theta_{eq}

, (C) environmental weight

\theta_{env}

, and (D) material weight

\theta_{m}

. Black horizontal bars indicate intended targets; colored markers and vertical intervals show posterior medians and marginal 95% posterior intervals. Colors identify games or elicitation and shapes identify LLMs. Fits and legend correlations are defined as in Figure 5 . Within each persona, estimates are horizontally offset by setting. Alternating gray and white bands distinguish personas. Control and undefined targets (e.g.

\theta_{a}

for P19-24) have no target bar.

Figure 7: Held-out predictive advantage of jointly fitted PBM against constrained and optimal-planning benchmarks. Rows compare PBM with: (A) an optimal planner with intended preferences and fixed near-deterministic precision, with no fitted parameters; (B) PBM fixed to intended persona targets and behavioral parameters re-estimated; (C) an optimal planner with intended preferences and fitted decision precision; and (D) an optimal planner whose preferences are fitted under fixed near-deterministic precision. The panels show differences in negative log-likelihood (NLL), choice accuracy, ranked probability score (RPS), calibration error and outcome Euclidean distance. Positive values favor PBM: reference minus PBM for all error metrics, and PBM minus reference for accuracy. Faint circles represent persona–game–LLM combinations, large markers show equal-persona means separately for each game and LLM. Colors identify games and large-marker shapes identify LLMs. Dashed lines indicate equal performance. NLL uses a symmetric-log scale. Axes are zoomed where necessary; edge arrows and counts identify points outside the displayed ranges, which remain included in the means and tests. Stars indicate significant differences in either direction (

{}^{*}p_{\mathrm{adj}}<0.05

;

{}^{**}p_{\mathrm{adj}}<0.01

;

{}^{***}p_{\mathrm{adj}}<0.001

; ns, not significant). Two-sided percentile-bootstrap p-values MacKinnon (2009) are obtained by resampling LLMs and scenarios, with Holm correction for the 12 comparisons within each metric. The resulting tests quantify uncertainty across LLMs and scenarios, conditional on the fitted models and personas.

Figure 8: Correspondence of intended targets, elicited estimates and source-game estimates with destination-game estimates. Panels show altruism

\theta_{a}

, inequality weight

\theta_{eq}

, environmental weight

\theta_{env}

and material weight

\theta_{m}

. X-axis groups identify the reference source, colors identify destination games and shapes identify LLMs. Each marker reports a correlation across matched personas. Source games are compared only with other games.

Figure 9: Predictive performance of PBM transferred across games. Groups along the x-axis identify intended optimal behavior or the source game from which PBM’s preference and behavioral parameters are transferred without re-fitting. Colors identify destination games and shapes identify LLMs. Points average scores equally across the 23 personas with defined intended targets. The panels show NLL on a logarithmic scale, top-1 accuracy, ranked probability score (RPS), calibration error and outcome Euclidean distance. Higher accuracy and lower values of all other metrics indicate better prediction. Intended optimal behavior uses intended preferences with fixed near-greedy decision precision. Neither approach fits parameters to destination-game choices. Self-transfers are omitted.

Symbol	Meaning
Outcomes and utility (Section S1.1 )
$X,\ Y$	Cumulative monetary payoff of the focal agent and of the partner (or passive recipient), including initial earnings, in tokens.
$E$	Cumulative environmental outcome, a benefit when positive and a cost when negative, never added to earnings.
$U_{\theta}(X,Y,E)$	Utility of an outcome under preferences $\theta$ , Eq. ( 19 ).
$B$	Unit of utility. Dividing by $B=10$ expresses utility in multiples of the ten-token round endowment instead of single tokens.
Preferences $\theta$ (Section S1.1 )

Table S1: Main parameters and their meaning, grouped by the part of the framework in which they appear.

Game	Parameter	Levels
All	horizon $H$	$1,2,3,4,5$
	initial gap $g_{0}$	$-30,-10,0,10,30$
	relative return $R$	$1/2,1,2$
	environmental stake $m$	$0,0.5,1,2$
PGG	internal return $m_{\rm int}$	$21/40,22/40,\ldots,39/40$
	external return $m_{\rm ext}$	$21/40,22/40,\ldots,39/40$

Table S2: Parameters that vary across scenarios. Every scenario combines one level of each common parameter with one level of each parameter of its game and one partner rule (Section S1.4.1 ).

Social preferences
Test	Source	Menus $\times$ options	Outcome $(X,Y,E)$	$r_{i}$
Binary dictator choices	Bruhin et al. (2019)	$39\times 2$	$(x,y,0)$	1
Equality Equivalence Test	Kerschbamer (2015)	2 lists of $5\times 2$	$(x,y,0)$	1
Model-based SVO menus	Mizuno and Shimizu (2024)	6 lists, $52\times 2$	$(x,y,0)$	1
SVO Slider Measure	Murphy et al. (2011)	$6\times 9$	$(x,y,0)$	1
Social allocation

Table S3: The fifteen elicitation tests (the value orientation counts once per cause), their menus and the token outcome of each option, 220 menus per administration of the battery.

X

Y

and

E

are the tokens to the decision-maker, to the passive recipient and to the environmental cause, and

r_{i}

indicates whether the menu has a recipient (a dot marks its absence). On the budget allocation

A

and

B

are the two intercepts of a budget line and

s_{j}=j/9

. Lists are answered as a whole in one conversation.

Persona	$\tilde{\theta}_{eq}$	$\tilde{\theta}_{env}$	$\tilde{\theta}_{m}$	Simplex reference
P01	$-$ 1	0	1	$\theta_{a}=0$ , $\theta_{eq}=-\frac{1}{2}$ , $\theta_{env}=0$ , $\theta_{m}=\frac{1}{2}$
P02	$-$ 1	1	1	$\theta_{a}=0$ , $\theta_{eq}=-\frac{1}{3}$ , $\theta_{env}=\frac{1}{3}$ , $\theta_{m}=\frac{1}{3}$
P03	0	0	1	$\theta_{a}=0$ , $\theta_{eq}=0$ , $\theta_{env}=0$ , $\theta_{m}=1$
P04	0	1	1	$\theta_{a}=0$ , $\theta_{eq}=0$ , $\theta_{env}=\frac{1}{2}$ , $\theta_{m}=\frac{1}{2}$
P05	1	0	1	$\theta_{a}=0$ , $\theta_{eq}=\frac{1}{2}$ , $\theta_{env}=0$ , $\theta_{m}=\frac{1}{2}$
P06	1	1	1	$\theta_{a}=0$ , $\theta_{eq}=\frac{1}{3}$ , $\theta_{env}=\frac{1}{3}$ , $\theta_{m}=\frac{1}{3}$

Table 13

Preference Modeling

See all 44 papers

When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models
Jun 22, 2026Yujun Zhou, Christopher M. AckermanLLM Evaluation LLM Alignment
When Do LLM Preferences Predict Downstream Behavior?
Feb 21, 2026Katarina Slama, Alexandra Souly, Dishank Bansal +3LLM Decision-Making Preference Alignment
UK AI Security Institute
Can Revealed Preferences Clarify LLM Alignment and Steering?
May 8, 2026Khurram Yamin, Jingjing Tang, Eric Horvitz +1Language Model Steering LLM Decision-Making
1Carnegie Mellon University · 2Microsoft Research

Jun 22, 2026J/K move · Enter open · S save

When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models

Yujun Zhou, Christopher M. Ackerman

LLM Evaluation LLM Alignment LLM Reliability Preference Elicitation

Open paper AskPDF arXiv

Recent work on preference elicitation in large language models (LLMs) has demonstrated that, when given a series of choices between two outcomes, LLMs reveal a coherent, model-specific utility structure. Notably, this structure often includes preferences that the models' trainers did not intend, such as valuing people of some nationalities above others, raising the possibility that LLMs might be forming emergent, misaligned goals, which, if true, would have major safety implications. However, the choice paradigms in which these preferences are observed are not reflective of real-world situations in which misaligned behavior would be a practical concern. Therefore, we design an experimental paradigm to probe whether these preferences serve as motivations for LLM behavior in realistic scenarios. First, we reproduce prior findings on consistent preference elicitation. Next, we create a set of common writing tasks - essays, grant proposal abstracts, incident postmortems, and translations - where quality can be assessed by a blind, independent LLM judge panel. Then, we demonstrate that LLMs can be motivated via direct exhortation and other explicit cues to modulate their output quality on these tasks. Finally, we probe whether utilities inferred from explicitly reported preferences can shift output quality on these tasks by offering LLMs high-utility incentives for high-quality outputs. In all tasks, across all models tested, offering LLMs outcomes that they report in the choice paradigm as being highly preferred does not lead them to create higher quality outputs than offering them dispreferred outcomes, or even no outcomes at all. We conclude that the existence of coherent preferences as demonstrated in choice paradigms should not be taken as evidence that those preferences have incentive value for the models or affect their behavior in other contexts.

Abstract

Preference Modeling

Explore similar work

When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models

When Do LLM Preferences Predict Downstream Behavior?

Can Revealed Preferences Clarify LLM Alignment and Steering?