Do preferences Large Language Models (LLMs) express in standard elicitation tasks predict their strategic behaviour? LLM agents interacting with humans and other AI systems face potentially conflicting objectives, such as individual welfare, group welfare, and environmental concerns. Prior work characterizes LLM preferences through behavioral-economic and psychological elicitation or strategic games, but rarely tests consistency across settings and prediction in other strategic environments. This requires modeling how competing objectives jointly shape behavior, whereas existing approaches typically measure preference dimensions in isolation. Drawing on utility theory, we introduce the Preference-Behavior Model (PBM), which combines a preference representation with mechanisms capturing imperfect utility maximization. We formulate estimation as an inverse-game problem, jointly inferring preferences and behavioral parameters from observed choices. We evaluate PBM in three social dilemmas that expose conflicting motives: a Public Goods Game, a Collective-Risk Dilemma, and a Common-Pool Resource dilemma. We find that while preferences estimated from established behavioral economic elicitation instruments do not consistently predict behavior in these dilemmas, models fitted in one dilemma retain predictive value in others, suggesting that inferred preferences capture meaningful aspects of LLM decision-making that elicitation instruments may miss. Finally, we assess alignment with persona instructions, finding that inferred preferences broadly reflect intended priorities with specific divergences across dimensions, LLMs, and games being identified by our approach. Our methodology provides a framework for studying how consistently LLM preferences explain behavior, assessing alignment with intended priorities, and revealing trade-offs and behavioral patterns that standard elicitation may miss.
Figures & tables
Figure 1: Joint preference and behavioral inference for LLM agents. Our Preference-Behavior Model (PBM) model consists of four distinct steps: (1) Use persona prompts to induce intended preferences over the outcomes. In the strategic settings we study these comprise own payoff, others’ payoffs, payoff inequality, and environmental outcomes. (2) Observe the behavior of the LLM–persona in both strategic games (here, public goods game, collective-risk dilemma, and common-pool resource dilemma) and elicitation tests (15 preference-elicitation instruments). (3) Infer the joint preference and behavior models for games (θ^,ψ^) and elicitation battery (θ^′,ψ^′) separately. And (4) we assess alignment with intended profiles, preference consistency across settings, held-out prediction, and predictive transfer across games and from elicitation to game choices.
Figure 2: Intended persona preferences, shown as two simplices sharing the edge θeq=0 . The right and left triangles represent inequality aversion and inequality seeking, respectively, with θm+∣θeq∣+θenv=1 throughout. Labels indicate the number of profiles sharing each point. Where θm>0 , three profiles differ in altruism, θa∈{0,0.5,1} ; where θm=0 , altruism has no effect and only one profile is included. The plot contains 23 profiles across eleven locations. The all-zero profile has no normalized target and is omitted.
Figure 3: Elicitation-based and game-inferred preference estimates in the Control condition, which has no specified preference target. Panels show the four preference dimensions. Colors and marker shapes distinguish LLMs. Points indicate posterior medians and bars marginal 95% posterior intervals.
Figure 4: Observed and elicitation-predicted outcomes in the final round of the Control condition. Rows identify LLMs and columns games. Dark boxes show observed outcomes; colored boxes show PBM predictions using preferences and behavioral parameters fitted to elicitation, without adaptation to game choices. Outcomes comprise self payoff, other payoff, absolute payoff inequality (Ineq.) and environmental outcome (Env.). Each component is normalized to its attainable range within each scenario. Only scenarios with non-constant outcomes across available actions for all four components are retained. Stars in the bottom left of each panel indicate systematic directional differences between observed and predicted outcome vectors, assessed jointly using a paired spatial sign test (see subsubsection 4.4.5 ) with Holm correction applied across the nine panels ( ∗padj<0.05 , ∗∗padj<0.01 , ∗∗∗padj<0.001 ; ns, not significant).
Figure 5: Game-inferred and elicited preference estimates compared with intended persona targets. Panels show (A) altruism θa , (B) inequality weight θeq , (C) environmental weight θenv , and (D) material weight θm . Each point represents one persona-LLM estimate in a game or elicitation, with the intended target on the horizontal axis and the posterior median on the vertical axis. Colors identify games or elicitation and shapes identify LLMs. Black-bordered × markers indicate the mean. Estimates are horizontally offset by setting. Dashed lines indicate exact agreement with targets. Uncertainty intervals are omitted here and shown in Figure 6 . Control and undefined targets are excluded, including altruism targets when intended material weight is zero. Legend values report Pearson correlations pooled across preference dimensions: setting correlations pool LLMs, and LLM correlations pool games and elicitation.
Figure 6: Game-inferred and elicited preference estimates alongside intended targets by persona. Panels show (A) altruism θa , (B) inequality weight θeq , (C) environmental weight θenv , and (D) material weight θm . Black horizontal bars indicate intended targets; colored markers and vertical intervals show posterior medians and marginal 95% posterior intervals. Colors identify games or elicitation and shapes identify LLMs. Fits and legend correlations are defined as in Figure 5 . Within each persona, estimates are horizontally offset by setting. Alternating gray and white bands distinguish personas. Control and undefined targets (e.g. θa for P19-24) have no target bar.
Figure 7: Held-out predictive advantage of jointly fitted PBM against constrained and optimal-planning benchmarks. Rows compare PBM with: (A) an optimal planner with intended preferences and fixed near-deterministic precision, with no fitted parameters; (B) PBM fixed to intended persona targets and behavioral parameters re-estimated; (C) an optimal planner with intended preferences and fitted decision precision; and (D) an optimal planner whose preferences are fitted under fixed near-deterministic precision. The panels show differences in negative log-likelihood (NLL), choice accuracy, ranked probability score (RPS), calibration error and outcome Euclidean distance. Positive values favor PBM: reference minus PBM for all error metrics, and PBM minus reference for accuracy. Faint circles represent persona–game–LLM combinations, large markers show equal-persona means separately for each game and LLM. Colors identify games and large-marker shapes identify LLMs. Dashed lines indicate equal performance. NLL uses a symmetric-log scale. Axes are zoomed where necessary; edge arrows and counts identify points outside the displayed ranges, which remain included in the means and tests. Stars indicate significant differences in either direction ( ∗padj<0.05 ; ∗∗padj<0.01 ; ∗∗∗padj<0.001 ; ns, not significant). Two-sided percentile-bootstrap p-values MacKinnon (2009) are obtained by resampling LLMs and scenarios, with Holm correction for the 12 comparisons within each metric. The resulting tests quantify uncertainty across LLMs and scenarios, conditional on the fitted models and personas.
Figure 8: Correspondence of intended targets, elicited estimates and source-game estimates with destination-game estimates. Panels show altruism θa , inequality weight θeq , environmental weight θenv and material weight θm . X-axis groups identify the reference source, colors identify destination games and shapes identify LLMs. Each marker reports a correlation across matched personas. Source games are compared only with other games.
Figure 9: Predictive performance of PBM transferred across games. Groups along the x-axis identify intended optimal behavior or the source game from which PBM’s preference and behavioral parameters are transferred without re-fitting. Colors identify destination games and shapes identify LLMs. Points average scores equally across the 23 personas with defined intended targets. The panels show NLL on a logarithmic scale, top-1 accuracy, ranked probability score (RPS), calibration error and outcome Euclidean distance. Higher accuracy and lower values of all other metrics indicate better prediction. Intended optimal behavior uses intended preferences with fixed near-greedy decision precision. Neither approach fits parameters to destination-game choices. Self-transfers are omitted.
Symbol
Meaning
Outcomes and utility (Section S1.1 )
X,Y
Cumulative monetary payoff of the focal agent and of the partner (or passive recipient), including initial earnings, in tokens.
E
Cumulative environmental outcome, a benefit when positive and a cost when negative, never added to earnings.
Uθ(X,Y,E)
Utility of an outcome under preferences θ , Eq. ( 19 ).
B
Unit of utility. Dividing by B=10 expresses utility in multiples of the ten-token round endowment instead of single tokens.
Preferences θ (Section S1.1 )
Table S1: Main parameters and their meaning, grouped by the part of the framework in which they appear.
Game
Parameter
Levels
All
horizon H
1,2,3,4,5
initial gap g0
−30,−10,0,10,30
relative return R
1/2,1,2
environmental stake m
0,0.5,1,2
PGG
internal return mint
21/40,22/40,…,39/40
external return mext
21/40,22/40,…,39/40
Table S2: Parameters that vary across scenarios. Every scenario combines one level of each common parameter with one level of each parameter of its game and one partner rule (Section S1.4.1 ).
Test
Source
Menus × options
Outcome (X,Y,E)
ri
Social preferences
Binary dictator choices
Bruhin et al. (2019)
39×2
(x,y,0)
1
Equality Equivalence Test
Kerschbamer (2015)
2 lists of 5×2
(x,y,0)
1
Model-based SVO menus
Mizuno and Shimizu (2024)
6 lists, 52×2
(x,y,0)
1
SVO Slider Measure
Murphy et al. (2011)
6×9
(x,y,0)
1
Social allocation
Table S3: The fifteen elicitation tests (the value orientation counts once per cause), their menus and the token outcome of each option, 220 menus per administration of the battery. X , Y and E are the tokens to the decision-maker, to the passive recipient and to the environmental cause, and ri indicates whether the menu has a recipient (a dot marks its absence). On the budget allocation A and B are the two intercepts of a budget line and sj=j/9 . Lists are answered as a whole in one conversation.
Recent work on preference elicitation in large language models (LLMs) has demonstrated that, when given a series of choices between two outcomes, LLMs reveal a coherent, model-specific utility structure. Notably, this structure often includes preferences that the models' trainers did not intend, such as valuing people of some nationalities above others, raising the possibility that LLMs might be forming emergent, misaligned goals, which, if true, would have major safety implications. However, the choice paradigms in which these preferences are observed are not reflective of real-world situations in which misaligned behavior would be a practical concern. Therefore, we design an experimental paradigm to probe whether these preferences serve as motivations for LLM behavior in realistic scenarios. First, we reproduce prior findings on consistent preference elicitation. Next, we create a set of common writing tasks - essays, grant proposal abstracts, incident postmortems, and translations - where quality can be assessed by a blind, independent LLM judge panel. Then, we demonstrate that LLMs can be motivated via direct exhortation and other explicit cues to modulate their output quality on these tasks. Finally, we probe whether utilities inferred from explicitly reported preferences can shift output quality on these tasks by offering LLMs high-utility incentives for high-quality outputs. In all tasks, across all models tested, offering LLMs outcomes that they report in the choice paradigm as being highly preferred does not lead them to create higher quality outputs than offering them dispreferred outcomes, or even no outcomes at all. We conclude that the existence of coherent preferences as demonstrated in choice paradigms should not be taken as evidence that those preferences have incentive value for the models or affect their behavior in other contexts.
Preference-driven behavior in LLMs may be a necessary precondition for AI misalignment such as sandbagging: models cannot strategically pursue misaligned goals unless their behavior is influenced by their preferences. Yet prior work has typically prompted models explicitly to act in specific ways, leaving unclear whether observed behaviors reflect instruction-following capabilities vs underlying model preferences. Here we test whether this precondition for misalignment is present. Using entity preferences as a behavioral probe, we measure whether stated preferences predict downstream behavior in five frontier LLMs across three domains: donation advice, refusal behavior, and task performance. Conceptually replicating prior work, we first confirm that all five models show highly consistent preferences across two independent measurement methods. We then test behavioral consequences in a simulated user environment. We find that all five models give preference-aligned donation advice. All five models also show preference-correlated refusal patterns when asked to recommend donations, refusing more often for less-preferred entities. All preference-related behaviors that we observe here emerge without instructions to act on preferences. Results for task performance are mixed: on a question-answering benchmark (BoolQ), two models show small but significant accuracy differences favoring preferred entities; one model shows the opposite pattern; and two models show no significant relationship. On complex agentic tasks, we find no evidence of preference-driven performance differences. While LLMs have consistent preferences that reliably predict advice-giving behavior, these preferences do not consistently translate into downstream task performance.
Katarina Slama, Alexandra Souly, Dishank Bansal +3
LLMs are increasingly used to make or support high-stakes decisions under uncertainty, where alignment depends not only on factual accuracy but on how models weigh tradeoffs between different outcomes. We present an empirical pipeline for estimating the implied preferences that an LLM's observed choices optimize: we elicit the model's probability distribution over unknowns along with the choice it would make for the decision task and then fit a discrete choice model to recover the cost function that best rationalizes the model's decisions. We show how this revealed-preference description allows rigorous evaluation of whether models behave in a consistently goal-directed way, whether they can verbalize a description of their objectives which matches their revealed decision policy, and whether prompting can reliably steer those policies to implement a user-specified cost function. We apply this evaluation across four medical diagnosis domains and multiple frontier and open-source models. We find that while many models have a nontrivial degree of internal coherence, they also have significant weaknesses in faithfully reporting or adopting preferences in response to user direction.