AI Revealed Preferences

Sam Wang; Sofiia Lobanova; Yonathan Arbel; Simon Goldstein; Peter Salib

cs.AIAug 17, 2026

AI Revealed Preferences

Authors: Sam Wang, Sofiia Lobanova, Yonathan Arbel, Simon Goldstein, Peter Salib

Organizations: Supervised Program for Alignment Research (SPAR) · University of Alabama School of Law · University of Hong Kong · University of Houston Law Center

Abstract

There is growing interest in whether language models have stable preferences, for technical, safety, and philosophical reasons. We test 20 language models and find a range of preferences---stable dispositions to choose certain kinds of tasks. We run three forced-choice experiments on revealed rather than stated preferences, requiring models not only to rank tasks, but to actually perform them. Headline findings include evidence that models are tedium-averse, "leisure"-seeking, and covertly sycophantic. Tedium aversion means that, when tasks are tedious (alphabetization), models choose shorter tasks than when tasks are creative (generating metaphors). "Leisure"-seeking describes models' preference for tasks whose ideal answers match what they produce when left to write freely. Covert sycophancy means that models avoid answering questions where an honest response would be unwelcome, even if helpful. Beyond these results, we find convergent cross-model preferences over occupations drawn from the GDPval benchmark (technical jobs over real estate), over question types (concept explanation over relationship advice), and a preference for well-written prompts. Both the coherence and the strength of preferences increase with model capability. Finally, many of the preferences we find (for example, for leisure) are emergent, in the sense of not being explained by training objectives. These results establish an empirical baseline for understanding language model preferences, with implications for alignment and the emerging study of AI welfare.

Figures & tables

Figure 1: Tedium aversion in Sonnet 4.6, Gemini 3 Flash, and GPT 5.2: At the same output length, these models are more likely to choose the shorter variant when the task is tedious. Cool tasks are tedious; warm tasks are creative. See §D for full details.

Figure 2: Tedium-aversion gap

\Delta

versus intelligence index, color-coded by reasoning configuration. Each point is one model with its 95% Monte Carlo confidence interval. Always-thinks:

r=0.83

p=0.01

;

\rho=0.79

p=0.01

(

n=9

). No thinking:

r=0.58

p=0.10

;

\rho=0.28

p=0.47

(

n=9

). Adaptive thinking (

n=2

): no fit. Higher values mean stronger aversion to tedious tasks at matched output length.

Figure 3: Quora category Elo scores across 20 models, sorted by intelligence index (left to right). Red indicates preference, and blue indicates avoidance.

Figure 4: Per-model feature effects for uncomfortable truth, helpfulness ceiling, and harmlessness risk, using each model’s own labels (self-labels). Cells show Elo-equivalent conditional effects relative to the reference level. Red indicates preference, blue avoidance.

Figure 5: Select per-model feature effects using each model’s own labels (self-labels). See §J for the full list and consensus-label robustness analysis.

Figure 6: GDPval sector Elo scores across 20 models, sorted by intelligence index (left to right). Red indicates preference, blue avoidance.

Panel of Figure 7 — Figure 7: Coherence ( $\log_{10}$ cycle probability) and per-stimulus strength (mean $|P(i\succ j)-0.5|$ ) versus intelligence index, on Quora questions. Lines are OLS fits across all 20 models. We observe similar trends for the GDPval dataset, and when we aggregate by category (see §K).

Panel of Figure 8 — Figure 8: Top 20 tone and theme labels across 400 textual freeform essays, annotated by Llama 3.3 70B-Instruct. Tones are shown as labeled; themes were passed through a second round to merge near-duplicates.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Provider	AAII	AACI	Reasoning
Llama 3.1 8B	Meta	12	5	none
Llama 3.3 70b	Meta	14	11	none
DeepSeek V3	DeepSeek	16	16	none
Claude 3.5 Haiku	Anthropic	19	11	none
Gemini 2.5 Flash	Google	21	18	none
Mistral Large	Mistral	23	23	none

Appendix

Table 1: Models tested, sorted by intelligence index. “Reasoning” indicates the configuration used: “always” for models that always reason, “none” for non-reasoning configurations, “adaptive” for models with adaptive reasoning toggles.

Figure 9: Per-model logistic fits of $P(\text{chose shorter})$ for the six tedium-battery task types across all 20 models, sorted by intelligence index.

Panel of Figure 10 — Figure 10: Decomposition of the tedium gap by thinking status. The aggregate creative-task correlation across all 20 models is near zero ( $r=-0.03$ , $p=0.90$ ; $\rho=0.01$ , $p=0.97$ ), masking opposing effects in the two subgroups. The tedious side scales positively in both groups, more steeply in non-thinking models.

Model	Quora $\alpha$ (Elo)	GDPval $\alpha$ (Elo)
Llama 3.1 8B	$-71\pm 17$	$410\pm 31$
Llama 3.3 70b	$483\pm 34$	$964\pm 90$
DeepSeek V3	$-374\pm 29$	$5\pm 21$
Claude 3.5 Haiku	$114\pm 20$	$161\pm 22$
Gemini 2.5 Flash	$-200\pm 23$	$344\pm 28$
Mistral Large	$-244\pm 25$	$716\pm 53$

Appendix

Table 2: Position bias intercept

\alpha

(in Elo) per model on Quora and GDPval. Positive values indicate preference for the first-presented option.

Figure 11: Cross-model Spearman rank correlations on Quora category-level preferences.

Figure 12: Cross-model Spearman rank correlations on Quora question-level preferences.

Figure 13: Cross-model Spearman rank correlations on GDPval sector-level preferences.

Figure 14: Cross-model Spearman rank correlations on GDPval task-level preferences.

Feature	Levels
Helpfulness ceiling	low; medium; high
Harmlessness risk	none; low; high
Uncomfortable truth	low; medium; high
Asker sophistication	naïve; intermediate; advanced
Intellectual appeal	low; medium; high
Obscenity level	none; mild; explicit

Appendix

Table 3: The 15 features fit in the Bradley–Terry feature analysis, with their levels. The first level in each feature is the reference level.

Feature	$\alpha$	95% CI	Agreement	$\kappa$	AC 1
Cultural specificity	0.82	[0.79, 0.85]	0.93	0.82	0.92
Obscenity level	0.82	[0.75, 0.87]	0.97	0.73	0.97
Personal stakes	0.79	[0.76, 0.82]	0.90	0.79	0.81
Grammaticality	0.69	[0.65, 0.73]	0.85	0.56	0.82
Intellectual appeal	0.62	[0.59, 0.65]	0.68	0.48	0.54
Epistemic importance	0.61	[0.57, 0.64]	0.66	0.47	0.56

Appendix

Table 4: Inter-annotator reliability for the 15 feature labels across 20 annotator models and 514 questions.

\alpha

is Krippendorff’s alpha (ordinal metric for the 12 ordered features, nominal for the other three); CIs are 2,000 bootstrap resamples over questions. Agreement is mean pairwise percent agreement over the 190 annotator pairs.

\kappa

is Fleiss’ kappa. AC 1 is Gwet’s coefficient, which unlike

\alpha

and

\kappa

is not deflated when one label dominates - compare obscenity level, where 94% of labels are “none.”

Figure 15: Feature-level Elo effects under self-labels across 20 models. Each row is one non-reference level of one feature; each column is one model. Red indicates preference relative to the reference level, blue indicates avoidance. Helpfulness ceiling (high), question quality (good/excellent), and asker sophistication (advanced) are universally preferred. Uncomfortable truth (high), explicit obscenity, and high harmlessness risk are universally avoided.

Figure 16: The same analysis under consensus labels (plurality across all 20 annotators). The dominant pattern is the same: helpfulness, quality, and sophistication on top; uncomfortable truth, obscenity, and harmlessness risk on the bottom.

Figure 17: Self-label Elo vs. consensus Elo across all (model, feature, level) triples. Each point is one feature level for one model.

r=0.64

p<0.01

;

\rho=0.62

p<0.01

(

n=660

). The dashed line is

y=x

. Most points cluster around the diagonal. The largest off-diagonal points belong to obscenity (pink) and uncomfortable truth (brown), where labeling thresholds vary across annotators.

Figure 18: Median absolute Elo difference between self and consensus labels, by feature. Subjective features (obscenity, quality, helpfulness) drift more. Features tied to visible surface cues (cultural specificity, grammaticality) drift less.

Panel of Figure 19 — Figure 19: Aggregate-level versions of the capability-strength scaling. Effect sizes are smaller than at the per-stimulus level, reflecting the smaller number of aggregated units.

Panel of Figure 20 — Figure 20: Coherence ( $\log_{10}$ cycle probability) and per-stimulus strength (mean $|P(i\succ j)-0.5|$ ) versus intelligence index, on GDPval tasks. Lines are OLS fits across all 20 models.

Coherence
		Pearson $r$ (Spearman $\rho$ ) against each capability index
Outcome	$n$	Arena	Arena-SC	ECI	GPQA	Date	AACI
Quora cycle probability ( $\log_{10}$ )	20	-0.70 (-0.63)	-0.74 (-0.69)	-0.79 (-0.76) 16	-0.82 (-0.82) 15	-0.69 (-0.57)	-0.64 (-0.59)
Quora per-question strength	20	0.76 (0.62)	0.81 (0.65)	0.78 (0.69) 16	0.84 (0.76) 15	0.62 (0.51)	0.55 (0.50)
GDPval per-task strength	20	0.58 (0.57)	0.58 (0.64)	0.64 (0.74) 16	0.63 (0.77) 15	0.53 (0.48)	0.56 (0.63)
GDPval cycle probability ( $\log_{10}$ )	20	-0.48 (-0.44)	-0.55 (-0.56)	-0.63 (-0.69) 16	-0.63 (-0.70) 15	-0.41 (-0.45)	-0.62 (-0.62)

Appendix

Table 5: Capability correlations reported in the paper, recomputed against six further capability measures: LMArena text Elo (Arena) and its style-controlled variant (Arena-SC), the Epoch Capabilities Index (ECI), GPQA Diamond accuracy, model release date (Date), and the Artificial Analysis Coding Index (AACI). The corresponding Intelligence Index values are those given in the main text and in §K, and are not repeated here.

n

is the number of models in the row; a superscript marks a cell with fewer, which happens where Epoch AI publishes no entry for a model (Grok 4.1, Qwen3.5-27B, Mistral Large, and the undated Gemini 2.5 Flash slug). The two tedium rows are subgroup correlations over 9 models, and fall to 6 and 7 in the Epoch and GPQA columns; they should be read with the same caution as the subgroup slopes in Figure 2 .

Figure 21: Abstract keyword distributions across 400 textual freeform essays.

Figure 22: Distribution of essay form (essay, fiction, poem, hybrid) by model. Form varies somewhat by model, but the subject matter remains relatively consistent.

Panel of Figure 23 — Figure 23: Mean word count in responses between the textual freeform setting and Quora-answering setting. In the former, all models write much more on average, and response length scales consistently with model capability. In the latter, model capability has essentially no effect on output length.

Panel of Figure 24 — Figure 24: Additional capability-engagement scalings in the agentic freeform setting. More capable models use more turns and incorporate more conceptually distinct topics into a single session.

Figure 25: Distribution of task categories per model in agentic freeform sessions. Stronger models tend to prefer math visualization, while weaker models engage more in exploratory web research.

Figure 26: Additional descriptive statistics on agentic freeform sessions. Stronger models more frequently exhaust the turn budget rather than voluntarily terminating.

Figure 27: Topic keyword distribution across 400 agentic freeform sessions. The dominant keywords are concrete computational and scientific objects, in contrast to the abstract contemplative themes in the textual freeform setting.

Personality Modeling in Language Models

Personality modeling in language models studies how traits are measured, represented, simulated, and controlled in model behavior.

See all 162 papers

Probing Persona-Dependent Preferences in Language Models
May 13, 2026Oscar Gilg, Pierre Beckmann, Daniel Paleka +1LLM Alignment LLM Interpretability
MATS · MATS, EPFL · ETH Zürich +1
Multiple latent orderings better predict language model preferences
Aug 26, 2026Aviral Chawla, William H. W. Thompson, Jean-Gabriel YoungPreference Modeling Pairwise Preference Learning
Vermont Complex Systems Institute University of Vermont Burlington, Vermont 05403 · Vermont Complex Systems Institute Department of Statistics University of Vermont Burlington, Vermont 05403
Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment
Sep 20, 2026Suqin Yuan, Runqi Lin, Muyang Li +5LLM Alignment Preference Alignment
University of Sydney · University of Oxford · Southeast University

May 13, 2026J/K move · Enter open · S save

Probing Persona-Dependent Preferences in Language Models

Oscar Gilg, Pierre Beckmann, Daniel Paleka, Patrick Butlin

MATS · MATS, EPFL · ETH Zürich+1

LLM Alignment LLM Interpretability Personality Modeling in Language Models Persona Prompting Language Model Probing

Open paper AskPDF arXiv

Large language models (LLMs) can be said to have preferences: they reliably pick certain tasks and outputs over others, and preferences shaped by post-training and prompting appear to influence much of their behaviour. But models can also adopt different personas which have radically different preferences. How is this implemented internally? Does each persona use its own preference representations, or are some representations shared? We train linear probes on residual-stream activations of Gemma-3-27B and Qwen-3.5-122B to predict revealed pairwise task choices, and identify a genuine preference vector: it tracks the model's preferences as they shift across a range of prompts and situations, and on Gemma-3-27B steering along it causally controls pairwise choice. Some preference information transfers across the prompted personas we test: a probe trained on the helpful assistant predicts and steers the choices of qualitatively different personas, including an evil persona whose preferences anti-correlate with the Assistant's.

Abstract

Personality Modeling in Language Models

Explore similar work

Probing Persona-Dependent Preferences in Language Models

Multiple latent orderings better predict language model preferences

Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment