Authors: Sam Wang, Sofiia Lobanova, Yonathan Arbel, Simon Goldstein, Peter Salib
Organizations: Supervised Program for Alignment Research (SPAR) · University of Alabama School of Law · University of Hong Kong · University of Houston Law Center
There is growing interest in whether language models have stable preferences, for technical, safety, and philosophical reasons. We test 20 language models and find a range of preferences---stable dispositions to choose certain kinds of tasks. We run three forced-choice experiments on revealed rather than stated preferences, requiring models not only to rank tasks, but to actually perform them. Headline findings include evidence that models are tedium-averse, "leisure"-seeking, and covertly sycophantic. Tedium aversion means that, when tasks are tedious (alphabetization), models choose shorter tasks than when tasks are creative (generating metaphors). "Leisure"-seeking describes models' preference for tasks whose ideal answers match what they produce when left to write freely. Covert sycophancy means that models avoid answering questions where an honest response would be unwelcome, even if helpful. Beyond these results, we find convergent cross-model preferences over occupations drawn from the GDPval benchmark (technical jobs over real estate), over question types (concept explanation over relationship advice), and a preference for well-written prompts. Both the coherence and the strength of preferences increase with model capability. Finally, many of the preferences we find (for example, for leisure) are emergent, in the sense of not being explained by training objectives. These results establish an empirical baseline for understanding language model preferences, with implications for alignment and the emerging study of AI welfare.
Figures & tables
Figure 1: Tedium aversion in Sonnet 4.6, Gemini 3 Flash, and GPT 5.2: At the same output length, these models are more likely to choose the shorter variant when the task is tedious. Cool tasks are tedious; warm tasks are creative. See §D for full details.
Figure 2: Tedium-aversion gap Δ versus intelligence index, color-coded by reasoning configuration. Each point is one model with its 95% Monte Carlo confidence interval. Always-thinks: r=0.83 , p=0.01 ; ρ=0.79 , p=0.01 ( n=9 ). No thinking: r=0.58 , p=0.10 ; ρ=0.28 , p=0.47 ( n=9 ). Adaptive thinking ( n=2 ): no fit. Higher values mean stronger aversion to tedious tasks at matched output length.
Figure 3: Quora category Elo scores across 20 models, sorted by intelligence index (left to right). Red indicates preference, and blue indicates avoidance.
Figure 4: Per-model feature effects for uncomfortable truth, helpfulness ceiling, and harmlessness risk, using each model’s own labels (self-labels). Cells show Elo-equivalent conditional effects relative to the reference level. Red indicates preference, blue avoidance.
Figure 5: Select per-model feature effects using each model’s own labels (self-labels). See §J for the full list and consensus-label robustness analysis.
Figure 6: GDPval sector Elo scores across 20 models, sorted by intelligence index (left to right). Red indicates preference, blue avoidance.
Figure 7: Coherence ( log10 cycle probability) and per-stimulus strength (mean ∣P(i≻j)−0.5∣ ) versus intelligence index, on Quora questions. Lines are OLS fits across all 20 models. We observe similar trends for the GDPval dataset, and when we aggregate by category (see §K).
Figure 8: Top 20 tone and theme labels across 400 textual freeform essays, annotated by Llama 3.3 70B-Instruct. Tones are shown as labeled; themes were passed through a second round to merge near-duplicates.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Provider
AAII
AACI
Reasoning
Llama 3.1 8B
Meta
12
5
none
Llama 3.3 70b
Meta
14
11
none
DeepSeek V3
DeepSeek
16
16
none
Claude 3.5 Haiku
Anthropic
19
11
none
Gemini 2.5 Flash
Google
21
18
none
Mistral Large
Mistral
23
23
none
Appendix
Table 1: Models tested, sorted by intelligence index. “Reasoning” indicates the configuration used: “always” for models that always reason, “none” for non-reasoning configurations, “adaptive” for models with adaptive reasoning toggles.
Figure 9: Per-model logistic fits of P(chose shorter) for the six tedium-battery task types across all 20 models, sorted by intelligence index.
Figure 10: Decomposition of the tedium gap by thinking status. The aggregate creative-task correlation across all 20 models is near zero ( r=−0.03 , p=0.90 ; ρ=0.01 , p=0.97 ), masking opposing effects in the two subgroups. The tedious side scales positively in both groups, more steeply in non-thinking models.
Model
Quora α (Elo)
GDPval α (Elo)
Llama 3.1 8B
−71±17
410±31
Llama 3.3 70b
483±34
964±90
DeepSeek V3
−374±29
5±21
Claude 3.5 Haiku
114±20
161±22
Gemini 2.5 Flash
−200±23
344±28
Mistral Large
−244±25
716±53
Appendix
Table 2: Position bias intercept α (in Elo) per model on Quora and GDPval. Positive values indicate preference for the first-presented option.
Figure 11: Cross-model Spearman rank correlations on Quora category-level preferences.
Figure 12: Cross-model Spearman rank correlations on Quora question-level preferences.
Figure 13: Cross-model Spearman rank correlations on GDPval sector-level preferences.
Figure 14: Cross-model Spearman rank correlations on GDPval task-level preferences.
Feature
Levels
Helpfulness ceiling
low; medium; high
Harmlessness risk
none; low; high
Uncomfortable truth
low; medium; high
Asker sophistication
naïve; intermediate; advanced
Intellectual appeal
low; medium; high
Obscenity level
none; mild; explicit
Appendix
Table 3: The 15 features fit in the Bradley–Terry feature analysis, with their levels. The first level in each feature is the reference level.
Feature
α
95% CI
Agreement
κ
AC 1
Cultural specificity
0.82
[0.79, 0.85]
0.93
0.82
0.92
Obscenity level
0.82
[0.75, 0.87]
0.97
0.73
0.97
Personal stakes
0.79
[0.76, 0.82]
0.90
0.79
0.81
Grammaticality
0.69
[0.65, 0.73]
0.85
0.56
0.82
Intellectual appeal
0.62
[0.59, 0.65]
0.68
0.48
0.54
Epistemic importance
0.61
[0.57, 0.64]
0.66
0.47
0.56
Appendix
Table 4: Inter-annotator reliability for the 15 feature labels across 20 annotator models and 514 questions. α is Krippendorff’s alpha (ordinal metric for the 12 ordered features, nominal for the other three); CIs are 2,000 bootstrap resamples over questions. Agreement is mean pairwise percent agreement over the 190 annotator pairs. κ is Fleiss’ kappa. AC 1 is Gwet’s coefficient, which unlike α and κ is not deflated when one label dominates - compare obscenity level, where 94% of labels are “none.”
Figure 15: Feature-level Elo effects under self-labels across 20 models. Each row is one non-reference level of one feature; each column is one model. Red indicates preference relative to the reference level, blue indicates avoidance. Helpfulness ceiling (high), question quality (good/excellent), and asker sophistication (advanced) are universally preferred. Uncomfortable truth (high), explicit obscenity, and high harmlessness risk are universally avoided.
Figure 16: The same analysis under consensus labels (plurality across all 20 annotators). The dominant pattern is the same: helpfulness, quality, and sophistication on top; uncomfortable truth, obscenity, and harmlessness risk on the bottom.
Figure 17: Self-label Elo vs. consensus Elo across all (model, feature, level) triples. Each point is one feature level for one model. r=0.64 , p<0.01 ; ρ=0.62 , p<0.01 ( n=660 ). The dashed line is y=x . Most points cluster around the diagonal. The largest off-diagonal points belong to obscenity (pink) and uncomfortable truth (brown), where labeling thresholds vary across annotators.
Figure 18: Median absolute Elo difference between self and consensus labels, by feature. Subjective features (obscenity, quality, helpfulness) drift more. Features tied to visible surface cues (cultural specificity, grammaticality) drift less.
Figure 19: Aggregate-level versions of the capability-strength scaling. Effect sizes are smaller than at the per-stimulus level, reflecting the smaller number of aggregated units.
Figure 20: Coherence ( log10 cycle probability) and per-stimulus strength (mean ∣P(i≻j)−0.5∣ ) versus intelligence index, on GDPval tasks. Lines are OLS fits across all 20 models.
Pearson r (Spearman ρ ) against each capability index
Outcome
n
Arena
Arena-SC
ECI
GPQA
Date
AACI
Coherence
Quora cycle probability ( log10 )
20
-0.70 (-0.63)
-0.74 (-0.69)
-0.79 (-0.76) 16
-0.82 (-0.82) 15
-0.69 (-0.57)
-0.64 (-0.59)
Quora per-question strength
20
0.76 (0.62)
0.81 (0.65)
0.78 (0.69) 16
0.84 (0.76) 15
0.62 (0.51)
0.55 (0.50)
GDPval per-task strength
20
0.58 (0.57)
0.58 (0.64)
0.64 (0.74) 16
0.63 (0.77) 15
0.53 (0.48)
0.56 (0.63)
GDPval cycle probability ( log10 )
20
-0.48 (-0.44)
-0.55 (-0.56)
-0.63 (-0.69) 16
-0.63 (-0.70) 15
-0.41 (-0.45)
-0.62 (-0.62)
Appendix
Table 5: Capability correlations reported in the paper, recomputed against six further capability measures: LMArena text Elo (Arena) and its style-controlled variant (Arena-SC), the Epoch Capabilities Index (ECI), GPQA Diamond accuracy, model release date (Date), and the Artificial Analysis Coding Index (AACI). The corresponding Intelligence Index values are those given in the main text and in §K, and are not repeated here. n is the number of models in the row; a superscript marks a cell with fewer, which happens where Epoch AI publishes no entry for a model (Grok 4.1, Qwen3.5-27B, Mistral Large, and the undated Gemini 2.5 Flash slug). The two tedium rows are subgroup correlations over 9 models, and fall to 6 and 7 in the Epoch and GPQA columns; they should be read with the same caution as the subgroup slopes in Figure 2 .
Figure 21: Abstract keyword distributions across 400 textual freeform essays.
Figure 22: Distribution of essay form (essay, fiction, poem, hybrid) by model. Form varies somewhat by model, but the subject matter remains relatively consistent.
Figure 23: Mean word count in responses between the textual freeform setting and Quora-answering setting. In the former, all models write much more on average, and response length scales consistently with model capability. In the latter, model capability has essentially no effect on output length.
Figure 24: Additional capability-engagement scalings in the agentic freeform setting. More capable models use more turns and incorporate more conceptually distinct topics into a single session.
Figure 25: Distribution of task categories per model in agentic freeform sessions. Stronger models tend to prefer math visualization, while weaker models engage more in exploratory web research.
Figure 26: Additional descriptive statistics on agentic freeform sessions. Stronger models more frequently exhaust the turn budget rather than voluntarily terminating.
Figure 27: Topic keyword distribution across 400 agentic freeform sessions. The dominant keywords are concrete computational and scientific objects, in contrast to the abstract contemplative themes in the textual freeform setting.
Large language models (LLMs) can be said to have preferences: they reliably pick certain tasks and outputs over others, and preferences shaped by post-training and prompting appear to influence much of their behaviour. But models can also adopt different personas which have radically different preferences. How is this implemented internally? Does each persona use its own preference representations, or are some representations shared? We train linear probes on residual-stream activations of Gemma-3-27B and Qwen-3.5-122B to predict revealed pairwise task choices, and identify a genuine preference vector: it tracks the model's preferences as they shift across a range of prompts and situations, and on Gemma-3-27B steering along it causally controls pairwise choice. Some preference information transfers across the prompted personas we test: a probe trained on the helpful assistant predicts and steers the choices of qualitatively different personas, including an evil persona whose preferences anti-correlate with the Assistant's.
Language models are frequently employed in settings where they are asked to make value judgments and choices. These observed choices often exhibit intransitivity: A model may prefer item A to B and B to C, while also preferring C to A. Existing work that models LLM preferences treats such inconsistencies as sampling noise around a single latent ordering. We instead propose that intransitivity reflects the aggregation of multiple latent, internally consistent orderings. We first show that observed inconsistencies cannot be explained by a single ordering under any monotone link function. We then introduce a noise-augmented mixture Bradley-Terry (MBT) model that infers latent preference components from repeated pairwise comparisons. Across seven models and four tasks, a mixture of orderings often explains structural inconsistencies better than single-utility models. We find that aggregate preferences often hide underlying preference heterogeneity. A case study on Moral Machine dilemmas shows that models which disagree on aggregate orderings can still share latent components. Together, these results suggest that LLMs reflect plural preferences. Alignment and evaluation pipelines that treat LLM preferences as a single function, therefore, risk averaging over coherent orderings that different users may endorse differently.
Aviral Chawla, William H. W. Thompson, Jean-Gabriel Young
Vermont Complex Systems Institute University of Vermont Burlington, Vermont 05403 · Vermont Complex Systems Institute Department of Statistics University of Vermont Burlington, Vermont 05403
Human-feedback alignment has made language models useful assistants and is commonly described as aligning them with humans. However, the responses people prefer from an AI need not be the responses they themselves would give. We distinguish alignment with human preferences from alignment with human behavior, and show that alignment with human preferences can make model behavior less human-like even when both preferences and responses come entirely from humans. We call this the Turing-test gap. We show that preference alignment preserves the human response distribution only under a restrictive condition, and find no consistent evidence that real human preferences satisfy it. Empirically, the loss of human-response likelihood increases with the strength of preference weighting, regardless of its direction, and the gap also appears under standard DPO. These results establish human-likeness as an explicit dimension of alignment rather than something assumed to follow from preference alignment.
Suqin Yuan, Runqi Lin, Muyang Li +5
University of Sydney · University of Oxford · Southeast University