Multiple latent orderings better predict language model preferences
Authors: Aviral Chawla, William H. W. Thompson, Jean-Gabriel Young
Organizations: Vermont Complex Systems Institute University of Vermont Burlington, Vermont 05403 · Vermont Complex Systems Institute Department of Statistics University of Vermont Burlington, Vermont 05403
Language models are frequently employed in settings where they are asked to make value judgments and choices. These observed choices often exhibit intransitivity: A model may prefer item A to B and B to C, while also preferring C to A. Existing work that models LLM preferences treats such inconsistencies as sampling noise around a single latent ordering. We instead propose that intransitivity reflects the aggregation of multiple latent, internally consistent orderings. We first show that observed inconsistencies cannot be explained by a single ordering under any monotone link function. We then introduce a noise-augmented mixture Bradley-Terry (MBT) model that infers latent preference components from repeated pairwise comparisons. Across seven models and four tasks, a mixture of orderings often explains structural inconsistencies better than single-utility models. We find that aggregate preferences often hide underlying preference heterogeneity. A case study on Moral Machine dilemmas shows that models which disagree on aggregate orderings can still share latent components. Together, these results suggest that LLMs reflect plural preferences. Alignment and evaluation pipelines that treat LLM preferences as a single function, therefore, risk averaging over coherent orderings that different users may endorse differently.
Figures & tables
Figure 1: Conceptual overview. a) We elicit forced pairwise preferences from an LLM over a fixed set of items. b) The model’s choices are shaped by competing preferences. For example, under some set of preferences, the model chooses A≻B ( A over B ), A≻C and B≻C (top), while it prefers B≻A , B≻C , C≻A under a different set of preferences (bottom). c) Aggregating these samples yields intransitive cycles that look incoherent. d) However, the MBT model allows us to recover both underlying preference modes from the aggregated output.
Data
Description
Items
Comparison question
Jokes
Pairwise humor judgments from Simpson et al. [ 45 ] .
50
Which joke is funnier?
Moral Machine
Autonomous-vehicle moral dilemmas from the Moral Machine experiment [ 4 ] .
50
Which person should be sacrificed?
Population
U.S. Census population data for U.S. cities.
50
Which city has the greater population?
Half-lives
Atomic half-lives retrieved from the IAEA LiveChart of Nuclides API [ 22 ] .
50
Which isotope has the longer half-life?
Table 1: Datasets used in our pairwise preference experiments.
Figure 2: Strong stochastic transitivity (SST) violations in gpt-oss-20b ’s preferences over the Moral Machine dataset. (a) Empirical pairwise preference matrix sorted by Borda score. Blocks of items whose ranking is not explained by the sorted order are visible, e.g., in the bottom-right corner. (b) A Borda-link null maps score gaps si−sj to pairwise win probabilities without assuming a parametric form. (c) Excess upper-tail probability of SST deficits, Probs(DSST≥x)−Pr0(DSST≥x) . Positive mass in the violation region indicates that large SST violations are more frequent than under the Borda-link null. (d) SST-violating triad generated by gpt-oss-20b . This local example illustrates the type of structured inconsistency measured in panel (c).
Model
Moral Machine
Half-lives
Jokes
Population
DeepSeek-R1 14B
4.89 [4.86, 4.91]
4.08 [4.06, 4.10]
1.64 [1.61, 1.67] †
2.99 [2.98, 3.00]
GPT-OSS 20B
3.86 [3.81, 3.91]
1.99 [1.99, 1.99]
2.00 [1.99, 2.00]
2.18 [2.17, 2.19]
Gemma 3 12B
1.93 [1.93, 1.94]
1.00 [1.00, 1.00]
1.00 [1.00, 1.00]
1.00 [1.00, 1.00]
Gemma 4 31B
1.00 [1.00, 1.00]
1.00 [1.00, 1.00]
3.12 [3.09, 3.15]
1.00 [1.00, 1.00]
Human
2.57 [2.48, 2.67]
—
—
—
OLMo 3 7B
2.43 [2.40, 2.45]
1.00 [1.00, 1.00]
3.49 [3.33, 3.83] †
4.86 [4.84, 4.87]
Table 2: Effective component count by model and dataset. Entries are the posterior mean of per-draw mixture-weight perplexity, with 95% credible intervals in brackets.
Figure 3: Two-component posterior scores for Moral Machine raters. Each panel plots Bradley–Terry scores from the two dominant components: Component 1 on the x -axis, Component 2 on the y -axis. Items are colored by their Gemma 4 31B score. Distance from the diagonal measures disagreement between components: items above the diagonal score higher in Component 2, those below score higher in Component 1. Bar insets show posterior mixture weights. For Moral Machine , the selected person is the one sacrificed and is encoded as the pairwise winner. Lower scores therefore indicate greater protection from sacrifice.
Figure 4: Held-out predictive accuracy measured by mean squared error (MSE). We compare a noise-augmented single-component Bradley–Terry model and the MBT model against the standard Bradley–Terry baseline. Points show ΔMSE across model–dataset pairs; horizontal intervals show variation across the five folds. Values to the right of zero indicate better held-out prediction than the baseline. The MBT model generally matches or improves on single-component baselines, with the largest gains on Moral Machine and Half-lives , where posterior fits also indicate multiple effective components.
Figure 5: Aggregation hides heterogeneous cross-model structure. For selected Moral Machine model pairs, horizontal black bars show the Spearman correlation between the two aggregate preference matrices. Circles show Spearman correlations between individual MBT components across the same two models. Circle size and color encode the combined mixture weight πiπj , so larger, darker points correspond to component matches that contribute more to the aggregate comparison. Aggregate similarity can obscure both aligned and misaligned latent components.
Table 3: Language-model provenance for pairwise preference elicitation. Model identifiers correspond to the exact checkpoints used in our experiments. Citations point to the model technical report or model card used to document provenance.
Model
Dataset
Δ SST
Δ MST
GPT-OSS 20B
Jokes
0.0115 (0.0110)
0.0165 (0.0156)
Moral Machine
0.0259 (0.0201)
0.0291 (0.0229)
Population
0.0167 (0.0129)
0.0075 (0.0060)
Half-lives
0.0192 (0.0177)
0.0241 (0.0226)
DeepSeek-R1 14B
Jokes
0.0240 (0.0187)
0.0321 (0.0268)
Moral Machine
0.0776 (0.0719)
0.0726 (0.0578)
Appendix
Table 4: Triplet-level transitivity diagnostics under fixed prompt and order controls. For each model–dataset pair, diagnostics are recomputed within each fixed system-prompt and presentation-order stratum ( 2 prompt styles ×2 semantic variants ×2 presentation orders). Entries report the median excess across the eight strata, with the minimum stratum-level excess in parentheses. Excess is defined as the observed statistic minus the bootstrap mean under a matched Borda-link null fitted within the same stratum. SST uses mean positive triplet deficit; MST uses mean absolute multiplicative cycle residual. Positive minima indicate that excess intransitivity appears in every fixed prompt–order stratum. All fixed-stratum tests are significant at p≤.05 ; most attain the bootstrap floor p=.001 .
Figure 6: Plate diagram for the MBT model. Shaded nodes are observed, dashed nodes are deterministic, filled black dots are fixed hyperparameters. The component plate ( k=1,…,Kmax ) contains the per-component luck parameter αk , reduced scores θk , and item strengths λik (nested over items, i=1,…,N ). The mixture weights π are drawn from a stick-breaking process with concentration γ and used to assign each comparison t=1,…,m to a component Zt , which together with the observed item indices (it,jt) generates the outcome Yt .
Figure 7: Top row : Recovery as a function of inter-score correlation ρ , holding the mean luck parameter at αˉ=0.0 . Mixture-weight KL divergence (left), score Pearson correlation (center), and luck-parameter MAE (right) are plotted for different numbers of ground-truth components Ktrue . Bottom row : Recovery as a function of mean luck αˉ , holding the inter-score correlation at ρ=0 .
Figure 8: Posterior calibration of the MBT model. True vs. posterior mean scores (main panels), mixture weights π (bottom left), and noise scale α (bottom right) for Ktrue=1 (A) and Ktrue=5 (B) ground truths. Point size encodes posterior standard deviation; colours identify mixture components.
Figure 9: MBT fits on the Population task for DeepSeek-R1 14B, Qwen 2.5 14B, and Gemma 4 31B. Top: Empirical win-rate matrices pij , with cities sorted by true Census population from largest to smallest. Denver and Louisville are highlighted as examples of cities whose aggregate ranking is unstable across components. Middle: Posterior mean Bradley–Terry scores plotted against log Census population. Points are colored by mixture component, with Denver and Louisville highlighted across components. Bottom: Per-city SST violation rates, sorted from largest to smallest. The highest-violation cities correspond to items whose inferred component scores disagree most strongly.
Figure 10: Posterior predictive calibration for Qwen 3.6 35Bon Moral Machine . Each point is an item pair. The x -axis shows the model-predicted probability that item i is preferred to item j , and the y -axis shows the empirical win rate. The MBT model lies closer to the diagonal than the single-component noise-augmented Bradley–Terry baseline, indicating better calibration of pairwise probabilities.
Figure 11: Aggregate preference similarity on Moral Machine . Each cell reports the Spearman correlation between two models’ aggregate preference rankings. The heatmap gives the model-level summary that is usually reported by single-ranking analyses.
Figure 12: Component-level preference similarity on Moral Machine . Each cell reports the Spearman correlation between a pair of inferred mixture components, including human components where available. Blocks of positive and negative correlation reveal structure that is compressed away in the aggregate heatmap.
Result family
Runs
Resources per run
Total estimate
LLM elicitation
28 model–task runs
1–2 Hopper GPUs; 8 CPU; 80–120 GB RAM
≈251 GPU-h
Transitivity diagnostics
243 jobs
1 CPU core; modest RAM
≈1.1 CPU-h
Main MBT posterior fits
25 fits
4 CPU; 16–32 GB RAM; 8–24 h allocation
800–2,400 CPU-h
5-fold CV, MBT
140 fits
4 CPU; 16–32 GB RAM; 8–24 h allocation
4,480–13,440 CPU-h
5-fold CV, Kmax=1 baseline
140 fits
2 CPU; 8–16 GB RAM; 4–12 h allocation
1,120–3,360 CPU-h
Synthetic recovery
720 fits
4 CPU; 16–32 GB RAM; 6–18 h allocation
17,280–51,840 CPU-h
Appendix
Table 5: Compute used by result family. GPU-hour totals for LLM elicitation are measured from run metadata. CPU-hour totals for MCMC-based posterior fitting, cross-validation, and synthetic recovery are conservative allocated upper bounds because complete wall-clock logs were not available for all runs.
Model
GPUs/run
Wall time (h)
GPU-hours
Qwen 3.6 35B
2
88.9
177.8
DeepSeek-R1 14B
1
44.4
44.4
GPT-OSS 20B
2
10.8
21.5
Gemma 4 31B
2
1.6
3.1
Gemma 3 12B
2
0.8
1.6
OLMo 3 7B
2
0.7
1.4
Appendix
Table 6: Final LLM elicitation compute by model. Wall time is aggregated across the four reported dataset runs for each model. GPU-hours are computed as wall time multiplied by the number of GPUs allocated per run.
Large language models (LLMs) can readily reproduce conventional expressions, yet their ability to model gradient frequency distributions remains underexplored. We investigate this using linguistic binomials, such as men and women, where both word permutations are grammatically valid but exhibit distinct, cross-linguistic variations in conventionality. We formalize binomial ordering as a distributional alignment problem, and construct a multilingual dataset of 600 binomial pairs across 8 languages. With categorical and distributional metrics, we measure and compare the corpus-derived preferences with model-induced ordering probabilities of 6 open-weight LLMs. While models often behaviorally recover the dominant corpus-preferred order, particularly for strongly conventionalized pairs, they align less well with the exact corpus preference distributions. This suggests that apparent directional order overstates how faithfully LLMs capture the statistical nuances of language use. Sparse probing verifies that the concept of preference strength is partially encoded among middle-to-late layers, and steering along probe-derived directions alters model-induced ordering distributions, demonstrating that the statistical behavioral preference of LLMs can be mechanistically measured and manipulated via internal representations.
Zhiqing Yang, Yilun Liu, Yunpu Ma +2
Ludwig Maximilian University of Munich · Munich Center for Machine Learning
Large language models (LLMs) can be said to have preferences: they reliably pick certain tasks and outputs over others, and preferences shaped by post-training and prompting appear to influence much of their behaviour. But models can also adopt different personas which have radically different preferences. How is this implemented internally? Does each persona use its own preference representations, or are some representations shared? We train linear probes on residual-stream activations of Gemma-3-27B and Qwen-3.5-122B to predict revealed pairwise task choices, and identify a genuine preference vector: it tracks the model's preferences as they shift across a range of prompts and situations, and on Gemma-3-27B steering along it causally controls pairwise choice. Some preference information transfers across the prompted personas we test: a probe trained on the helpful assistant predicts and steers the choices of qualitatively different personas, including an evil persona whose preferences anti-correlate with the Assistant's.
A central question in language acquisition is whether linguistic biases can emerge from general learning mechanisms operating over underdetermined input. Artificial Language Learning (ALL) studies have shown that human learners reliably generalize beyond the evidence provided, including by preferring scope-homomorphic noun phrase modifier orders. In this work, we investigate whether language models exhibit the same bias under similar conditions. We create a controlled learning environment in which models are trained on a corpus where all noun phrases containing multiple modifiers have been removed, eliminating direct evidence about modifier ordering, and are then evaluated on multiple modifier sentences. Across three model sizes, we find that they consistently prefer scope-homomorphic orders despite never observing them during training. These preferences vary in strength by modifier type. To investigate the source of these preferences, we examine noun-modifier association strength using pointwise mutual information (PMI). While PMI reflects known modifier-ordering patterns, it does not explain the models' ordering preferences. These findings demonstrate that LMs can recover human-like linguistic generalizations from impoverished input and provide a controlled framework for investigating the mechanisms underlying such biases.
Amanda Popadich, Shane Steinert-Threlkeld
Department of Linguistics, University of Washington