cs.LGAug 26, 2026
SaveMultiple latent orderings better predict language model preferences
Organizations: Vermont Complex Systems Institute University of Vermont Burlington, Vermont 05403 · Vermont Complex Systems Institute Department of Statistics University of Vermont Burlington, Vermont 05403
Abstract
Language models are frequently employed in settings where they are asked to make value judgments and choices. These observed choices often exhibit intransitivity: A model may prefer item to and to , while also preferring to . Existing work that models LLM preferences treats such inconsistencies as sampling noise around a single latent ordering. We instead propose that intransitivity reflects the aggregation of multiple latent, internally consistent orderings. We first show that observed inconsistencies cannot be explained by a single ordering under any monotone link function. We then introduce a noise-augmented mixture Bradley-Terry (MBT) model that infers latent preference components from repeated pairwise comparisons. Across seven models and four tasks, a mixture of orderings often explains structural inconsistencies better than single-utility models. We find that aggregate preferences often hide underlying preference heterogeneity. A case study on Moral Machine dilemmas shows that models which disagree on aggregate orderings can still share latent components. Together, these results suggest that LLMs reflect plural preferences. Alignment and evaluation pipelines that treat LLM preferences as a single function, therefore, risk averaging over coherent orderings that different users may endorse differently.
Figures & tables
Figure 1: Conceptual overview. a) We elicit forced pairwise preferences from an LLM over a fixed set of items. b) The model’s choices are shaped by competing preferences. For example, under some set of preferences, the model chooses ( over ), and (top), while it prefers , , under a different set of preferences (bottom). c) Aggregating these samples yields intransitive cycles that look incoherent. d) However, the MBT model allows us to recover both underlying preference modes from the aggregated output.
| Data | Description | Items | Comparison question |
|---|---|---|---|
| Jokes | Pairwise humor judgments from Simpson et al. [ 45 ] . | 50 | Which joke is funnier? |
| Moral Machine | Autonomous-vehicle moral dilemmas from the Moral Machine experiment [ 4 ] . | 50 | Which person should be sacrificed? |
| Population | U.S. Census population data for U.S. cities. | 50 | Which city has the greater population? |
| Half-lives | Atomic half-lives retrieved from the IAEA LiveChart of Nuclides API [ 22 ] . | 50 | Which isotope has the longer half-life? |
Table 1: Datasets used in our pairwise preference experiments.
Figure 2: Strong stochastic transitivity (SST) violations in gpt-oss-20b ’s preferences over the Moral Machine dataset. (a) Empirical pairwise preference matrix sorted by Borda score. Blocks of items whose ranking is not explained by the sorted order are visible, e.g., in the bottom-right corner. (b) A Borda-link null maps score gaps to pairwise win probabilities without assuming a parametric form. (c) Excess upper-tail probability of SST deficits, . Positive mass in the violation region indicates that large SST violations are more frequent than under the Borda-link null. (d) SST-violating triad generated by gpt-oss-20b . This local example illustrates the type of structured inconsistency measured in panel (c).
| Model | Moral Machine | Half-lives | Jokes | Population |
|---|---|---|---|---|
| DeepSeek-R1 14B | 4.89 [4.86, 4.91] | 4.08 [4.06, 4.10] | 1.64 [1.61, 1.67] | 2.99 [2.98, 3.00] |
| GPT-OSS 20B | 3.86 [3.81, 3.91] | 1.99 [1.99, 1.99] | 2.00 [1.99, 2.00] | 2.18 [2.17, 2.19] |
| Gemma 3 12B | 1.93 [1.93, 1.94] | 1.00 [1.00, 1.00] | 1.00 [1.00, 1.00] | 1.00 [1.00, 1.00] |
| Gemma 4 31B | 1.00 [1.00, 1.00] | 1.00 [1.00, 1.00] | 3.12 [3.09, 3.15] | 1.00 [1.00, 1.00] |
| Human | 2.57 [2.48, 2.67] | — | — | — |
| OLMo 3 7B | 2.43 [2.40, 2.45] | 1.00 [1.00, 1.00] | 3.49 [3.33, 3.83] | 4.86 [4.84, 4.87] |
Table 2: Effective component count by model and dataset. Entries are the posterior mean of per-draw mixture-weight perplexity, with 95% credible intervals in brackets.
Figure 3: Two-component posterior scores for Moral Machine raters. Each panel plots Bradley–Terry scores from the two dominant components: Component 1 on the -axis, Component 2 on the -axis. Items are colored by their Gemma 4 31B score. Distance from the diagonal measures disagreement between components: items above the diagonal score higher in Component 2, those below score higher in Component 1. Bar insets show posterior mixture weights. For Moral Machine , the selected person is the one sacrificed and is encoded as the pairwise winner. Lower scores therefore indicate greater protection from sacrifice.
Figure 4: Held-out predictive accuracy measured by mean squared error (MSE). We compare a noise-augmented single-component Bradley–Terry model and the MBT model against the standard Bradley–Terry baseline. Points show across model–dataset pairs; horizontal intervals show variation across the five folds. Values to the right of zero indicate better held-out prediction than the baseline. The MBT model generally matches or improves on single-component baselines, with the largest gains on Moral Machine and Half-lives , where posterior fits also indicate multiple effective components.
Figure 5: Aggregation hides heterogeneous cross-model structure. For selected Moral Machine model pairs, horizontal black bars show the Spearman correlation between the two aggregate preference matrices. Circles show Spearman correlations between individual MBT components across the same two models. Circle size and color encode the combined mixture weight , so larger, darker points correspond to component matches that contribute more to the aggregate comparison. Aggregate similarity can obscure both aligned and misaligned latent components.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Display name | Model identifier | License / terms |
|---|---|---|
| DeepSeek-R1 14B | deepseek-ai/DeepSeek-R1 14B-R1-Distill-Qwen-14B [ 15 ] | MIT |
| GPT-OSS 20B | openai/gpt-oss-20b [ 33 ] | Apache 2.0 |
| Gemma 3 12B | google/gemma-3-12b-it [ 50 ] | Gemma terms |
| Gemma 4 31B | google/gemma-4-31b-it [ 19 ] | Gemma terms |
| OLMo 3 7BThink | allenai/Olmo-3-7B-Think [ 32 ] | Apache 2.0 |
| Qwen 2.5 14B | Qwen/Qwen2.5-14B-Instruct [ 51 , 65 ] | Apache 2.0 |
Table 3: Language-model provenance for pairwise preference elicitation. Model identifiers correspond to the exact checkpoints used in our experiments. Citations point to the model technical report or model card used to document provenance.
| Model | Dataset | SST | MST |
|---|---|---|---|
| GPT-OSS 20B | Jokes | 0.0115 (0.0110) | 0.0165 (0.0156) |
| Moral Machine | 0.0259 (0.0201) | 0.0291 (0.0229) | |
| Population | 0.0167 (0.0129) | 0.0075 (0.0060) | |
| Half-lives | 0.0192 (0.0177) | 0.0241 (0.0226) | |
| DeepSeek-R1 14B | Jokes | 0.0240 (0.0187) | 0.0321 (0.0268) |
| Moral Machine | 0.0776 (0.0719) | 0.0726 (0.0578) |
Table 4: Triplet-level transitivity diagnostics under fixed prompt and order controls. For each model–dataset pair, diagnostics are recomputed within each fixed system-prompt and presentation-order stratum ( prompt styles semantic variants presentation orders). Entries report the median excess across the eight strata, with the minimum stratum-level excess in parentheses. Excess is defined as the observed statistic minus the bootstrap mean under a matched Borda-link null fitted within the same stratum. SST uses mean positive triplet deficit; MST uses mean absolute multiplicative cycle residual. Positive minima indicate that excess intransitivity appears in every fixed prompt–order stratum. All fixed-stratum tests are significant at ; most attain the bootstrap floor .
Figure 6: Plate diagram for the MBT model. Shaded nodes are observed, dashed nodes are deterministic, filled black dots are fixed hyperparameters. The component plate ( ) contains the per-component luck parameter , reduced scores , and item strengths (nested over items, ). The mixture weights are drawn from a stick-breaking process with concentration and used to assign each comparison to a component , which together with the observed item indices generates the outcome .
Figure 7: Top row : Recovery as a function of inter-score correlation , holding the mean luck parameter at . Mixture-weight KL divergence (left), score Pearson correlation (center), and luck-parameter MAE (right) are plotted for different numbers of ground-truth components . Bottom row : Recovery as a function of mean luck , holding the inter-score correlation at .
Figure 8: Posterior calibration of the MBT model. True vs. posterior mean scores (main panels), mixture weights (bottom left), and noise scale (bottom right) for (A) and (B) ground truths. Point size encodes posterior standard deviation; colours identify mixture components.
Figure 9: MBT fits on the Population task for DeepSeek-R1 14B, Qwen 2.5 14B, and Gemma 4 31B. Top: Empirical win-rate matrices , with cities sorted by true Census population from largest to smallest. Denver and Louisville are highlighted as examples of cities whose aggregate ranking is unstable across components. Middle: Posterior mean Bradley–Terry scores plotted against log Census population. Points are colored by mixture component, with Denver and Louisville highlighted across components. Bottom: Per-city SST violation rates, sorted from largest to smallest. The highest-violation cities correspond to items whose inferred component scores disagree most strongly.
Figure 10: Posterior predictive calibration for Qwen 3.6 35Bon Moral Machine . Each point is an item pair. The -axis shows the model-predicted probability that item is preferred to item , and the -axis shows the empirical win rate. The MBT model lies closer to the diagonal than the single-component noise-augmented Bradley–Terry baseline, indicating better calibration of pairwise probabilities.
Figure 11: Aggregate preference similarity on Moral Machine . Each cell reports the Spearman correlation between two models’ aggregate preference rankings. The heatmap gives the model-level summary that is usually reported by single-ranking analyses.
Figure 12: Component-level preference similarity on Moral Machine . Each cell reports the Spearman correlation between a pair of inferred mixture components, including human components where available. Blocks of positive and negative correlation reveal structure that is compressed away in the aggregate heatmap.
| Result family | Runs | Resources per run | Total estimate |
|---|---|---|---|
| LLM elicitation | 28 model–task runs | 1–2 Hopper GPUs; 8 CPU; 80–120 GB RAM | GPU-h |
| Transitivity diagnostics | 243 jobs | 1 CPU core; modest RAM | CPU-h |
| Main MBT posterior fits | 25 fits | 4 CPU; 16–32 GB RAM; 8–24 h allocation | 800–2,400 CPU-h |
| 5-fold CV, MBT | 140 fits | 4 CPU; 16–32 GB RAM; 8–24 h allocation | 4,480–13,440 CPU-h |
| 5-fold CV, baseline | 140 fits | 2 CPU; 8–16 GB RAM; 4–12 h allocation | 1,120–3,360 CPU-h |
| Synthetic recovery | 720 fits | 4 CPU; 16–32 GB RAM; 6–18 h allocation | 17,280–51,840 CPU-h |
Table 5: Compute used by result family. GPU-hour totals for LLM elicitation are measured from run metadata. CPU-hour totals for MCMC-based posterior fitting, cross-validation, and synthetic recovery are conservative allocated upper bounds because complete wall-clock logs were not available for all runs.
| Model | GPUs/run | Wall time (h) | GPU-hours |
|---|---|---|---|
| Qwen 3.6 35B | 2 | 88.9 | 177.8 |
| DeepSeek-R1 14B | 1 | 44.4 | 44.4 |
| GPT-OSS 20B | 2 | 10.8 | 21.5 |
| Gemma 4 31B | 2 | 1.6 | 3.1 |
| Gemma 3 12B | 2 | 0.8 | 1.6 |
| OLMo 3 7B | 2 | 0.7 | 1.4 |
Table 6: Final LLM elicitation compute by model. Wall time is aggregated across the four reported dataset runs for each model. GPU-hours are computed as wall time multiplied by the number of GPUs allocated per run.