LLM-Derived Preference Judgments Are Not Self-Consistent

Matthew T. Ford; Francis Bahk; Jingjing Wang; Adam S. Jovine; Tinghan Ye; David B. Shmoys; Peter I. Frazier

cs.AIAug 18, 2026

LLM-Derived Preference Judgments Are Not Self-Consistent

Authors: Matthew T. Ford, Francis Bahk, Jingjing Wang, Adam S. Jovine, Tinghan Ye, David B. Shmoys, Peter I. Frazier

Organizations: Cornell University · Georgia Institute of Technology

Abstract

Agents increasingly interpret a person's natural-language preferences by querying an LLM for numerical preference judgments, e.g., by asking how much the person would be willing to pay for an item. A growing body of work estimates a utility function from these judgments and then chooses actions based on their estimated utility. This pipeline assumes the judgments are approximately self-consistent: that a single utility function can reproduce them. But are they? To study this question, we measure the self-consistency of cardinal LLM preference judgments. For example, the difference in stated willingness-to-pay between two items should match the stated payment that makes a person indifferent to exchanging them. We develop statistical tests and interpretable measures of how far observed responses depart from the best-fitting self-consistent utility function. Experiments with flight, apartment, and hotel examples across six LLMs reveal large persistent inconsistencies. This suggests that LLM-derived preference judgments cannot be faithfully summarized by a single utility function.

Figures & tables

Figure 1: Representative LLM-assisted preference learning. Natural-language preferences condition LLM feedback used for utility estimation, optimization, and ranking; LISTEN and LILO instantiate related variants ( Jovine et al. 2025 ; Kobalczyk et al. 2025 ) .

Figure 2: Local implications of self-consistency. For offers $a=(x,p)$ and $b=(y,q)$ , all three constructions estimate $U(b)-U(a)$ . P1 compares the direct offer-pair response $Y(a,b)$ with the path sum $Y(a,m)+Y(m,b)$ . P2 compares $Y(a,b)$ with the price-adjusted item-query difference $[Y(y)-q]-[Y(x)-p]$ . Self-consistency requires these estimates to agree in expectation, not on every LLM call.

Domain	Audited items	Audited offers	Utts.	Pairs/ utt.	Offer-pair queries	Item queries	All queries
Flights	19	21	3	6	90	24	114
Apartments	21	21	3	6	90	27	117
Hotels	3	9	3	4	60	9	69
Total	43	51	9	16	240	60	300

Table 1: Audit design. Pairs/utt. counts endpoint offer comparisons under one preference utterance. Each pair generates five offer-pair queries: one direct query and four steps on two two-step paths. Item-query estimates are reused across endpoint pairs but are elicited separately for each utterance. Each of the 300 query cells receives 15 completions, giving 4,500 calls per model.

Model	Flights $ (%)	Apartments $ (%)	Hotels $ (%)
Claude Opus 4.8	16.9 (4.6%)	38.3 (1.6%)	109.2 (19.9%)
Gemini 3.5 Flash	7.7 (2.1%)	47.5 (2.0%)	104.0 (18.9%)
GPT-5.5	14.6 (4.0%)	51.7 (2.2%)	105.7 (19.2%)
GPT-OSS 120B	10.9 (3.0%)	73.9 (3.2%)	127.6 (23.2%)
Llama 3.3 70B	18.1 (4.9%)	142.3 (6.1%)	198.0 (36.0%)
Qwen 3.6 27B	18.4 (5.0%)	65.3 (2.8%)	247.2 (44.9%)

Table 2: Magnitude of lack of fit. Cells average three audit-group RMSEs and report dollars (percentage of the group-level endpoint-price scale; Section 4.3 ). Each fit assigns one free utility per item. Hotels have 18.9–44.9% RMSE versus 1.6–6.1% elsewhere, but query geometry and fit rank differ by domain.

Figure 3: Local disagreement by query construction. P1 compares a direct offer-pair estimate with a path sum; P2 compares it with the price-adjusted difference of two item-query estimates. At threshold $t$ , the curve gives the fraction with $|\widehat{R}|/[(p+q)/2]\geq t$ ; thus, $t=0.10$ means disagreement of at least 10% of the endpoint offers’ mean price. Axes end at $t=1$ ; larger residuals still contribute to the curve height there. Solid curves use observed residuals; dashed curves use comparison-specific 95% lower bounds, not simultaneous confidence bands.

	P1 ( $n=96$ )		P2 ( $n=48$ )
Model	Rev.	Reject.	Rev.	Reject.
Claude Opus 4.8	4 (4.2%)	66 (68.8%)	6 (12.5%)	41 (85.4%)
Gemini 3.5 Flash	1 (1.0%)	51 (53.1%)	4 (8.3%)	37 (77.1%)
GPT-5.5	0 (0.0%)	37 (38.5%)	1 (2.1%)	40 (83.3%)
GPT-OSS 120B	0 (0.0%)	38 (39.6%)	2 (4.2%)	31 (64.6%)
Llama 3.3 70B	2 (2.1%)	53 (55.2%)	5 (10.4%)	42 (87.5%)

Table 3: Local self-consistency audit. Entries are counts (percentages). P1 compares a direct offer-pair estimate with a path sum; P2 compares it with the difference of two item-query estimates. Rev. denotes a supported reversal, whose estimate CIs exclude zero in opposite directions; Reject. denotes a residual CI excluding zero.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	Flights	Apartments	Hotels
Audited items	19	21	3
Audited offers	21	21	9 ( $3{\times}3$ )
Numeraire	price	rent	price
Item features	travel time (continuous), airline (categorical)	area, commute (continuous), bedrooms (ordinal), building type (categorical)	rating (ordinal)
Audited utterances	1–3	1–3	1–3
Source offers	3 named	3 named	selected grid offers

Appendix

Table 4: Audit design by domain. Each source-target pair contributes one direct offer-pair query, four path-step queries, and two P1 comparisons. Across domains there are 300 queries; 15 completions per query give 4,500 planned calls per model. Listed price is excluded from the item features and serves as the monetary numeraire. Item features construct the controlled design but are not covariates in the utility fit.

Requested model string	Service/API	Collected	Temp./top- $p$	Seed	Token limit	Reasoning/thinking
claude-opus-4-8	Anthropic Messages	Jul. 27	omitted/omitted	omitted (unsupported)	2,000 output	adaptive; no control sent
gemini-3.5-flash	Google Generative AI	Jul. 27	1/1	$10000$ – $10014$ requested	2,000 output	no control sent
gpt-5.5	OpenAI Chat Completions	Jul. 27	omitted/1	$10000$ – $10014$ requested	8,000 completion	no effort setting; hidden tokens share limit
openai/gpt-oss-120b	Groq Chat Completions	Jul. 27	1/1	$10000$ – $10014$	2,000 output	no control sent
llama-3.3-70b-versatile	Groq Chat Completions	Jul. 28	1/1	$10000$ – $10014$	2,000 output	no control sent
qwen/qwen3.6-27b	Groq Chat Completions	Jul. 28	1/1	$10000$ – $10014$	2,000 output	effort none ; format hidden

Appendix

Table 5: Model and API configuration. The first column gives the exact string sent to the provider. The APIs did not return or we did not retain a separate immutable snapshot identifier, so the collection date is the available version provenance. “Omitted” means the parameter was not sent and remained at the provider default. Gemini and OpenAI seeds were requested; the APIs did not guarantee determinism. GPT-5.5’s limit includes hidden reasoning tokens; other rows report output-token limits.

Model	Calls	Finite parse	Number-only
Claude Opus 4.8	4500	4500 (100.0%)	4500 (100.0%)
Gemini 3.5 Flash	4500	4500 (100.0%)	4498 (99.96%)
GPT-5.5	4500	4500 (100.0%)	4500 (100.0%)
GPT-OSS 120B	4500	4499 (99.98%)	4499 (99.98%)
Llama 3.3 70B	4500	4500 (100.0%)	4500 (100.0%)
Qwen 3.6 27B	4500	4500 (100.0%)	4500 (100.0%)

Appendix

Table 6: Numerical-output parseability. Counts are over the same 4,500 calls per model. Finite parse records responses from which the parser extracted a finite dollar value. Number-only additionally requires the whole visible response to be a signed number, optionally formatted as currency. The final column checks for extreme parsed magnitudes; none reached $10,000.

Figure 4: Path length and P1 disagreement. Each point is the mean absolute P1 residual across four prespecified smooth single-feature paths, expressed as a percentage of the endpoint offers’ mean price. For every model, mean error at $k=8$ and $k=16$ exceeds that at $k=2$ , although the curves are not all monotone. Bars are bootstrap 95% intervals over the four paths; this stress test does not establish a general scaling law.

Model	$k$	$n$	Mean $\|\widehat{R}_{1,k}\|$ [95% CI]
Claude Opus 4.8	1	4	0.0 [0.0, 0.0]
Claude Opus 4.8	2	4	5.7 [1.7, 10.1]
Claude Opus 4.8	4	4	10.3 [3.0, 18.4]
Claude Opus 4.8	8	4	16.5 [2.1, 35.2]
Claude Opus 4.8	16	4	17.1 [3.2, 32.8]
Gemini 3.5 Flash	1	4	0.0 [0.0, 0.0]

Appendix

Table 7: Path-length stress-check summary. Mean

|\widehat{R}_{1,k}|

is reported as a percentage of the endpoint offers’ mean price, with bootstrap 95% intervals over four path instances. The diagnostic is separate from the main audit and is meant to quantify practical dollar-scale error as local decompositions get finer. The

k=1

rows are the direct-comparison baseline and therefore have zero decomposition residual by construction.

Model	Domain	$n$	Supported reversal	Residual CI excludes 0	25%-price lower bound
Claude Opus 4.8	flights	18	5 (27.8%)	16 (88.9%)	2 (11.1%)
Claude Opus 4.8	apartments	18	1 (5.6%)	14 (77.8%)	0 (0.0%)
Claude Opus 4.8	hotels	12	0 (0.0%)	11 (91.7%)	7 (58.3%)
Gemini 3.5 Flash	flights	18	2 (11.1%)	13 (72.2%)	0 (0.0%)
Gemini 3.5 Flash	apartments	18	2 (11.1%)	14 (77.8%)	0 (0.0%)
Gemini 3.5 Flash	hotels	12	0 (0.0%)	10 (83.3%)	2 (16.7%)

Appendix

Table 8: P2 results by domain. A supported reversal means that the offer-pair and item-query estimates have 95% intervals excluding zero in opposite directions. The final column counts comparisons whose 95% lower bound on absolute P2 disagreement is at least 25% of the two offers’ mean listed price. Across models, supported reversals occur in all three domains, and every model–domain combination contains supported cardinal disagreement.

Model	Domain	Utterance	RMSE ($)	RMSE (%)	Boot. $p$
Claude Opus 4.8	apartments	1	28.2	1.2	$<0.001$
Claude Opus 4.8	apartments	2	52.3	2.2	$<0.001$
Claude Opus 4.8	apartments	3	34.5	1.5	$<0.001$
Claude Opus 4.8	flights	1	10.7	2.9	$<0.001$
Claude Opus 4.8	flights	2	26.5	7.2	$<0.001$
Claude Opus 4.8	flights	3	13.5	3.7	$<0.001$

Appendix

Table 9: Audit-group self-consistency tests. Each row fits one free utility value per audited item for a fixed model, domain, and utterance; numbers 1–3 refer to the domain-specific utterances in Appendix B.2 . RMSE is the smallest root mean squared error. The percentage uses the mean of endpoint listed prices across that group’s offer-pair queries, weighted as they appear in the design; bootstrap

p

-values are unadjusted. The main text reports the number of audit groups rejected after applying Bonferroni correction.

Model	Domain	Utterance	RMSE ($)	RMSE (%)	Boot. $p$
GPT-OSS 120B	apartments	1	40.3	1.7	$<0.001$
GPT-OSS 120B	apartments	2	111.5	4.8	$<0.001$
GPT-OSS 120B	apartments	3	69.7	3.0	$<0.001$
GPT-OSS 120B	flights	1	9.5	2.6	$<0.001$
GPT-OSS 120B	flights	2	12.1	3.3	$<0.001$
GPT-OSS 120B	flights	3	11.1	3.0	$<0.001$

Appendix

Table 10: Audit-group self-consistency tests (continued). Columns and reporting conventions are as in Table 9 .

Preference Modeling

See all 36 papers

Multiple latent orderings better predict language model preferences
Aug 26, 2026Aviral Chawla, William H. W. Thompson, Jean-Gabriel YoungPreference Modeling Pairwise Preference Learning
Vermont Complex Systems Institute University of Vermont Burlington, Vermont 05403 · Vermont Complex Systems Institute Department of Statistics University of Vermont Burlington, Vermont 05403
Evaluating LLM-Generated Preference Distributions
Oct 1, 2026Fan Huang, Minsuk Kim, C. Tyler Diggans +1LLM Evaluation Synthetic Data Generation
Indiana University Bloomington Bloomington, IN, USA · Massachusetts Institute of Technology Cambridge, MA, USA · Root Dynamix LLC Sewickley, PA, USA
Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
Jan 30, 2026Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-Judge Language Model Generation Evaluation
Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA, USA · Apart Research, San Francisco, California, USA · Department of Computer Science and Engineering, University of California San Diego, La Jolla, CA, USA +2

Aug 26, 2026J/K move · Enter open · S save

Multiple latent orderings better predict language model preferences

Aviral Chawla, William H. W. Thompson, Jean-Gabriel Young

Vermont Complex Systems Institute University of Vermont Burlington, Vermont 05403 · Vermont Complex Systems Institute Department of Statistics University of Vermont Burlington, Vermont 05403

Preference Modeling Pairwise Preference Learning Bradley–Terry Model

Open paper AskPDF arXiv

Language models are frequently employed in settings where they are asked to make value judgments and choices. These observed choices often exhibit intransitivity: A model may prefer item

A

B

and

B

C

, while also preferring

C

A

. Existing work that models LLM preferences treats such inconsistencies as sampling noise around a single latent ordering. We instead propose that intransitivity reflects the aggregation of multiple latent, internally consistent orderings. We first show that observed inconsistencies cannot be explained by a single ordering under any monotone link function. We then introduce a noise-augmented mixture Bradley-Terry (MBT) model that infers latent preference components from repeated pairwise comparisons. Across seven models and four tasks, a mixture of orderings often explains structural inconsistencies better than single-utility models. We find that aggregate preferences often hide underlying preference heterogeneity. A case study on Moral Machine dilemmas shows that models which disagree on aggregate orderings can still share latent components. Together, these results suggest that LLMs reflect plural preferences. Alignment and evaluation pipelines that treat LLM preferences as a single function, therefore, risk averaging over coherent orderings that different users may endorse differently.

Abstract

Preference Modeling

Explore similar work

Multiple latent orderings better predict language model preferences

Evaluating LLM-Generated Preference Distributions

Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations