Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models

Daniel Robert Kling Alexander; Catherine Louise Kling

cs.AIOct 7, 2026

Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models

Authors: Daniel Robert Kling Alexander, Catherine Louise Kling

Organizations: University of Michigan School of Information · Tisch University Professor, SC Johnson College of Business, Cornell University

Abstract

Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades. It judges survey responses without knowing the true value, through a framework of validity and related concepts: content, construct, and criterion validity, reliability, incentive compatibility, and consequentiality. We argue that this framework is a general method for evaluating language models, and we set out what each concept means for LLM evaluation. We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models. In this economic application, the validity tests take the form of predictions from economic theory: demand should slope down, and willingness to pay should respond to the scope of the good and to income. The tests separate the models sharply. Two older models fail the most basic test at a household income level of $75,000, and the two newest pass every test of theoretical validity we can score, but diverge on convergent validity. Passing validity tests shows that a model's answers are coherent, not that they are correct.

Figures & tables

Concept	Question in stated preference	Question for LLM evaluation	In our demonstration
Content validity	Does the instrument describe the good, payment, and decision clearly and credibly?	Does the task pose the question it claims to, and could the model be recalling the benchmark rather than answering it?	Published survey text, verbatim; the scale’s published name withheld
Construct: theoretical	Do answers move as economic theory predicts?	Do answers respond to what should matter, in the right direction, and ignore what should not?	Demand slopes down; scope; income
Construct: convergent	Do different measures of the same value agree?	Do different prompts, formats, and scoring methods reach the same conclusion?	Two WTP estimators; referendum vs. open-ended; human estimates
Criterion validity	Do stated values match real payments?	Does what the model says match what it does when it acts?	Not assessed: the model never pays
Reliability	Does the answer survive repetition?	Are results stable across samples, reruns, and model versions?	Ten independent draws per question
Incentive compatibility	Does the format reward truthful answers?	Does the setup reward telling the user what they want to hear?	Referendum format; not tested

Table 1: Validity concepts from stated-preference economics ( Mitchell and Carson, 1989 ; Kling et al., 2012 ; Bishop and Boyle, 2019 ; Carson and Groves, 2007 ; Carson et al., 2014 ) and what each asks of an LLM evaluation.

Figure 1: Share of yes votes by annual tax at a $75,000 household income, pooled over the six watershed scenarios (60 draws per point). A valid valuation requires a downward slope.

	Local watershed			Non-local watershed			Full study region
Model	One-level	Min L2	Min L3	One-level	Min L2	Min L3	One-level	Min L2	Min L3
Claude Sonnet 5	1,155	1,808	1,220	462	651	485	1,463	1,753	1,058
	(90)	(114)	(100)	(53)	(51)	(54)	(119)	(105)	(68)
GPT-5.6 Terra	1,623	3,156 †	1,184	925	801	541	2,066	3,156 †	1,747
	(135)	(301)	(100)	(95)	(74)	(60)	(158)	(301)	(134)
GPT-4o	3,156 †	–	2,945	936	957	617	2,524	–	2,657

Table A1: Estimated WTP at a

75,000 household income:

\alpha/\beta

from a Firth-penalized logit of the vote on the tax, fitted to 100 votes per cell, dollars per year for five years; delta-method standard errors in parentheses. “–” means the price coefficient is not significantly positive (a flat curve). † marks an estimate above the

3,000 top bid. Human means are from Vossler et al., (2023) , Model 1 (mixed logit), in 2021 dollars, and average over respondents’ own watersheds, which vary in size; the comparison is approximate.

Explore similar work

Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study
Jul 22, 2026Andrei Chetvergov, Stepan Ukolov, Timofei Sivoraksha +5Large Language Model Evaluation Large Language Models Fail
Ivannikov Institute for System Programming of the Russian Academy of Sciences, Moscow, Russia · Russian Presidential Academy of National Economy and Public Administration, Moscow, Russia
Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance
Jul 6, 2026Robert Morabito, Tyler McDonald, Charitra Viswanath +4Large Language Model Evaluation Review
Brock University · Emory University · University of Southern California +2
Ideological Bias in LLMs' Economic Causal Reasoning
Apr 23, 2026Donggyu Lee, Hyeok Yun, Jungwon Kim +4Large Language Model Bias Ideology
Graduate School of Data Science, KAIST, Daejeon, South Korea · College of Business, KAIST, Daejeon, South Korea · School of Computing, KAIST, Daejeon, South Korea +1

Jul 22, 2026, cs.CLJ/K move · Enter open · S save

Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study

Andrei Chetvergov, Stepan Ukolov, Timofei Sivoraksha, Alexander Evseev+4

Ivannikov Institute for System Programming of the Russian Academy of Sciences, Moscow, Russia · Russian Presidential Academy of National Economy and Public Administration, Moscow, Russia

Large Language Model Evaluation Large Language Models Fail Value Intrinsic Motivation Intermediate Checkpoints

Open paper AskPDF arXiv

Large language models are increasingly evaluated through the values they endorse, but such evaluations presuppose that models can identify the value expressed in a concrete situation. We study this prerequisite as controlled top-1 recognition over Schwartz's ten basic values. Our evaluation set contains 1,000 Russian situational texts, balanced across the ten values and independently labeled by two human annotators per item. We evaluate 21 instruction-tuned LLM runs under a fixed ranked-response protocol; 20 runs with reliable outputs form the semantic panel. Pooled Acc@1 is 0.683 and Acc@3 is 0.892, showing that models often locate the correct motivational region while ranking close alternatives unstably. Adjacent values account for 50.9% of semantic errors, compared with 24.4% under a checkpoint-specific null. Eight directed confusions recur across checkpoints and human-confirmed subsets. Several are strongly asymmetric, including Universalism to Benevolence, Tradition to Conformity, and Security to Power, whereas Stimulation-Hedonism forms a bidirectional boundary. Their severity is checkpoint-specific and can bias higher-order value profiles. The results motivate value-recognition evaluation that combines exact accuracy, ranked recovery, and directed error analysis.

Abstract

Explore similar work

Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study

Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance

Ideological Bias in LLMs' Economic Causal Reasoning