econ.GNNov 3, 2026

Decision Models as Scalable Economic Surrogates that Capture Human Variation

Authors: Maxime Saxena, Fabio Caccioli, R. Maria del Rio-Chanona

Organizations: Department of Computer Science, University College London, UK · Systemic Risk Centre, London School of Economics and Political Science, UK · Google DeepMind

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Decision Models (DMs), a new class of AI models, give fast responses and return a probability distribution over a fixed set of options. These properties make them candidate human surrogates in economic simulations and surveys, a role increasingly filled by Large Language Models (LLMs), which are costly at scale and under-represent the variation in human behaviour. We present a first evaluation of DMs as human surrogates in economics, comparing 10 DMs with 8 LLMs in three economic settings: a household spending survey on consumption, several classic economic games and a labour-market field experiment. Frontier DMs, those that score highest on general decision-making benchmarks, reproduce the variation in human responses more closely than LLMs in all three settings. They capture the modal human response as well as LLMs in the spending survey, but less well in the economic games. Frontier DMs are also roughly ten times faster than the fastest LLM and can be over a hundred times cheaper. Together, these results suggest that DMs capture the heterogeneity of human behaviour better than LLMs and make large-scale economic simulations more feasible.

Figures & tables

Figure 1: Frontier DMs better represent humans than LLMs in two consumer spending survey questions. KL divergence is compared to the Turing Test. DMs are solid circles while LLMs are hollow squares. Scores are averaged over the two consumer spending survey questions. Qwen (NT) is Qwen with no thinking, and Gemini Flash (LT) is Gemini Flash with low thinking.
Figure 2: Frontier DMs capture the full human distribution better. Mean results from the 7 positions in Mei et al. (2024) . Shows the Wasserstein-1 distance and the Turing Test results against the Human results and a Uniform benchmark. DMs are circles and LLMs are hollow squares. The frontier DMs are: Jev, D1 Ultra, Mammoth and Perplexity. Qwen (NT) is Qwen with no thinking, and Gemini Flash (LT) is Gemini Flash with low thinking.
Figure 3: Frontier DMs better capture the smooth curve in the human results compared to LLMs on labour choices. Left panel shows DM results for the Work From Home treatment on top and the LLM results below, with the human data in black. The right panel shows the Wasserstein-1 distance between models and the human results across all 5 treatments. DMs are circled, with frontier DMs in bold. The full fitted curves for all five treatments are in Appendix B.3 . Qwen (NT) is Qwen with no thinking, and Gemini Flash (LT) is Gemini Flash with low thinking.
Figure 4: DMs are faster and cheaper than LLMs. Averaged across all calls to the models across the three experiments with equal weight to each experiment. Mean seconds per call and mean dollars per 1000 calls are on log scales. Qwen (with thinking) is omitted as it was only run for the economic games, where its average time per call was 657 seconds and cost was 0 as it was run locally. Gemma is included in the locally run models, but was in fact run using an API, where it is free to use. Qwen (NT) is Qwen with no thinking, while Gemini Flash (LT) is Gemini Flash with low thinking.
Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Model NameModelProviderCost $/M in/outDevice
Jevjev-1.13.0TypeSafe0.042 / 0.00API
Perplexitypplx-decider-v1.1-27bPerplexity0.02 / 0.00API
Mammothbaseten-mammoth-27bBaseten0.10 / 0.00API
D1 Ultrad1-ultraLiquid0.042 / 0.00API
Luna Decisionsgpt-6-lunaOpenAI0.10 / 0.00API
Mercurymercury-decideInception0.02 / 0.00API
Appendix
Table 1: Decision Models used. The GPU used was a Quadro RTX 6000.
Model NameModelProviderCost $/M in/outDevice
Gemini Flashgemini-3.8-flashGoogle0.75 / 3.75API
Gemini Argongemini-4-argonGoogle2.0 / 10.0API
Gemmagemma-4-26b-a4b-itGoogle-API
Luna LLMgpt-6-lunaOpenAI0.1 / 0.5API
Solgpt-6-solOpenAI2.0 / 10.0API
Haikuclaude-haiku-5-5Anthropic0.1 / 0.5API
Appendix
Table 2: LLMs used. APIs are all the provider’s standard API interface. The GPU used was a Quadro RTX 6000 except for Qwen with thinking, where a mixture of a Quadro RTX 6000 and an Apple M4 were used to increase the speed of the runs. The Quadro RTX 6000 averaged 361 seconds per call and the M4 around 713 on similar length outputs. Both are several orders of magnitude slower than any of the DMs.
Model NameArtificial Analysis Decision Index
Jev57
Perplexity56
Mammoth65.9
D1 Ultra65
Luna Decisions48
Mercury53
Appendix
Table 3: Decision Index scores. Jev score is in fact Jev 1.14, a slightly newer version than we use. Artificial Analysis have not run Jev 1.13 on their benchmark, but TypeSafe, the creators of Jev, find little difference in score between the two models on their own benchmark. We therefore make an assumption that the same holds true on the Artificial Analysis benchmark. The results for Mammoth are those reported by the creators of Mammoth rather than those reported by Artificial Analysis (since they have not been run by Artificial Analysis).
LLMJevPerplexityMammothD1 Ultra
Panel A: KL divergence from humans (bits)
Qwen (NT)2.08∗2.08^{*}2.04∗2.04^{*}2.23∗2.23^{*}2.13∗2.13^{*}
Gemini Flash0.89∗0.89^{*}0.84∗0.84^{*}1.03∗1.03^{*}0.93∗0.93^{*}
Gemini Flash (LT)0.84∗0.84^{*}0.79∗0.79^{*}0.98∗0.98^{*}0.88∗0.88^{*}
Luna LLM1.23∗1.23^{*}1.18∗1.18^{*}1.37∗1.37^{*}1.27∗1.27^{*}
Sol1.54∗1.54^{*}1.49∗1.49^{*}1.68∗1.68^{*}1.58∗1.58^{*}
Appendix
Table 4: Paired bootstrap tests of each DM against each LLM, averaged over the two SCE spending questions. Panel A tests which model has a lower KL divergence while Panel B tests which model has a higher Turing Test score. Each cell is in bits (Panel A) or percentage points (Panel B), averaged over the 2 questions with equal weights: positive if the decision model is lower in KL divergence, higher in Turing Test. ∗ Significant at the 1% level (two-sided paired bootstrap test, 10,000 draws, Holm-corrected over each panel’s tests). Each draw resamples the 1,000 personas with replacement.
Decision modelKL divergence from humans (bits)Turing test (%)
Jev0.28∗0.28^{*}12.4∗12.4^{*}
D1 Ultra0.32∗0.32^{*}14.3∗14.3^{*}
Perplexity0.23∗0.23^{*}15.4∗15.4^{*}
Mammoth0.42∗0.42^{*}9.9∗9.9^{*}
Appendix
Table 5: Paired bootstrap tests of the frontier DMs against the uniform distribution, averaged over the two SCE spending questions. Values are as above, with a higher number meaning DMs have a lower KL divergence or higher Turing test scores. ∗ Significant at the 1% level (two-sided paired bootstrap test, 10,000 draws, Holm-corrected). Each draw resamples the 1,000 personas with replacement.
Figure 5: Our results are robust to three different robustness checks. DMs are solid while LLMs are hollow. Human is the test where models are told to act like a human. Nosystem removes the system prompt for the LLMs while wording changes the wording for the loss of income question. Scores are averaged over the two consumer spending survey questions with equal weight.
LLMJevPerplexityMammothD1 Ultra
Panel A: Wasserstein-1 distance (% of range)
Qwen1.01.010.5∗10.5^{*}9.5∗9.5^{*}9.4∗9.4^{*}
Qwen (NT)9.8∗9.8^{*}19.4∗19.4^{*}18.3∗18.3^{*}18.3∗18.3^{*}
Gemini Flash8.3∗8.3^{*}17.9∗17.9^{*}16.9∗16.9^{*}16.8∗16.8^{*}
Gemini Flash (LT)10.3∗10.3^{*}19.9∗19.9^{*}18.9∗18.9^{*}18.8∗18.8^{*}
Luna LLM2.7∗2.7^{*}12.2∗12.2^{*}11.2∗11.2^{*}11.1∗11.1^{*}
Appendix
Table 6: Bootstrap tests of frontier DMs, using the choice answer, against each LLM, averaged over Mei et al. [2024] games. Each cell is how much lower the Wasserstein-1 distance or higher the Turing Test the decision model is than the LLM, in percentage points, averaged over the 7 games with equal weights: positive if the decision model is lower Wasserstein-1 or higher Turing test. ∗ Significant at the 1% level (two-sided bootstrap test, 10,000 draws, Holm-corrected over each panel’s tests). Each model’s calls are resampled with replacement in each game for the bootstrap.
LLMJevPerplexityMammothD1 Ultra
Panel A: Wasserstein-1 distance (% of range)
Qwen5.6∗5.6^{*}9.1∗9.1^{*}9.2∗9.2^{*}7.5∗7.5^{*}
Qwen (NT)14.4∗14.4^{*}17.9∗17.9^{*}18.1∗18.1^{*}16.3∗16.3^{*}
Gemini Flash13.0∗13.0^{*}16.4∗16.4^{*}16.6∗16.6^{*}14.9∗14.9^{*}
Gemini Flash (LT)15.0∗15.0^{*}18.4∗18.4^{*}18.6∗18.6^{*}16.8∗16.8^{*}
Luna LLM7.3∗7.3^{*}10.8∗10.8^{*}10.9∗10.9^{*}9.2∗9.2^{*}
Appendix
Table 7: Bootstrap tests of frontier DMs, using the score answer, against each LLM, averaged over Mei et al. [2024] games. Each cell is how much lower the Wasserstein-1 distance or higher the Turing Test the decision model is than the LLM, in percentage points, averaged over the 7 games with equal weights: positive if the decision model is lower Wasserstein-1 or higher Turing test. ∗ Significant at the 1% level (two-sided bootstrap test, 10,000 draws, Holm-corrected over each panel’s tests). Each model’s calls are resampled with replacement in each game for the bootstrap.
LLMJevPerplexityMammothD1 Ultra
Panel A: KL divergence (bits)
Qwen2.24∗2.24^{*}2.83∗2.83^{*}2.75∗2.75^{*}2.89∗2.89^{*}
Qwen (NT)3.81∗3.81^{*}4.40∗4.40^{*}4.32∗4.32^{*}4.46∗4.46^{*}
Gemini Flash3.51∗3.51^{*}4.10∗4.10^{*}4.02∗4.02^{*}4.16∗4.16^{*}
Gemini Flash (LT)3.80∗3.80^{*}4.38∗4.38^{*}4.30∗4.30^{*}4.44∗4.44^{*}
Luna LLM2.66∗2.66^{*}3.25∗3.25^{*}3.16∗3.16^{*}3.31∗3.31^{*}
Appendix
Table 8: Bootstrap tests of frontier DMs, using the choice answer, against each LLM, averaged over Mei et al. [2024] games. Each cell is how much lower the KL divergence of the decision model is than the LLM, averaged over the 7 games with equal weights: positive if the decision model is lower KL divergence. ∗ Significant at the 1% level (two-sided bootstrap test, 10,000 draws, Holm-corrected). Each model’s calls are resampled with replacement in each game for the bootstrap.
LLMJevPerplexityMammothD1 Ultra
Panel A: KL divergence (bits)
Qwen2.61∗2.61^{*}2.95∗2.95^{*}2.96∗2.96^{*}2.93∗2.93^{*}
Qwen (NT)4.18∗4.18^{*}4.52∗4.52^{*}4.53∗4.53^{*}4.50∗4.50^{*}
Gemini Flash3.88∗3.88^{*}4.22∗4.22^{*}4.23∗4.23^{*}4.20∗4.20^{*}
Gemini Flash (LT)4.16∗4.16^{*}4.50∗4.50^{*}4.52∗4.52^{*}4.48∗4.48^{*}
Luna LLM3.03∗3.03^{*}3.37∗3.37^{*}3.38∗3.38^{*}3.35∗3.35^{*}
Appendix
Table 9: Bootstrap tests of frontier DMs, using the score answer, against each LLM, averaged over Mei et al. [2024] games. Each cell is how much lower the KL divergence of the decision model is than the LLM, averaged over the 7 games with equal weights: positive if the decision model is lower KL divergence. ∗ Significant at the 1% level (two-sided bootstrap test, 10,000 draws, Holm-corrected). Each model’s calls are resampled with replacement in each game for the bootstrap.
Decision modelWasserstein-1 distance (% of range)Turing test (%)
Jev (choice)−7.2∗-7.2^{*}5.3∗5.3^{*}
Jev (score)−2.5∗-2.5^{*}8.0∗8.0^{*}
D1 Ultra (choice)1.3∗1.3^{*}6.3∗6.3^{*}
D1 Ultra (score)−0.7∗-0.7^{*}7.9∗7.9^{*}
Perplexity (choice)2.4∗2.4^{*}4.3∗4.3^{*}
Perplexity (score)0.9∗0.9^{*}7.5∗7.5^{*}
Appendix
Table 10: Bootstrap tests of frontier DMs against the uniform distribution. Each cell is how much lower the Wasserstein-1 distance or higher the Turing Test the decision model is than the uniform distribution, averaged over the 7 games with equal weights: positive if the decision model is lower Wasserstein-1 or higher Turing test. ∗ Significant at the 1% level (two-sided bootstrap test, 10,000 draws, Holm-corrected). Each model’s calls are resampled with replacement in each game for the bootstrap.
Figure 6: Fitted inattention-corrected curves for all 5 treatments. Job A is the more flexible alternative, so in 4 treatments it is the alternative and when the treatment is employer discretion, it is the standard 9-5 40 hour job. The left column shows frontier DMs, while the right column shows all LLMs.
LLMJevPerplexityMammothD1 Ultra
Qwen (NT)0.80∗0.80^{*}0.96∗0.96^{*}0.79∗0.79^{*}0.89∗0.89^{*}
Gemini Flash0.52∗0.52^{*}0.68∗0.68^{*}0.51∗0.51^{*}0.61∗0.61^{*}
Gemini Flash (LT)0.52∗0.52^{*}0.68∗0.68^{*}0.51∗0.51^{*}0.61∗0.61^{*}
Luna LLM0.85∗0.85^{*}1.01∗1.01^{*}0.84∗0.84^{*}0.94∗0.94^{*}
Sol0.26∗0.26^{*}0.42∗0.42^{*}0.25∗0.25^{*}0.35∗0.35^{*}
Gemma0.40∗0.40^{*}0.56∗0.56^{*}0.39∗0.39^{*}0.49∗0.49^{*}
Appendix
Table 11: Bootstrap tests of frontier DMs against each LLM. Each cell is how much closer to the applicants’ fitted WTP distribution the decision model’s is than the LLM’s, in $ per hour of Wasserstein-1 distance (the area between the fitted choice curves without inattention), averaged over the 5 treatments with equal weights: positive if the decision model is closer. ∗ Significant at the 1% level (two-sided bootstrap test, 10,000 draws, Holm-corrected over the tests).
 FSEDWFHFHCFMean
Laya0.49†0.49^{\dagger}0.49†0.49^{\dagger}0.43†0.43^{\dagger}0.49†0.49^{\dagger}0.41†0.41^{\dagger}0.460.46
Jev0.000.000.000.000.000.000.000.000.000.000.000.00
AnyJev0.000.000.030.030.000.000.010.010.000.000.010.01
Luna Decisions0.000.000.010.010.000.000.000.000.000.000.000.00
Perplexity0.000.000.010.010.000.000.000.000.000.000.000.00
Mammoth0.030.030.070.070.020.020.080.080.030.030.050.05
Appendix
Table 12: Estimated inattention in each of Mas and Pallais [2017] ’s treatments. Inattention α\alpha in the inattention-corrected logit P(A∣gap)=α+(1−2α)F(b gap+c)P(A\mid\text{gap})=\alpha+(1-2\alpha)F(b\,\text{gap}+c) : the share not choosing A when it pays 5more,fromalinearfitofthechoicesatwagegapsof5 more, from a linear fit of the choices at wage gaps of 2 to 5.Comparisontothehumanaverageof25−305. Comparison to the human average of 25-30% then requires the above estimate of \alpha$ to be doubled. Mean is the average over the 5 treatments with equal weights. † Above 0.4, so no WTP is fitted. We use the following abbreviations to save space: FS (Flexible Schedule), ED (Employer Discretion), WFH (Work From Home), FH (Flexible Hours), CF (Combined Flexible).
 Mean seconds per callMean $ per 1,000 calls
 SpendingGamesLabourMeanSpendingGamesLabourMean
Laya0.030.030.030.030.030.030.030.030.0000.0000.0000.0000.0000.0000.0000.000
Jev0.260.260.240.240.210.210.240.240.0240.0240.0220.0220.0200.0200.0220.022
AnyJev2.812.811.941.940.750.751.831.830.0000.0000.0000.0000.0000.0000.0000.000
Luna Decisions0.220.220.220.220.240.240.230.230.0330.0330.0310.0310.0280.0280.0300.030
Perplexity0.330.330.310.310.330.330.320.320.0060.0060.0060.0060.0050.0050.0060.006
Appendix
Table 13: Mean time and cost per call in each experiment. Seconds exclude model loading and a warm-up call. Cost is worked out from each call’s tokens and the API’s list prices; models run locally cost 0, and Gemma is free. Mean is the average over the 3 experiments with equal weights, blank for a model not run in all of them.
 AveragePurchasesOcc. mobilityRationing & Markup
 CostTimeCostTimeCostTimeCostTime
Frontier DM
Jev0.0220.0220.240.240.410.410.120.1232.9932.999.899.8911.0011.003.303.30
Perplexity0.0060.0060.320.320.110.110.170.178.658.6513.4313.432.882.884.484.48
Mammoth0.0310.0310.290.290.570.570.150.1545.8845.8812.1712.1715.2915.294.064.06
D1 Ultra0.0090.0090.270.270.170.170.140.1413.4013.4011.1911.194.474.473.733.73
Appendix
Table 14: Projected total cost and time of each experiment. Occ. mobility refers to the occupational mobility ABM in del Rio-Chanona et al. [2021] . Average is the mean cost (per1,000calls)andtime(secondspercall)overthethreeexperimentsofTable13.Eachexperiment’scost( per 1,000 calls) and time (seconds per call) over the three experiments of Table 13 . Each experiment’s cost () is its calls times the mean cost per call, and its time (hours) its calls times the mean seconds per call divided by the calls run at once. Purchase Intent, 18,600 calls, 10 at once; Occupational mobility, 500,000 calls (3 for each by a decision model), 10 at once; Rationing, 500,000 calls, 10 at once.
More in this topic

Human Behavior Simulation

Human behavior simulation models how individuals and groups decide, interact, move, and adapt in social, economic, educational, and physical settings.

See all 247 papers

Explore similar work

CardsList

  1. Evaluating LLMs as Human Surrogates in Controlled Experiments

    Mar 8, 2026Adnan Hoq, Tim WeningerHuman Behavior SimulationLarge Language Model-Based Social Simulation

    University of Notre Dame Notre Dame, Indiana, USA

  2. HouseholdBench: Evaluating Large Language Models as Predictors of Household Economic Behavior

    Oct 6, 2026Jin Huang, Diego Ferreras Garrucho, Yutong Xie +4LLM EvaluationLLM Fine-Tuning

    University of Michigan, Ann Arbor · London School of Economics · MobLab Inc +1

  3. Large language models replicate and predict human cooperation across experiments in game theory

    Nov 6, 2025Andrea Cera Palatsi, Samuel Martin-Gutierrez, Ana S. Cardenal +1LLM EvaluationGame Theory

    Department for Computational Social Sciences and Humanities, Barcelona Supercomputing Center · Grupo de Sistemas Complejos, Universidad Polit´ecnica de Madrid · School of Law and Political Science, Universitat Oberta de Catalunya +1