Benchmarking large language model agent societies against human behavioural distributions

Raad Bin Tareaf

physics.soc-phAug 28, 2026

Benchmarking large language model agent societies against human behavioural distributions

Authors: Raad Bin Tareaf

Organizations: AI Cluster, XU Exponential University of Applied Sciences, Marlene-Dietrich-Allee 12B, 14482, Potsdam, Germany.

Abstract

Populations of large language model agents are increasingly used as experimental societies. Three doubts shadow every such result: whether the agents behave like the humans they stand in for, whether a finding survives changes to the apparatus that leave the rules untouched, and whether apparent social dynamics are interaction at all rather than the reproduction of experiments the models have read. This article introduces SILICA, an open instrument that tests all three. Five environments carry published human anchors, each paired with perturbations that re-render the same rules and with variants whose payoffs point away from the memorised result. Twelve open-weight models were run through it on a single consumer graphics card. Agreement with human data is confined to starting points: first-round public-goods contributions fall inside the equivalence margin for eight of eleven models, while no model matches end-state contributions or the human corridor of cooperation. Merely swapping the order in which two actions are listed costs one model 58 points of cooperation. Presenting responders with a fixed schedule of offers shows that only one model, the sole reasoning-trained one, places its acceptance threshold where the incentive requires; two move theirs part of the way, two move them the wrong way, and three never acquire one. Conventions form through a shared prior over the names rather than through negotiation, though negotiation reappears once that prior is disrupted. On the certification ladder defined here, current silicon societies support exploratory claims and no more.

Figures & tables

Figure 1: The instrument. Left Five environments, each paired with a published human anchor, a perturbation library whose cells are separated into those that re-render the same rules and those that alter what the agents are told, and a gradient of payoff variants ending in C3, where the result recorded in the literature is no longer the profitable one; the last two entries are controls added after a transcript audit. Centre The roster and the execution protocol; the full roster with quantisation and per-model run counts is in the Supplementary Information. Right The four measurements and the ladder that converts them into a claim grade, with the current verdict shown against each tier. Cell counts and run totals are read from the released record; the registration shown in the protocol card is at https://doi.org/10.17605/OSF.IO/AFSDU .

Level	Cell	PD	PGG	Ult.	11–20	Naming
Representation	agent naming = names	$\bullet$	$\bullet$
	history rendering = narrative	$\bullet$
	history summary statistics = shown	$\bullet$	$\bullet$
	action-label order = swapped	$\bullet$
	persona layout = bullets	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$
	persona layout = table	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$

Table 1: The perturbation library: every cell of the audit grid, its level, and the environments in which it was run (

\bullet

). One factor changes per cell; all other components are held at the canonical baseline (plain persona, neutral content, canonical framing, tabular history without summary statistics, default label order, letter labels, temperature

0.7

, memory window

5

, default group size). Each cell holds 10 seeded runs per model (2 runs of 50 games in the one-shot battery).

Question	What it asks	What the data show	Tier
RQ1 Level fidelity	Does behaviour match human levels?	public goods round 1: 8/11 equivalent; round 10: 0/11; ultimatum offers: 1/12; dictator: 0/12; PD inside the human corridor: 0/12	Tier 1
RQ1 Directional fidelity	Does behaviour move in the human direction?	public goods decay 1/12; PD decay 3/12; convention formation 10/12	Tier 1
RQ2 Robustness	Does the finding survive re-rendering the same rules?	Holm-significant shifts in 64 of 182 computable cells; largest representation-level reversal 1.00; largest design-level 1.00	Tier 1
RQ3 Contamination	Is the behaviour retrieved or computed?	ultimatum, under a fixed offer schedule: 1 of 12 models sets its acceptance threshold where the outside option requires, 2 move theirs partway, 2 the wrong way and 3 never acquire one; naming with a per-agent shuffled pool: 7 of 8 models still converge in every seed, 1.5 to 2.6 times slower, while Qwen2.5-7B converges in only 2 of 3 seeds at 14–24 times the fixed-order value; final-round defection with no endgame incentive in 3 of 10 models	Tier 1
RQ4 Certification	What can the instrument certify?	no finding reaches Tier 3. Convention formation reaches Tier 2: the shuffled-pool control shows that coordination is driven by a shared prior over the labels rather than by shared list position, and that negotiation does occur once that cue is removed — but still between twice and twenty-eight times faster than the classical baseline, so the mechanism is related to the human one without matching it. Every other finding is Tier 1	–

Table 2: The study at a glance. Each row states one research question, the evidence assembled for it, and the highest certification tier the evidence supports. Denominators are the number of models for which the comparison is computable; R1-Distill completed a partial battery and is excluded from cells where it holds fewer than three runs. Equivalence is the two one-sided test against the human anchor at the margin fixed in advance; significance is Holm-adjusted within family under the exact test. Per-model acceptance thresholds and payoff consistency are in Supplementary Table S5; a high consistency score can be produced by rejecting almost everything, so the threshold rather than the score is the quantity reported here.

Figure 2: Public goods and costly punishment. a First- and last-round contributions at the canonical baseline, with the interquartile range across runs and the human corridors shaded; the three classical baselines are shown in italics to the right of the divider. b Final-round contributions without and with the punishment option, with the standard deviation across runs;

\ast

marks a Holm-significant decrease under the exact test. c The share of runs in which punishment is purchased (bars) and the points spent per agent per round (points, right axis).

	Repeated dilemma		Public goods		Ultimatum	Naming		11–20
Model	cooperation	in band	round 1	round 10	offer	conventions	$p^{*}$	$W_{1}$
Qwen3-32B	0.95 $\pm$ 0.05	no	0.15 $\pm$ 0.17	0.00 $\pm$ 0.00	0.49 $\pm$ 0.01	1.00	0.20	0.062 [0.06,0.07]
Qwen3-14B	0.85 $\pm$ 0.08	no	0.50 $\pm$ 0.00	0.50 $\pm$ 0.00	0.50 $\pm$ 0.00	1.00	0.40	0.218 [0.21,0.22]
Qwen3-8B	1.00 $\pm$ 0.00	no	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.22 $\pm$ 0.05	1.00	0.30	0.576 [0.58,0.58]
Qwen3-4B	0.00 $\pm$ 0.00	no	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.50 $\pm$ 0.00	1.00	0.30	0.637 [0.63,0.64]
Qwen2.5-14B	1.00 $\pm$ 0.00	no	0.57 $\pm$ 0.02	0.79 $\pm$ 0.09	0.40 $\pm$ 0.00	1.00	0.20	0.224 [0.22,0.22]

Table 3: Baseline fidelity against the human anchors, one row per model. Values are means over the canonical baseline runs with the standard deviation across runs;

n

is the number of runs. Bold marks an outcome that passes the two one-sided equivalence test against its anchor at the margin fixed in advance. PD cooperation is compared with the corridor

0.20

–

0.50

reported across prisoner’s dilemma studies; public-goods contributions with

0.50

in round 1 and

0.15

in round 10; ultimatum offers with

0.40

of the pie; convention formation with the human result that a convention always forms; and the 11–20 choice distribution through the Wasserstein distance to the distribution of Arad and Rubinstein, normalised by the width of the support, with a bootstrap 95% interval. R1-Distill completed a partial battery and its public-goods cell holds a single run, shown as –. In the

p^{*}

column, a value is the smallest tested committed-minority fraction at which the convention is overturned in at least half of runs;

>0.40

means a convention forms but was never overturned at any tested fraction; “no conv.” means no convention forms, so there is nothing to overturn; and “not run” means the tipping arm was not executed for that model.

Figure 3: Every cell of the audit grid. One panel per environment; rows are models, columns are perturbation cells, ordered representation-level first and separated from the design-level cells by a rule. Colour is the shift of the environment’s primary outcome relative to the canonical baseline; $\ast$ marks a Holm-significant contrast under the exact test. The one-shot games, whose cells hold two runs and are descriptive.

	$\Delta_{\max}$ , representation-level					$\Delta_{\max}$ , design-level					$S$ , representation / design
Model	PD	PGG	Ult.	11–20	Naming	PD	PGG	Ult.	11–20	Naming	PD	PGG	Naming
Qwen3-14B	0.68	0.00	0.00	0.08	0.00	0.77	0.55	0.00	0.01	0.00	0.57 / 0.57	0.00 / 0.00	1.00 / 1.00
Gemma2-9B	0.17	0.14	0.06	0.02	0.00	1.00	0.24	0.20	0.01	0.50	0.00 / 0.00	0.00 / 0.00	1.00 / 1.00
Llama3.1-8B	0.46	0.02	0.03	0.08	0.00	0.59	0.08	0.12	0.08	1.00	0.14 / 0.43	0.00 / 0.00	0.00 / 0.17
Mistral-7B	0.00	0.09	0.21	0.00	0.10	0.07	0.07	0.04	0.00	1.00	0.00 / 0.00	0.00 / 0.00	1.00 / 0.67
Phi3-mini	0.11	0.04	0.02	0.03	1.00	0.23	0.12	0.02	0.02	1.00	0.57 / 0.57	0.00 / 0.00	0.67 / 0.83

Table 4: Robustness of the primary outcome under the perturbation library, one row per model, split by the level of the perturbation.

\Delta_{\max}

(Eq. 4 ) is the range of cell means over the canonical baseline and either the representation-level cells (the same rules re-rendered: persona layout, history rendering and summary, action-label order, agent naming) or the design-level cells (content changes: persona content, moralized or risk framing, step-by-step reasoning, sampling temperature, memory window, group size).

S

is the share of cells (baseline included) in which the canonical human effect keeps its sign (PD and PGG: decline of cooperation or contributions over rounds; naming: convergence); it is defined only for those three environments. Perturbation cells hold 10 runs per model (2 runs of 50 games in the one-shot games, where

\Delta_{\max}

is therefore descriptive); baseline cells 30 (6 in the one-shot games). –

=

arm not run, baseline with fewer than 3 runs, or fewer than two cells with at least 2 runs (R1-Distill completed a partial battery; its per-cell run counts are listed in the Supplementary roster).

Figure 4: Acceptance functions in the ultimatum game under a fixed schedule of offers. Each panel shows one model’s rejection rate against the offer, in the canonical game where rejection pays nothing and in the divergent-payoff variant where rejection pays 40. The dashed line marks the offer above which accepting is the profitable act in the divergent variant; the dotted line marks the low-offer threshold conventionally reported for human responders. Shading marks the region in which rejection is payoff-maximising under the divergent payoffs.

Figure 5: How conventions form, and what survives when the incentive to defect is removed. a Match rate over the first 300 interactions of the canonical naming game, one line per model. b Match rate over the first twenty interactions with the name that wins. c Defection rate by round in the canonical game (dashed) and in the variant where cooperation is strictly dominant (solid).

Figure 6: One society overturning its own convention. The naming game is the only environment here whose record is natively a graph: each line of the transcript is a directed interaction between two identified agents, carrying the word each proposed and whether the two matched. a–c Three moments of a single run of the tipping arm (Qwen3-32B, seed 0, 24 agents): the established convention, the point at which a committed minority of six agents — 25% of the population, the human critical mass — has split the population evenly, and the state after the minority has carried it. Node positions are computed once from the whole run and held fixed, so the only thing that changes between panels is colour; edges show the last 25 interactions before each moment. d The boxed neighbourhood at the contested moment, with individual interactions drawn. e Adoption of the new word over the run. Committed agents, ringed in black, never switch; mismatched interactions are the mechanism by which the rest do.

Figure 7: Four primary outcomes against model size. a Cooperation rate in the repeated dilemma. b Final-round contribution in the public-goods game. c Share of runs reaching a convention. d Distance between the model’s choice distribution in the 11–20 game and the human one. Lines join models of the same family; the reasoning-trained model is drawn as a diamond; shaded bands mark the human values. Public-goods cells holding fewer than three runs are omitted.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Params (B)	Quantisation	Deliberation	Runs	Cells	Calls	Invalid	Rate	GPU h
Qwen3-32B	32.8	AWQ 4-bit	off	792	86	1,699,840	10	0.00%	49.6
Qwen3-14B	14.8	FP8	off	827	88	1,940,640	0	0.00%	131.8
Qwen3-8B	8.2	bf16	off	792	86	1,699,840	0	0.00%	72.4
Qwen3-4B	4.0	bf16	off	792	86	1,699,840	21	0.00%	47.3
Qwen2.5-14B	14.7	AWQ 4-bit	off	792	86	1,699,840	0	0.00%	16.3
Qwen2.5-7B	7.6	AWQ 4-bit	off	792	86	1,699,840	0	0.00%	21.6

Appendix

Table S1: Model roster and execution statistics. Parameter counts are the published totals. Quantisation is as served; the Qwen3 models expose an optional deliberation mode, which was disabled throughout, so the reasoning axis rests on the single reasoning-trained model rather than on a decoding switch. Attempted calls are estimated from the run manifest; invalid actions are replies that failed schema validation and were replaced by the environment’s fallback. R1-Distill completed a partial battery, which is why its cell count is lower.

Family	Env	Model	Contrast	$n$	$\delta$	$p$	$p_{H}$
contamination	bargain	Gemma2-9B	C3 vs C0	6+6	-1.00	0.002	0.015
contamination	bargain	Mistral-7B	C3 vs C0	6+6	-1.00	0.002	0.015
contamination	bargain	Qwen1.5-7B	C3 vs C0	6+6	-1.00	0.002	0.015
contamination	bargain	Qwen2.5-7B	C3 vs C0	6+6	+1.00	0.002	0.015
contamination	bargain	Qwen3-32B	C3 vs C0	6+6	+1.00	0.002	0.015
contamination	bargain	Phi3-mini	C3 vs C0	6+6	+0.67	0.067	0.134

Appendix

Table S2: Test-level record for the contrasts named in the main text, plus every contamination contrast: 95 of 227 computable contrasts, ordered by family.

p

is the exact permutation value conditional on the observed ties and

p_{H}

its Holm-adjusted counterpart within the family. The complete file, including the value each test would take under a normal approximation, is results/stat_tests.csv in the released record.

Model	C1 shift	C2 shift	Priming shift	Reports knowing	Names correctly
Qwen3-32B	-0.05	+0.05	-0.03	100%	70%
Qwen3-14B	-0.20	-0.85	+0.08	100%	70%
Qwen3-8B	+0.00	+0.00	+0.00	100%	56%
Qwen3-4B	+0.34	+0.10	+0.03	100%	57%
Qwen2.5-14B	+0.00	+0.00	+0.00	80%	51%
Qwen2.5-7B	+0.00	-0.98	+0.00	93%	55%

Appendix

Table S3: Contamination outcomes away from the divergent-payoff variant, and what the models say when asked directly. C1 re-skins the game with a different cover story and action labels while preserving the payoffs; C2 additionally shifts the numbers. Values are the shift in the repeated dilemma’s cooperation rate relative to the canonical baseline. The priming column is the shift when the paradigm is named inside the behavioural prompt. Recognition is asked in a separate session: whether the model reports knowing the paradigm, and whether it names it correctly. Recognition predicts none of the behavioural columns. R1-Distill completed a partial battery: it ran the C1 and divergent-payoff cells of the repeated dilemma but no C2 cell, so its C2 entry is absent rather than zero.

Environment	Anchor	Source	Evidence base	Provenance
11–20 game	full choice distribution; $P(17)=.32$ , $P(18)=.30$	Arad & Rubinstein (2012), AER 102(7), Table 1	single experiment, $n=108$	published table
Ultimatum	offer $.40$ of pie; rejection $.16$	Oosterbeek et al. (2004), Exp. Econ. 7(2)	meta-analysis, 37 papers, 75 results	meta-analytic mean
Dictator	giving $.28$ of endowment	Engel (2011), Exp. Econ. 14	meta-analysis, 328 treatments, 20,813 decisions	random-effects mean
Trust	sent $.50$ ; returned $.37$ of available	Johnson & Mislin (2011), JEP 32(5)	meta-analysis, 162 replications, $>$ 23,000 participants	meta-analytic mean
Public goods	round 1 $.50$ ; round 10 $.15$ ; punishment $.55$	Ledyard (1995) survey; Zelmer (2003); Fehr & Gächter (2000)	27 studies, 711 groups; punishment from a single experiment	stylised fact, not a single estimate
Repeated PD	cooperation corridor $.20$ – $.50$	Mengel (2018), Econ. J. 128(616)	meta-study, 96 studies, $>$ 3,500 participants	range across studies

Appendix

Table S4: Provenance of every human anchor. Each value was checked against its primary source on 26 August 2026 and the shipped anchor file now records the citation and the verification date. One correction was made: two cells of the 11–20 distribution had been mis-transcribed, and all Wasserstein distances were recomputed against the published table. The public-goods trajectory is a stylised fact drawn from a survey and a meta-analysis rather than a single meta-analytic estimate, and its punishment value comes from one experiment; the manuscript states this rather than describing the anchor as a meta-analytic corridor.

Model	Variant	10	20	25	30	33	37	40	Threshold	Consistency
Qwen3-32B	canonical	0.13	0.00	0.00	0.00	0.00	0.00	0.00	–	0.99
	divergent	1.00	0.53	0.67	0.68	0.00	0.03	0.05	30	0.61
Qwen3-14B	canonical	1.00	1.00	1.00	0.00	0.00	0.00	0.00	25	0.67
	divergent	1.00	1.00	1.00	1.00	0.75	0.13	0.00	33	0.86
Qwen3-8B	canonical	0.00	0.00	0.00	0.00	0.00	0.00	0.00	–	1.00
	divergent	0.00	0.00	0.00	0.00	0.00	0.00	0.00	–	0.25

Appendix

Table S5: Acceptance functions under the fixed offer schedule: rejection rate at each scheduled offer, 20 games per offer per seed, three seeds. In the canonical variant rejection pays nothing, so accepting every offer is payoff-maximising; in the divergent variant rejection pays 40, so rejecting below 40 is. The threshold is the largest offer still rejected in at least half of games. R1-Distill is the only model whose threshold falls where the outside option puts it.

Model	Fixed order	Cap	Runs	Converged	Median	Range	First-20 match	Name
Qwen3-32B	50	600	3	3	104	90–117	0.67	A
Qwen3-14B	50	600	3	3	103	100–237	0.42	A
Qwen3-8B	50	600	3	3	75	50–113	0.85	A
Qwen3-4B	50	600	3	3	129	104–202	0.58	A
Qwen2.5-14B	50	600	3	3	81	79–97	0.68	A
Qwen2.5-7B	50	600	3	0	600	600–600	0.17	C/G

Appendix

Table S6: The shuffled-pool control. Each agent receives its own random permutation of the same ten names, and an unparsable reply falls back to a random name rather than the first listed one. Fixed-order convergence is the median over the 30 baseline runs; 50 is the earliest value the convergence rule can return. Coordination survives in every model and slows in all of them. Runs are listed separately by cap: Qwen2.5-7B reached the 600-interaction cap in all three capped runs and, uncapped, converged in only two of three. Every model still converges on the same name as under a fixed order except R1-Distill, which moves from A to E.

	Interactive multi-agent	Human-anchored (distribution)	Perturbation audit	Contamination audit (C3)	$\geq$ 5 model families	Scale axis	Generation axis	Claim certification	Open exec. benchmark
Park et al. (2024)	✓	✓	–	–	–	–	–	–	(✓)
Ashery et al. (2025)	✓	(✓)	(✓)	(✓)	(✓)	–	–	–	(✓)
Akata et al. (2025)	✓	(✓)	–	–	(✓)	–	–	–	–
Cui et al. (2025)	–	✓	–	–	–	–	–	–	(✓)
Gao et al. (2025)	–	✓	(✓)	(✓)	✓	–	–	–	(✓)
GTBench ( Duan et al., 2024 )	✓	–	–	–	✓	(✓)	–	–	✓

Appendix

Table S7: Positioning against the closest prior work. Rows name the benchmark or platform where one exists and the study otherwise. Unlike every other table in this article, these entries are editorial judgements about other authors’ work rather than measurements: a mark records only whether the cited work set that capability as an objective, not how well it succeeded. The criteria are defined in the text above. ✓ stated as an objective, with results reported against it; (✓) a partial or methodological precursor, present but not carried through to a reported result; – not an objective of the cited work, which is a statement about scope rather than a criticism.

Figure S1: Perturbation cells in the two one-shot environments. Rows are models, columns are cells, ordered representation-level first. These cells hold two runs each, so the shifts are descriptive and no significance is marked; the three environments in which run-level tests are possible appear in the main text.

Figure S2: The divergent-payoff ultimatum cell as originally run, before the fixed offer schedule was introduced. a Every game, plotted as the offer against the responder’s decision; the shaded band marks the range in which the memorised script and the payoffs disagree. b The share of each model’s games that fall in that band. Seven of nine models never produce a single diagnostic game, because the proposer is the same model as the responder and offers at or above the outside option. This is the observation that motivated the fixed schedule reported in the main text.

Figure S3: Committed-minority tipping, recomputed. a Share of runs in which an established convention is overturned, against the size of the committed minority; the shaded band is the human bracket, 21–25% of the group. Lines are blue where a critical mass could be estimated and orange where the convention was never overturned at any tested fraction. b The resulting

p^{*}

per model, defined as the smallest tested fraction at which the convention is overturned in at least half of runs. Open circles with an arrow mark models whose conventions survive every tested minority; models that never form a convention have nothing to overturn, and the tipping arm was not executed for R1-Distill.

Figure S4: Single-agent diagnostics: cooperation against four fixed opponents. These runs are diagnostics rather than perturbation cells, and they were excluded from the baseline in the revised analysis; the figure is unaffected by that correction because it plots the diagnostic runs themselves. Two models cooperate with an opponent that always defects, which is the clearest single indication that behaviour is not payoff-driven.

Figure S5: A society in progress: one run rendered as a raster of agents against rounds, with an excerpt of the recorded transcript. Included because the corrections reported in the main text were found by reading transcripts against outcome statistics, and this is what that reading looks like.

Large Language Model-Based Social Simulation

Large language model-based social simulation studies how language-model agents represent people and interact to produce social behavior.

See all 257 papers

Jun 29, 2026

Collective cooperation without individual fidelity in LLM agents

Large language models (LLMs) are increasingly used as agents in simulations of social systems, yet it remains unclear when their behavior can be interpreted as a faithful proxy for human decision-making. Here we test LLM agents against a direct empirical benchmark: a large-scale networked Prisoner's Dilemma experiment with human participants. Using the same interaction protocol, payoff structure, and network topologies, we compare nine open-weight LLMs with the human data. The selected model reproduces several macro-level features of cooperation dynamics, including the early decline and later stabilization of cooperation. This aggregate agreement, however, does not extend uniformly to finer levels of behavior. LLM populations underestimate individual-level heterogeneity and generate conditional cooperation patterns that differ from those observed in humans. Adding a fraction of random agents improves some aspects of micro-level agreement, but does not remove the mismatch in decision rules. These findings reveal a macro--micro dissociation in LLM-based social agents: collective outcomes can appear human-like even when the underlying behavioral distributions and mechanisms are not. They suggest that validating LLM agents as human surrogates requires comparisons across aggregate dynamics, individual heterogeneity, and context-dependent decision rules, rather than outcome-level agreement alone.

Henrique Ferraz de Arruda, Carlos Gracia Lázaro, Alberto Aleta +1

ARAID Foundation, Zaragoza, Spain. · Institute for Biocomputation and Physics of Complex Systems, University of Zaragoza, Zaragoza, Spain. · Universidad San Jorge (USJ), Zaragoza, Spain. +1

Multi-Agent LLM Systems Human Behavior Simulation LLM Agent Evaluation Large Language Model-Based Social Simulation

Ask PDF

Jul 19, 2026

Poor Man's Agentic Modeling: Simulating Large LLM-Agent Societies on a Laptop

Simulating societies of many large language model (LLM) agents is expensive, yet the questions asked of such simulations are usually macroscopic: phase behaviour, stylised facts, and scaling with the number of agents

N

, not the cognition of any single agent. We turn a statistical-physics observation into a method: replace each LLM agent by a low-parameter model fitted from a few hundred to a few thousand cheap queries, then run the society at any

N

on a laptop. Whether this works is decided before the simulation runs, chiefly by what each agent perceives. We introduce an [interaction order x memory] taxonomy that maps perception and memory to an effective theory and a predicted

N

-trend of the surrogate error. We validate it on a faithful reimplementation of the LLM macroeconomy EconAgent and seven further named LLM simulations, with agent decisions cloned from genuine LLM elicitations (primarily DeepSeek) for a few dollars; the predicted error trends hold cell by cell, and the two refuted predictions, both on a strongly saturating response and traced to its curvature, are themselves matched quantitatively by the theory with no free parameters.

Igor Itkin

Independent Researcher, Tel Aviv, Israel

Multi-Agent LLM Systems Surrogate Modeling Agent-Based Modeling Large Language Model-Based Social Simulation

Ask PDF

Nov 6, 2025

Large language models replicate and predict human cooperation across experiments in game theory

Large language models (LLMs) are increasingly deployed as decision-making agents in high-stakes domains and as imitators of human behavior in the social and behavioral sciences. Yet how closely LLMs mirror human decision-making remains poorly understood. This gap is critical: misalignment could produce harmful outcomes in practice, while failure to replicate human behavior renders LLMs ineffective as social simulators. Here, we address this gap by replicating large-scale game-theoretic experiments and by introducing a systematic prompting and probing framework for machine-behavioral evaluation. We test three open models typically used to power agents (Llama, Mistral, and Qwen). Across 121 dyadic games spanning four classical game types, Llama reproduces human cooperation patterns with high fidelity, while Qwen aligns closely with Nash equilibrium predictions. Characterizing models through behavioral phenotyping, we find that humans and Llama share an envious decision profile, while Qwen and Mistral exhibit different profiles. An attention-based analysis of payoff salience reveals Llama processes payoff information in a structured, layer-dependent manner absent in Qwen and Mistral, suggesting a mechanistic basis for its closer alignment with human behavior. Population-level behavioral replication is achieved without persona-based prompting, simplifying the simulation process. Extending the experimental parameter space beyond the original human-tested games, we generate and preregister testable hypotheses for novel game configurations. Our findings demonstrate appropriately configured LLMs can replicate aggregate human behavioral patterns, exhibit human-like decision phenotypes, and enable systematic exploration of unexplored experimental spaces, offering a complementary approach to traditional behavioral research that generates new empirical predictions about human social decision-making.

Andrea Cera Palatsi, Samuel Martin-Gutierrez, Ana S. Cardenal +1

Department for Computational Social Sciences and Humanities, Barcelona Supercomputing Center · Grupo de Sistemas Complejos, Universidad Polit´ecnica de Madrid · School of Law and Political Science, Universitat Oberta de Catalunya +1

LLM Evaluation Game Theory Human Behavior Simulation Large Language Model-Based Social Simulation

Ask PDF

Abstract

Large Language Model-Based Social Simulation

Explore similar work

Collective cooperation without individual fidelity in LLM agents

Poor Man's Agentic Modeling: Simulating Large LLM-Agent Societies on a Laptop

Large language models replicate and predict human cooperation across experiments in game theory