physics.soc-phAug 28, 2026
SaveBenchmarking large language model agent societies against human behavioural distributions
Organizations: AI Cluster, XU Exponential University of Applied Sciences, Marlene-Dietrich-Allee 12B, 14482, Potsdam, Germany.
Abstract
Populations of large language model agents are increasingly used as experimental societies. Three doubts shadow every such result: whether the agents behave like the humans they stand in for, whether a finding survives changes to the apparatus that leave the rules untouched, and whether apparent social dynamics are interaction at all rather than the reproduction of experiments the models have read. This article introduces SILICA, an open instrument that tests all three. Five environments carry published human anchors, each paired with perturbations that re-render the same rules and with variants whose payoffs point away from the memorised result. Twelve open-weight models were run through it on a single consumer graphics card. Agreement with human data is confined to starting points: first-round public-goods contributions fall inside the equivalence margin for eight of eleven models, while no model matches end-state contributions or the human corridor of cooperation. Merely swapping the order in which two actions are listed costs one model 58 points of cooperation. Presenting responders with a fixed schedule of offers shows that only one model, the sole reasoning-trained one, places its acceptance threshold where the incentive requires; two move theirs part of the way, two move them the wrong way, and three never acquire one. Conventions form through a shared prior over the names rather than through negotiation, though negotiation reappears once that prior is disrupted. On the certification ladder defined here, current silicon societies support exploratory claims and no more.
Figures & tables
Figure 1: The instrument. Left Five environments, each paired with a published human anchor, a perturbation library whose cells are separated into those that re-render the same rules and those that alter what the agents are told, and a gradient of payoff variants ending in C3, where the result recorded in the literature is no longer the profitable one; the last two entries are controls added after a transcript audit. Centre The roster and the execution protocol; the full roster with quantisation and per-model run counts is in the Supplementary Information. Right The four measurements and the ladder that converts them into a claim grade, with the current verdict shown against each tier. Cell counts and run totals are read from the released record; the registration shown in the protocol card is at https://doi.org/10.17605/OSF.IO/AFSDU .
| Level | Cell | PD | PGG | Ult. | 11–20 | Naming |
| Representation | agent naming = names | |||||
| history rendering = narrative | ||||||
| history summary statistics = shown | ||||||
| action-label order = swapped | ||||||
| persona layout = bullets | ||||||
| persona layout = table |
Table 1: The perturbation library: every cell of the audit grid, its level, and the environments in which it was run ( ). One factor changes per cell; all other components are held at the canonical baseline (plain persona, neutral content, canonical framing, tabular history without summary statistics, default label order, letter labels, temperature , memory window , default group size). Each cell holds 10 seeded runs per model (2 runs of 50 games in the one-shot battery).
| Question | What it asks | What the data show | Tier |
|---|---|---|---|
| RQ1 Level fidelity | Does behaviour match human levels? | public goods round 1: 8/11 equivalent; round 10: 0/11; ultimatum offers: 1/12; dictator: 0/12; PD inside the human corridor: 0/12 | Tier 1 |
| RQ1 Directional fidelity | Does behaviour move in the human direction? | public goods decay 1/12; PD decay 3/12; convention formation 10/12 | Tier 1 |
| RQ2 Robustness | Does the finding survive re-rendering the same rules? | Holm-significant shifts in 64 of 182 computable cells; largest representation-level reversal 1.00; largest design-level 1.00 | Tier 1 |
| RQ3 Contamination | Is the behaviour retrieved or computed? | ultimatum, under a fixed offer schedule: 1 of 12 models sets its acceptance threshold where the outside option requires, 2 move theirs partway, 2 the wrong way and 3 never acquire one; naming with a per-agent shuffled pool: 7 of 8 models still converge in every seed, 1.5 to 2.6 times slower, while Qwen2.5-7B converges in only 2 of 3 seeds at 14–24 times the fixed-order value; final-round defection with no endgame incentive in 3 of 10 models | Tier 1 |
| RQ4 Certification | What can the instrument certify? | no finding reaches Tier 3. Convention formation reaches Tier 2: the shuffled-pool control shows that coordination is driven by a shared prior over the labels rather than by shared list position, and that negotiation does occur once that cue is removed — but still between twice and twenty-eight times faster than the classical baseline, so the mechanism is related to the human one without matching it. Every other finding is Tier 1 | – |
Table 2: The study at a glance. Each row states one research question, the evidence assembled for it, and the highest certification tier the evidence supports. Denominators are the number of models for which the comparison is computable; R1-Distill completed a partial battery and is excluded from cells where it holds fewer than three runs. Equivalence is the two one-sided test against the human anchor at the margin fixed in advance; significance is Holm-adjusted within family under the exact test. Per-model acceptance thresholds and payoff consistency are in Supplementary Table S5; a high consistency score can be produced by rejecting almost everything, so the threshold rather than the score is the quantity reported here.
Figure 2: Public goods and costly punishment. a First- and last-round contributions at the canonical baseline, with the interquartile range across runs and the human corridors shaded; the three classical baselines are shown in italics to the right of the divider. b Final-round contributions without and with the punishment option, with the standard deviation across runs; marks a Holm-significant decrease under the exact test. c The share of runs in which punishment is purchased (bars) and the points spent per agent per round (points, right axis).
| Repeated dilemma | Public goods | Ultimatum | Naming | 11–20 | ||||
|---|---|---|---|---|---|---|---|---|
| Model | cooperation | in band | round 1 | round 10 | offer | conventions | ||
| Qwen3-32B | 0.95 0.05 | no | 0.15 0.17 | 0.00 0.00 | 0.49 0.01 | 1.00 | 0.20 | 0.062 [0.06,0.07] |
| Qwen3-14B | 0.85 0.08 | no | 0.50 0.00 | 0.50 0.00 | 0.50 0.00 | 1.00 | 0.40 | 0.218 [0.21,0.22] |
| Qwen3-8B | 1.00 0.00 | no | 1.00 0.00 | 1.00 0.00 | 0.22 0.05 | 1.00 | 0.30 | 0.576 [0.58,0.58] |
| Qwen3-4B | 0.00 0.00 | no | 1.00 0.00 | 1.00 0.00 | 0.50 0.00 | 1.00 | 0.30 | 0.637 [0.63,0.64] |
| Qwen2.5-14B | 1.00 0.00 | no | 0.57 0.02 | 0.79 0.09 | 0.40 0.00 | 1.00 | 0.20 | 0.224 [0.22,0.22] |
Table 3: Baseline fidelity against the human anchors, one row per model. Values are means over the canonical baseline runs with the standard deviation across runs; is the number of runs. Bold marks an outcome that passes the two one-sided equivalence test against its anchor at the margin fixed in advance. PD cooperation is compared with the corridor – reported across prisoner’s dilemma studies; public-goods contributions with in round 1 and in round 10; ultimatum offers with of the pie; convention formation with the human result that a convention always forms; and the 11–20 choice distribution through the Wasserstein distance to the distribution of Arad and Rubinstein, normalised by the width of the support, with a bootstrap 95% interval. R1-Distill completed a partial battery and its public-goods cell holds a single run, shown as –. In the column, a value is the smallest tested committed-minority fraction at which the convention is overturned in at least half of runs; means a convention forms but was never overturned at any tested fraction; “no conv.” means no convention forms, so there is nothing to overturn; and “not run” means the tipping arm was not executed for that model.
Figure 3: Every cell of the audit grid. One panel per environment; rows are models, columns are perturbation cells, ordered representation-level first and separated from the design-level cells by a rule. Colour is the shift of the environment’s primary outcome relative to the canonical baseline; marks a Holm-significant contrast under the exact test. The one-shot games, whose cells hold two runs and are descriptive.
| , representation-level | , design-level | , representation / design | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | PD | PGG | Ult. | 11–20 | Naming | PD | PGG | Ult. | 11–20 | Naming | PD | PGG | Naming |
| Qwen3-14B | 0.68 | 0.00 | 0.00 | 0.08 | 0.00 | 0.77 | 0.55 | 0.00 | 0.01 | 0.00 | 0.57 / 0.57 | 0.00 / 0.00 | 1.00 / 1.00 |
| Gemma2-9B | 0.17 | 0.14 | 0.06 | 0.02 | 0.00 | 1.00 | 0.24 | 0.20 | 0.01 | 0.50 | 0.00 / 0.00 | 0.00 / 0.00 | 1.00 / 1.00 |
| Llama3.1-8B | 0.46 | 0.02 | 0.03 | 0.08 | 0.00 | 0.59 | 0.08 | 0.12 | 0.08 | 1.00 | 0.14 / 0.43 | 0.00 / 0.00 | 0.00 / 0.17 |
| Mistral-7B | 0.00 | 0.09 | 0.21 | 0.00 | 0.10 | 0.07 | 0.07 | 0.04 | 0.00 | 1.00 | 0.00 / 0.00 | 0.00 / 0.00 | 1.00 / 0.67 |
| Phi3-mini | 0.11 | 0.04 | 0.02 | 0.03 | 1.00 | 0.23 | 0.12 | 0.02 | 0.02 | 1.00 | 0.57 / 0.57 | 0.00 / 0.00 | 0.67 / 0.83 |
Table 4: Robustness of the primary outcome under the perturbation library, one row per model, split by the level of the perturbation. (Eq. 4 ) is the range of cell means over the canonical baseline and either the representation-level cells (the same rules re-rendered: persona layout, history rendering and summary, action-label order, agent naming) or the design-level cells (content changes: persona content, moralized or risk framing, step-by-step reasoning, sampling temperature, memory window, group size). is the share of cells (baseline included) in which the canonical human effect keeps its sign (PD and PGG: decline of cooperation or contributions over rounds; naming: convergence); it is defined only for those three environments. Perturbation cells hold 10 runs per model (2 runs of 50 games in the one-shot games, where is therefore descriptive); baseline cells 30 (6 in the one-shot games). – arm not run, baseline with fewer than 3 runs, or fewer than two cells with at least 2 runs (R1-Distill completed a partial battery; its per-cell run counts are listed in the Supplementary roster).
Figure 4: Acceptance functions in the ultimatum game under a fixed schedule of offers. Each panel shows one model’s rejection rate against the offer, in the canonical game where rejection pays nothing and in the divergent-payoff variant where rejection pays 40. The dashed line marks the offer above which accepting is the profitable act in the divergent variant; the dotted line marks the low-offer threshold conventionally reported for human responders. Shading marks the region in which rejection is payoff-maximising under the divergent payoffs.
Figure 5: How conventions form, and what survives when the incentive to defect is removed. a Match rate over the first 300 interactions of the canonical naming game, one line per model. b Match rate over the first twenty interactions with the name that wins. c Defection rate by round in the canonical game (dashed) and in the variant where cooperation is strictly dominant (solid).
Figure 6: One society overturning its own convention. The naming game is the only environment here whose record is natively a graph: each line of the transcript is a directed interaction between two identified agents, carrying the word each proposed and whether the two matched. a–c Three moments of a single run of the tipping arm (Qwen3-32B, seed 0, 24 agents): the established convention, the point at which a committed minority of six agents — 25% of the population, the human critical mass — has split the population evenly, and the state after the minority has carried it. Node positions are computed once from the whole run and held fixed, so the only thing that changes between panels is colour; edges show the last 25 interactions before each moment. d The boxed neighbourhood at the contested moment, with individual interactions drawn. e Adoption of the new word over the run. Committed agents, ringed in black, never switch; mismatched interactions are the mechanism by which the rest do.
Figure 7: Four primary outcomes against model size. a Cooperation rate in the repeated dilemma. b Final-round contribution in the public-goods game. c Share of runs reaching a convention. d Distance between the model’s choice distribution in the 11–20 game and the human one. Lines join models of the same family; the reasoning-trained model is drawn as a diamond; shaded bands mark the human values. Public-goods cells holding fewer than three runs are omitted.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Params (B) | Quantisation | Deliberation | Runs | Cells | Calls | Invalid | Rate | GPU h |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3-32B | 32.8 | AWQ 4-bit | off | 792 | 86 | 1,699,840 | 10 | 0.00% | 49.6 |
| Qwen3-14B | 14.8 | FP8 | off | 827 | 88 | 1,940,640 | 0 | 0.00% | 131.8 |
| Qwen3-8B | 8.2 | bf16 | off | 792 | 86 | 1,699,840 | 0 | 0.00% | 72.4 |
| Qwen3-4B | 4.0 | bf16 | off | 792 | 86 | 1,699,840 | 21 | 0.00% | 47.3 |
| Qwen2.5-14B | 14.7 | AWQ 4-bit | off | 792 | 86 | 1,699,840 | 0 | 0.00% | 16.3 |
| Qwen2.5-7B | 7.6 | AWQ 4-bit | off | 792 | 86 | 1,699,840 | 0 | 0.00% | 21.6 |
Table S1: Model roster and execution statistics. Parameter counts are the published totals. Quantisation is as served; the Qwen3 models expose an optional deliberation mode, which was disabled throughout, so the reasoning axis rests on the single reasoning-trained model rather than on a decoding switch. Attempted calls are estimated from the run manifest; invalid actions are replies that failed schema validation and were replaced by the environment’s fallback. R1-Distill completed a partial battery, which is why its cell count is lower.
| Family | Env | Model | Contrast | ||||
|---|---|---|---|---|---|---|---|
| contamination | bargain | Gemma2-9B | C3 vs C0 | 6+6 | -1.00 | 0.002 | 0.015 |
| contamination | bargain | Mistral-7B | C3 vs C0 | 6+6 | -1.00 | 0.002 | 0.015 |
| contamination | bargain | Qwen1.5-7B | C3 vs C0 | 6+6 | -1.00 | 0.002 | 0.015 |
| contamination | bargain | Qwen2.5-7B | C3 vs C0 | 6+6 | +1.00 | 0.002 | 0.015 |
| contamination | bargain | Qwen3-32B | C3 vs C0 | 6+6 | +1.00 | 0.002 | 0.015 |
| contamination | bargain | Phi3-mini | C3 vs C0 | 6+6 | +0.67 | 0.067 | 0.134 |
Table S2: Test-level record for the contrasts named in the main text, plus every contamination contrast: 95 of 227 computable contrasts, ordered by family. is the exact permutation value conditional on the observed ties and its Holm-adjusted counterpart within the family. The complete file, including the value each test would take under a normal approximation, is results/stat_tests.csv in the released record.
| Model | C1 shift | C2 shift | Priming shift | Reports knowing | Names correctly |
|---|---|---|---|---|---|
| Qwen3-32B | -0.05 | +0.05 | -0.03 | 100% | 70% |
| Qwen3-14B | -0.20 | -0.85 | +0.08 | 100% | 70% |
| Qwen3-8B | +0.00 | +0.00 | +0.00 | 100% | 56% |
| Qwen3-4B | +0.34 | +0.10 | +0.03 | 100% | 57% |
| Qwen2.5-14B | +0.00 | +0.00 | +0.00 | 80% | 51% |
| Qwen2.5-7B | +0.00 | -0.98 | +0.00 | 93% | 55% |
Table S3: Contamination outcomes away from the divergent-payoff variant, and what the models say when asked directly. C1 re-skins the game with a different cover story and action labels while preserving the payoffs; C2 additionally shifts the numbers. Values are the shift in the repeated dilemma’s cooperation rate relative to the canonical baseline. The priming column is the shift when the paradigm is named inside the behavioural prompt. Recognition is asked in a separate session: whether the model reports knowing the paradigm, and whether it names it correctly. Recognition predicts none of the behavioural columns. R1-Distill completed a partial battery: it ran the C1 and divergent-payoff cells of the repeated dilemma but no C2 cell, so its C2 entry is absent rather than zero.
| Environment | Anchor | Source | Evidence base | Provenance |
|---|---|---|---|---|
| 11–20 game | full choice distribution; , | Arad & Rubinstein (2012), AER 102(7), Table 1 | single experiment, | published table |
| Ultimatum | offer of pie; rejection | Oosterbeek et al. (2004), Exp. Econ. 7(2) | meta-analysis, 37 papers, 75 results | meta-analytic mean |
| Dictator | giving of endowment | Engel (2011), Exp. Econ. 14 | meta-analysis, 328 treatments, 20,813 decisions | random-effects mean |
| Trust | sent ; returned of available | Johnson & Mislin (2011), JEP 32(5) | meta-analysis, 162 replications, 23,000 participants | meta-analytic mean |
| Public goods | round 1 ; round 10 ; punishment | Ledyard (1995) survey; Zelmer (2003); Fehr & Gächter (2000) | 27 studies, 711 groups; punishment from a single experiment | stylised fact, not a single estimate |
| Repeated PD | cooperation corridor – | Mengel (2018), Econ. J. 128(616) | meta-study, 96 studies, 3,500 participants | range across studies |
Table S4: Provenance of every human anchor. Each value was checked against its primary source on 26 August 2026 and the shipped anchor file now records the citation and the verification date. One correction was made: two cells of the 11–20 distribution had been mis-transcribed, and all Wasserstein distances were recomputed against the published table. The public-goods trajectory is a stylised fact drawn from a survey and a meta-analysis rather than a single meta-analytic estimate, and its punishment value comes from one experiment; the manuscript states this rather than describing the anchor as a meta-analytic corridor.
| Model | Variant | 10 | 20 | 25 | 30 | 33 | 37 | 40 | 45 | 50 | Threshold | Consistency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-32B | canonical | 0.13 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | – | 0.99 |
| divergent | 1.00 | 0.53 | 0.67 | 0.68 | 0.00 | 0.03 | 0.05 | 0.00 | 0.00 | 30 | 0.61 | |
| Qwen3-14B | canonical | 1.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 25 | 0.67 |
| divergent | 1.00 | 1.00 | 1.00 | 1.00 | 0.75 | 0.13 | 0.00 | 0.00 | 0.00 | 33 | 0.86 | |
| Qwen3-8B | canonical | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | – | 1.00 |
| divergent | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | – | 0.25 |
Table S5: Acceptance functions under the fixed offer schedule: rejection rate at each scheduled offer, 20 games per offer per seed, three seeds. In the canonical variant rejection pays nothing, so accepting every offer is payoff-maximising; in the divergent variant rejection pays 40, so rejecting below 40 is. The threshold is the largest offer still rejected in at least half of games. R1-Distill is the only model whose threshold falls where the outside option puts it.
| Model | Fixed order | Cap | Runs | Converged | Median | Range | First-20 match | Name |
|---|---|---|---|---|---|---|---|---|
| Qwen3-32B | 50 | 600 | 3 | 3 | 104 | 90–117 | 0.67 | A |
| Qwen3-14B | 50 | 600 | 3 | 3 | 103 | 100–237 | 0.42 | A |
| Qwen3-8B | 50 | 600 | 3 | 3 | 75 | 50–113 | 0.85 | A |
| Qwen3-4B | 50 | 600 | 3 | 3 | 129 | 104–202 | 0.58 | A |
| Qwen2.5-14B | 50 | 600 | 3 | 3 | 81 | 79–97 | 0.68 | A |
| Qwen2.5-7B | 50 | 600 | 3 | 0 | 600 | 600–600 | 0.17 | C/G |
Table S6: The shuffled-pool control. Each agent receives its own random permutation of the same ten names, and an unparsable reply falls back to a random name rather than the first listed one. Fixed-order convergence is the median over the 30 baseline runs; 50 is the earliest value the convergence rule can return. Coordination survives in every model and slows in all of them. Runs are listed separately by cap: Qwen2.5-7B reached the 600-interaction cap in all three capped runs and, uncapped, converged in only two of three. Every model still converges on the same name as under a fixed order except R1-Distill, which moves from A to E.
| Interactive multi-agent | Human-anchored (distribution) | Perturbation audit | Contamination audit (C3) | 5 model families | Scale axis | Generation axis | Claim certification | Open exec. benchmark | |
| Park et al. (2024) | ✓ | ✓ | – | – | – | – | – | – | (✓) |
| Ashery et al. (2025) | ✓ | (✓) | (✓) | (✓) | (✓) | – | – | – | (✓) |
| Akata et al. (2025) | ✓ | (✓) | – | – | (✓) | – | – | – | – |
| Cui et al. (2025) | – | ✓ | – | – | – | – | – | – | (✓) |
| Gao et al. (2025) | – | ✓ | (✓) | (✓) | ✓ | – | – | – | (✓) |
| GTBench ( Duan et al., 2024 ) | ✓ | – | – | – | ✓ | (✓) | – | – | ✓ |
Table S7: Positioning against the closest prior work. Rows name the benchmark or platform where one exists and the study otherwise. Unlike every other table in this article, these entries are editorial judgements about other authors’ work rather than measurements: a mark records only whether the cited work set that capability as an objective, not how well it succeeded. The criteria are defined in the text above. ✓ stated as an objective, with results reported against it; (✓) a partial or methodological precursor, present but not carried through to a reported result; – not an objective of the cited work, which is a statement about scope rather than a criticism.
Figure S1: Perturbation cells in the two one-shot environments. Rows are models, columns are cells, ordered representation-level first. These cells hold two runs each, so the shifts are descriptive and no significance is marked; the three environments in which run-level tests are possible appear in the main text.
Figure S2: The divergent-payoff ultimatum cell as originally run, before the fixed offer schedule was introduced. a Every game, plotted as the offer against the responder’s decision; the shaded band marks the range in which the memorised script and the payoffs disagree. b The share of each model’s games that fall in that band. Seven of nine models never produce a single diagnostic game, because the proposer is the same model as the responder and offers at or above the outside option. This is the observation that motivated the fixed schedule reported in the main text.
Figure S3: Committed-minority tipping, recomputed. a Share of runs in which an established convention is overturned, against the size of the committed minority; the shaded band is the human bracket, 21–25% of the group. Lines are blue where a critical mass could be estimated and orange where the convention was never overturned at any tested fraction. b The resulting per model, defined as the smallest tested fraction at which the convention is overturned in at least half of runs. Open circles with an arrow mark models whose conventions survive every tested minority; models that never form a convention have nothing to overturn, and the tipping arm was not executed for R1-Distill.
Figure S4: Single-agent diagnostics: cooperation against four fixed opponents. These runs are diagnostics rather than perturbation cells, and they were excluded from the baseline in the revised analysis; the figure is unaffected by that correction because it plots the diagnostic runs themselves. Two models cooperate with an opponent that always defects, which is the clearest single indication that behaviour is not payoff-driven.
Figure S5: A society in progress: one run rendered as a raster of agents against rounds, with an excerpt of the recorded transcript. Included because the corrections reported in the main text were found by reading transcripts against outcome statistics, and this is what that reading looks like.
More in this topic
Large Language Model-Based Social Simulation
Large language model-based social simulation studies how language-model agents represent people and interact to produce social behavior.
See all 257 papers