Local Predictability and Collective Fidelity in LLM-Agent Societies

Igor Itkin

physics.soc-phSep 20, 2026

Local Predictability and Collective Fidelity in LLM-Agent Societies

Authors: Igor Itkin

Organizations: Tel Aviv, Israel

Abstract

Compact surrogates could reduce the cost of simulating large language model societies, but must reproduce collective behavior. We compare individual predictions and collective forecasts using 9,455 published trajectories and new experiments on opinion dynamics. Neighbor information improves individual prediction in all 16 public-data settings and pooled collective forecasts on held-out questions, although collective gains depend on transfer conditions. Tests on 24 new statements do not confirm earlier contrasting history effects in forecasts from the initial state. Qwen benefits from history after three observed rounds. These findings motivate direct collective validation, explicit limits on available observations, and comparisons with simple baselines.

Figures & tables

Study	Complete episodes	Test population and horizon	Held out and scope of inference
Public histories	9,455/9,600	$N=32$ ; $T=8$	18 held-out questions shared across four backbones. Two overlapping transfer settings: new questions, or new questions plus unsigned graphs. Reanalysis, not a prospective experiment or exact source-model reproduction.
New finite grid	430/440	$N=16,32,64,128$ ; $T=3$	New initializations and sizes, but calibrated questions. Five backends; balanced evaluation retains 22 backend/question panels and 304 test episodes. Small or single-class panels limit inference.
Long follow-up	192/192	$N=16,64$ ; $T=3,6,12$	Five test initializations per size, four known subjective questions, two local backends. Short-frozen versus long-refit calibration; nested horizons are not independent tests. One transport-failed episode replaced.
New statements	479/480	$N=16,64$ ; $T=3,6$	20 fixed new statements, two local backends. Untouched source policies versus topic-specific training/validation; two test initializations per size. Balanced panels retain 20 Gemma and 19 Qwen questions after one technical replay.
API extension	120/120	$N=16$ ; $T=3,6$	The same 20 new statements on Qwen Max and GLM, one test initialization each; both calibration arms. Sixteen separate source episodes precede the listed grid. One validation replay; no API population-size scaling test.
Prospective replication	192/192	$N=16,64$ ; $T=6$	24 fixed additional statements, two local backends and two test initializations per size. Primary scores use updates 4–6. Original source policies remain fixed; conditional forecasts observe states 0–3 with prespecified adaptation rules and simple baselines. No failures or exclusions (Sec. XI ).

Table 1: Scope of the six LLM-trajectory studies. Episode counts include designated training and validation data, not only tests. Completed technical replacements count once; failed originals are preserved.

N

and

T

describe held-out evaluation. Counts and overlapping questions must not be interpreted as independent replications or pooled into a common effect.

Figure 1: Graph-field policy versus global-mean policy on the same test questions. Positive values indicate lower error with graph fields. Points are paired reductions; bars are 95% question-cluster bootstrap intervals. O and S denote objective and subjective questions (12 and 6 test clusters per backbone). Upper panels retain available graph types; lower panels train on signed graphs and test on unsigned lattices as well as new questions. Local log-loss gains are more consistent than collective-error gains. Intervals are not multiplicity-adjusted.

Setting	Comparison	$\Delta\ell$	$\Delta E_{m}$
Questions	Global to graph	$0.1518\;[0.1370,0.1681]$	$0.0315\;[0.0188,0.0465]$
Questions	Degree to graph	$0.1515\;[0.1361,0.1683]$	$0.0318\;[0.0188,0.0478]$
Questions	Graph to normalized	$0.0057\;[0.0031,0.0084]$	$-0.0026\;[-0.0114,0.0052]$
Questions	Graph to history	$0.0116\;[0.0085,0.0151]$	$-0.0005\;[-0.0066,0.0049]$
Questions + lattices	Global to graph	$0.1412\;[0.1216,0.1635]$	$0.0238\;[-0.0059,0.0556]$
Questions + lattices	Degree to graph	$0.1424\;[0.1224,0.1656]$	$0.0363\;[-0.0013,0.0758]$

Table 2: Pooled paired reductions with descriptive 95% intervals. Each question is averaged across the four backbones before bootstrap resampling;

n=18

questions per setting. Positive numbers favor the second policy.

\Delta\ell

is in nats per node-transition;

\Delta E_{m}

is in binary-magnetization units.

Figure 2: Local and collective changes relative to the graph-sum policy. Top: replace graph sums by degree-normalized neighborhood means. Bottom: add the validation-selected history. Rightward and upward changes improve local and collective scores, respectively; a lower-right point improves local prediction but worsens the collective observable. Colors identify backbones and point labels identify legend entries; circle/square markers denote objective/subjective questions, with backbone and mode identified in the legend. All sixteen cells appear for each comparison. These are point estimates; pooled intervals appear in Table 2 .

Figure 3: A post-hoc diagnostic of the Llama subjective-question combined-transfer cell, showing all six test questions rather than selected individual successes or failures. Black curves are observed binary magnetizations; colored curves are autonomous surrogate means from the same accepted initial states. All available unsigned graphs are averaged equally for each question. Question labels are fixed hash prefixes. This illustration has only two to four observed replicas per graph and should not be interpreted as an exact reference expectation.

Setting	Semantic comparison	$\Delta\ell$	$\Delta E_{m}$
Questions	Global to graph	$0.1510\;[0.1361,0.1677]$	$0.0309\;[0.0171,0.0464]$
Questions	Graph to history	$0.0112\;[0.0082,0.0145]$	$-0.0022\;[-0.0092,0.0037]$
Questions + lattices	Global to graph	$0.1420\;[0.1225,0.1643]$	$0.0308\;[0.0038,0.0608]$
Questions + lattices	Graph to history	$0.0097\;[0.0054,0.0148]$	$0.0045\;[-0.0066,0.0148]$

Table 3: Post-primary matched semantic augmentation. Paired reductions and descriptive 95% question-cluster intervals; positive favors the second policy. Each setting uses the original 18 held-out questions and accepted histories.

Figure 4: New LLM trajectories: balanced eight-question panels for the two local backends, with two test initializations per question/size/regime. Curves show three-round error against observed magnetization for the primary pooled-regime fits; shading gives descriptive 95% question-cluster intervals. The same questions enter every size. Larger populations do not uniformly resolve policy error. Bands overlap and are not tests of paired differences; paired contrasts are reported in Table 4 . These four sizes do not establish asymptotic scaling.

E_{m}

abbreviates

E_{m}^{\mathrm{new}}

in Eq. ( 12 ).

Backend	Information	Questions	$\Delta\ell$	$\Delta E_{m}$
Qwen local 27.3B	global	8	$-0.0168\;[-0.0384,-0.0013]$	$-0.0012\;[-0.0306,0.0262]$
Qwen local 27.3B	local	8	$0.0039\;[-0.0206,0.0265]$	$-0.0101\;[-0.0343,0.0121]$
Gemma local 8.0B	global	8	$-0.0121\;[-0.0278,0.0012]$	$-0.0048\;[-0.0097,0.0003]$
Gemma local 8.0B	local	8	$-0.0057\;[-0.0244,0.0098]$	$-0.0005\;[-0.0056,0.0035]$
Qwen Flash API	global	2	$0.0000$	$0.0000$
Qwen Flash API	local	2	$0.0000$	$0.0000$

Table 4: New-campaign balanced-panel graph-versus-scalar reductions, averaged equally across the four sizes. Fits pool information regimes. Positive favors graph features. Intervals condition on retained questions; API panels have only two questions and receive point estimates only. Different API question composition precludes interpreting this table as a model leaderboard.

E_{m}

abbreviates

E_{m}^{\mathrm{new}}

in Eq. ( 12 ).

Figure 5: Completed 12-round follow-up, local-input regime. Top: observed-transition log loss; bottom: autonomous mean-magnetization error. Dashed lines use the original three-round fits; solid lines use independently generated 12-round calibration data with validation-only selection. Circle, square, and triangle markers denote scalar, graph, and graph-plus-history policies, respectively. Points average five test initializations, two sizes (

N=16,64

), and the same four known questions per backend. Horizons are nested, not independent experiments. Curves are descriptive point estimates, not uncertainty bands. This figure uses the technical-replay overlay; paired graph contrasts and replay sensitivity are given in Tables E.1 and E.1 .

E_{m}

abbreviates

E_{m}^{\mathrm{new}}

in Eq. ( 12 ).

Figure 6: Completed new-question transfer test, local input at

T=6

. Top: graph-versus-scalar reductions; bottom: two-lag-versus-graph reductions. Left: observed-transition log loss; right: autonomous mean-magnetization error. Positive favors the alternative. Open squares are untouched source-frozen policies; filled circles use topic-specific training and validation, never test fitting. Points average two test realizations, two sizes and 20 Gemma or 19 Qwen questions; bars are descriptive 95% whole-question bootstrap intervals. The primary topic-calibrated graph comparisons are unresolved. The secondary source-frozen Gemma history comparison improves the local score but worsens the collective score, while its Qwen counterpart improves both. This is the technical-replay overlay, not an additional independent experiment. The Gemma collective contrast does not exclude zero under the broader multiplicity sensitivity in Sec. XII.4 ; the error is Eq. ( 12 ).

Model	$q$	$\Delta\ell$	$\Delta E_{m}$
GLM 5.2	20	$0.0122\;[-0.0242,0.0408]$	$0.0444\;[-0.0030,0.1158]$
Qwen 3.8 Max	20	$0.0959\;[0.0313,0.1524]$	$-0.0035\;[-0.0466,0.0349]$

Table 5: Prespecified API-model comparison: topic-calibrated graph versus scalar policy, local input,

T=6

, held-out

N=16

. Entries are baseline-minus- alternative error reductions with descriptive 95% whole-question bootstrap intervals;

q

counts complete question panels after the validation replay. Local log loss is measured in nats per node transition.

E_{m}

abbreviates

E_{m}^{\mathrm{new}}

in Eq. ( 12 ).

Figure 7: The same prespecified API contrasts as Table 5 . The dashed line marks zero improvement. Both local and collective scores are evaluated on the same held-out trajectories; a gain in one coordinate does not establish a gain in the other.

Study/backend ( $n_{q}$ )	Arm	Baseline $\to$ alternative	$\Delta E_{m}$	$\Delta E_{2,\mathrm{f}}$ [95% interval]	$\Delta C_{\mathrm{f}}$ [95% interval]
Topics: Gemma (20)	TC	scalar $\to$ graph	$-0.0056$	$-0.0084\;[-0.0358,+0.0116]$	$-0.0043\;[-0.0283,+0.0158]$
Topics: Qwen (19)	TC	scalar $\to$ graph	$+0.0037$	$+0.0041\;[-0.0114,+0.0226]$	$+0.0072\;[-0.0145,+0.0324]$
API: GLM (20)	TC	scalar $\to$ graph	$+0.0444$	$+0.0360\;[-0.0021,+0.1002]$	$+0.0409\;[-0.0021,+0.1125]$
API: Qwen Max (20)	TC	scalar $\to$ graph	$-0.0035$	$-0.0166\;[-0.0584,+0.0150]$	$-0.0016\;[-0.0337,+0.0294]$
Topics: Gemma (20)	SF	graph $\to$ H2	$-0.0149$	$-0.0300\;[-0.0439,-0.0171]^{\dagger}$	$-0.0221\;[-0.0312,-0.0117]^{\dagger}$
Topics: Qwen (19)	SF	graph $\to$ H2	$+0.0186$	$+0.0226\;[+0.0147,+0.0318]^{\dagger}$	$+0.0188\;[+0.0141,+0.0234]^{\dagger}$

Table 6: Post-hoc collective-score sensitivity on amended balanced panels. All ten displayed comparisons were fixed before computing these scores. Input is local, sizes are pooled, and horizons are six rounds for topics/API and twelve for the long study. Differences are baseline minus alternative; positive values favor the alternative. Here

n_{q}

counts questions, H2 denotes two retained lags, SF source-frozen, TC topic-calibrated, and short the short-frozen long-study arm. The last row compares short-frozen with long-refitted scalar policies. Original MAE is retained; brackets are paired 95% descriptive intervals for fair MSE and CRPS. A dagger marks intervals also excluding zero in the corresponding 24-contrast post-hoc sensitivity; that adjustment does not apply to the last row. All policies, horizons, sizes, original/amended versions and uncorrected MSE are retained in the machine-readable analysis.

Backend	Contrast	$\Delta E_{2,\mathrm{f}}$ [adjusted interval]	$\Delta C_{\mathrm{f}}$ [adjusted interval]
Gemma	AH	$-0.00117\;[-0.01496,+0.00643]$	$+0.00783\;[-0.01580,+0.02864]$
Gemma	AC	$-0.00053\;[-0.00195,+0.00024]$	$-0.00048\;[-0.00308,+0.00188]$
Gemma	AB	$+0.00115\;[-0.02309,+0.01428]$	$+0.02042\;[-0.01977,+0.05442]$
Gemma	CH	$+0.00224\;[-0.00318,+0.00649]$	$+0.01264\;[-0.00702,+0.03080]$
Gemma	CO	$+0.01006\;[-0.00859,+0.04740]$	$+0.02939\;[-0.00298,+0.08119]$
Gemma	CB	$+0.00756\;[+0.00001,+0.01789]$	$+0.01441\;[-0.00134,+0.03320]$

Table 7: All 28 primary contrasts in the precollection-frozen 24-statement test, updates 4–6, with equal question and population-size weights. For either error score

S

, AH is autonomous

S(\mathrm{graph})-S(\mathrm{history2})

; AC is autonomous

S(\mathrm{history2})-S(\mathrm{cap\ history2})

; AB is autonomous

S(\mathrm{two\ state})-S(\mathrm{cap\ history2})

; CH is conditional

S(\mathrm{graph})-S(\mathrm{history2})

; CO is conditional

S(\mathrm{graph})-S(\mathrm{offset\ graph})

; CB is conditional

S(\mathrm{two\ state})-S(\mathrm{offset\ graph})

; GC is CH minus AH. Conditional forecasts observe states 0–3. Positive AH/CH favors history; positive AC/CO favors the intervention; negative AB/CB favors the simple baseline. All brackets use the same 28-contrast adjustment. Signs relative to zero are evaluated before rounding.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Objective	Subjective	Excluded
GPT-4o-mini	1600	800	0
Qwen3.5-9B	1597	800	3
Gemma-3n-E4B-it	1520	738	142
Llama-3.1-8B	1600	800	0
Total	6317	3138	145

Appendix

Table 8: Complete episodes accepted by the integrity filter. Original counts are 1,600 objective and 800 subjective episodes for each backbone.

Held-out questions
	One-step log loss			Mean-trajectory MAE			Terminal $W_{1}$
Backbone / mode	Global	Graph	History	Global	Graph	History	Global	Graph	History
GPT / O	0.567	0.215	0.208	0.370	0.336	0.327	0.593	0.495	0.470
GPT / S	0.506	0.197	0.178	0.206	0.166	0.162	0.279	0.238	0.234
Qwen / O	0.318	0.199	0.190	0.519	0.489	0.510	0.604	0.559	0.578
Qwen / S	0.460	0.302	0.285	0.176	0.127	0.130	0.249	0.202	0.206

Appendix

Table 9: Absolute primary-run test scores; lower is better. History is the validation-selected depth (two lags in all cells). O/S denote objective/subjective questions. These node-resolved policies share current spin, initial spin, and round. Results for interaction-free, static-degree, normalized-neighbor, one-lag, and persistence controls, including Brier and absolute-magnetization scores, are retained in the accompanying machine-readable results.

Policy / split seed	Transfer	$\Delta\ell$	$\Delta E_{m}$
Majority / 260906	Questions + lattices	$0.1412\;[0.1216,0.1635]$	$0.0238\;[-0.0059,0.0556]$
Majority / 260906	Questions	$0.1518\;[0.1370,0.1681]$	$0.0315\;[0.0188,0.0465]$
Recorded binary / 260906	Questions + lattices	$0.1416\;[0.1219,0.1641]$	$0.0240\;[-0.0060,0.0557]$
Recorded binary / 260906	Questions	$0.1516\;[0.1369,0.1679]$	$0.0319\;[0.0192,0.0469]$
Strict / 260910	Questions + lattices	$0.1333\;[0.0905,0.1711]$	$0.0250\;[-0.0281,0.0735]$
Strict / 260910	Questions	$0.1580\;[0.1303,0.1816]$	$0.0078\;[-0.0037,0.0191]$

Appendix

Table 10: Post-primary split and integrity sensitivity: graph-versus-global paired reductions with 95% question-cluster intervals. Each setting has 18 test questions; different seeds reuse the underlying 60-question corpus. “Recorded binary” omits the majority-agreement requirement; “Majority” yields the same accepted histories as the primary strict rule on this snapshot.

		Pooled-regime fit		Separate-regime fit
Backend	Information	$\Delta\ell$	$\Delta E_{m}$	$\Delta\ell$	$\Delta E_{m}$
Qwen local 27.3B	global	-0.0168	-0.0012	-0.0119	-0.0123
Qwen local 27.3B	local	0.0039	-0.0101	0.0160	0.0016
Gemma local 8.0B	global	-0.0121	-0.0048	-0.0066	-0.0033
Gemma local 8.0B	local	-0.0057	-0.0005	0.0169	-0.0022
Qwen Flash API	global	0.0000	0.0000	0.0000	0.0000

Appendix

Table 11: Post-primary calibration sensitivity on the unchanged balanced test panel. Entries are graph-versus-scalar point reductions; positive favors graph features. Separate-regime fitting can change both the sign and magnitude of a difference.

\Delta\ell

is the log-loss reduction and

\Delta E_{m}

the reduction in Eq. ( 12 ).

Backend	Input	Fit horizon	$\Delta\ell$	$\Delta E_{m}(12)$
Qwen	global	3 rounds	$1.1655\;[-0.0309,3.2847]$	$0.0073\;[-0.0172,0.0381]$
Qwen	global	12 rounds	$-0.0290\;[-0.0545,-0.0036]$	$0.0202\;[-0.0368,0.1037]$
Qwen	local	3 rounds	$0.0282\;[-0.0060,0.0708]$	$-0.0087\;[-0.0296,0.0027]$
Qwen	local	12 rounds	$-0.0102\;[-0.0456,0.0203]$	$-0.0228\;[-0.0628,0.0171]$
Gemma	global	3 rounds	$-0.0010\;[-0.0196,0.0162]$	$-0.0049\;[-0.0328,0.0153]$
Gemma	global	12 rounds	$0.0056\;[-0.0023,0.0134]$	$-0.0028\;[-0.0057,0.0002]$

Appendix

Table 12: Twelve-round graph-versus-scalar paired reductions, averaged equally over

N=16,64

. Positive favors graph features. All entries use the technically amended four-question panel for each backend and five test initializations per size. Brackets are descriptive 95% whole-question bootstrap intervals. The same questions occur in both calibration arms; no test outcome selects fits or regularization.

E_{m}

abbreviates

E_{m}^{\mathrm{new}}

in Eq. ( 12 ).

		Original ( $q=3$ )		Replay ( $q=4$ )
Input	Fit rounds	$\Delta\ell$	$\Delta E_{m}$	$\Delta\ell$	$\Delta E_{m}$
global	3	-0.0020	-0.0067	-0.0010	-0.0049
global	12	0.0044	-0.0024	0.0056	-0.0028
local	3	0.0628	-0.0036	0.0497	-0.0020
local	12	0.0078	0.0065	-0.0007	0.0027

Appendix

Table 13: Technical-replay sensitivity for Gemma at

T=12

: graph-versus-scalar point reductions before and after restoring the transport-failed case. Original estimates omit all s2 tests from the comparison; the amended analysis restores them. The question set changes from three to four, so the difference does not isolate the effect of replaying one trajectory. Original three-question intervals are suppressed; amended intervals appear in Table E.1 . Qwen is unchanged.

E_{m}

abbreviates

E_{m}^{\mathrm{new}}

in Eq. ( 12 ).

Backend	Input	Calibration	$\Delta\ell$	$\Delta E_{m}$
Gemma	global	Source-frozen	$0.0260\;[0.0094,0.0462]$	$0.0000\;[-0.0015,0.0017]$
Gemma	global	Topic-specific	$-0.0197\;[-0.0315,-0.0085]$	$-0.0084\;[-0.0180,-0.0001]$
Gemma	local	Source-frozen	$-0.0640\;[-0.1287,-0.0062]$	$-0.0244\;[-0.0757,0.0290]$
Gemma	local	Topic-specific	$0.0099\;[-0.0049,0.0259]$	$-0.0056\;[-0.0311,0.0150]$
Qwen	global	Source-frozen	$-0.0350\;[-0.0409,-0.0285]$	$-0.0704\;[-0.1060,-0.0325]$
Qwen	global	Topic-specific	$-0.0578\;[-0.0925,-0.0280]$	$-0.0224\;[-0.0525,0.0019]$

Appendix

Table 14: New-question graph-versus-scalar reductions at

T=6

, averaged equally over

N=16,64

. Positive favors graph. Brackets are descriptive 95% whole-question intervals, without multiplicity adjustment. All entries use the technical-replay overlay: 20 Gemma and 19 Qwen questions. Primary comparisons are the topic-specific local-input rows; other rows are secondary.

E_{m}

abbreviates

E_{m}^{\mathrm{new}}

in Eq. ( 12 ).

			One-step log loss			Magnetization MAE
Backend	Input	Calibration	Scalar	Graph	History	Scalar	Graph	History
Gemma	global	Source-frozen	0.8639	0.8379	0.8494	0.4614	0.4614	0.4796
Gemma	global	Topic-specific	0.4029	0.4226	0.3946	0.1372	0.1456	0.1533
Gemma	local	Source-frozen	0.5351	0.5990	0.5790	0.5467	0.5711	0.5860
Gemma	local	Topic-specific	0.4595	0.4496	0.4289	0.2234	0.2290	0.2338
Qwen	global	Source-frozen	0.1622	0.1973	0.2092	0.3363	0.4067	0.4983

Appendix

Table 15: Absolute new-question errors on the same overlay. Lower is better; “History” means the two-lag policy. Source-frozen policies receive no new-topic calibration. Topic-specific policies receive designated training and validation episodes, never held-out test observations. Full one-lag and persistence outcomes are retained in the machine-readable results.

E_{m}

abbreviates

E_{m}^{\mathrm{new}}

in Eq. ( 12 ).

		Original ( $q=19$ )		Replay ( $q=20$ )
Gemma input	Calibration	$\Delta\ell$	$\Delta E_{m}$	$\Delta\ell$	$\Delta E_{m}$
global	Source-frozen	0.0271	0.0001	0.0260	0.0000
global	Topic-specific	-0.0198	-0.0078	-0.0197	-0.0084
local	Source-frozen	-0.0662	-0.0212	-0.0640	-0.0244
local	Topic-specific	0.0093	-0.0097	0.0099	-0.0056

Appendix

Table 16: Gemma replay sensitivity: graph-versus-scalar point reductions before and after restoring u15. Question composition changes from 19 to 20; these are not paired estimates of a replay effect. Qwen is unchanged. Original and amended per-question values and uncertainty intervals are archived separately.

E_{m}

abbreviates

E_{m}^{\mathrm{new}}

in Eq. ( 12 ).

Model	$q$	$\Delta\ell$	$\Delta E_{m}$
GLM 5.2	19	$0.0128\;[-0.0282,0.0431]$	$0.0468\;[-0.0024,0.1240]$
Qwen 3.8 Max	20	$0.0959\;[0.0313,0.1524]$	$-0.0035\;[-0.0466,0.0349]$

Appendix

Table 17: Original complete-panel API analysis, before the GLM validation replay. The corresponding amended estimates appear in Table 5 . Both use the same prespecified primary comparison and interval convention. The replay restores a missing validation panel; it does not create another held-out realization.

E_{m}

abbreviates

E_{m}^{\mathrm{new}}

in Eq. ( 12 ).

Model	Calibration	Comparison	$\Delta\ell$	$\Delta E_{m}$
GLM 5.2	Source-frozen	Graph vs scalar	$0.0326\;[0.0177,0.0490]$	$0.0101\;[-0.0411,0.0624]$
GLM 5.2	Source-frozen	One lag vs graph	$-0.0536\;[-0.0867,-0.0249]$	$0.0097\;[-0.0189,0.0387]$
GLM 5.2	Source-frozen	Two lags vs graph	$-0.0572\;[-0.0883,-0.0277]$	$0.0088\;[-0.0206,0.0376]$
GLM 5.2	Topic-specific	One lag vs graph	$0.0111\;[-0.0030,0.0305]$	$-0.0071\;[-0.0158,-0.0003]$
GLM 5.2	Topic-specific	Two lags vs graph	$0.0293\;[0.0021,0.0674]$	$-0.0083\;[-0.0195,0.0012]$
Qwen 3.8 Max	Source-frozen	Graph vs scalar	$0.1076\;[0.0696,0.1454]$	$0.0759\;[0.0034,0.1458]$

Appendix

Table 18: All prespecified secondary six-round API policy contrasts. Positive error reductions favor the named alternative. History means additional spin/field lags, not the agent’s full semantic history. Intervals are descriptive whole-question bootstrap intervals and are not multiplicity-adjusted.

E_{m}

abbreviates

E_{m}^{\mathrm{new}}

in Eq. ( 12 ).

Model	Calibration	Policy	$\ell$	$E_{m}$
GLM 5.2	Control	Persistence	$2.0723\;[1.8133,2.3170]$	$0.5844\;[0.4448,0.7271]$
GLM 5.2	Source-frozen	Graph	$0.2249\;[0.1728,0.2804]$	$0.5894\;[0.3993,0.7910]$
GLM 5.2	Source-frozen	One lag	$0.2784\;[0.2051,0.3536]$	$0.5797\;[0.4067,0.7792]$
GLM 5.2	Source-frozen	Two lags	$0.2821\;[0.2084,0.3580]$	$0.5806\;[0.4068,0.7862]$
GLM 5.2	Source-frozen	Scalar	$0.2575\;[0.2022,0.3129]$	$0.5995\;[0.4288,0.7791]$
GLM 5.2	Topic-specific	Graph	$0.2441\;[0.1501,0.3488]$	$0.1638\;[0.1006,0.2366]$

Appendix

Table 19: Absolute API-model errors at

T=6

, local input, held-out

N=16

, after the separately audited validation replay. Smaller is better.

\ell

is observed-transition log loss and

E_{m}

is autonomous mean-magnetization trajectory MAE. All policy families and persistence are retained, with descriptive 95% question-bootstrap intervals.

E_{m}

abbreviates

E_{m}^{\mathrm{new}}

in Eq. ( 12 ).

Backend	Forecast variant	Graph $E_{2,\mathrm{f}}$	H2 $E_{2,\mathrm{f}}$	$G_{E_{2,\mathrm{f}}}$	$G_{C_{\mathrm{f}}}$
Gemma	Autonomous	0.6668	0.7328	-0.0660	-0.0438
Gemma	Observed prefix	0.4989	0.3891	+0.1098	+0.0497
Gemma	Prefix + offset	0.0276	0.0347	-0.0071	-0.0114
Gemma	Autonomous + clock cap	0.6711	0.7397	-0.0687	-0.0465
Qwen	Autonomous	0.3740	0.3306	+0.0434	+0.0369
Qwen	Observed prefix	0.2429	0.2045	+0.0384	+0.0411

Appendix

Table 20: Exploratory controls on the common 19-topic local panel, original short-source graph and history-2 (H2) policies, and updates 4–6. Values are point estimates;

G

is graph minus H2 error. “Observed prefix” conditions on states 0–3; “prefix + offset” additionally adapts one scalar intercept to updates 1–3; “clock cap” changes only the raw time feature beyond source support. Conditional and autonomous rows do not have the same information budget. Full per-question scores and separate-size sensitivities are retained.

Responder	Native agreement (%)	Stance-only agreement (%)	Difference [adjusted interval], p.p.
Gemma	57.895	68.421	$+10.5263\;[+3.9474,+17.1053]$
Qwen	54.605	54.605	$+0.0000\;[-3.9474,+3.9474]$

Appendix

Table 21: New text-intervention responses on identical contexts. The difference is stance-only minus native agreement, in percentage points (p.p.). Brackets are the six-contrast post-hoc sensitivity intervals, not new-domain validation.

Backend	Initialization	Forecast	$E_{2,\mathrm{f}}$	$C_{\mathrm{f}}$
Gemma	Autonomous	Graph	0.02822	0.08093
Gemma	Autonomous	History-2	0.02940	0.07310
Gemma	Autonomous	Capped graph	0.02808	0.07984
Gemma	Autonomous	Capped history-2	0.02993	0.07358
Gemma	Autonomous	Persistence	0.84942	0.90636
Gemma	Autonomous	Two-state	0.03108	0.09400

Appendix

Table 22: All 56 primary-window absolute scores: two backends, 14 forecasts and two scores, on all 24 questions. Values average updates 4–6, then replicas, sizes and questions with equal weights. Lower is better. These are descriptive point estimates, not an additional ranking selected for significance. Autonomous forecasts observe state 0 only; conditional forecasts observe states 0–3. Thus the two initialization groups have different information budgets. Cap and offset denote the fixed time-feature bound and prefix-only intercept adaptation, respectively; persistence and two-state are specified simple controls.

Explore similar work

Collective cooperation without individual fidelity in LLM agents
Jun 29, 2026Henrique Ferraz de Arruda, Carlos Gracia Lázaro, Alberto Aleta +1Multi-Agent LLM Systems Human Behavior Simulation
ARAID Foundation, Zaragoza, Spain. · Institute for Biocomputation and Physics of Complex Systems, University of Zaragoza, Zaragoza, Spain. · Universidad San Jorge (USJ), Zaragoza, Spain. +1
Collective Opinion Dynamics in Structured LLM Populations
Apr 13, 2026Erica Cau, Andrea Failla, Giulio RossettiMulti-Agent LLM Systems Opinion Dynamics
Department of Computer Science, University of Pisa · ISTI-CNR
Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts
Sep 27, 2026Cheng Peng, Ruixi Luo, Zhi Chen +1Language Model Ensembles Probabilistic Forecasting
Chinese University of Hong Kong

Jun 29, 2026, physics.soc-phJ/K move · Enter open · S save

Collective cooperation without individual fidelity in LLM agents

Henrique Ferraz de Arruda, Carlos Gracia Lázaro, Alberto Aleta, Yamir Moreno

ARAID Foundation, Zaragoza, Spain. · Institute for Biocomputation and Physics of Complex Systems, University of Zaragoza, Zaragoza, Spain. · Universidad San Jorge (USJ), Zaragoza, Spain.+1

Multi-Agent LLM Systems Human Behavior Simulation LLM Agent Evaluation Large Language Model-Based Social Simulation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as agents in simulations of social systems, yet it remains unclear when their behavior can be interpreted as a faithful proxy for human decision-making. Here we test LLM agents against a direct empirical benchmark: a large-scale networked Prisoner's Dilemma experiment with human participants. Using the same interaction protocol, payoff structure, and network topologies, we compare nine open-weight LLMs with the human data. The selected model reproduces several macro-level features of cooperation dynamics, including the early decline and later stabilization of cooperation. This aggregate agreement, however, does not extend uniformly to finer levels of behavior. LLM populations underestimate individual-level heterogeneity and generate conditional cooperation patterns that differ from those observed in humans. Adding a fraction of random agents improves some aspects of micro-level agreement, but does not remove the mismatch in decision rules. These findings reveal a macro--micro dissociation in LLM-based social agents: collective outcomes can appear human-like even when the underlying behavioral distributions and mechanisms are not. They suggest that validating LLM agents as human surrogates requires comparisons across aggregate dynamics, individual heterogeneity, and context-dependent decision rules, rather than outcome-level agreement alone.

Abstract

Explore similar work

Collective cooperation without individual fidelity in LLM agents

Collective Opinion Dynamics in Structured LLM Populations

Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts