The ODD protocol is the de facto standard for describing agent-based models, but it lacks vocabulary for the components that determine LLM-agent behavior: prompt assembly, memory serialization, parsing, and retry logic. We propose ODD+LLM, a backward-compatible extension adding five documentation blocks grounded in the CoALA cognitive-architecture framework. To show that the gap matters, we conduct a multiverse analysis [Steegen et al., 2016] of a published LLM naming-game simulation, varying six specification choices a reimplementer must make. Variants that are defensible readings of the paper stay within statistical noise of our baseline reconstruction. Missing one instruction that the paper states but is easy to overlook (that the model answers as an outside observer rather than as a player) raises the share of runs settling on the convention agents individually favor from 0.35 to 1.00 and makes convergence three times faster, in a reimplementation that uses a different model from the original. We complement the demonstration with a reconstruction-sufficiency review of 28 published LLM-agent simulations, finding that the details of how prompts are built and how model outputs are parsed are undetermined in 77% of cases, and that no paper fully specifies its main LLM call. ODD+LLM gives the field a standard home for information it already produces but currently scatters across code repositories, supplementary files, and undocumented defaults.
Figures & tables
ID
Axis
Main-text ambiguity
Levels
SI
M1
Memory format
“dynamic memory prompt”
table / narr. / JSON
SI
M2
Memory order
Order of H entries unstated
recent / oldest-first
SI
M3
Score semantics
“accumulated score over H ”
window / cumulative
SI
M4
Person framing
“external observer”
3rd / 1st-person
Spec.
M5
Pairing protocol
“two agents randomly selected”
random / perfect
SI
M6
Parse failure
“consistent format”
resample+uniform / uniform-only
Open
Table 1: Branch points (multiverse axes). SI = determinable from the supplementary materials; Spec. = specified in the main text but non-obvious; Open = genuinely underspecified. Four of six choices are resolvable from supplementary materials, one is specified in the main text but non-obvious, and a reader working from the methods section alone would likely resolve them differently.
Arm
P(M∣empty)
R4 Asymmetry
Baseline (table/recent/3rd)
0.680
+0.200
M1: Narrative
0.653
+0.063
M1: JSON
0.730
−0.040
M2: Oldest-first
0.690
+0.021
M3: Cumulative score
0.707
+0.217
M4: First-person
0.587
−0.243
Table 2: Stage 1 probe results. R4 asymmetry measures the difference P(M∣{1:M,Q;2:Q,M})−P(Q∣{1:Q,M;2:M,Q}) (matching Flint Ashery et al.’s interaction indexing): positive means M is the “strong” convention. M4 (person framing) reverses the sign; M1 (format) and M2 (order) attenuate or reverse it. M5 and M6 cannot logically affect a direct probe (they are population-level mechanisms); their variation is sampling noise.
Arm
P(M)
95% Wilson CI
Conv. round
Baseline
0.350
[0.18, 0.57]
16.1
M1: Narrative
0.450
[0.26, 0.66]
15.6
M1: JSON
0.300
[0.15, 0.52]
17.6
M2: Oldest-first
0.450
[0.26, 0.66]
19.9
M3: Cumulative
0.350
[0.18, 0.57]
14.9
M4: First-person
1.000
[0.84, 1.00]
5.3
Table 3: Stage 2 results. Conv. round = first population round at which mean success rate crosses the 0.95 threshold; arms may subsequently dip below it (see Figure 2). First-person framing (M4) produces convergence to M in all 20 seeds, three times faster than any third-person arm.
Figure 1: Specification curve ( Simonsohn et al., 2020 ) : P(M) across multiverse arms (sorted), with Wilson CIs. The dot matrix below shows which branch level each arm uses. The first-person arm (M4, rightmost) is the only arm whose CI does not overlap the others; a single specification choice qualitatively determines the outcome.
Figure 2: Mean success rate by population round. First-person framing converges by round 5; most third-person arms converge by rounds 15–20, though one arm (M2: oldest-first) shows a slower trajectory reaching full coordination only near round 35. All arms produce naming-game-like convergence (they are all plausible implementations), but they land on different conventions.
Block
Scope
Mean
Items at 0
n
A (Architecture)
system
1.63
n/a
209
B (Identity)
system
1.57
n/a
131
C spine
system
0.90
n/a
48
D spine
system
1.73
n/a
56
E (Validation)
system
1.46
E3a (37%)
269
C per-role
repeated
0.48
CR2 (100%)
186
Table 4: Mean sufficiency by block (0–2 scale). System-level items that existing documentation conventions cover (A, D-spine: mean 1.68) score roughly double the repeated-record items (C per-role, D call-site: mean 0.69) that have no standard home. n<28×items because NA items (design makes the item inapplicable) and UNVERIFIED items (supplementary not retrieved) are excluded from scoring.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Paper
Str.
Year
Sys
D-site
DPs
Roles
Claim
Akata et al. 2025
S1
2025
1.86
1.73
10
1
mechanism
Flint Ashery et al. 2025a
S1
2025
1.64
0.64
21
1
mechanism
Fontana et al. 2025
S1
2025
1.89
1.17
13
1
n/a
Li and Shirado 2025
S1
2025
1.77
1.29
14
1
mechanism
Piatti et al. 2024
S1
2024
1.78
0.93
20
2
mechanism
Chuang et al. 2024
S2
2024
1.69
1.23
17
1
n/a
Appendix
Table 5: Per-paper sufficiency scores (0–2 scale). Strata: S1 minimal-choice, S2 opinion dynamics, S3 platform-scale, S4 economic/market, S5 generative architectures, S6 policy testbeds, S7 norms/emergence. Claim: mechanism = mechanistic causal claim; policy = policy-relevant conclusion; n/a = exploratory or unclassified. Wang et al. 2025a (COLING) and 2025c (ACM TOIS) are corpus papers; Wang et al. 2025b is YuLan-OneSim, cited only as related work.
Section
Item
Mean
n(0)
n(1)
n(2)
n
Signature
DS_1 (input bindings)
1.81
0
5
22
27
DS_2 (output type)
1.60
0
8
12
20
Binding
DB_1 (static text)
1.45
3
6
13
22
DB_2 (serialization)
1.27
5
6
11
22
DB_3 (order/placement)
0.86
5
15
2
22
DB_4 (history policy)
1.36
2
14
12
28
Appendix
Table 6: Block D per-call-site item scores. Items are grouped by section within the call-site record. The resolution layer is nearly absent from the literature: DR 3 –DR 5 have never scored 2 in any paper.
LLMs are increasingly deployed to simulate social interactions, yet many of the existing simulators remain ad hoc and monolithic. This lack of architectural standardization prevents reproducible research and complicates downstream evaluation. We advance a rigorous science of LLM-based multi-agent simulation by modularizing core components into Environments, Agents, Simulation engines, and Evaluation metrics (EASE). We demonstrate the utility of EASE configuration by wrapping it in an experimental study schema for orchestrating workflows centered around answering explicit research questions in generated scenarios. We contribute SiliSocS, an open-source, research-ready Silicon Society Sandbox implementing a study-structured EASE configuration to enable highly configurable and reproducible LLM-based social simulations. Using SiliSocS and EASE, we present three case studies, showcasing the system's comprehensive assessment of existing questions, ability to dive deeper into complex questions, and elaboration of existing studies, respectively. Together, these case studies highlight the limitations of current modeling approaches and isolate the impacts of design choices on key results.
Sneheel Sarangi, Maximilian Puelma Touzel, Aurélien Bück-Kaeffer +3
McGill University · Mila - Quebec Artificial Intelligence Institute · Université de Montréal +1
LLM-agent simulations make natural-language social scenarios easy to instantiate, but their outputs can be overread as predictions and are often difficult to compare with explicit social dynamics. We present AgoraSim, a hybrid agent-based modeling framework for scenario-oriented social reaction analysis. AgoraSim resolves textual or multimodal artifacts into editable ABM configurations, runs ratio-controlled populations that mix LLM, vision-language, custom-endpoint, random, and classical agents, and compares the same scenario against matched classical reference dynamics. All agents emit a shared structured decision object, enabling common action spaces, interaction protocols, metrics, and audit records. Exposed through a local UI, Python SDK/CLI, and REST API, AgoraSim helps users inspect scenario trajectories, compare modeling assumptions, and identify cases that warrant empirical validation.
Chung-Chi Chen
Human-Agent Ally Lab (HAA Lab) National Institute of Informatics, Japan
Recent work has used LLM agents to reproduce empirical social science results with access to both the data and code. We broaden this scope by asking: Can they reproduce results given only a paper's methods description and original data? We develop an agentic reproduction system that extracts structured methods descriptions from papers, runs reimplementations under strict information isolation -- agents never see the original code, results, or paper -- and enables deterministic, cell-level comparison of reproduced outputs to the original results. An error attribution step traces discrepancies through the system chain to identify root causes. Evaluating four agent scaffolds and four LLMs on 48 papers with human-verified reproducibility, we find that agents can largely recover published results, but performance varies substantially between models, scaffolds, and papers. Root cause analysis reveals that failures stem both from agent errors and from underspecification in the papers themselves.
Benjamin Kohler, David Zollikofer, Johanna Einsiedler +2