ODD+LLM: Specifying LLM-Agent Behavior for Reproducible Social Simulation
Organizations: Amazon Web Services
Abstract
The ODD protocol is the de facto standard for describing agent-based models, but it lacks vocabulary for the components that determine LLM-agent behavior: prompt assembly, memory serialization, parsing, and retry logic. We propose ODD+LLM, a backward-compatible extension adding five documentation blocks grounded in the CoALA cognitive-architecture framework. To show that the gap matters, we conduct a multiverse analysis [Steegen et al., 2016] of a published LLM naming-game simulation, varying six specification choices a reimplementer must make. Variants that are defensible readings of the paper stay within statistical noise of our baseline reconstruction. Missing one instruction that the paper states but is easy to overlook (that the model answers as an outside observer rather than as a player) raises the share of runs settling on the convention agents individually favor from 0.35 to 1.00 and makes convergence three times faster, in a reimplementation that uses a different model from the original. We complement the demonstration with a reconstruction-sufficiency review of 28 published LLM-agent simulations, finding that the details of how prompts are built and how model outputs are parsed are undetermined in 77% of cases, and that no paper fully specifies its main LLM call. ODD+LLM gives the field a standard home for information it already produces but currently scatters across code repositories, supplementary files, and undocumented defaults.
Figures & tables
| ID | Axis | Main-text ambiguity | Levels | SI |
|---|---|---|---|---|
| M1 | Memory format | “dynamic memory prompt” | table / narr. / JSON | SI |
| M2 | Memory order | Order of entries unstated | recent / oldest-first | SI |
| M3 | Score semantics | “accumulated score over ” | window / cumulative | SI |
| M4 | Person framing | “external observer” | 3rd / 1st-person | Spec. |
| M5 | Pairing protocol | “two agents randomly selected” | random / perfect | SI |
| M6 | Parse failure | “consistent format” | resample+uniform / uniform-only | Open |
| Arm | R4 Asymmetry | |
|---|---|---|
| Baseline (table/recent/3rd) | 0.680 | |
| M1: Narrative | 0.653 | |
| M1: JSON | 0.730 | |
| M2: Oldest-first | 0.690 | |
| M3: Cumulative score | 0.707 | |
| M4: First-person | 0.587 |
| Arm | 95% Wilson CI | Conv. round | |
|---|---|---|---|
| Baseline | 0.350 | [0.18, 0.57] | 16.1 |
| M1: Narrative | 0.450 | [0.26, 0.66] | 15.6 |
| M1: JSON | 0.300 | [0.15, 0.52] | 17.6 |
| M2: Oldest-first | 0.450 | [0.26, 0.66] | 19.9 |
| M3: Cumulative | 0.350 | [0.18, 0.57] | 14.9 |
| M4: First-person | 1.000 | [0.84, 1.00] | 5.3 |
| Block | Scope | Mean | Items at 0 | |
| A (Architecture) | system | 1.63 | n/a | 209 |
| B (Identity) | system | 1.57 | n/a | 131 |
| C spine | system | 0.90 | n/a | 48 |
| D spine | system | 1.73 | n/a | 56 |
| E (Validation) | system | 1.46 | E3a (37%) | 269 |
| C per-role | repeated | 0.48 | CR2 (100%) | 186 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Paper | Str. | Year | Sys | D-site | DPs | Roles | Claim |
|---|---|---|---|---|---|---|---|
| Akata et al. 2025 | S1 | 2025 | 1.86 | 1.73 | 10 | 1 | mechanism |
| Flint Ashery et al. 2025a | S1 | 2025 | 1.64 | 0.64 | 21 | 1 | mechanism |
| Fontana et al. 2025 | S1 | 2025 | 1.89 | 1.17 | 13 | 1 | n/a |
| Li and Shirado 2025 | S1 | 2025 | 1.77 | 1.29 | 14 | 1 | mechanism |
| Piatti et al. 2024 | S1 | 2024 | 1.78 | 0.93 | 20 | 2 | mechanism |
| Chuang et al. 2024 | S2 | 2024 | 1.69 | 1.23 | 17 | 1 | n/a |
| Section | Item | Mean | ||||
| Signature | DS_1 (input bindings) | 1.81 | 0 | 5 | 22 | 27 |
| DS_2 (output type) | 1.60 | 0 | 8 | 12 | 20 | |
| Binding | DB_1 (static text) | 1.45 | 3 | 6 | 13 | 22 |
| DB_2 (serialization) | 1.27 | 5 | 6 | 11 | 22 | |
| DB_3 (order/placement) | 0.86 | 5 | 15 | 2 | 22 | |
| DB_4 (history policy) | 1.36 | 2 | 14 | 12 | 28 |
Large Language Model-Based Social Simulation
Large language model-based social simulation studies how language-model agents represent people and interact to produce social behavior.
See all 292 papers