In this paper, we present SOCIA-∇, a diagnosis-driven multi-agent framework for automated simulator-code construction. SOCIA-∇ iteratively refines executable simulator code within a textual computation graph, where LLM-driven agents synthesize, execute, evaluate, diagnose, and repair simulators. A grounded simulator blueprint attributes behavioral residuals to candidate mechanisms and their spatial or interaction contexts, which are then converted into localized textual repair signals with history-aware memory and constraint projection. We evaluate SOCIA-∇ on the real-world LLMob personal-mobility benchmark and a controlled semi-synthetic geo-social mask-adoption benchmark. Across in-distribution simulation, out-of-distribution what-if generation, and geo-social diffusion, SOCIA-∇ achieves the strongest overall simulator fidelity. SOCIA-∇'s code and data are publicly available at https://github.com/cruiseresearchgroup/SOCIA-Nabla.
Table 1. Evaluation results ( ± denotes 95% CIs) on the LLMob personal mobility simulation benchmark. Mob. N → N and Mob. A → A are in-distribution settings, while Mob. N → A is an out-of-distribution what-if setting. Lower is better for JSD/WD and higher is better for POI. Best and second-best results are shown in bold and underlined , respectively.
Method
RMSE ↓
Brier ↓
TD ↓
AI Scientist-v2
0.551±0.010
0.424±0.002
0.523±0.004
YuLan-OneSim
0.413±0.009
0.391±0.006
0.410±0.004
Reflexion
0.263±0.005
0.334±0.003
0.321±0.005
G-Sim-ES
0.089±0.002
0.298 ± 0.003
0.279±0.002
G-Sim-SBI
0.199±0.003
0.285±0.003
0.194±0.002
SOCIA - ∇
0.075 ± 0.007
0.255 ± 0.002
0.161 ± 0.007
Table 2. Evaluation results ( ± denotes 95% CIs) on the Mask Adoption task. RMSE measures global adoption-curve fidelity, Brier evaluates individual-level probabilistic calibration of mask-wearing decisions, and TD (Transition Divergence) measures day-to-day adoption-state transition fidelity under information exposure and social influence. Lower values indicate better performance.
Models ↓
Mask.
Mob. N → A
Δ RMSE ↓
Δ JSD ↓
Δ WD ↓
SOCIA - ∇
-
-
-
w/o mom.
+0.123 ± 0.003
+0.056 ± 0.002
+0.093 ± 0.006
w/o proj.
+0.173 ± 0.003
+0.098 ± 0.005
+0.131 ± 0.006
w/o data-analysis
+0.307 ± 0.005
+0.203 ± 0.008
+0.247 ± 0.003
w/o eval.
+0.362 ± 0.004
+0.207 ± 0.006
+0.235 ± 0.004
Table 3. Ablation and backbone-sensitivity results on Mask Adoption and Personal Mobility N → A . Deltas are relative to the full GPT-5.2-based SOCIA - ∇ configuration; positive values indicate degradation and negative values indicate improvement.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 2. Evidence-based refinement trajectories over outer-loop iterations. We compare SOCIA - ∇ and G-Sim-SBI under the same calibration setting to examine how each method improves simulator fidelity during iterative refinement. The top row shows SOCIA - ∇ , and the bottom row shows G-Sim-SBI. Across columns, we track three macro-level metrics: Mask Adoption RMSE, and OOD Mobility (N → A) measured by JSD and Wasserstein distance (WD). Lower values indicate better performance.
Automated simulator construction requires distributional fidelity, distinguishing it from generic code generation. We identify two failure modes in long-horizon LLM agents: contextual drift and optimization instability arising from conflating structural and parametric errors. We propose SOCIA-EVO, a dual-anchored evolutionary framework. SOCIA-EVO introduces: (1) a static blueprint to enforce empirical constraints; (2) a bi-level optimization to decouple structural refinement from parameter calibration; and (3) a self-curating Strategy Playbook that manages remedial hypotheses via Bayesian-weighted retrieval. By falsifying ineffective strategies through execution feedback, SOCIA-EVO achieves robust convergence, generating simulators that are statistically consistent with observational data. The code and data of SOCIA-EVO are available here: https://github.com/cruiseresearchgroup/SOCIA/tree/evo.
Yuncheng Hua, Sion Weatherhead, Mehdi Jafari +2
School of Computer Science and Engineering, University of New South Wales, Australia
LLMs are increasingly deployed to simulate social interactions, yet many of the existing simulators remain ad hoc and monolithic. This lack of architectural standardization prevents reproducible research and complicates downstream evaluation. We advance a rigorous science of LLM-based multi-agent simulation by modularizing core components into Environments, Agents, Simulation engines, and Evaluation metrics (EASE). We demonstrate the utility of EASE configuration by wrapping it in an experimental study schema for orchestrating workflows centered around answering explicit research questions in generated scenarios. We contribute SiliSocS, an open-source, research-ready Silicon Society Sandbox implementing a study-structured EASE configuration to enable highly configurable and reproducible LLM-based social simulations. Using SiliSocS and EASE, we present three case studies, showcasing the system's comprehensive assessment of existing questions, ability to dive deeper into complex questions, and elaboration of existing studies, respectively. Together, these case studies highlight the limitations of current modeling approaches and isolate the impacts of design choices on key results.
Sneheel Sarangi, Maximilian Puelma Touzel, Aurélien Bück-Kaeffer +3
McGill University · Mila - Quebec Artificial Intelligence Institute · Université de Montréal +1
LLM-agent simulations make natural-language social scenarios easy to instantiate, but their outputs can be overread as predictions and are often difficult to compare with explicit social dynamics. We present AgoraSim, a hybrid agent-based modeling framework for scenario-oriented social reaction analysis. AgoraSim resolves textual or multimodal artifacts into editable ABM configurations, runs ratio-controlled populations that mix LLM, vision-language, custom-endpoint, random, and classical agents, and compares the same scenario against matched classical reference dynamics. All agents emit a shared structured decision object, enabling common action spaces, interaction protocols, metrics, and audit records. Exposed through a local UI, Python SDK/CLI, and REST API, AgoraSim helps users inspect scenario trajectories, compare modeling assumptions, and identify cases that warrant empirical validation.
Chung-Chi Chen
Human-Agent Ally Lab (HAA Lab) National Institute of Informatics, Japan