SOCIA-: Textual Gradient Meets Multi-Agent Orchestration for Automated Simulator Generation
Organizations: School of Computer Science Engineering, University of New South Wales, Australia
Abstract
In this paper, we present SOCIA-, a diagnosis-driven multi-agent framework for automated simulator-code construction. SOCIA- iteratively refines executable simulator code within a textual computation graph, where LLM-driven agents synthesize, execute, evaluate, diagnose, and repair simulators. A grounded simulator blueprint attributes behavioral residuals to candidate mechanisms and their spatial or interaction contexts, which are then converted into localized textual repair signals with history-aware memory and constraint projection. We evaluate SOCIA- on the real-world LLMob personal-mobility benchmark and a controlled semi-synthetic geo-social mask-adoption benchmark. Across in-distribution simulation, out-of-distribution what-if generation, and geo-social diffusion, SOCIA- achieves the strongest overall simulator fidelity. SOCIA-'s code and data are publicly available at https://github.com/cruiseresearchgroup/SOCIA-Nabla.
Figures & tables
| Methods | Mob. N N (ID) | Mob. A A (ID) | Mob. N A (OOD) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| JSD | WD | POI | JSD | WD | POI | JSD | WD | POI | |
| AI Scientist-v2 | 0.887 0.006 | 3.224 0.006 | 0.003 0.001 | 0.451 0.006 | 3.373 0.006 | 0.190 0.003 | 0.773 0.007 | 2.045 0.006 | 0.070 0.007 |
| YuLan-OneSim | 0.793 0.004 | 2.191 0.004 | 0.200 0.008 | 0.366 0.005 | 2.485 0.005 | 0.310 0.004 | 0.639 0.006 | 1.541 0.006 | 0.091 0.008 |
| Reflexion | 0.423 0.005 | 0.783 0.006 | 0.410 0.006 | 0.187 0.006 | 0.713 0.006 | 0.550 0.006 | 0.553 0.007 | 0.612 0.007 | 0.220 0.003 |
| G-Sim-ES | 0.546 0.002 | 0.225 0.003 | 0.802 0.002 | 0.272 0.004 | 0.467 0.004 | 0.601 0.002 | 0.534 0.005 | 0.456 0.005 | 0.105 0.002 |
| G-Sim-SBI | 0.635 0.003 | 1.638 0.003 | 0.097 0.001 | 0.012 0.003 | 1.690 0.006 | 0.990 0.002 | 0.045 0.005 | 0.633 0.003 | 0.214 0.004 |
| Method | RMSE | Brier | TD |
|---|---|---|---|
| AI Scientist-v2 | |||
| YuLan-OneSim | |||
| Reflexion | |||
| G-Sim-ES | 0.298 0.003 | ||
| G-Sim-SBI | |||
| SOCIA - | 0.075 0.007 | 0.255 0.002 | 0.161 0.007 |
| Models | Mask. | Mob. N A | |
|---|---|---|---|
| RMSE | JSD | WD | |
| SOCIA - | - | - | - |
| w/o mom. | +0.123 0.003 | +0.056 0.002 | +0.093 0.006 |
| w/o proj. | +0.173 0.003 | +0.098 0.005 | +0.131 0.006 |
| w/o data-analysis | +0.307 0.005 | +0.203 0.008 | +0.247 0.003 |
| w/o eval. | +0.362 0.004 | +0.207 0.006 | +0.235 0.004 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.