Separating Diagnosis from Control: Auditable Policy Adaptation in Agent-Based Simulations with LLM-Based Diagnostics

Shaoxin Zhong; Yuchen Su; Michael Witbrock

cs.AIMar 24, 2026

Separating Diagnosis from Control: Auditable Policy Adaptation in Agent-Based Simulations with LLM-Based Diagnostics

Authors: Shaoxin Zhong, Yuchen Su, Michael Witbrock

Organizations: University of Auckland, Auckland, New Zealand

Abstract

Mitigating elderly loneliness requires policy interventions that achieve both adaptability and auditability. Existing methods struggle to reconcile these objectives: traditional agent-based models suffer from static rigidity, while direct large language model (LLM) controllers lack essential traceability. This work proposes a three-layer framework that separates diagnosis from control to achieve both properties simultaneously. LLMs operate strictly as diagnostic instruments that assess population state and generate structured risk evaluations, while deterministic formulas with explicit bounds translate these assessments into traceable parameter updates. This separation ensures that every policy decision can be attributed to inspectable rules while maintaining adaptive response to emergent needs. We validate the framework through systematic ablation across five experimental conditions in elderly care simulation. Results demonstrate that explicit control rules outperform end-to-end black-box LLM approaches by 11.7% while preserving full auditability, confirming that transparency need not compromise adaptive performance.

Figures & tables

Figure 1: Three-layer architecture for auditable policy adaptation. The simulation layer models agent dynamics, the diagnosis layer uses LLM-based diagnostics for risk assessment, and the control layer applies deterministic formulas. Solid arrows indicate data flow; the dashed arrow indicates the policy feedback loop.

Condition	LLM Diagnosis	Adaptation	Control Type
Baseline	–	–	None
Fixed Policy	–	–	Static $\theta$
LLM Mapping	✓	–	Fixed response
Closed-loop (Ours)	✓	✓	Deterministic
Black-box LLM	✓	✓	LLM decides $\theta$

Table 1: Experimental conditions.

Method	Mean	SD
Baseline	0.717	0.018
Fixed Policy	0.674	0.012
LLM Mapping	0.680	0.007
Closed-loop (Ours)	0.607	0.020
Black-box LLM	0.687	0.025

Table 2: Holdout performance across four seeds.

${}^{*}p<.001$ , ${}^{}p<.01$
Comparison	Mean Diff.( $\Delta$ )	Improve(%)	Cohen’s( $d$ )	$p$ -value
Closed vs. LLM Mapping	$-0.073$	$10.7\%$	$-4.87$	$<.001^{***}$
Closed vs. Black-box	$-0.081$	$11.7\%$	$-3.54$	$.002^{**}$
Closed vs. Fixed	$-0.067$	$10.0\%$	$-4.05$	$.001^{**}$
Black-box vs. Fixed	$+0.013$	–	$+0.66$	$.385$

Table 3: Pairwise comparisons. Large effect sizes reflect deterministic simulation dynamics.

Figure 2: Loneliness trajectories over 200 days ( $N=30$ ). Shaded regions: $\pm 1$ SD across 4 seeds.

Figure 3: Parameter trajectories. Shaded regions: $\pm 1$ SD across 4 seeds. (Left) Visit threshold $\theta_{t}$ . (Right) Visit probability $\theta_{p}$ .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value	Mean	SD	Min	Max	$\Delta$ %
Baseline	–	0.636	0.016	0.623	0.648	–
Risk threshold ( $r$ )	0.30	0.636	0.016	0.623	0.648	+0.0%
	0.50	0.636	0.016	0.623	0.648	+0.0%
Priority ( $p$ )	0.65	0.636	0.016	0.623	0.648	+0.0%
	0.85	0.680	0.007	0.677	0.682	+6.4%
Update cap ( $\Delta\theta$ )	0.03	0.636	0.016	0.623	0.648	+0.0%

Appendix

Table 4: Sensitivity on holdout seeds 300, 400 (Baseline:

r=0.4

p=0.75

, cap

=0.05

Agent	Risk (L)	Risk (F)	Primary Driver	Priority
Agent 8	High	Low	Social isolation	Network: 0.80
Agent 13	High	Medium	Health & isolation	Network: 0.80
Agent 0	Medium	Low	Minor mood dip	Visit: 0.70
7/30 classified as High risk ( $r=0.23$ )

Appendix

Table 5: Example diagnosis outputs (Day 120, Seed 300; 30 agents diagnosed).

Agent-Based Modeling

Agent-based modeling simulates how interacting, heterogeneous entities and their environments produce patterns in social, economic, biological, and engineered systems.

See all 160 papers

LLM-powered reasoning in agent-based modeling
Jul 7, 2026Sifat Afroj Moon, Dakotah Maguire, Adam Spannaus +5Agent-Based Modeling Large Language Model-Based Social Simulation
Oak Ridge National Laboratory · Computational Science and Engineering Division, Oak Ridge National Laboratory, Oak Ridge, TN 37830, USA · Geospatial Science and Human Security Division, Oak Ridge National Laboratory, Oak Ridge, TN 37830, USA +1
A Safety-Aware Role-Orchestrated Multi-Agent LLM Framework for Behavioral Health Communication Simulation
Mar 31, 2026Ha Na ChoMulti-Agent LLM Systems Large Language Model-Based Role-Play Simulation
Department of Informatics, University of California, Irvine, USA
PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents
Oct 15, 2026Kshitij Mishra, Abhijith Sharma, Nils Lukas +1AI Agent Auditing LLM Agent Evaluation
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

Jul 7, 2026J/K move · Enter open · S save

LLM-powered reasoning in agent-based modeling

Sifat Afroj Moon, Dakotah Maguire, Adam Spannaus, Joe Tuccillo+4

Oak Ridge National Laboratory · Computational Science and Engineering Division, Oak Ridge National Laboratory, Oak Ridge, TN 37830, USA · Geospatial Science and Human Security Division, Oak Ridge National Laboratory, Oak Ridge, TN 37830, USA+1

Agent-Based Modeling Large Language Model-Based Social Simulation

Open paper AskPDF arXiv

Agent-based modeling (ABM) has the capability to model millions of individuals and their interactions, which is useful for policy making. However, ABMs have traditionally relied on static prior, which prevents the models from adapting to real-time changes. Our research provides a novel approach to addressing this information gap. Large language models (LLMs) offer new opportunities to predict human decision-making. Here, we introduce a scalable Hybrid Agent-based and Language-driven Epidemic (HALE) modeling framework that leverages LLMs to predict human decision-making in an ABM simulation. As a proof-of-concept, we use HALE to simulate COVID-19 and its effects in Salt Lake County, UT.

Abstract

Agent-Based Modeling

Explore similar work

LLM-powered reasoning in agent-based modeling

A Safety-Aware Role-Orchestrated Multi-Agent LLM Framework for Behavioral Health Communication Simulation

PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents