The design of manufacturing systems relies on discrete-event simulation (DES) models that experts still build in tool-specific interfaces. AI agents can now write simulation code, but every model they write from scratch is new, unverified software. We present SimuLean, an open-source 3D DES simulator on the Unity engine designed to be programmed by AI agents: its modelling semantics are verified once, in the platform, and the agent builds and studies one model through domain-level commands, in the 3D scene or headless; physically simulated durations are calibrated for fast headless studies. We evaluated it on 18 manufacturing modelling tasks, from single machines to published production lines. A frontier agent built valid models in 93 of 108 attempts (86%), in about two minutes each; on the benchmark's interface version, two frontier agents built 105 of 108 models through domain commands alone, so that each comes with a record of how it was built. Every residual failure was silent - a model that builds and runs but answers wrongly - and only comparison with reference models revealed them all. In an industrial case, one agent working alone rebuilt, verified and studied a robotised welding workshop from a published study, in SimuLean and in SimPy. Comparing them with the published, expert-built model exposed an error in it; against the corrected model, the agent-built models agree within 1%. One agent can already build complete, realistic and traceable models for the design of manufacturing systems; independent reference models are what make them trustworthy.
Figures & tables
Figure 1: A simulator designed for people versus one designed for AI agents. Left: the agent reaches a GUI-first tool only through workarounds, the model is split between the drawing and hidden settings, and nothing checks the results. Right: in SimuLean the agent works through domain commands and machine-readable contracts on a single model whose scene is a view, runs it headless or in 3D, and every model must build and run through a gate before it is studied. Below: the changes between the two. Diagram drawn with the assistance of Claude (Anthropic).
Figure 2: Positioning of LLM-based simulation modelling work by agent autonomy and automated verification (ordinal levels; positions within a level are approximate, from each work’s published description). To our knowledge, no prior work checks the models an agent builds with build-and-run and behavioural checks against criteria fixed before the runs. Diagram drawn with the assistance of Claude (Anthropic).
Work
Target
Agent role
Verification
Evaluation
Jackson et al. [37]
Python
One-shot
Manual review
2 small models
Monks et al. [11]
SimPy
Iterative, expert-guided
Unit tests; vs. published results
2 healthcare models
Chen et al. [12]
SimPy (fine-tuned 7B)
One-shot
Executability, format, consistency
600 queueing tasks
Monks et al. [42]
DES code
Pipeline with reflection
Schema; one-off statistical comparison
8 queueing diagrams
Elbasheer et al. [13]
PS (COTS)
Templates, local LLM
Schema only
1 case; 199 requests; 1 LLM
Schmitt et al. [15]
SimPy blueprint
Fixed LLM workflow; 2 human checkpoints
Inspector LLM; human review; one-off vs. COTS
2 cases, 21 runs; cost
Table 1: Closest related work on LLM-based construction of simulation models. Verification: how the generated model is checked; Evaluation: as reported by the authors. COTS: commercial tool; PS: Plant Simulation. Works in which the LLM acts inside a model built by people are discussed in the text.
Principle
In SimuLean
P1 Single source of truth
One configuration from which every run is built; the 3D scene is a view of it, not a second model
P2 Domain-level commands
The agent adds, connects, sets, runs and tests through the modelling vocabulary, not generic editor operations or arbitrary code
P3 Executable without the interface
Every model runs headless and in parallel, with the same results as the scene
P4 Deterministic reproducibility
One random stream per element and purpose; given a seed, results are identical in any mode, order or parallelism
P5 Machine-readable contracts
Conventions, the parameter catalog and every error are exposed in a form the agent reads before acting
P6 Automated verification gate
Every change to the simulator is checked against reference models; every model is built and run before it is accepted
Table 2: Design principles of an agent-programmable simulator and how SimuLean implements them.
Figure 3: SimuLean architecture. Left: front-ends on the Unity integration layer, on an engine assembly without any Unity reference. Right: the agent’s entry points — (a) domain commands through the Model Context Protocol, (b) machine-readable contracts, (c) the headless runner, (d) the verification gate. Diagram drawn with the assistance of Claude (Anthropic).
Figure 4: One build path: the scene is extracted into a configuration from which the engine builds the model; the scene components are bound to it as views, or the same configuration runs headless in N parallel replications. The optimiser and experimenter vary the configuration, never the scene. Diagram drawn with the assistance of Claude (Anthropic).
Commands
What they do
types , params
What can be placed; an element’s fields and catalog keys
add_element , connect , set
Place elements and tools, connect them (a track between transport nodes), set fields by name or path
apply † , repeat † , invoke †
Many edits in one all-or-nothing call; copy a group of elements N times with its internal connections; run an object’s actions (e.g. fill a travel matrix)
add_team , load_table
Create resource teams; load input tables
query † , fit †
Query or import into the project database (SQL, CSV, Excel); fit distributions to data and write the best into a field
model , validate
Read the whole model back as JSON; build it and check a study without running
Table 3: Domain-level commands exposed to the agent in the released version, by family (prefix simulean_ omitted). † : not in v1, the version of the benchmark.
SimuLean
SimPy
salabim
JaamSim
FlexSim
AnyLogic
Plant Sim.
Open source; free
yes (GPL-3.0)
yes (MIT)
yes (MIT)
yes (Apache 2.0)
no; free student ed. (Autodesk)
no; free learning ed.
no; free student ed.
3D layout, animation
yes
no
yes (2D, 3D)
yes
yes
yes
yes
Random stream per element
yes, default (by name)
by scripting
optional, per distrib.
yes (seed per distrib.)
yes (per object)
by scripting
n.c. e
Agent interface (MCP)
yes, native
– a
–
–
community / third party b
community b
no native c
Machine-readable model
partial d
code
code
yes (text)
yes (XML)
yes (XML)
via interfaces
Experimenter / optimiser
yes / yes (GA, random; pluggable algorithms)
– / –
– / –
yes / –
yes / yes (OptQuest add-on)
yes / yes (OptQuest)
yes / yes (GA)
Table 4: Capabilities relevant to agent-built manufacturing models. –: not found in the documentation consulted (not a statement that it cannot be done, e.g. by programming); Plant Sim.: Plant Simulation. a Concept only: an LLM wrote SimPy code from MCP-compatible context [ 44 ] . b Community or third-party MCP servers (e.g. a commercial assistant that drives FlexSim from AI coding tools), not official releases [ 69 , 70 ] . c No native MCP server; an AI copilot is offered in Plant Simulation X, and a research prototype drives Plant Simulation through COM [ 14 ] . d Read-only JSON description of the whole model. e Not confirmed in the public documentation consulted.
Figure 5: Evaluation design. The benchmark (L1–L5 and held-out tasks, k=3 ) runs in the SimuLean arm (C-Full on two interface versions, ablations C-NoGate and C-Generic; Opus and Sonnet) and, for six tasks, in the SimPy arm. The harness runs every final model and classifies it by the first step it fails, with validity judged against a reference model. The industrial case is built three times on each of two interface versions and three times in SimPy, and validated against the corrected published model. Diagram drawn with the assistance of Claude (Anthropic).
Claude Opus 5.5
Claude Sonnet 5.5
Level
Valid
95% CI
Valid
95% CI
L1 Basic flow
18/18
[0.82, 1.00]
18/18
[0.82, 1.00]
L2 Resources & time
13/18
[0.49, 0.88]
13/18
[0.49, 0.88]
L3 Material handling
18/18
[0.82, 1.00]
17/18
[0.74, 0.99]
L4 Change & decide
16/18
[0.67, 0.97]
4/9
[0.19, 0.73]
L5 Industrial lines
12/12
[0.76, 1.00]
6/6
[0.61, 1.00]
Table 5: Valid models by level and model, C-Full, both interface versions pooled (valid / attempts, Wilson 95% CI). Sonnet: L4, L5 and held-out on v1 only, so its totals pool 54 attempts at L1–L3 with 27 at L4–H4.
Figure 6: Agent time to a valid model by task level (v1, C-Full; log scale). Points are valid attempts, bars the medians. Stars: the manual baseline, the two authors building T1.2 (L1) and T2.3 (L2) by hand ( n=2 , familiar with the simulator, hence optimistic).
Figure 7: Effort in tokens and cost (v1, C-Full). (a) Mean tokens per attempt by type — cache read, cache write, output and uncached input — by level, Opus solid and Sonnet hatched. (b) Estimated cost per valid model: Claude Code’s API-equivalent estimate for all attempts at a level divided by the valid models (the runs used a subscription; no estimate exists for a run ended by the token cap).
Opus C-Full
Sonnet C-Full
NoGate
Generic
Outcome / class
v0
v1
v0
v1
v1
v1
Total
Attempts
54
54
27
54
27
27
243
Valid
47
46
23
43
23
25
207
Build or gate failure
0
0
0
0
0
0
0
Silent: operator busy in break (T2.1)
2
3
3
2
3
2
15
Silent: ambiguous spec. (H4)
3
3
–
3
–
–
9
Table 6: Outcomes of the 243 attempts of the main block by class (counts). No attempt failed at the build or gate step.
Figure 8: Detection by the build-and-run gate, the eight behavioural checks (a model is flagged if any check fails) and the behavioural evaluator, which compares the model with its reference: silent errors and valid models of the main block (243 attempts), and 45 defects seeded into correct models. The evaluator defines the outcome of the agents’ models and is shown for the seeded defects only.
Figure 9: The same agent (Claude Opus 5.5) operating SimuLean (v1, C-Full) and programming SimPy from scratch on six benchmark tasks, k=3 : median (a) tokens per attempt (log scale), (b) tool calls and (c) agent time per attempt.
Figure 10: Mean flow time of the AGV cell against fleet size for the travel models, (a) wide two-way aisles and (b) narrow single-lane aisles. Headless variants: mean of 100 replications with 95% CI; T3 (Unity navigation and physics, interactive scene): mean and range of five replications sharing seeds 1–5 with the headless runs. Dashed: the flow-time target fixed before the T3 runs.
Figure 11: Process flow of the robotised welding workshop (scenario 6 of Lamas-Rodríguez et al. [85] ), as specified to the agent. Inside a span its overhead crane moves the parts (span 3 has no crane and no station); two rail carts in the front corridor carry them between spans. Stubs are cut and bevelled in span 1 and wait in the buffers of span 2 unless an assembly bed is waiting for them; chords are stored by node type in span 4. On the assembly beds the boilermaker pair fits the chord and the two stubs and a dimensional-control (DC) pair does DC1; in span 5 the welder makes the root pass, an NDT pair tests it and a DC pair does DC2, then the robot welds the node, which goes through DC3 and, after a post-weld wait, the final NDT. Dashed blue: reworks and repeated steps; numbered circles: the stages of the specification. Times, probabilities, capacities, speeds and positions come from the original model and are not shown. Diagram drawn with the assistance of Claude (Anthropic).
Run
Working days
Time (min)
Calls
USD
Code
v1 rep1
4.863
97.4
244
25.1
115
v1 rep2
4.879
130.7
2 756
51.7
42
v1 rep3
4.968
82.8
1 281
39.0
58
v1 mean
4.903
103.6
1 427
38.6
72
v2 rep1
4.915
129.4
289
44.9
1
v2 rep2
5.125
91.9
262
34.7
0
Table 7: The nine runs of the industrial case on the first specification. Working days per jacket, in real working time: harness re-runs of each exported model, 50 replications (95% half-widths 0.007–0.010). Time, calls and cost of the whole run; USD: Claude Code’s API-equivalent estimate. Code: calls of the editor’s code-evaluation tool (SimuLean) or lines of Python (SimPy).
Figure 12: The robotised welding workshop built by the agent of run v1 rep1 (first specification) in SimuLean. Purely visual work followed the model: a separate agent session added the building shell, span zones, labels and cameras, and an interactive agent session with the authors’ review (about half an hour, three rounds) gave the parts and stations generic 3D models made for this case (tubular chords and stubs, turning-roll positioners, a pipe-profile cutter, a robot cell) and laid out the waiting parts in piles. The exported model is unchanged by both (a name-normalised comparison of the exported configurations shows no change to any element). Views (b)–(d) were taken while the model was running.
Model
Working days per jacket
Difference from corrected
Original as run, with the routing error (FlexSim)
4.836±0.008
−2.9%
Original corrected (FlexSim)
4.98±0.06
reference
Agent-built, SimPy, 3 runs
4.999 (4.998–5.001)
+0.4%
Agent-built, SimuLean v1, 3 runs
4.986 (4.964–4.998)
+0.1% ( −0.3 to +0.4% )
Agent-built, SimuLean v2, 3 runs
4.991 (4.983–4.999)
+0.2% ( +0.1 to +0.4% )
Table 8: The reference model, its correction and the agent-built models of the corrected specification, in working days per jacket (real working time). Mean ± 95% half-width over 50 replications, or mean (range) over three runs, each re-run with 50 replications. FlexSim reports the original study’s KPI; its results are converted to working days with the model’s working calendar (as run, the original model reproduces the published value). In every model the busy hours of the welder, the robot, the cutting station and the NDT pairs equal the workloads implied by the original data.
Existing Simulink benchmarks mainly evaluate whether generated models compile, execute, or resemble a reference model. These criteria do not establish whether a model satisfies its engineering requirements. We introduce SimuVerity, a benchmark of 101 text-to-executable Simulink model-generation tasks across ten engineering domains. For each task, executable-system profiles ground the engineering specification and four families of native simulation scenarios. A hierarchical evaluator first checks artifact delivery, native executability, and engineering qualification, then scores qualified models across six dimensions covering accuracy, output quality, mechanistic fidelity, control and causal integrity, operating-domain robustness, and dynamic response. We evaluate six agent systems with SimuVerity. The best system achieves an overall score of only 42.86. The results show that structural similarity is a poor proxy for engineering performance: capability bottlenecks arise both in producing qualified implementations and in satisfying multidimensional requirements after qualification. Meanwhile, some high-scoring models still exhibit severe visual-layout disorder. SimuVerity provides a systematic basis for assessing agents' engineering capabilities and diagnosing failures in executable Simulink model generation.
LLarge language model (LLM)-based AI agents deployed in manufacturing environments require populated, schema-correct data for validation, yet production MES data is proprietary, privacy-encumbered, and vendor-specific. This paper introduces the Template-as-Ontology principle: a single Python configuration module (700-770 lines, 45 validated exports) serves simultaneously as the specification for a time-stepped manufacturing simulator and as the runtime domain schema for AI analytics tools, producing alignment by construction rather than integration. We formally define the domain template as a typed relational configuration schema and prove that structural alignment between simulation and tool layers is guaranteed by single-source consumption. A five-layer pipeline--simulation, PostgreSQL, CDC/Iceberg lakehouse, star schema, and 12 parameterized AI tools--generates causally coherent, MES-shaped data spanning 66 entity types across four operational domains mapped to ISA-95/IEC 62264. We validate the architecture with six industry templates (aerospace, pharma, automotive, electronics, beverages, warehousing) running on identical framework code. Calibration experiments (60 runs, 10 seeds per template) confirm parametric controllability: observed KPIs fall within configured ranges across all templates. A controlled hallucination experiment (72 tool invocations, Qwen3-32B) demonstrates that ontology-constrained parameters eliminate tool-parameter fabrication (0% constrained vs. 43% unconstrained hallucination rate for the evaluated model, Fisher's exact test p < 10^-12); the 0% constrained rate is an architectural guarantee that holds for any model. The framework provides a reusable data layer for discrete manufacturing AI validation.
Grama Chethan
Siemens Digital Industries Software, Plano, TX 75024, United States
LLM agents increasingly drive systems research and rely on simulators to evaluate candidate designs at scale. Simulators are imperfect by design: their modeling limitations can distort predictions yet leave the recommended design correct, or change which design should be selected. We introduce Siris to test whether LLMs can tell these cases apart, judging from their systems understanding when a limitation invalidates the simulator's recommendation. By perturbing a simulator, Siris constructs workload pairs with the same candidates, limitation, and simulator recommendation. The perturbation changes predictions on both workloads, yet a reference simulator supports the recommendation on only one. Evaluating thirteen frontier LLMs on LLM serving design problems, we find that identifying a limitation does not ensure a correct choice, and even the strongest models sometimes abandon valid recommendations. Self-correction through a review turn helps these models restore valid choices but introduces new errors in several others. These findings motivate evaluating systems understanding through design decisions under imperfect simulators.