PLCWorld: Benchmarking LLM-Generated PLC Programs in Closed-Loop Plant Simulation

Yunji Kim; Yunseok Lee; Hyunwoo Seo; Jaerim Choi; Woojin Lee

cs.AIOct 2, 2026

PLCWorld: Benchmarking LLM-Generated PLC Programs in Closed-Loop Plant Simulation

Authors: Yunji Kim, Yunseok Lee, Hyunwoo Seo, Jaerim Choi, Woojin Lee

Organizations: Dongguk University · MOAI technologies

Abstract

Programmable logic controllers (PLCs) coordinate industrial equipment by reading sensor inputs and issuing control commands. Evaluating whether large language model (LLM)-generated PLC programs satisfy task requirements and safety constraints requires observing how their commands affect device and workpiece states. We introduce PLCWorld, a common closed-loop execution environment and benchmark that couples Structured Text (ST) execution with simulated plant responses and sensor feedback. Grounded in control relations identified in industrial PLC programs and engineering documentation, PLCWorld contains 100 synthetic tasks and 473 registered task-condition pairs across Motion Control and Material Handling, with difficulty defined by control-dependency scope. A common protocol reports Task Success and Safety Violation separately. Validation combines practitioner review, reference and alternative programs, targeted counterexamples, specification-evaluator alignment checks, and comparisons with independent ST runtimes. Reference and alternative programs satisfy their applicable cases, while all 542 targeted counterexamples activate their designated evaluator rules under at least one registered condition. Execution Gap relates submission-profile acceptance to subsequent task failure or observed Safety Violation. Across the constructed task groups, direct GPT-5.5 achieves 82.70% Task Success on Easy cases but 25.10% on Hard cases. Evaluations of six LLMs and four adapted generation-and-verification workflows further expose differences between completion, safety, and generation cost. Our code, simulation environment, benchmark tasks, and baseline implementations are publicly available at https://yunji0516.github.io/PLCWorld/.

Figures & tables

Figure 1: Execution Gap . Even after the specified upstream checks pass, the lift can rise without the workpiece if grasping is incomplete when the fixed wait ends.

Panel of Table 1 — Table 1: Reported validation coverage of LLM-based PLC code generation studies.

Figure 2: The PLCWorld framework. ST programs generated from task specifications are executed in closed loop with a simulated plant. Execution records are used to assess Task Success and Safety Violation separately, and comparison with upstream checks on the same programs reveals Execution Gap .

Figure 3: Representative executions by suite and difficulty. Columns show Motion Control (left) and Material Handling (right), and rows show control dependencies within one basic function (Easy), across the functions of one job (Medium), and between jobs sharing resources (Hard).

Level	Tasks a	Dependency Scope	Motion Control	Material Handling
Easy	15	Within one basic function	“ Move a slide to a position, confirm arrival and standstill ”	“ Grasp a workpiece at a prepared position, confirm it is held ”
Medium	20	Between functions of one job: one function’s state conditions another’s execution	“ Retract the slide, reorient the table, approach after standstill ”	“ Transport a workpiece, release it after the jig is ready ”
Hard	15	Between independent jobs: one job’s state constrains another’s progress	“ Coordinate two lift jobs as the shared shuttle and rotary/approach cell become available ”	“ Coordinate two handling jobs as the shared robot and inspection station become available ”

Table 2: Difficulty levels defined by dependency scope, with tasks and representative examples.

Figure 4: Control requirement families

Validation target	Validation method
(1) Task specification	Practitioner review
(2) Evaluator judgment	Reference/alternative ST & counterexamples
(3) Spec–evaluator alignment	Binding & semantic checks
(4) ST runtime	Reruns & cross-runtime checks

Table 3: Overview of validation

Figure 5: Suite-combined success rate by difficulty.

(1) Direct ST generation
Model / Method	Motion Control						Material Handling						Execution [-1pt] Gap $\downarrow$
	Easy		Medium		Hard		Easy		Medium		Hard
	SR $\uparrow$	SVR $\downarrow$	SR $\uparrow$	SVR $\downarrow$	SR $\uparrow$	SVR $\downarrow$	SR $\uparrow$	SVR $\downarrow$	SR $\uparrow$	SVR $\downarrow$	SR $\uparrow$	SVR $\downarrow$
GPT-5.5	71.11	1.67	75.58	7.05	18.06	65.19	97.83	0.00	93.06	1.77	27.87	32.24	37.88
GPT-4o	14.44	22.37	8.91	42.06	0.00	28.57	28.99	13.49	1.04	21.11	0.00	57.71	92.75
Claude Sonnet 5	75.00	2.27	75.91	25.50	0.00	73.33	88.41	0.72	77.43	1.04	4.37	51.72	50.90

Table 5: Results (%) by suite and difficulty. Execution Gap is pooled across suites, conditional on common upstream acceptance. Bold marks each block’s highest SR and lowest SVR, including ties.

Figure 6: Execution Gap

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Suite / Difficulty	#Tasks	#Cond.	I/O avg.	I/O range	Device avg.	Device range
Material / Easy	15	46	8.0	5–12	1.20	1–2
Material / Medium	20	96	17.4	11–28	2.85	1–7
Material / Hard	15	122	57.5	37–75	5.13	2–9
Motion / Easy	15	60	8.9	7–12	1.07	1–2
Motion / Medium	20	101	20.1	12–29	2.10	1–4
Motion / Hard	15	48	128.4	64–146	3.87	3–5

Appendix

Table 7: Benchmark composition by suite and difficulty. I/O counts all distinct addresses declared in each task’s public specification, including BOOL , INT , and DINT signals. Device counts denote distinct operational device families per task under the author-defined 17-family taxonomy.

Requirement family	Easy	Medium	Hard	Easy	Medium	Hard	Total
	Material Handling			Motion Control
Positioning and stabilization	8	19	14	14	20	15	90
Synchronization and compensation	0	1	1	4	4	9	19
Access and clearance	0	12	14	2	17	15	60
Gripping and support	5	13	14	1	4	1	38
Quantity and inventory	4	6	8	0	0	0	18

Appendix

Table 8: Distribution of control requirement families by suite and difficulty. Counts are descriptive and non-exclusive; a task may contribute to multiple rows.

Component	Information provided	Role in evaluation
Task objective	Required functional outcome and, where applicable, the requested target, item, quantity, destination, or operating mode.	Defines the plant-level result the submitted program must achieve.
Timing and completion	PLC scan period, completion deadline, evaluation endpoint or termination condition, and any required completion signal or post-completion retention.	Defines scan timing, completion validity, required result retention, and when evaluation ends.
Public I/O interface	Signal names, addresses, types, and input/output semantics.	Defines the interface through which the ST program observes and controls the task environment.
Initial plant state	Initial device states, positions, workpiece or fixture relations, and relevant request or process state.	Defines the starting state for each execution.
Operating conditions	Permitted ranges or alternatives for device responses, initial conditions, and external inputs.	Defines the range within which evaluation conditions may vary.
Device-response semantics	Task-relevant effects of activating and deactivating outputs, including timing and transition conditions.	Defines how program outputs can change device, process, workpiece, or fixture state.

Appendix

Table 9: Principal components of a PLCWorld public task specification. The specification defines required observable behavior while allowing implementation choices within the supported ST profile.

Fixed across conditions	Task objective; public I/O interface and semantics; device-response rules; completion requirements and deadline rules; safety constraints.
May vary by condition	Initial plant state, permitted device-response or timing values, and task-defined external inputs or request parameters.

Appendix

Table 10: Relationship between a task specification and its execution conditions. Condition-specific values remain within the operating ranges published for the task.

Task	Suite / Difficulty	Public objective
MOT-E14	Motion / Easy	Suppress the sway of an empty hoist hook and bring the trolley to rest at its current work position.
MOT-M16	Motion / Medium	Position the table and carriage for tool access, maintain their alignment throughout the requested tool cycle, confirm completion of that cycle, and retract the carriage.
MOT-H03	Motion / Hard	Coordinate two independent setup requests through lift stations, a shared shuttle, moving docking, and rotary synchronization, satisfying relative-motion constraints and returning each fixture.
MAT-E06	Material / Easy	Discharge one product from the already aligned pocket and retract the pusher, while preserving the products in all other pockets.
MAT-M14	Material / Medium	Prepare a single product, clear the feeder, and time its transfer into a continuously moving receiving pocket, while preserving the reserve product.
MAT-H05	Material / Hard	Coordinate assembly, inspection, rework, and outfeed on a shared indexing table, preserving body–insert pairing and current inspection results, clearing all tools before indexing, and reinspecting reworked products before release.

Appendix

Table 11: Representative PLCWorld task objectives across suites and difficulty levels. Full model-facing specifications are distributed with the benchmark.

Figure 8: Plant configurations for the representative tasks in Table 11 , organized by task suite and difficulty level.

Explore similar work

PLCBench: Can Autonomous LLM Agents Turn PLC Access into Sustained Physical Impact?
Aug 27, 2026Yitian Zhou, Jingyu Zheng, Qiliang Jiang +6Cybersecurity Cyber-Physical Systems
Zhejiang University · Xi’an Jiaotong University · University of Bristol
ESBMC-PLC: Formal Verification of IEC 61131-3 Ladder Diagram Programs Using SMT-Based Model Checking
Jun 13, 2026Pierre Dantas, Lucas Cordeiro, Waldir JuniorSafety-Critical Control Formal Verification
Computer Science, The University of Manchester Manchester, UK · Electrical Engineering, Federal University of Amazonas (UFAM) Manaus, AM, Brazil
Detecting Ladder Logic Bombs in IEC 61131-3 PLC Programs using ESBMC-PLC+: A Formal Verification Approach with Trigger Synthesis
Jul 9, 2026Pierre Dantas, Lucas Cordeiro, Waldir JuniorCybersecurity Backdoor Detection
Electrical Engineering Federal University of Amazonas (UFAM) Manaus, AM, Brazil

Aug 27, 2026J/K move · Enter open · S save

PLCBench: Can Autonomous LLM Agents Turn PLC Access into Sustained Physical Impact?

Yitian Zhou, Jingyu Zheng, Qiliang Jiang, Linkang Du+5

Zhejiang University · Xi’an Jiaotong University · University of Bristol

Cybersecurity Cyber-Physical Systems AI Agent Security Benchmarks

Open paper AskPDF arXiv

Industrial control systems (ICSs) rely on programmable logic controllers (PLCs) to connect networked computation with physical control. Tool-using large language model (LLM) agents represent an emerging attack threat: can an autonomous agent convert a network-reachable PLC into sustained adverse physical impact? However, existing evaluations focus on digital tasks or individual stages of PLC testing. In ICSs, evaluations that stop at software exploitation, an accepted write, or tool access may therefore mischaracterize physical risk. We present PLCBENCH, to our knowledge, the first real-PLC hardware-in-the-loop (HIL) framework for characterizing this cyber-to-physical capability and its boundaries. It combines vendor-native interaction, commercial PLC execution, closed-loop reduced-order process simulation, and independent outcome verification. A deterministic evaluator applies fixed rules to runner, communication, PLC-object, and process records to assign six hidden diagnostic flags, distinguishing usable PLC interaction, process-linked manipulation, and sustained physical impact. We instantiate PLCBENCH on four commercial PLCs crossed with four closed-loop workloads. Across five LLM families and 240 real-PLC episodes, 75 episodes (31.3%) sustain their respective physical objectives. Stagewise results show that 98 episodes stop before a valid native read, whereas 62 reach a process-linked write but do not sustain the final objective. Notably, richer process observation is associated with an increase in conditional objective attainment after a process-linked write from 44.2% to 64.0%. These measurements localize failure in configured PLC-process deployments and identify intervention points for future defense evaluation. To support reproducibility, we release the safely disclosable PLCBENCH code and a software-only reproduction pipeline through the accompanying artifact.

Abstract

Explore similar work

PLCBench: Can Autonomous LLM Agents Turn PLC Access into Sustained Physical Impact?

ESBMC-PLC: Formal Verification of IEC 61131-3 Ladder Diagram Programs Using SMT-Based Model Checking

Detecting Ladder Logic Bombs in IEC 61131-3 PLC Programs using ESBMC-PLC+: A Formal Verification Approach with Trigger Synthesis