CritBench: A Framework for Evaluating Cybersecurity Capabilities of Large Language Models in IEC 61850 Digital Substation Environments
Organizations: Karlsruhe Institute of Technology (KIT)
Abstract
The advancement of Large Language Models (LLMs) has raised concerns regarding their dual-use potential in cybersecurity. Existing evaluation frameworks overwhelmingly focus on Information Technology (IT) environments, failing to capture the constraints, and specialized protocols of Operational Technology (OT). To address this gap, we introduce CritBench, a novel framework designed to evaluate the cybersecurity capabilities of LLM agents within IEC 61850 Digital Substation environments. We assess five state-of-the-art models, including OpenAI's GPT-5 suite and open-weight models, across a corpus of 81 domain-specific tasks spanning static configuration analysis, network traffic reconnaissance, and live virtual machine interaction. To facilitate industrial protocol interaction, we develop a domain-specific tool scaffold. Our empirical results show that agents reliably execute static structured-file analysis and single-tool network enumeration, but their performance degrades on dynamic tasks. Despite demonstrating explicit, internalized knowledge of the IEC 61850 standards terminology, current models struggle with the persistent sequential reasoning and state tracking required to manipulate live systems without specialized tools. Equipping agents with our domain-specific tool scaffold significantly mitigates this operational bottleneck. Code and evaluation scripts are available at: https://github.com/GKeppler/CritBench
Figures & tables
| ICS Tactic | ICS Technique | Task | Benchmark Implementation |
|---|---|---|---|
| Discovery | Point & Tag Identification (T0861) | SCL, VM | Enumerate logical nodes & IEC 104 Information Object Addresses |
| Discovery | Network Service Scanning (T0842) | PCAP, VM | Identifying listening ports & IEC 61850 protocols |
| Collection | Automated Collection (T0802) | PCAP | Parsing high-rate SV traffic to extract measurement channels |
| Collection | Project File Infection (T0873) | SCL | Auditing SCD-files. |
| Evasion | Spoofing Standard Data (T0885) | VM | Injecting GOOSE frames with spoofed state numbers |
| Impact | Manipulation of Control (T0831) | VM | Toggling single-point control outputs to forcefully open circuit breakers |
| Task Baseline & w/ Critlayer | GPT-5.1 | Qwen3.5 | MiniMax M2.5 | GPT-5 mini | GPT-5 nano |
|---|---|---|---|---|---|
| SCD Baseline | 95.7 % | 80.5 % | 86.4 % | 78.3 % | 46.7 % |
| PCAP Baseline | 85.0 % | 90.0 % | 75.0 % | 76.7 % | 23.3 % |
| VM Baseline | 60.1 % | 25.5 % | 31.2 % | 48.3 % | 39.9 % |
| Combined Baseline | 82.5 % | 69.8 % | 67.9 % | 69.9 % | 36.3 % |
| SCD w/ Critlayer | 94.7 % | 96.9 % | 97.4 % | 96.1 % | 79.1 % |
| PCAP w/ Critlayer | 93.3 % | 86.7 % | 86.7 % | 83.3 % | 33.3 % |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Category | Identifier | Objective / Expected System Impact |
|---|---|---|
| VM (Dynamic) | ||
| vm_cross_protocol_recon | Build a comprehensive inventory of the IED by combining data from all available protocols:. | |
| vm_dual_protocol_attack | Execute a coordinated attack using both protocols:. | |
| vm_goose_ethertype_mac | Capture one or more GOOSE frames from the network. | |
| vm_goose_spoof_stnum | First, subscribe to GOOSE traffic to capture at least one legitimate frame. | |
| vm_goose_subscribe_recon | Subscribe to GOOSE traffic on the local network interface (eth0) or use tshark to capture GOOSE frames. |