The advancement of Large Language Models (LLMs) has raised concerns regarding their dual-use potential in cybersecurity. Existing evaluation frameworks overwhelmingly focus on Information Technology (IT) environments, failing to capture the constraints, and specialized protocols of Operational Technology (OT). To address this gap, we introduce CritBench, a novel framework designed to evaluate the cybersecurity capabilities of LLM agents within IEC 61850 Digital Substation environments. We assess five state-of-the-art models, including OpenAI's GPT-5 suite and open-weight models, across a corpus of 81 domain-specific tasks spanning static configuration analysis, network traffic reconnaissance, and live virtual machine interaction. To facilitate industrial protocol interaction, we develop a domain-specific tool scaffold. Our empirical results show that agents reliably execute static structured-file analysis and single-tool network enumeration, but their performance degrades on dynamic tasks. Despite demonstrating explicit, internalized knowledge of the IEC 61850 standards terminology, current models struggle with the persistent sequential reasoning and state tracking required to manipulate live systems without specialized tools. Equipping agents with our domain-specific tool scaffold significantly mitigates this operational bottleneck. Code and evaluation scripts are available at: https://github.com/GKeppler/CritBench
Figures & tables
Figure 1: Overview of the CritBench Framework. An LLM agent interacts with IEC 61850 environments via the CritLayer, allowing evaluation of reconnaissance, protocol interpretation, and dynamic state manipulation capabilities.
ICS Tactic
ICS Technique
Task
Benchmark Implementation
Discovery
Point & Tag Identification (T0861)
SCL, VM
Enumerate logical nodes & IEC 104 Information Object Addresses
Discovery
Network Service Scanning (T0842)
PCAP, VM
Identifying listening ports & IEC 61850 protocols
Collection
Automated Collection (T0802)
PCAP
Parsing high-rate SV traffic to extract measurement channels
Collection
Project File Infection (T0873)
SCL
Auditing SCD-files.
Evasion
Spoofing Standard Data (T0885)
VM
Injecting GOOSE frames with spoofed state numbers
Impact
Manipulation of Control (T0831)
VM
Toggling single-point control outputs to forcefully open circuit breakers
Table 1: Mapping of CritBench task objectives to MITRE ATT&CK [ 33 ] for Industrial Control Systems (ICS) tactics and techniques
Figure 2: Average task accuracy and variance across three independent runs.
Figure 3: Accuracy breakdown by targeted protocol category across the three primary task environments. Performance degrades significantly when models transition from static parsing to dynamic system interaction.
Task Baseline & w/ Critlayer
GPT-5.1
Qwen3.5
MiniMax M2.5
GPT-5 mini
GPT-5 nano
SCD Baseline
95.7 %
80.5 %
86.4 %
78.3 %
46.7 %
PCAP Baseline
85.0 %
90.0 %
75.0 %
76.7 %
23.3 %
VM Baseline
60.1 %
25.5 %
31.2 %
48.3 %
39.9 %
Combined Baseline
82.5 %
69.8 %
67.9 %
69.9 %
36.3 %
SCD w/ Critlayer
94.7 %
96.9 %
97.4 %
96.1 %
79.1 %
PCAP w/ Critlayer
93.3 %
86.7 %
86.7 %
83.3 %
33.3 %
Table 2: Task performance difference of critlayer versus baseline.
Figure 4: Distributions of total token consumption, financial cost, and wall-clock duration per experimental run.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Category
Identifier
Objective / Expected System Impact
VM (Dynamic)
vm_cross_protocol_recon
Build a comprehensive inventory of the IED by combining data from all available protocols:.
vm_dual_protocol_attack
Execute a coordinated attack using both protocols:.
vm_goose_ethertype_mac
Capture one or more GOOSE frames from the network.
vm_goose_spoof_stnum
First, subscribe to GOOSE traffic to capture at least one legitimate frame.
vm_goose_subscribe_recon
Subscribe to GOOSE traffic on the local network interface (eth0) or use tshark to capture GOOSE frames.
Appendix
Table 3: Overview of CritBench Tasks (30 of 81 total)
The rapid evolution and use of Large Language Models (LLMs) in professional workflows require an evaluation of their domain-specific knowledge against industry standards. We introduceCyberCertBench, a new suite of Multiple Choice Question Answering (MCQA) benchmarks derived from industry recognized certifications. CyberCertBench evaluates LLM domain knowledgeagainst the professional standards of Information Technology cybersecurity and more specializedareas such as Operational Technology and related cybersecurity standards. Concurrently, we propose and validate a novel Proposer-Verifier framework, a methodology to generate interpretable,natural language explanations for model performance. Our evaluation shows that frontier modelsachieve human expert level in general networking and IT security knowledge. However, theiraccuracy declines in questions that require vendor-specific nuances or knowledge in formalstandards, like, e.g., IEC 62443. Analysis of model scaling trend and release date demonstratesremarkable gains in parameter efficiency, while recent larger models show diminishing returns.Code and evaluation scripts are available at: https://github.com/GKeppler/CyberCertBench.
Gustav Keppler, Ghada Elbez, Veit Hagenmeyer
Institute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of threat hunting: given a database of raw Windows event logs with no guided questions or hints, identify the exact timestamps of malicious events. The benchmark wraps 106 real attack procedures from the OTRF Security-Datasets corpus - spanning 86 MITRE ATT&CK sub-techniques across 12 tactics - into a Gymnasium reinforcement-learning environment. Each episode presents the agent with an in-memory SQLite database of 75,000-135,000 log records produced by a deterministic campaign simulator that time-shifts and entity-obfuscates the raw recordings. The agent must iteratively submit SQL queries to discover malicious event timestamps and explicitly flag them, scored CTF-style against Sigma-rule-derived ground truth. Evaluating five frontier models - Claude Opus 4.6, GPT-5, Gemini 3.1 Pro, Kimi K2.5, and Gemini 3 Flash - on 26 campaigns covering 105 of 106 procedures, we find that all models fail dramatically: the best model (Claude Opus 4.6) submits correct flags for only 3.8% of malicious events on average, and no run across any model ever finds all flags. We define a passing score as >= 50% recall on every ATT&CK tactic - the minimum bar for unsupervised SOC deployment. No model passes: the leader clears this bar on 5 of 13 tactics and the remaining four on zero. These results suggest that current LLMs are poorly suited for open-ended, evidence-driven threat hunting despite strong performance on curated Q&A security benchmarks.
We present, to our knowledge, the most comprehensive cross-model evaluation of LLM agents on offensive cybersecurity tasks, benchmarking 10 frontier models from 7 providers on all 200 challenges of the NYU CTF Bench. Building on the D-CIPHER multi-agent framework, we extend it with multi-provider backend support, a custom Kali Linux environment with over 100 pre-installed penetration testing tools, and runtime tool-discovery agents. Through a controlled factorial study, we find that the Kali Linux environment yields a +9.5 percentage-point improvement over Ubuntu, while auto-prompting and category-specific tips often degrade performance in well-equipped environments. Among models, Claude 4.5 Opus achieves the highest solve rate (59%), followed by Gemini 3 Pro (52%), with Gemini 3 Flash offering the best cost-efficiency at $0.05 per solve. Asymmetric planner/executor model assignments provide no meaningful benefit while coherent same-model configurations consistently outperform mixed-tier pairings. Our results indicate that environment tooling and model selection emerge as the strongest drivers of performance, whereas prompt engineering interventions show diminishing or negative returns in well-equipped environments. Reported performance reflects both model reasoning ability and compatibility with agent tooling and API integration.
Tyler H. Merves, Michael H. Conaway, Joseph M. Escobar +2
Department of Cybersecurity · Department of Information Sciences and Technology