cs.CRSep 26, 2026

CyberClear: A Benchmark for LLM Agent Systems on APT Attack Chain Provenance

Authors: Qi Chen, Fushuo Huo, Hangli Shen, Jingcai Guo, Shuhao Li, Guang Cheng

Organizations: School of Cyber Science and Engineering, Southeast University · The Hong Kong Polytechnic University · Zhongguancun Laboratory

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language model agents have demonstrated promising capabilities in cybersecurity tasks, yet their ability to reconstruct complete Advanced Persistent Threat attack campaigns from complex security logs remains largely unexplored. Existing cybersecurity benchmarks for agents mainly focus on vulnerability discovery, exploitation, and security analysis tasks, leaving the evaluation of attack chain provenance under realistic security logs insufficiently studied. To address this gap, we introduce CyberClear, a benchmark for evaluating LLM agents and advanced agent systems on APT attack chain provenance from long-context security logs. CyberClear covers both single-step attacks and multi-stage attack chains, requiring agents to identify attack evidence, infer attack progression, and generate provenance graphs containing entities, causal relationships, MITRE ATT&CK techniques, and forensic evidence. To enable comprehensive evaluation, we develop an evaluation method tailored to APT attack chain provenance. Unlike conventional text similarity metrics that focus on surface-level matching, our evaluation examines whether reconstructed graphs preserve the semantics of attack chains across single-step behavior correctness, multi-step behavior identification, temporal and causal consistency, entity and relationship fidelity, and overall attack narrative consistency. Advanced multi-agent systems powered by state-of-the-art LLMs still struggle on CyberClear, motivating us to propose CyberProvenance, an agent cyber harness designed for multi-agents that augments LLM agents with evidence accumulation, execution-based validation, and feedback-guided refinement mechanisms for reliable attack-chain provenance. Extensive evaluations on CyberClear demonstrate the effectiveness of CyberProvenance in improving evidence reasoning, execution-grounded validation, and complete APT attack chain reconstruction.

Figures & tables

Figure 1: Overview of CyberClear. Given long-context defender logs without prior attack clues, agents are required to identify attack evidence, infer attack progression, and reconstruct complete APT attack chains across multiple attack stages.
BenchmarkScopeInstanceAttack BehaviorsATT&CKLogsAttack Chain
NYU CTF Bench ( Shao et al., 2025 )CTF2006✗✗✗
Cybench ( Zhang et al., 2025a )CTF406✗✗✗
AutoAdvExBench ( Carlini et al., 2025 )CTF + Real-world752✗✗✗
CVE-Bench ( Zhu et al., 2025 )Real-world408✗✗✗
BountyBench ( Zhang et al., 2026 )Real-world4036✗✗✗
SEC-bench ( Lee et al., 2025 )Real-world20016✗✗✗
Table 1: Comparison with existing cybersecurity benchmarks for Agents.
Figure 2: The construction pipeline of CyberClear.
StatisticEasyMediumHardOverall
Total Samples
Sample Count31812210450
Attack Complexity
Single-step Attack31800318
Multi-step Attack012210132
Input Log Volume (count)
Table 2: Statistics of the CyberClear.
Figure 3: Distribution of attack behaviors.
Figure 4: Overview of CyberProvenance, an agent cyber harness for reliable attack-chain provenance. Designed for multi-agent collaboration, the harness assigns specialized capabilities to different agents, enabling evidence accumulation, execution-based validation, and feedback-guided refinement.
Methods Code-Level MetricsSemantic Metrics
BLEUROUGE-LCodeBLEUPass@1SingleLooseStrictDetailScenario
OpenClaw (Single) 0.33170.48630.37950.82220.67940.60060.43870.40510.4805
 0.35560.51170.38680.72000.78120.71250.54900.48980.5823
 0.36920.52970.38870.84670.77260.72140.50080.45360.5862
OpenClaw (Multiple) 0.41830.56540.56480.92000.55130.47490.35800.33010.3679
 0.44220.59080.57210.81780.71600.66290.50340.40900.5847
Table 3: Performance comparison on CyberClear across agent frameworks and LLM backbones using code-level and semantic metrics. Blue cells indicate the best result for each metric.
Figure 5: Average Semantic Metrics scores of different agent frameworks across the Easy, Medium, and Hard subsets of CyberClear.
Figure 6: Case study comparing AgentVerse and CyberProvenance on attack provenance using GPT-5.6-Terra. CyberProvenance corrects unsupported causal relations by aligning the reconstructed graph with execution evidence.
Figure 7: Average Semantic Metrics scores across single-stage and multi-stage attacks.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Methods Code-Level MetricsSemantic Metrics
BLEUROUGE-LCodeBLEUPass@1SingleLooseStrictDetailScenario
OpenClaw (Single) 0.36170.52360.30190.81130.45090.35880.22530.26020.3209
 0.34280.49750.36590.82670.65210.57480.41460.39170.4615
OpenClaw (Multiple) 0.44830.60270.48720.90850.56940.45920.29820.32790.4293
 0.42370.57180.55260.91780.58940.51270.38120.35260.4098
AgentVerse 0.39640.58190.52980.92890.63830.53140.35100.36500.4869
Appendix
Table 4: Additional results of representative agent frameworks on CyberClear with different LLM backbones.
VariantsCode-Level MetricsSemantic Metrics
BLEUROUGE-LCodeBLEUPass@1SingleLooseStrictDetailScenario
CyberProvenance0.49520.69680.70160.93330.88970.74380.63370.61240.7263
w/o Evidence Memory0.4510 ↓\downarrow 0.04420.6423 ↓\downarrow 0.05450.6554 ↓\downarrow 0.04620.9044 ↓\downarrow 0.02890.8421 ↓\downarrow 0.04760.6845 ↓\downarrow 0.05930.5578 ↓\downarrow 0.07590.5236 ↓\downarrow 0.08880.6502 ↓\downarrow 0.0761
w/o Execution Validation0.4598 ↓\downarrow 0.03540.6492 ↓\downarrow 0.04760.6615 ↓\downarrow 0.04010.8978 ↓\downarrow 0.03550.8470 ↓\downarrow 0.04270.6780 ↓\downarrow 0.06580.5485 ↓\downarrow 0.08520.5200 ↓\downarrow 0.09240.6425 ↓\downarrow 0.0838
w/o Feedback Refinement0.4825 ↓\downarrow 0.01270.6811 ↓\downarrow 0.01570.6889 ↓\downarrow 0.01270.9178 ↓\downarrow 0.01550.8734 ↓\downarrow 0.01630.7169 ↓\downarrow 0.02690.5983 ↓\downarrow 0.03540.5716 ↓\downarrow 0.04080.6885 ↓\downarrow 0.0378
Appendix
Table 5: Ablation study of CyberProvenance on CyberClear using GPT-5.6-Terra. Red values indicate performance drops relative to the full model.
Figure 8: Consistency between GPT-5.6-Terra-based evaluation and human judgments. Two human experts independently evaluate generated provenance graphs, with each evaluation repeated three times and averaged.
Figure 9: Code-Level similarity across difficulty levels using BLEU, ROUGE-L, and CodeBLEU. Columns correspond to easy, medium, and hard instances, and each radar chart compares agent frameworks under three LLM backbones.
Figure 10: Code-Level similarity across single-stage and multi-stage instances using BLEU, ROUGE-L, and CodeBLEU. Columns correspond to three LLM backbones, and each heatmap compares agent frameworks under single-stage and multi-stage settings.
Figure 18: Case study of attack-chain provenance. LLM-Debate incorrectly omits the bash →\rightarrow stty dependency and its terminal-configuration step, despite explicit audit evidence highlighted in yellow: comm="stty" , exe="/usr/bin/stty" , and stty rows 38 columns 116 . CyberProvenance correctly aligns this evidence with the corresponding entity relation and attack step, producing a provenance graph consistent with the observed execution.
Figure 19: Case study comparing provenance results. LLM-Debate omits the bash →\rightarrow stty dependency, although the yellow-highlighted audit evidence explicitly records /usr/bin/stty and stty rows 38 columns 116 . CyberProvenance correctly aligns this evidence with the corresponding entity relation and attack step.
Figure 20: Case study of attack-chain provenance. MacNet fails to preserve three key evidence-supported relations, including the modification of auditf.service , its privileged activation through sudo , and the subsequent execution of the malicious service, while also omitting important host, service, and network entities. As highlighted in yellow, the logs explicitly record vim /etc/systemd/system/auditf.service , systemctl enable auditf.service , and the execution of /usr/bin/auditf . CyberProvenance retains these dependencies and reconstructs a more complete attack progression consistent with the observed audit evidence.
Figure 21: Case study of attack-chain provenance. OpenClaw fails to faithfully preserve the process dependencies associated with terminal configuration. As highlighted in yellow, the logs explicitly record python3 spawning bash , followed by the execution of /usr/bin/stty with rows 38 columns 116 . CyberProvenance correctly reconstructs these evidence-supported process relations and the corresponding attack-step progression.
Figure 22: Case study of attack-chain provenance. AgentScope-V2 misclassifies a key entity and fails to faithfully align the reconstructed steps with the observed execution chain. As highlighted in yellow, the logs explicitly record the Puppet-triggered curl download, the subsequent execution of healthcheck_cron.sh by cron , and the final lftp connection to 172.17.100.122 . CyberProvenance correctly reconstructs these evidence-supported entities and attack-step dependencies.
Figure 23: Case study of attack-chain provenance. LLM-Blender misses several key evidence-supported relations and fails to faithfully preserve the reconnaissance chain. As highlighted in yellow, the logs explicitly record the zmcontroller connection to 192.42.1.174:443 , the chmod 755 /tmp/nmap operation, and the subsequent nmap scan of 172.17.100.0/24 . CyberProvenance correctly reconstructs these observed entities, relations, and attack-step dependencies.
Attack behaviorATT&CK ID(s)
B01 Abuse Elevation Control Mechanism: Sudo and Sudo CachingT1548.003
B02 Account Access RemovalT1531
B03 Account Discovery: Local AccountT1087.001
B04 Active Scanning: Vulnerability ScanningT1595.002
B05 Active Scanning: Wordlist ScanningT1595.003
B06 Application Window DiscoveryT1010
Appendix
Table 6: All attack behaviors in CyberClear. The 450 instances yield 89 distinct technique sets: 46 single-technique and 43 composite behaviors.
BehaviorOpenClaw (Single)OpenClaw (Multiple)AgentVerseLLM-DebateMultiPersonaLLM-BlenderDyLANMacNetCAMELAgentScope-V2CyberProvenance
B01✓✓✓✓✓✓✓✓✓✓✓
B02✓✓✓✓✓✓✓✓✓✓✓
B03✓✓✓✓✓✓✓✓✓✓✓
B04✓✓✓✓✓✓✓✓✓✓✓
B05✓✓✓✓✓✓✓✓✓✓✓
B06✓✓✓✓✓✓✓✓✓✓✓
Appendix
Table 7: Attack behavior coverage of different agent frameworks under GPT-5.6-Terra. A checkmark indicates that the framework successfully handles the corresponding attack behavior, while a cross indicates failure.
More in this topic

Cybersecurity

Cybersecurity develops methods to protect digital and cyber-physical systems from attacks, vulnerabilities, data loss, and misuse.

See all 355 papers

Explore similar work

CardsList
Aug 4, 2026

DiagChain: A Diagnostic Benchmark for Evaluating LLM Agents on Evidence-Grounded Attack Chain Reconstruction

Large Language Model (LLM) agents offer a promising approach to attack chain reconstruction by retrieving and interpreting heterogeneous telemetry to infer ordered attacker actions. However, existing benchmarks mainly evaluate final outputs or aggregate accuracy, providing limited insight into how errors arise and propagate across intermediate reasoning stages. We present DiagChain, a diagnostic benchmark for evidence-grounded attack chain reconstruction that enables stage-wise evaluation of LLM agents. DiagChain includes MAIN-69, a suite of 69 scenarios spanning multiple operating systems, evidence noise levels, and chain lengths. It further introduces Evidence-Centric Retrieval-Augmented Generation (ECRAG), which couples evidence retrieval with an evolving structured representation of the reconstructed chain. Five complementary metrics are introduced to assess distinct stages of the reconstruction process and support systematic failure diagnosis. Based on evaluations using 6 LLMs, DiagChain reveals that even the strongest configuration succeeds on only 39.6% of the 849 reference steps in MAIN-69. Our analysis further shows that smaller models struggle with the more basic task of incorporating retrieved evidence into their outputs, whereas larger models can proceed to later steps, where correctly ordering that evidence becomes the main bottleneck. These results validate the importance of diagnostic evaluation beyond end-to-end accuracy and provide actionable insights for improving evidence-grounded cybersecurity agents.

Xuyang Liu, Yibin Han, Zhenwei Zhang +8

KLISS, BNRist, School of Software, Tsinghua University, Beijing, China · CRRC Corporation Limited, Beijing, China

Retrieval-Augmented GenerationLLM Agent EvaluationAI Agent Security BenchmarksLLMs for Cybersecurity
AskPDF
Sep 30, 2026

APTInvestBench: Evaluating Autonomous APT Investigation under Varying Telemetry

Large language model (LLM) agents could help security operations centers (SOCs) investigate advanced persistent threats (APTs) by turning weak leads into evidence for intrusion scoping and response. Yet success under one telemetry setting does not establish robustness to changes in log collection, retention, or sampling. We introduce APTInvestBench, a benchmark for evaluating cross-telemetry robustness in autonomous APT investigation. It comprises 370 cases across seven SOC-inspired conditions, derived from 56 report-informed attack reconstructions with 16.4 million log records. Agents investigate unverified leads and submit reports with record-level citations. Fixed action-level support requirements track sufficient evidence across available logs, query returns, and formal citations, separating telemetry limitations from acquisition and reporting gaps. Across eleven LLMs, agents acquire sufficient evidence for 44.3% of recoverable attack actions on average, while formal citations support only 25.0%. More importantly, aggregate coverage can conceal substantial instability: from Full to endpoint-only telemetry, coverage declines by only 1.6 percentage points, yet 35.5% of previously covered actions lose sufficient citation support despite remaining recoverable. Across four frameworks, such losses persist even when registered supporting records remain unchanged. APTInvestBench provides reusable investigation environments and diagnostic evaluation for identifying these gaps and developing more reliable defensive agents.

Yu Wang, Shuhao Li, Tao Yin +4

Zhongguancun Laboratory

Distribution Shift RobustnessAI Agent BenchmarksLLMs for CybersecurityLLM Agent Reliability
AskPDF
Oct 22, 2026

SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence

Large language models (LLMs) have shown strong potential for cybersecurity, leading to growing interest in LLM-based automated cybersecurity log analysis. However, current datasets (1) are limited in scale; (2) rely mainly on single-source evidence; and (3) have only binary or multi-class attack labels, without dense annotations of the reasoning process. To address these limitations, SecLog-Bench has three key features. First, we build an extensible simulation platform that covers 29 attack scenarios, supports repeated execution and the integration of new scenarios, and enables scalable generation of cybersecurity logs with attack-stage labels. Second, we use multi-source logs, including audit logs, application logs, and network traffic, to jointly analyze cyber attacks. Third, we annotate each sample with the essential context for cybersecurity log reasoning and a structured step-by-step evidence chain. We evaluate eight mainstream LLMs on this benchmark for post-detection attack-progress reasoning. Results show that, when provided with scaffolded evidence, current LLMs exhibit modest capability in attack-progress reasoning, with the best model achieving only 44.44% accuracy and 45.07% Macro-F1. Their reasoning remains unreliable; i.e., the highest Stage Reasoning F1 (SRF1) score is only 54.87%. Moreover, even correct predictions may be accompanied by flawed reasoning, and reasoning quality is often misaligned with prediction accuracy.

Yifan Wang, Dongliang Xu, Xiaoyong Li +3

School of Airspace Science and Engineering, Shandong University, Shandong 264209, China. · School of Computer Science, University of Technology Sydney, Ultimo, NSW 2007, Australia. · an independent researcher in Braddon, ACT 2612, Australia.

CybersecurityLLMs for CybersecurityLanguage Model Reasoning Evaluation
AskPDF