CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?
Authors: Xiangsen Chen, Xuan Feng, Shuo Chen, Matthieu Maitre, Sudipto Rakshit, Diana Duvieilh, Ashley Picone, Nan Tang
Organizations: Microsoft Research · Hong Kong University of Science and Techonology (Guangzhou) · Microsoft · Hong Kong University of Science and Techonology
Analyzing Open Source Intelligence (OSINT) from large volumes of data is critical for drafting and publishing comprehensive CTI reports. This process usually follows a three-stage workflow -- triage, deep search and TI drafting. While Large Language Models (LLMs) offer a promising route toward automation, existing benchmarks still have limitations. These benchmarks often consist of tasks that do not reflect real-world analyst workflows. For example, human analysts rarely receive tasks in the form of multiple-choice questions. Also, existing benchmarks often rely on model-centric metrics that emphasize lexical overlap rather than actionable, detailed insights essential for security analysts. Moreover, they typically fail to cover the complete three-stage workflow. To address these issues, we introduce CyberThreat-Eval, which is collected from the daily CTI workflow of a world-leading company. This expert-annotated benchmark assesses LLMs on practical tasks across all three stages as mentioned above. It utilizes analyst-centric metrics that measure factual accuracy, content quality, and operational costs. Our evaluation using this benchmark reveals important insights into the limitations of current LLMs. For example, LLMs often lack the nuanced expertise required to handle complex details and struggle to distinguish between correct and incorrect information. To address these challenges, the CTI workflow incorporates both external ground-truth databases and human expert knowledge. TRA allows human experts to iteratively provide feedback for continuous improvement. The code is available at GitHub and HuggingFace.
Figures & tables
Figure 1: Illustration of the threat analysis workflow.
Threat Report Summarization; Threat-related QA and Category Classification
ROUGE; Accuracy; Micro/Macro-F 1
Table 1: Existing CTI benchmarks and approaches: tasks and metrics
Benchmark
Triage
Deep Search
TI Drafting
IoCs
TTPs
Threat Actors
Root Cause
CTIBench Alam et al. (2024)
✗
✗
✗
✓
✓
✓
CTISum Peng et al. (2024)
✗
✗
✗
✗
✗
✗
SEvenLLM-Bench Ji et al. (2024)
✗
✗
✓
✓
✓
✓
CyberMetric Tihanyi et al. (2024)
✗
✗
✗
✗
✗
✗
SECURE Bhusal et al. (2024)
✗
✗
✗
✓
✗
✓
Table 2: Workflow-stage coverage of existing benchmarks and approaches
Category
Task / Component
Data Size
Triage
Triaging & Assigning Priority Scores
488 articles
Deep Search
Identifying Related Articles
55 articles
TI Drafting
IoCs Extraction
1310 IoCs
TTPs Identification
1565 TTPs
Threat Actor Generation
412 articles
Root Cause Generation
412 articles
Table 3: Data Description for Benchmark Tasks by Workflow Stage.
Category
Task
Metric
GPT-4o
o3-mini
GPT-4o (FT)
GPT-4o-mini (FT)
Triage
Priority Scoring (Description)
Precision (Accepted)
0.3590
0.3982
0.3392
0.3477
Recall (Accepted)
0.9899
0.9091
0.9798
0.9798
Pass Rate (Score)
0.566
0.550
0.600
0.590
Avg. Bias (Score)
0.505
0.680
0.470
0.520
Total Time (s)
708.23
5178.38
3939.35
4424.71
Total Tokens
430124
450093
453280
555547
Table 4: Performance Metrics for Vanilla LLMs by Task Category, Task, Metric, and Model
Figure 2: Illustrative examples of LLM performance gaps: sparse content generation for threat actors/root causes, and hallucination of IoCs and TTPs compared to ground truth or expert expectations.
Figure 3: The illustration of how TRA works
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Threat Category
Unknown/NA
Singular System
Singular Company
Singular Country
Multiple Countries
Defacement / Spam
5
3
3
3
3
Mobile Malware
3
3
3
3
3
Malware Updates
2
3
3
3
2
New Malware
3
3
3
3
2
Vulnerability Exploitation (CVE < 9)
5
2
2
5
5
Cryptominer / Resource Hijacking
3
3
3
3
3
Appendix
Table 5: Priority Scoring for Threat Categories
Figure 9Figure 10
Task Category
Metric
GPT-4o
o3-mini
GPT-4o (FT)
GPT-4o-mini (FT)
Vanilla
TRA
Vanilla
TRA
Vanilla
TRA
Vanilla
TRA
IoC Ext.
Precision
0.8240
0.8496
0.8503
0.8864
0.8846
0.9034
0.6944
0.7469
Time (s)
580.13
2285.65
7534.92
9517.04
726.95
1938.19
808.67
1966.16
Tokens
339080
458093
448152
599989
331766
452677
319700
395396
TTP Identification
Precision
0.2787
0.3065
0.3480
0.4190
0.2387
0.2508
0.1771
0.2111
Recall
0.2270
0.2163
0.1759
0.1558
0.1846
0.1839
0.1414
0.1358
Appendix
Table 6: Performance Overview: Vanilla LLMs vs. TRA (with Corresponding Base LLMs) on Key Metrics
Summary
BERT-Score F 1
ROUGE-L F 1
Human Expert verdict
S - 1 : Researchers uncovered the attack was carried out by the Russian threat actor Water Gamayun, also known as EncryptHub and Larva-208. The group, exploiting a zero-day vulnerability in Microsoft Management Console (MMC), tracked as CVE-2025-26633, utilised a technique called MSC EvilTwin to manipulate two .msc files with the same name, one legitimate and one malicious, and abused the MUIPath feature to fetch and execute the malicious file, maintain persistence, and exfiltrate sensitive data.
0.710
0.342
Sparse / Not actionable
S - 2 : The attack was carried out by the Russian threat actor Water Gamayun, also known as EncryptHub and Larva-208. Water Gamayun is a Russian threat actor known for sophisticated cyber espionage tactics. Recently, they have been linked to the exploitation of a zero-day vulnerability in the Microsoft Management Console (MMC), identified as CVE-2025-26633. This vulnerability allows attackers to manipulate .msc files and the Multilingual User Interface Path (MUIPath) to execute malicious code, enabling data theft and persistence on compromised systems. The group employs advanced techniques, such as using a trojan loader to introduce both clean and malicious .msc files, tricking the MMC into executing the harmful version. This method helps them bypass traditional security measures. The MSC EvilTwin loader, written in PowerShell, facilitates the download and execution of various malware payloads, including EncryptHub stealer and DarkWisp backdoor, which are designed to steal sensitive information and maintain control over infected systems. Water Gamayun’s operations are characterized by their ability to blend malicious activities with normal system operations, making detection challenging. They have been active since at least April 2024, continuously refining their methods and leveraging legitimate Windows components to execute their payloads stealthily. Microsoft has released security updates to address this vulnerability, and organizations are urged to apply these updates and implement additional security measures to mitigate the threat.
0.715
0.306
Preferred / Actionable
Appendix
Table 7: Metrics vs. human assessment for two Water Gamayun summaries.