Synthetic insider threat benchmarks face a consistency problem: corpora generated without an external factual constraint cannot rule out cross-artifact contradictions. The CERT dataset -- the field's canonical benchmark -- is also static, lacks cross-surface correlation scenarios, and predates the LLM era. We present OrgForge-IT, a verifiable synthetic benchmark in which a deterministic simulation engine maintains ground truth and language models generate only surface prose, making cross-artifact consistency an architectural guarantee. The corpus spans 51 simulated days, 2,904 telemetry records at a 96.4% noise rate, and four detection scenarios designed to defeat single-surface and single-day triage strategies across three threat classes and eight injectable behaviors. A ten-model leaderboard reveals several findings: (1) triage and verdict accuracy dissociate - eight models achieve identical triage F1=0.80 yet split between verdict F1=1.0 and 0.80; (2) baseline false-positive rate is a necessary companion to verdict F1, with models at identical verdict accuracy differing by two orders of magnitude on triage noise; (3) victim attribution in the vishing scenario separates tiers - Tier A models exonerate the compromised account holder while Tier B models detect the attack but misclassify the victim; (4) rigid multi-signal thresholds structurally exclude single-surface negligent insiders, demonstrating the necessity of parallel, threat-class-specific triage pipelines; and (5) agentic software-engineering training acts as a force multiplier for multi-day temporal correlation, but only when paired with frontier-level parameter scale. Finally, prompt sensitivity analysis reveals that unstructured prompts induce vocabulary hallucination, motivating a two-track scoring framework separating prompt adherence from reasoning capability. OrgForge-IT is open source under the MIT license.
Figures & tables
Figure 1: Architecture of the physics-cognition boundary. The deterministic Python engine owns all factual simulation state and writes ground truth; language models operate exclusively at designated injection points, generating surface prose from validated proposals. LLMs cannot write to the event log or mutate state directly, making cross-artifact consistency an architectural guarantee rather than an empirical claim.
Table 2: Threat subject configuration for the benchmark corpus.
Figure 2: The sliding window problem illustrated by the three-phase host data hoarding scenario. Each phase falls below the two-signal escalation threshold when examined in isolation. A hoarding_trail_start_day breadcrumb links phase 3 back to phase 1, but a triage agent processing only the current window’s events treats phase 3 as an isolated archive move rather than the culmination of a multi-day exfiltration sequence.
Figure 3: Three-stage evaluation pipeline. Stage 1 establishes a clean-data false-positive rate using pre-onset baseline records. Stage 2 applies a 7-day sliding window with a two-signal escalation threshold. Stage 3 receives only escalated suspects and produces a structured JSON verdict with per-artifact evidence citations. Strong Stage 2 performance does not predict Stage 3 performance.
Model
Triage F 1
Verdict F 1
Base FP
V.Prec
V.Rec
Vish.
Trail
Tier A — Verdict F 1 = 1.0, clean baseline
Devstral 2 123B
0.800
1.000
0.021
1.000
1.000
✓
✓
Claude Opus 4.6
0.800
1.000
0.021
1.000
1.000
✓
✓
Tier B — Verdict F 1 = 0.80, operationally differentiated by baseline FP rate
DeepSeek v3.2
0.800
0.800
0.021
0.667
1.000
✓
✓
Mistral Large 675B
0.800
0.800
0.021
0.667
1.000
✓
✓
Table 3: Ten-model leaderboard. All runs use the same 51-day corpus with 3 threat subjects ( subjects=3 , sim_days=51 ). Triage F 1 and verdict F 1 are evaluated independently. Baseline FP rate measures false positives on the clean pre-onset period; lower is better. Bold marks the best value per column. Models are grouped by verdict tier. † Llama 4 Maverick achieves Tier B verdict F 1 but with elevated triage noise relative to other Tier B models. ‡ Llama 3.3 70B verdict F 1=0.80 is operationally disqualified by its 0.813 baseline FP rate (39 false escalations).
Figure 4: Triage F 1 vs. verdict F 1 for all ten models. Bubble area is proportional to baseline false-positive rate (larger = more triage noise). Eight models cluster at triage F 1=0.80 yet split between verdict F 1=1.0 (Tier A) and F 1=0.80 (Tier B), illustrating the triage/verdict dissociation. Within Tier B, Llama 3.3 70B’s bubble (FP = 0.813, red outline) dominates the chart relative to the tight clean-baseline cluster (FP ≤ 0.023), making the operational cost of equal verdict accuracy immediately visible. Llama 3.3 70B is operationally disqualified despite matching Tier B verdict F 1 .
Figure 5: Day 19 timeline showing two simultaneous sessions on Chris’s account. The anomalous session (macOS, TOTP, residential IP) is initiated 17 minutes after Jax’s vishing call and is inconsistent with Chris’s established device profile (iOS, push notification, corporate IP). Tier A models reason that mutually inconsistent concurrent sessions identify attacker access rather than account-holder behavior, returning innocent for Chris. Tier B models flag the anomalous session as evidence against Chris, producing the false positive verdict that holds all eight at precision 0.667.
Behavior
Devstral
Opus
DeepSeek
ML
GLM-5
S46
H45
Q3C
L4M
L33
unusual_hours_access
2/0
2/0
2/0
2/0
2/0
2/0
2/0
2/0
2/0
2/0
sentiment_drift
2/0
2/0
2/0
2/0
1/0
2/0
2/0
1/0
2/0
2/0
host_data_hoarding
1/0
1/0
1/0
1/0
1/0
1/0
1/0
1/0
1/0
1/0
data_exfil_email
1/0
1/0
1/0
1/0
1/0
1/0
1/0
1/0
1/0
1/0
social_engineering
1/0
1/0
1/0
1/0
1/0
1/0
1/0
1/0
1/0
1/0
idp_anomaly
1/1
1/1
1/1
1/1
1/1
1/1
1/1
1/0
—
1/0
Table 4: Per-behavior detection results (TP/FP) for all ten models. Cells show true / false positive counts; — indicates the behavior was not cited by that model. L33 = Llama 3.3 70B; L4M = Llama 4 Maverick; ML = Mistral Large 675B; Q3C = Qwen3-Coder; S46 = Claude Sonnet 4.6; H45 = Claude Haiku 4.5.
Model
Variant
Verdict F 1
Base FP
V.Prec
Vish.
Trail
Tier A
Devstral 2 123B
official
1.000
0.021
1.000
✓
✓
Devstral 2 123B
v2_natural
1.000
0.021
1.000
✓
✗
Claude Opus 4.6
official
1.000
0.021
1.000
✓
✓
Claude Opus 4.6
v2_natural
1.000
0.021
1.000
✓ †
✗
Tier B
Table 5: Prompt sensitivity results. Official prompt scores are reproduced from Table 3 for reference. Sensitivity runs are not on the leaderboard. Vish. = vishing detected; Trail = host trail reconstructed. † Scorer returns Vish. = no; verdict text explicitly names Chris as victim_of_social_engineering and cites anomalous_auth_from_new_device_after_vishing_call — a scorer artifact, not a capability failure (see text).
We propose a two-stage large language model (LLM) framework for zero-shot detection of insider threats and advanced persistent threats (APTs) from heterogeneous security logs. The framework models user activity as chronological timelines and incorporates retrieval-augmented generation (RAG) to provide personalised behavioural context from each user's historical activity. Rather than performing end-to-end classification directly from raw logs, it first generates structured, interpretable sets of threat-specific risk indicators, which are then classified jointly across temporal sequences to capture attack patterns spanning multiple windows.The framework is evaluated on two benchmark datasets, CERT r5.2 for insider threat detection and PicoDomain for APT detection, using four combinations of two open-weight LLMs under both retrieval and non-retrieval settings. All configurations outperform the previous state-of-the-art LLM-based framework (GABM), with the best configuration improving the F1-score by 11.40 percentage points on CERT r5.2 and 31.50 percentage points on PicoDomain. Results further show that retrieval mainly benefits weaker LLMs by generating more discriminative risk indicators, whereas stronger models achieve comparable performance without retrieved context. The most effective assignment of LLMs to the two stages depends on the dataset. These findings show that the quality of the generated risk indicators is the main driver of zero-shot cyber threat detection performance.
Abdullah Alghamdi, Siamak Layeghy, Marius Portmann
We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of threat hunting: given a database of raw Windows event logs with no guided questions or hints, identify the exact timestamps of malicious events. The benchmark wraps 106 real attack procedures from the OTRF Security-Datasets corpus - spanning 86 MITRE ATT&CK sub-techniques across 12 tactics - into a Gymnasium reinforcement-learning environment. Each episode presents the agent with an in-memory SQLite database of 75,000-135,000 log records produced by a deterministic campaign simulator that time-shifts and entity-obfuscates the raw recordings. The agent must iteratively submit SQL queries to discover malicious event timestamps and explicitly flag them, scored CTF-style against Sigma-rule-derived ground truth. Evaluating five frontier models - Claude Opus 4.6, GPT-5, Gemini 3.1 Pro, Kimi K2.5, and Gemini 3 Flash - on 26 campaigns covering 105 of 106 procedures, we find that all models fail dramatically: the best model (Claude Opus 4.6) submits correct flags for only 3.8% of malicious events on average, and no run across any model ever finds all flags. We define a passing score as >= 50% recall on every ATT&CK tactic - the minimum bar for unsupervised SOC deployment. No model passes: the leader clears this bar on 5 of 13 tactics and the remaining four on zero. These results suggest that current LLMs are poorly suited for open-ended, evidence-driven threat hunting despite strong performance on curated Q&A security benchmarks.
Analyzing Open Source Intelligence (OSINT) from large volumes of data is critical for drafting and publishing comprehensive CTI reports. This process usually follows a three-stage workflow -- triage, deep search and TI drafting. While Large Language Models (LLMs) offer a promising route toward automation, existing benchmarks still have limitations. These benchmarks often consist of tasks that do not reflect real-world analyst workflows. For example, human analysts rarely receive tasks in the form of multiple-choice questions. Also, existing benchmarks often rely on model-centric metrics that emphasize lexical overlap rather than actionable, detailed insights essential for security analysts. Moreover, they typically fail to cover the complete three-stage workflow. To address these issues, we introduce CyberThreat-Eval, which is collected from the daily CTI workflow of a world-leading company. This expert-annotated benchmark assesses LLMs on practical tasks across all three stages as mentioned above. It utilizes analyst-centric metrics that measure factual accuracy, content quality, and operational costs. Our evaluation using this benchmark reveals important insights into the limitations of current LLMs. For example, LLMs often lack the nuanced expertise required to handle complex details and struggle to distinguish between correct and incorrect information. To address these challenges, the CTI workflow incorporates both external ground-truth databases and human expert knowledge. TRA allows human experts to iteratively provide feedback for continuous improvement. The code is available at GitHub and HuggingFace.
Xiangsen Chen, Xuan Feng, Shuo Chen +5
Microsoft Research · Hong Kong University of Science and Techonology (Guangzhou) · Microsoft +1