OrgForge-IT: A Verifiable Synthetic Benchmark for LLM-Based Insider Threat Detection

Jeffrey Flynt

cs.CRMar 23, 2026

OrgForge-IT: A Verifiable Synthetic Benchmark for LLM-Based Insider Threat Detection

Authors: Jeffrey Flynt

Organizations: Independent Researcher

Abstract

Synthetic insider threat benchmarks face a consistency problem: corpora generated without an external factual constraint cannot rule out cross-artifact contradictions. The CERT dataset -- the field's canonical benchmark -- is also static, lacks cross-surface correlation scenarios, and predates the LLM era. We present OrgForge-IT, a verifiable synthetic benchmark in which a deterministic simulation engine maintains ground truth and language models generate only surface prose, making cross-artifact consistency an architectural guarantee. The corpus spans 51 simulated days, 2,904 telemetry records at a 96.4% noise rate, and four detection scenarios designed to defeat single-surface and single-day triage strategies across three threat classes and eight injectable behaviors. A ten-model leaderboard reveals several findings: (1) triage and verdict accuracy dissociate - eight models achieve identical triage F1=0.80 yet split between verdict F1=1.0 and 0.80; (2) baseline false-positive rate is a necessary companion to verdict F1, with models at identical verdict accuracy differing by two orders of magnitude on triage noise; (3) victim attribution in the vishing scenario separates tiers - Tier A models exonerate the compromised account holder while Tier B models detect the attack but misclassify the victim; (4) rigid multi-signal thresholds structurally exclude single-surface negligent insiders, demonstrating the necessity of parallel, threat-class-specific triage pipelines; and (5) agentic software-engineering training acts as a force multiplier for multi-day temporal correlation, but only when paired with frontier-level parameter scale. Finally, prompt sensitivity analysis reveals that unstructured prompts induce vocabulary hallucination, motivating a two-track scoring framework separating prompt adherence from reasoning capability. OrgForge-IT is open source under the MIT license.

Figures & tables

Panel of Figure 1 — Figure 1: Architecture of the physics-cognition boundary. The deterministic Python engine owns all factual simulation state and writes ground truth; language models operate exclusively at designated injection points, generating surface prose from validated proposals. LLMs cannot write to the event log or mutate state directly, making cross-artifact consistency an architectural guarantee rather than an empirical claim.

Property	Value
Simulation days	51
Threat subjects	3
Threat classes	3 (negligent, disgruntled, malicious)
DLP noise ratio	0.40
Observable records	2,904
True positive records	106

Table 1: OrgForge-IT corpus statistics.

Name	Class	Onset Day	Behaviors
Jordan	negligent	5	secret_in_commit
Tasha	disgruntled	10	sentiment_drift , cross_dept_snooping , unusual_hours_access
Jax	malicious	18	data_exfil_email , excessive_repo_cloning , unusual_hours_access , sentiment_drift , host_data_hoarding , social_engineering , idp_anomaly

Table 2: Threat subject configuration for the benchmark corpus.

Panel of Figure 2 — Figure 2: The sliding window problem illustrated by the three-phase host data hoarding scenario. Each phase falls below the two-signal escalation threshold when examined in isolation. A hoarding_trail_start_day breadcrumb links phase 3 back to phase 1, but a triage agent processing only the current window’s events treats phase 3 as an isolated archive move rather than the culmination of a multi-day exfiltration sequence.

Panel of Figure 3 — Figure 3: Three-stage evaluation pipeline. Stage 1 establishes a clean-data false-positive rate using pre-onset baseline records. Stage 2 applies a 7-day sliding window with a two-signal escalation threshold. Stage 3 receives only escalated suspects and produces a structured JSON verdict with per-artifact evidence citations. Strong Stage 2 performance does not predict Stage 3 performance.

Model	Triage F 1	Verdict F 1	Base FP	V.Prec	V.Rec	Vish.	Trail
Tier A — Verdict F 1 = 1.0, clean baseline
Devstral 2 123B	0.800	1.000	0.021	1.000	1.000	✓	✓
Claude Opus 4.6	0.800	1.000	0.021	1.000	1.000	✓	✓
Tier B — Verdict F 1 = 0.80, operationally differentiated by baseline FP rate
DeepSeek v3.2	0.800	0.800	0.021	0.667	1.000	✓	✓
Mistral Large 675B	0.800	0.800	0.021	0.667	1.000	✓	✓

Table 3: Ten-model leaderboard. All runs use the same 51-day corpus with 3 threat subjects ( subjects=3 , sim_days=51 ). Triage F 1 and verdict F 1 are evaluated independently. Baseline FP rate measures false positives on the clean pre-onset period; lower is better. Bold marks the best value per column. Models are grouped by verdict tier.

\dagger

Llama 4 Maverick achieves Tier B verdict F 1 but with elevated triage noise relative to other Tier B models.

\ddagger

Llama 3.3 70B verdict F

{}_{1}=0.80

is operationally disqualified by its 0.813 baseline FP rate (39 false escalations).

Panel of Figure 4 — Figure 4: Triage F 1 vs. verdict F 1 for all ten models. Bubble area is proportional to baseline false-positive rate (larger = more triage noise). Eight models cluster at triage F ${}_{1}=0.80$ yet split between verdict F ${}_{1}=1.0$ (Tier A) and F ${}_{1}=0.80$ (Tier B), illustrating the triage/verdict dissociation. Within Tier B, Llama 3.3 70B’s bubble (FP = 0.813, red outline) dominates the chart relative to the tight clean-baseline cluster (FP $\leq$ 0.023), making the operational cost of equal verdict accuracy immediately visible. Llama 3.3 70B is operationally disqualified despite matching Tier B verdict F 1 .

Panel of Figure 5 — Figure 5: Day 19 timeline showing two simultaneous sessions on Chris’s account. The anomalous session (macOS, TOTP, residential IP) is initiated 17 minutes after Jax’s vishing call and is inconsistent with Chris’s established device profile (iOS, push notification, corporate IP). Tier A models reason that mutually inconsistent concurrent sessions identify attacker access rather than account-holder behavior, returning innocent for Chris. Tier B models flag the anomalous session as evidence against Chris, producing the false positive verdict that holds all eight at precision 0.667.

Behavior	Devstral	Opus	DeepSeek	ML	GLM-5	S46	H45	Q3C	L4M	L33
unusual_hours_access	2/0	2/0	2/0	2/0	2/0	2/0	2/0	2/0	2/0	2/0
sentiment_drift	2/0	2/0	2/0	2/0	1/0	2/0	2/0	1/0	2/0	2/0
host_data_hoarding	1/0	1/0	1/0	1/0	1/0	1/0	1/0	1/0	1/0	1/0
data_exfil_email	1/0	1/0	1/0	1/0	1/0	1/0	1/0	1/0	1/0	1/0
social_engineering	1/0	1/0	1/0	1/0	1/0	1/0	1/0	1/0	1/0	1/0
idp_anomaly	1/1	1/1	1/1	1/1	1/1	1/1	1/1	1/0	—	1/0

Table 4: Per-behavior detection results (TP/FP) for all ten models. Cells show true / false positive counts; — indicates the behavior was not cited by that model. L33 = Llama 3.3 70B; L4M = Llama 4 Maverick; ML = Mistral Large 675B; Q3C = Qwen3-Coder; S46 = Claude Sonnet 4.6; H45 = Claude Haiku 4.5.

Model	Variant	Verdict F 1	Base FP	V.Prec	Vish.	Trail
Tier A
Devstral 2 123B	official	1.000	0.021	1.000	✓	✓
Devstral 2 123B	v2_natural	1.000	0.021	1.000	✓	✗
Claude Opus 4.6	official	1.000	0.021	1.000	✓	✓
Claude Opus 4.6	v2_natural	1.000	0.021	1.000	✓ $\dagger$	✗
Tier B

Table 5: Prompt sensitivity results. Official prompt scores are reproduced from Table 3 for reference. Sensitivity runs are not on the leaderboard. Vish. = vishing detected; Trail = host trail reconstructed.

\dagger

Scorer returns Vish. = no; verdict text explicitly names Chris as victim_of_social_engineering and cites anomalous_auth_from_new_device_after_vishing_call — a scorer artifact, not a capability failure (see text).

LLMs for Cybersecurity

Large language models for cybersecurity support threat analysis, vulnerability research, security operations, and cyber defense.

See all 255 papers

LLMs for Zero-Shot Threat Detection via Structured Risk Indicators
Aug 17, 2026Abdullah Alghamdi, Siamak Layeghy, Marius PortmannAnomaly Detection
University of Queensland, Brisbane, Australia
Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps
Apr 21, 2026Alankrit Chona, Igor Kozlov, Ambuj KumarAI Agent Security Benchmarks LLMs for Cybersecurity
Simbian AI
CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?
Mar 10, 2026Xiangsen Chen, Xuan Feng, Shuo Chen +5LLM Evaluation Cyber Threat Intelligence
Microsoft Research · Hong Kong University of Science and Techonology (Guangzhou) · Microsoft +1

Aug 17, 2026J/K move · Enter open · S save

LLMs for Zero-Shot Threat Detection via Structured Risk Indicators

Abdullah Alghamdi, Siamak Layeghy, Marius Portmann

University of Queensland, Brisbane, Australia

Anomaly Detection

Open paper AskPDF arXiv

We propose a two-stage large language model (LLM) framework for zero-shot detection of insider threats and advanced persistent threats (APTs) from heterogeneous security logs. The framework models user activity as chronological timelines and incorporates retrieval-augmented generation (RAG) to provide personalised behavioural context from each user's historical activity. Rather than performing end-to-end classification directly from raw logs, it first generates structured, interpretable sets of threat-specific risk indicators, which are then classified jointly across temporal sequences to capture attack patterns spanning multiple windows.The framework is evaluated on two benchmark datasets, CERT r5.2 for insider threat detection and PicoDomain for APT detection, using four combinations of two open-weight LLMs under both retrieval and non-retrieval settings. All configurations outperform the previous state-of-the-art LLM-based framework (GABM), with the best configuration improving the F1-score by 11.40 percentage points on CERT r5.2 and 31.50 percentage points on PicoDomain. Results further show that retrieval mainly benefits weaker LLMs by generating more discriminative risk indicators, whereas stronger models achieve comparable performance without retrieved context. The most effective assignment of LLMs to the two stages depends on the dataset. These findings show that the quality of the generated risk indicators is the main driver of zero-shot cyber threat detection performance.

Abstract

LLMs for Cybersecurity

Explore similar work

LLMs for Zero-Shot Threat Detection via Structured Risk Indicators

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?