SCRIPTIOC-BENCH: A Benchmark for Recognizing Actionable Threat Intelligence from Script-Based Malware using LLMs

Hanna Kim; Jian Cui; Minkyoo Song; Hwanjo Heo; Seungwon Shin; Kimin Lee; Xiaojing Liao

cs.CRSep 5, 2026

SCRIPTIOC-BENCH: A Benchmark for Recognizing Actionable Threat Intelligence from Script-Based Malware using LLMs

Authors: Hanna Kim, Jian Cui, Minkyoo Song, Hwanjo Heo, Seungwon Shin, Kimin Lee, Xiaojing Liao

Organizations: KAIST · University of Illinois Urbana-Champaign · ETRI

Abstract

Script-based malware remains a prevalent attack technique. These scripts often contain indicators of compromise (IOCs) that provide actionable threat intelligence. However, statically recovering such indicators is challenging, as relevant values may be dispersed or transformed within code. Although large language models (LLMs) have shown promise in security analysis, their ability to recover IOCs from malicious scripts remains underexplored. We present SCRIPTIOC-BENCH, a benchmark for measuring static IOC extraction capability on real-world malicious scripts. The benchmark comprises 634 manually verified JavaScript, PowerShell, and VBScript malware samples covering four IOC types (URLs, domains, IP addresses, and filesystem artifacts). We further stratify ground-truth IOCs by recovery level, distinguishing directly exposed indicators from those requiring decoding or reconstruction. Using this benchmark, we evaluate a broad range of proprietary and open-weight LLMs and show that IOC recovery without execution remains challenging across model scales: the strongest model reaches only 65.4 F1. To characterize how recovery fails, we introduce a false-positive taxonomy and use it to compare the error profiles of the evaluated models. We further study two mitigations on a small open-weight model, deterministic string utilities and task-specific adaptation, finding that they provide complementary recovery gains, raise precision, and shift errors toward sample-grounded mismatches.

Figures & tables

Figure 1: Overview of ScriptIOC-bench . Given a task description and a malicious script, an LLM extracts statically obtainable IOCs, which are evaluated against ground-truth IOCs. We analyze model performance along four dimensions corresponding to our research questions: overall performance (RQ1), IOC recovery level (RQ2), input complexity (RQ3), and failure modes (RQ4).

Source	Period	Lang.	# Samples	Dataset
Filescan.io	Nov. 2025–Feb. 2026	JS	97	JS f
	Jan. 2026–Mar. 2026	PS	75	PS f
	Jan. 2026–Mar. 2026	VBS	79	VBS f
Hynek Petrak	2015–2017	JS	383	JS hp

Table 1: Dataset composition by source, collection period, and script language.

Panel of Figure 2 — Figure 2: Distributions of (a) code token length and (b) gzip compression ratio across the benchmark dataset.

Dataset	Type	% Pos.	Inst. (Uniq.)	Avg.	Rec. dist. (L0/L1/L2)
JS f	URL	99.0	110 (104)	1.13	56/3/41
	Domain	57.7	63 (46)	0.65	44/5/51
	IP	18.6	18 (11)	0.19	78/0/22
	File	6.2	10 (10)	0.10	40/30/30
PS f	URL	92.0	147 (140)	1.96	65/13/22
	Domain	64.0	66 (58)	0.88	53/17/30

Table 2: IOC distribution and recovery-level composition across script categories. Columns report prevalence, total and unique IOC counts, average instances per sample, and the share of instances at each recovery level: L0 = Direct, L1 = Fixed-transform, and L2 = Program-dependent.

Model	Size	Model ID	Context
Qwen3-8B [ 45 ]	8.2B	Qwen3-8B	32K
Ministral3-8B [ 36 ]	8B	Ministral-3-8B-Instruct-2512	256K
Gemma4-8B [ 16 ]	8B	Gemma-4-E4B-it	128K
Qwen3-C80B ∗ [ 46 ]	80B	Qwen3-Coder-Next	262K
Qwen3-C480B ∗ [ 44 ]	480B	Qwen3-Coder-480B-A35B-Instruct	262K
Qwen3.6-35B [ 47 ]	35B	Qwen3.6-35B-A3B	262K

Table 3: LLM configurations used for evaluation.

Model	$JS_{f}$				$PS_{f}$				$VBS_{f}$				$JS_{hp}$				Overall
Model	RFail	P	R	F1	RFail	P	R	F1	RFail	P	R	F1	RFail	P	R	F1	RFail	P	R	F1
Qwen3-8B	8.4	38.5	45.3	41.6	17.9	44.4	43.7	44.0	12.9	30.4	27.1	28.7	14.4	28.0	22.0	24.7	13.7	31.3	26.5	28.7
Ministral3-8B	3.8	23.7	36.7	28.7	4.9	36.2	44.1	39.8	30.0	28.4	27.5	27.9	5.1	22.6	30.0	25.8	8.0	24.3	31.6	27.5
Gemma4-8B	10.0	72.4	40.0	51.5	5.8	59.6	12.2	20.2	0.4	48.9	7.3	12.8	6.2	45.4	12.3	19.3	6.0	50.8	13.9	21.8
Qwen3-C80B ∗	11.3	58.4	46.8	51.9	8.0	45.2	54.1	49.3	34.2	44.4	34.3	38.7	36.8	54.9	35.6	43.2	29.2	52.5	38.1	44.2
Qwen3.6-35B	6.8	56.3	49.8	52.8	16.9	63.9	48.7	55.2	17.3	51.5	24.9	33.6	14.4	55.9	41.3	47.6	13.9	56.5	41.0	47.5

Table 4: Main results across models. For each script category and overall, we report the response-failure rate (RFail) and micro-averaged precision (P), recall (R), and F1 over IOC instances. Response failures are counted as empty predictions.

Model	Direct	Fixed-transf.				Program-dep.
		1 (R)	1 (T)	2	$\geq 3$
# IOC instances	(858)	(230)	( 0 85)	(156)	( 0 21)	(1,230)
Qwen3-8B	66.9	27.8	16.5	0 2.6	0 0.0	0 1.5
Ministral3-8B	78.7	33.0	16.5	0 3.8	0 0.0	0 2.8
Qwen3.6-35B	80.2	59.1	37.6	16.0	0 0.0	14.7
Qwen3-C480B	88.2	68.3	63.5	54.5	0 9.5	13.8

Table 5: Micro recall (%) by IOC recovery level. Fixed-transform stratified by recovery-operation count, with single-step cases split into reassembly (R) vs. transcode (T).

Figure 3: Effect of input complexity on LLM-based IOC extraction. Columns are binned by token length or gzip compression ratio. The top row reports Response-failure rate, where darker is worse; the bottom row reports Regex-missed Recall, where darker is better.

Group		Category	Classification criterion	Example	Disp.
\Block 6-1 Sample [-0.2ex]-traceable	\Block 4-1 IOC	Type mismatch	The normalized value equals a reference IOC of a different type.	$G_{ip}$ : 1.2.3.4 $P_{dom}$ : 1.2.3.4
		Normalization error	Malformed, yet deterministically normalizes to a reference IOC.	G: https://a.co P: htps:/a.co
		Granularity	The right entity at a coarser or finer level.	G: a.b.com/x/y P: a.b.com/x
		Near-copy	A distorted copy of a reference IOC: a mis-cut boundary or altered characters.	G: 24sports.ca P: sports.ca
	\Block 2-1 non- IOC	Benign shared service	A benign shared platform in the sample, tied to no reference indicator.	P: ifconfig.me/ip
		Source non-IOC	A source string that is neither an indicator nor a benign service.	P: fileserver

Table 6: False-positive taxonomy, grouped by provenance; the assignment precedence is deterministic (Appendix H ). Disposition (operational handling if the error survives triage): Filterable , Recoverable , Review-inducing , Misleading . G / P denote the reference IOC and prediction.

Figure 4: Per-model false-positive composition within each IOC recovery level. Numbers above bars indicate total false positives.

Figure 5: Recall by IOC recovery level for the variants.

Figure 6: False-positive composition for the variants.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Screenshot of ScriptIOC-bench leaderboard for tracking static IOC extraction performance across LLMs.

Symbol	Value	Description
$\lambda_{f}$	0.5	Format weight
$\lambda_{c}$	1.0	Classification weight
$\lambda_{s}$	2.0	Self-consistency weight
$\lambda_{p}$	4.0	IOC precision weight
$\lambda_{r}$	4.0	IOC recall weight
$w_{\text{both}}$	1.00	Gold $\cap$ silver match weight

Appendix

Table 7: Reward hyperparameters.

Model	Endpoint			Endpoint + keys			Full URL
Model	P	R	F1	P	R	F1 ( $\Delta$ )	P	R	F1 ( $\Delta$ )
Qwen3-8B	33.6	27.9	30.5	31.2	26.1	28.4 ( $-2.1$ )	23.7	23.5	23.6 ( $-6.9$ )
Ministral3-8B	23.2	33.5	27.4	19.3	28.7	23.1 ( $-4.4$ )	13.1	25.7	17.3 ( $-10.1$ )
Gemma4-8B	45.2	12.2	19.2	43.3	11.9	18.7 ( $-0.5$ )	39.5	10.7	16.9 ( $-2.3$ )
Qwen3-C80B ∗	54.2	38.8	45.2	50.4	36.0	42.0 ( $-3.2$ )	44.3	32.4	37.4 ( $-7.8$ )
Qwen3.6-35B	51.6	39.2	44.6	50.5	38.3	43.5 ( $-1.0$ )	47.9	34.2	39.9 ( $-4.7$ )

Appendix

Table 8: URL extraction performance by matching strictness, micro-averaged over IOC instances.

Figure 8: F1 score across models, datasets, and IOC types. Each cell shows the model performance for a specific IOC type. Darker colors indicate higher F1.

Operator	What it does
Reassembly : reorders characters already in the source
Literal concat	Joins adjacent string literals
Const. propagation	Single-assignment substitution of var $\,\leftarrow\,$ literal or prior-variable concat; no control flow, loops, or computed values
Separator strip	Removes the most frequent repeated $n$ -gram, $n\in[3,16]$
Split/join	Rewrites split / join with literal arguments
NUL strip	Removes embedded NUL bytes

Appendix

Table 9: Transform set applied by the fixed decoder. Operators are applied recursively to depth

4

with hash-based de-duplication, a

4

MB per-blob cap, and

\leq 60

matches per operator, yielding candidate decoded blobs per script.

Transform	$n$	GPT-Terra	Gem.-Pro	Gem.-Flash
Reassembly (single)
Literal concat	116	91.4	93.1	91.4
Split/join	83	100.0	85.5	83.1
Constant propagation	24	54.2	79.2	91.7
NUL strip	0 7	100.0	71.4	85.7
Reversal	0 2	100.0	100.0	100.0

Appendix

Table 10: Per-transform recall (%) for the three frontier models over Fixed-transform IOCs.

Mechanism	$n$	GPT-Terra	Gem.-Pro	Gem.-Flash
Multi-variable dataflow	362	30.9	49.7	29.6
Execution-mediated	522	38.5	49.8	38.9
Loop-constructed	150	0 9.3	16.7	0 4.0
Nested encoding / compression	143	51.7	44.1	31.5
Computed-arithmetic / lookup	53	66.0	73.6	37.7
Overall Program-dependent	1230	35.4	46.1	31.0

Appendix

Table 11: Program-dependent recall (%) for the three frontier models over Program-dependent indicators, grouped by dominant construction mechanism (source-pattern heuristics).

	GPT-Terra			Gem.-Pro			Gem.-Flash
Dataset	L0	L1	L2	L0	L1	L2	L0	L1	L2
$JS_{f}$	95.4	33.3	10.7	89.8	44.4	13.1	51.9	22.2	11.9
$PS_{f}$	75.6	58.8	20.0	77.5	50.0	10.0	75.0	47.1	0 6.0
$VBS_{f}$	90.1	47.8	31.4	86.1	54.3	39.7	87.1	52.2	0 6.6
$JS_{hp}$	96.5	86.1	38.9	93.9	83.9	51.6	96.1	75.7	36.9

Appendix

Table 12: Micro recall (%) by recovery level within each dataset. L0 = Direct, L1 = Fixed-transform, L2 = Program-dependent.

Figure 9: Effect of string utilities and task-specific adaptation on IOC extraction (Qwen3-8B variants), by script categories.

Malware Analysis

Malware analysis studies how to detect, classify, attribute, and understand malicious software and its behavior.

See all 58 papers

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps
Apr 21, 2026Alankrit Chona, Igor Kozlov, Ambuj KumarAI Agent Security Benchmarks LLMs for Cybersecurity
Simbian AI
REBENCH: A Procedural, Fair-by-Construction Benchmark for LLMs on Stripped-Binary Types and Names (Extended Version)
Apr 30, 2026Jun Yeon Won, Xin Jin, Shiqing Ma +1LLM Evaluation Software Reverse Engineering
Ohio State University Columbus, OH, United States · University of Massachusetts Amherst Amherst, MA, United States
CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?
Mar 10, 2026Xiangsen Chen, Xuan Feng, Shuo Chen +5LLM Evaluation Cyber Threat Intelligence
Microsoft Research · Hong Kong University of Science and Techonology (Guangzhou) · Microsoft +1

Apr 21, 2026J/K move · Enter open · S save

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

Alankrit Chona, Igor Kozlov, Ambuj Kumar

Simbian AI

AI Agent Security Benchmarks LLMs for Cybersecurity

Open paper AskPDF arXiv

We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of threat hunting: given a database of raw Windows event logs with no guided questions or hints, identify the exact timestamps of malicious events. The benchmark wraps 106 real attack procedures from the OTRF Security-Datasets corpus - spanning 86 MITRE ATT&CK sub-techniques across 12 tactics - into a Gymnasium reinforcement-learning environment. Each episode presents the agent with an in-memory SQLite database of 75,000-135,000 log records produced by a deterministic campaign simulator that time-shifts and entity-obfuscates the raw recordings. The agent must iteratively submit SQL queries to discover malicious event timestamps and explicitly flag them, scored CTF-style against Sigma-rule-derived ground truth. Evaluating five frontier models - Claude Opus 4.6, GPT-5, Gemini 3.1 Pro, Kimi K2.5, and Gemini 3 Flash - on 26 campaigns covering 105 of 106 procedures, we find that all models fail dramatically: the best model (Claude Opus 4.6) submits correct flags for only 3.8% of malicious events on average, and no run across any model ever finds all flags. We define a passing score as >= 50% recall on every ATT&CK tactic - the minimum bar for unsupervised SOC deployment. No model passes: the leader clears this bar on 5 of 13 tactics and the remaining four on zero. These results suggest that current LLMs are poorly suited for open-ended, evidence-driven threat hunting despite strong performance on curated Q&A security benchmarks.

Abstract

Malware Analysis

Explore similar work

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

REBENCH: A Procedural, Fair-by-Construction Benchmark for LLMs on Stripped-Binary Types and Names (Extended Version)

CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?