Ranked by the Matcher: A Reproducibility Audit of Knowledge Graph Extraction from Threat Reports

Safayat Bin Hakim; Houbing Herbert Song

cs.CRSep 1, 2026

Ranked by the Matcher: A Reproducibility Audit of Knowledge Graph Extraction from Threat Reports

Authors: Safayat Bin Hakim, Houbing Herbert Song

Organizations: University of Maryland, Baltimore County Baltimore, Maryland, USA

Abstract

Security teams and researchers choose knowledge-graph extraction tooling for threat reports on the strength of published triple-F1 scores, yet those scores depend on how predicted triples are matched to gold annotations. We could reimplement the stated matching rule for only five of twelve inspected systems. Re-scoring ten system outputs on shared documents under eight protocols reverses eleven of forty-five pairwise orderings; one fixed prediction set spans 0.16--0.70 F1. On an external, human-adjudicated set, no mechanical matcher---lexical, embedding, or entailment---agrees with the reviewers more than seven times in ten; an LLM judge agrees far more often. To separate component effects from matcher rewards, we build CTIForge, whose deterministic validation layer can vary while extraction is held byte-identical. Across seven tested deployment configurations, no hosted backbone loses precision under validation and every offline one does. Backbone, decoding, and prompting covary across those configurations; on the one backbone we could serve both ways, serving alone reproduces the split. It coincides with a roughly 2.8-fold increase in actions explicitly disputing entity type, consistent with hand-written rules encoding the conventions of the extractor against which they were developed. We release the pipeline, protocol suite, and per-triple audit records.

Figures & tables

Fig. 1 : CTIForge pipeline architecture. Extraction is a single LLM call per chunk; every stage after it is deterministic, which is what allows all ablation arms to be derived from one extraction pass.

System	Reported matching criterion	Reprod.
Rule specified precisely enough to reimplement
CTIBench [ 6 ]	exact match on named entities	✓
GRID [ 5 ]	explicit rule set, LLM judge	✓
TACTIC-KG [ 41 ]	embedding similarity, $T{=}0.6$	✓
ANCHOR [ 40 ]	exact match plus partial credit	✓
Fieblinger et al. [ 3 ]	rouge $n$ -gram overlap	✓

TABLE I : Matching criteria reported by twelve CTI extraction systems and benchmarks. ✓ marks a rule reimplementable from the publication;

\times

marks one that is not. Shaded rows are systems also re-scored under eight protocols in Section VI-A .

Fig. 2 : Pairwise orderings among ten published systems under eight matching protocols. Each cell counts the protocols in which the row system outranks the column system; shading diverges from the four-four split at which an ordering is unstable. Outlined cells are the eleven pairs whose ordering is not stable across protocols.

Panel of Figure 4 — Fig. 3 : The matching protocol as a free parameter. No mechanical protocol reaches the LLM judge, whose figure is the rate at which reviewers agreed with it in [ 5 ] rather than an independent re-measurement, and the choice among them moves reported F1 by more than half a point.

Metric	Precision	Recall	F1
Triple (S-R-O)	0.5331	0.4282	0.4749
S-O pair (relation-free)	0.5959	0.4786	0.5309
Entity type classification	0.5479 accuracy / 0.5082 micro-F1
Predicted triples	1,465
Processing time	1,586.2 s total / 10.6 s per document

TABLE II : Full-pipeline results on the CTI-Nexus benchmark (149 reports, 1,824 gold triples). Triple matching compares subject, relation, and object under the three-pass protocol of Appendix E ; entity types are not compared. S-O pair ignores the relation.

Fig. 4 : Validation-induced precision change against the share of actions disputing entity type, one point per backbone. The dashed line marks no change. Hosted bars are

\pm

1 SD of

\Delta

P over five independent extraction passes on MiniMax-M3; the same interval is drawn on the other three hosted points, for which repeated passes were not run, and the two smallest cross zero. The offline backbones decode greedily and reproduce byte-identically, so they carry no sampling noise. Deployment groups are descriptive because backbone, decoding, and serving stack covary.

Backbone	Serving	B F1	B+C F1	$\Delta$ P	Collat.	Type
MiniMax-M3	hosted	0.4440	0.4371	$+$ 0.0125	31%	12%
GPT-4o	hosted	0.5076	0.4907	$+$ 0.0078	46%	18%
Haiku 4.5	hosted	0.4418	0.4255	$+$ 0.0024	43%	11%
Sonnet 4.5	hosted	0.4239	0.4075	$+$ 0.0002	47%	9%
Qwen2.5-7B	local	0.4104	0.3706	$-$ 0.0102	55%	37%
Gemma-2-9B	local	0.4305	0.3834	$-$ 0.0226	69%	35%

TABLE III : Module ablation over seven backbones. Within each backbone, all arms derive from one extraction pass. B is extraction alone; C adds validation and D canonicalization.

\Delta

P is B+C minus B; collateral is the share of removals that discard a correct triple, and type is the share of actions disputing entity type.

Category	CTI-Nexus (149 docs)	CTIKG (255 sent.)
Confidence lowered	716 (67.0%)	173 (31.5%)
Impossible type-pair	104 (9.7%)	72 (13.1%)
Repaired identifier	104 (9.7%)	23 (4.2%)
direction swap	58	12
identifier/relation	46	11
Type misassignment	75 (7.0%)	162 (29.5%)

TABLE IV : Error taxonomy distribution. Counts are validator actions, not triples; a single triple may generate more than one action. Seven of the fourteen declared categories fire at benchmark scale; the remainder are listed in Appendix Table G2 .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Entity type	Description
ThreatActor	Groups, APTs, nation-state actors (e.g., APT29, Lazarus Group)
Malware	Malicious software families (e.g., LockBit, Emotet)
Tool	Legitimate software misused in attacks (e.g., Cobalt Strike, Mimikatz)
Campaign	Named attack campaigns (e.g., Operation Cobalt Kitty)
Vulnerability	CVEs and security flaws (e.g., CVE-2023-4966)
Technique	ATT&CK techniques (e.g., T1566 Phishing)

Appendix

TABLE B1 : CTIForge entity types.

Relation type	Semantic scope
uses	Actor/malware uses tool, technique, or infrastructure
targets	Actor/campaign targets organization, sector, or location
exploits	Actor/malware exploits vulnerability or software
delivers	Campaign/malware delivers payload
communicates_with	Malware/tool communicates with IOC or C2
drops	Malware drops file or secondary payload

Appendix

TABLE B2 : CTIForge relation types.

Relation	Valid subject types	Valid object types
uses	ThreatActor, Malware, Campaign, Tool	Malware, Tool, Technique, Infrastructure, IOC, Software, File
targets	ThreatActor, Malware, Campaign, Tool, Vulnerability	Organization, Location, Software, Infrastructure, IOC, File, Vulnerability, Other
exploits	ThreatActor, Malware, Campaign, Tool, Vulnerability	Vulnerability, Software, Infrastructure
delivers	ThreatActor, Malware, Campaign	Malware, File, Tool
communicates_with	Malware, Tool	IOC, Infrastructure
drops	Malware, ThreatActor, Campaign	Malware, File, Tool

Appendix

TABLE C1 : Type-pair constraint table.

Panel of Figure 12 — Fig. D1 : Skeleton of the extraction prompt ( prompts/extraction.jinja ). Grey italics summarize blocks elided for space; rules are numbered as in the source.

Backbone	Arm	P	R	Triplet F1	S-O F1	Triples
MiniMax-M3	B	0.4598	0.4293	0.4440	0.4928	1,703
	B+C	0.4723	0.4068	0.4371	0.4819	1,571
GPT-4o	B	0.5431	0.4764	0.5076	0.5485	1,600
	B+C	0.5509	0.4424	0.4907	0.5448	1,465
Haiku 4.5	B	0.4628	0.4227	0.4418	0.4865	1,666
	B+C	0.4652	0.3920	0.4255	0.4755	1,537

Appendix

TABLE F1 : Full module ablation across all seven extraction backbones, hosted above the rule and offline below it. Each backbone’s arms derive from one shared extraction pass; B is extraction alone, C adds validation and D canonicalization. B+C+D is omitted: it is identical to B+C on every backbone to four decimals, which is the subject of Finding 7. Triples counts predictions emitted. Extraction for this table is a separate pass from Table II , so absolute values differ from it by run-to-run sampling; arms within a backbone share one pass and are exactly comparable.

Fig. F1 : Per-document triplet F1 for seven backbones, colored by deployment configuration. Within-backbone spread exceeds the gap between group means, and the groups overlap substantially.

Resource	Format	Scale	Role
CTI-Nexus annotations	JSON	149 reports	Primary document-level benchmark (1,824 gold triples)
CTIKG benchmark	CSV	255 sent.	Secondary sentence-level benchmark (693 gold triples)
MITRE ATT&CK	JSONL	1,076 rec.	Technique grounding and alias lookup
CVE database	JSONL	3,011 rec.	Vulnerability identifier validation
CWE/CAPEC	JSONL	1,957 rec.	Weakness and attack pattern reference
CTI blog corpus	JSONL	219 articles	Few-shot example retrieval

Appendix

TABLE G1 : Datasets and resources.

Category	Severity	Action	Description
empty_field	critical	reject	Empty subject, object, or relation
impossible_type_pair	critical	reject	Violates type-pair constraints
self_loop	critical	reject	Subject equals object
hallucinated_entity	high	reject	Entity absent from evidence
malformed_identifier	high	repair/reject	Malformed CVE, hash, or ID
type_misassignment	high	repair	Entity type inconsistent with name

Appendix

TABLE G2 : Validator taxonomy by category, severity, and default action. Shaded categories are rejected rather than flagged; names match the identifiers in error_taxonomy/error_log.jsonl .

Parameter	Value
Primary LLM (149-doc run)	GPT-4o
Same-backbone head-to-head	GPT-4o via OpenRouter
Development slice	Mistral Small 3.1 24B via OpenRouter
Temperature	0.1
Max tokens	4,096
Retry attempts	3

Appendix

TABLE G3 : Impact of configuration on the reported operating point: full hyperparameter set.

Category	CTI-Nexus	CTIKG	Default action
Confidence lowered	716	173	flag
Impossible type-pair	104	72	reject
Repaired identifier	104	23	repair
Type misassignment	75	162	flag
Generic placeholder	38	96	flag
Self-loop	17	12	reject

Appendix

TABLE H1 : Impact of each validation rule at benchmark scale: logged actions by category and outcome.

System	Precision	Recall	F1	Predicted
Uniform soft-match evaluation (both systems, same matcher)
CTIKG	0.5122	0.6955	0.5900	941
CTIForge (ours)	0.5722	0.4459	0.5012	540
Strict triple match (both systems, same matcher)
CTIKG	0.3560	0.4834	0.4100	941
CTIForge (ours)	0.0648	0.0505	0.0568	540

Appendix

TABLE I1 : Head-to-head comparison on the CTIKG benchmark (255 sentences, 693 gold triples) under both a relaxed S-O pair match and a strict triple match. The strict rows show the cost of mapping free-text gold into 12 relations, which is why the relaxed match is the headline here. Numbers in the “uniform” rows use the same matcher applied to both systems’ outputs; the “CTIKG (reported)” row restates the numbers published in the CTIKG paper, which use a different evaluation protocol.

Metric (scored against CTIForge’s constraint table)	CTIForge	CTI-Nexus
Relation specificity	0.809	0.532
Entity-type specificity	0.980	0.795
Semantic validity	0.837	0.560
Schema constraint compliance	0.854	0.585
Evidence alignment rate	0.462	0.000
Cross-document entity fusion	0.107	0.085

Appendix

TABLE I2 : Impact of scoring against the system’s own schema (definitional; not a cross-system comparison).

Gold-matched (per-document scoping)
Metric	CTIForge	CTI-Nexus	Winner
Triplet F1	0.4877	0.5449	CTI-Nexus
Triplet precision	0.5536	0.5100	CTIForge
Triplet recall	0.4359	0.5850	CTI-Nexus
Documents won	57	85	CTI-Nexus
Schema-independent (comparable across systems)

Appendix

TABLE J1 : Matched-backbone comparison on 149 documents using GPT-4o via OpenRouter. Gold-matched metrics depend on the stated matcher; schema-independent metrics do not use either system’s ontology. Constraint-table metrics appear separately in Appendix I . Winners are bold .

Serving	Decoding	B P	B $+$ C P	$\Delta$ P	Rem.	Collat.
local	greedy	0.4788	0.4686	$-$ 0.0102	175	54.9%
local	sampled	0.4913	0.4783	$-$ 0.0130	163	58.9%
hosted	greedy	0.4562	0.4703	$+$ 0.0141	294	42.2%
hosted	sampled	0.4496	0.4552	$+$ 0.0056	289	43.6%

Appendix

TABLE M1 : Qwen2.5-7B under crossed serving and decoding. Removed is B

+

C’s reduction in emitted triples; collateral is the share of those removals that discard a correct triple. Both local arms lose precision and both hosted arms gain it.

Knowledge Graphs

Knowledge graphs represent entities and relationships as structured, queryable networks for integrating data and supporting reasoning.

See all 232 papers

Context-aware Entity-Relation Extraction for Threat Intelligence Knowledge Graphs
May 15, 2026Inoussa Mouiche, sherif SaadRelation Extraction KG Construction
School of Computer Science, University of Windsor, ON, Canada
TRACE-CTI: Auditable Post-Extraction Governance of TTP Claims with Knowledge Graphs
Jul 27, 2026Federico Valletta, Giacomo Longo, Enrico Russo +1KG Construction AI Assurance
CASD – School of Advanced Defense Studies, Piazza della Rovere 83, 00165 Rome, Italy
GRID: Graph Representation of Intelligence Data for Security Text Knowledge Graph Construction
May 15, 2026Liangyi Huang, Zichen Liu, Fei Shao +5KG Construction Cyber Threat Intelligence
Arizona State University · Case Western Reserve University · University of Notre Dame +2

May 15, 2026J/K move · Enter open · S save

Context-aware Entity-Relation Extraction for Threat Intelligence Knowledge Graphs

Inoussa Mouiche, sherif Saad

School of Computer Science, University of Windsor, ON, Canada

Relation Extraction KG Construction Named Entity Recognition Cyber Threat Intelligence Cybersecurity

Open paper AskPDF arXiv

Cybersecurity Knowledge Graphs (CKGs) unify diverse Cyber Threat Intelligence (CTI) sources into structured, queryable formats, offering scalable solutions for automating proactive and real-time security responses. Their increasing adoption has significantly enhanced the workflow and decision-making efficiency of security professionals. However, constructing CKGs requires extracting entity-relation triples from unstructured CTI reports, a task hindered by complex report structure, domain-specific language, and semantic ambiguity. As a result, existing pipeline-based approaches often suffer from error propagation, reducing extraction accuracy and limiting generalizability. This paper introduces the Context-aware Threat Intelligence Knowledge Graph (CTiKG) framework, a pipeline architecture designed to accurately extract and classify threat entities and their relationships from CTI reports. CTiKG incorporates hybrid NLP models that leverage SecureBERT+ contextual embeddings and expert knowledge from a domain ontology to reduce misclassifications and mitigate cascading errors. Experiments on the DNRTI-AUG-STIX2 dataset, which comprises 21 entity types aligned with STIX 2.1, demonstrate significant improvements over state-of-the-art baselines, yielding 3-4% gains in NER and up to 8% in RE performance, based on precision, recall, and F1-score. Additional validation on DNRTI and STUCCO benchmarks confirms the framework's robustness and practical applicability. All datasets, including the curated DNRTI-AUG-STIX2, are released on GitHub to foster reproducibility and further research.

Abstract

Knowledge Graphs

Explore similar work

Context-aware Entity-Relation Extraction for Threat Intelligence Knowledge Graphs

TRACE-CTI: Auditable Post-Extraction Governance of TTP Claims with Knowledge Graphs

GRID: Graph Representation of Intelligence Data for Security Text Knowledge Graph Construction