ARBITRA: Robust Document-Level ATT&CK Extraction via Multi-Expert Argumentation

Youngjin Jin; Seungwon Shin

cs.CLOct 29, 2026

ARBITRA: Robust Document-Level ATT&CK Extraction via Multi-Expert Argumentation

Authors: Youngjin Jin, Seungwon Shin

Organizations: School of Electrical Engineering, KAIST

Abstract

Cyber threat intelligence (CTI) reports describe adversary behavior in natural language, but converting them into structured MITRE ATT&CK mappings remains difficult: existing methods decompose reports to sentence-level granularity for extraction, while modern LLMs can work with full reports in a single pass but with noticeable performance degradation. We present ARBITRA, which orchestrates multiple role-specialized LLM experts that propose candidate ATT&CK entities from the full report, which are then verified through a per-candidate quantitative bipolar argumentation framework (QBAF) grounded in document evidence and rationales. We additionally construct reasoning-specialized corpora for fine-tuning so experts can act as effective participants during this verification process. Across ATT&CK extraction benchmarks, ARBITRA reaches a stronger precision-oriented operating point than sentence-level pipelines, recent LLM-based extraction systems, and frontier models including GPT-5.5. Under adversarial perturbations simulating conflicting TTP statements, cross-report contamination, and semantic poisoning, ARBITRA limits mean precision degradation to roughly half that of frontier-model baselines for a more robust extraction.

Figures & tables

Figure 1: Overview of ARBITRA as described in Section 4 .

System	Setting	TRAM2 †			AnnoCTR ‡
		P	R	F1	P	R	F1
Sentence-level baseline
Full NER pipeline [ 3 ]	Sentence-level extraction	0.654	0.578	0.614	0.435	0.709	0.539
Document-level LLM baselines
GPT-4o-mini	Document-level extraction	0.317	0.322	0.299	0.293	0.256	0.263
GPT-4.1-mini	Document-level extraction	0.292	0.392	0.319	0.340	0.432	0.362

Table 1: Document-level extraction results on TRAM2 and AnnoCTR ( † one GPT-4.1-mini and two GPT-4o-mini experts; ‡ three GPT-4.1-mini experts). GPT-5.4 and GPT-5.5 are document-level baselines only (not used as ARBITRA experts). Deltas (in pp – percentage points) below each ARBITRA row are relative to the corresponding standalone expert: GPT-4o-mini for No-FT,

e_{\mathrm{basic}}

for Basic FT, and the best-F1 reasoning expert (

e_{\mathrm{inference}}

) for Reasoning FT. Bold: best per column.

Figure 2: TRAM2 precision–recall curve as

\tau

varies from 1.0 to 0.1. Fixed points denote single-pass baselines. ISO-F1 contours use aggregate precision/recall, whereas Table 1 reports document-level macro-F1, so the values need not coincide.

System	Backbone	TRAM2			AnnoCTR (techniques only)
		P	R	F1	P	R	F1
Document-level LLM baselines
Single-pass prompting	GPT-4o-mini	0.317	0.322	0.299	0.501	0.302	0.354
Single-pass prompting	GPT-5.5	0.335	0.561	0.411	0.648	0.651	0.635
Existing CTI extraction systems
TTPXHunter [ 33 ]	SecureBERT (finetuned)	0.245	0.531	0.328	0.220	0.356	0.265

Table 2: Document-level technique-only extraction against recent CTI systems and LLMs. CTINexus uses our ATT&CK-mapping bridge; TTPrint is reimplemented from its published prompts. We report matched-backbone, headline-backbone, and best-performing Reasoning-FT variants (

e_{\mathrm{anchor}}

e_{\mathrm{contrast}}

e_{\mathrm{inference}}

) for each applicable system. Bold: best per column among existing CTI systems and ARBITRA ; italics: document-level LLM scores exceeding ARBITRA . Gray cells represent scores from the best-performing backbone variant for that specific system.

System	Output tok.	Runtime	Output cost
GPT-5.4	2,865	43.6s	$0.0430
GPT-5.5	2,534	41.1s	$0.0760
ARBITRA	26,513	146.0s	$0.0214

Table 3: Per-document median runtime and output-token API cost on the TRAM2 evaluation (over 31 documents).

System	Verification Mechanism	TRAM2			AnnoCTR
		P	R	F1	P	R	F1
Condition 1: No fine-tuning (Phase 1: 3 $\times$ GPT-4o-mini)
ARBITRA (QBAF)	QBAF Discussion	0.348	0.342	0.327	0.350	0.449	0.357
ARBITRA (Verifier LLM)	Verifier LLM (GPT-4o-mini)	0.338	0.379	0.337	0.365	0.438	0.370
	$\Delta$ QBAF $-$ Verifier	+1.0 pp	$-$ 3.7 pp	$-$ 1.0 pp	$-$ 1.5 pp	+1.1 pp	$-$ 1.3 pp
Condition 2: Reasoning FT (Phase 1: $e_{\mathrm{anchor}}$ , $e_{\mathrm{contrast}}$ , $e_{\mathrm{inference}}$ ; verifier: GPT-4o-mini)

Table 4: Ablation of the Phase 2 verification mechanism on TRAM2 and AnnoCTR: per-candidate QBAF Discussion ( ARBITRA ) vs. a single Verifier LLM. Conditions are detailed in the section headers; ARBITRA (QBAF) Reasoning FT values in conditions 2 and 3 are identical and repeated for direct comparison.

\Delta

rows show ARBITRA minus Verifier LLM in percentage points (pp); green / red = QBAF advantage/disadvantage. Bold : higher value within each condition block.

Figure 3: QBAF scores for disputed TRAM2 candidates under Reasoning FT and Basic FT. TP/FP denote candidates that do/do not match a gold label; the dashed line marks

\tau=0.5

Figure 4: TRAM2 verification results at

\tau=0.5

with the candidate union phase fixed. Only the verification phase varies between ARBITRA ’s QBAF and single-call generic or domain-finetuned LLM verifiers.

Model	Clean	Conflicting ( $\oplus$ FP )	Cross-report ( $\oplus$ FP )	Semantic ( $\ominus$ TP )	Mean $\Delta$ P
	P/R	P/R	P/R	P/R
GPT-4o-mini (CoT)	0.317 / 0.322	0.289 / 0.302	0.284 / 0.284	0.285 / 0.251	$-9.8\%$
GPT-5.4	0.341 / 0.510	0.284 / 0.461	0.286 / 0.446	0.328 / 0.457	$-12.2\%$
GPT-5.5	0.335 / 0.561	0.266 / 0.512	0.278 / 0.504	0.320 / 0.511	$-14.0\%$
$e_{\mathrm{basic}}$ (finetuned GPT-4o-mini)	0.676 / 0.530	0.600 / 0.557	0.596 / 0.533	0.655 / 0.510	$-8.7\%$
$e_{\mathrm{anchor}}$ (finetuned GPT-4.1-mini)	0.605 / 0.500	0.543 / 0.503	0.502 / 0.517	0.633 / 0.495	$-7.5\%$

Table 5: Robustness summary on clean TRAM2 and three adversarial variants. Cells report precision/recall (P/R). Adversarial variants are annotated by their primary degradation objective:

\oplus

FP denotes false-positive injection and

\ominus

TP denotes true-positive suppression. Mean

\Delta

P reports the average relative precision change from each model’s own clean baseline across the three variants (negative values indicate degradation). Shaded cells range from white (least degradation) to red (most degradation).

Figure 5: Example case study of ARBITRA on the New Horabot campaign CTI report.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Canonical (3 diverse reasoning experts)
Expert Configuration	P	R	F1
$e_{\mathrm{anchor}}$ , $e_{\mathrm{contrast}}$ , $e_{\mathrm{inference}}$	0.722	0.635	0.649
2-expert subsets
$e_{\mathrm{anchor}}$ , $e_{\mathrm{inference}}$	0.691	0.644	0.640
$e_{\mathrm{anchor}}$ , $e_{\mathrm{contrast}}$	0.707	0.589	0.602
$e_{\mathrm{contrast}}$ , $e_{\mathrm{inference}}$	0.756	0.561	0.617

Appendix

Table 6: Expert pool combination ablation on TRAM2 (

\tau=0.5

). All configurations use the same Phase 2 QBAF orchestrator (GPT-4o-mini). The canonical ARBITRA Reasoning FT pool is repeated from Table 1 for reference. Bold : best; italic : second best per column.

	TRAM2 [ 26 ]			AnnoCTR [ 18 ]			CTIBench-ATE [ 1 ]
Statistic	Train	Test	Full	Train	Test	Full	Test
(a) Corpus Statistics
# Documents	120	31	151	83	34	117	60
# Sentences	15,358	3,820	19,178	3,133	2,017	5,150	401
Sent. / Doc Mean $\pm$ SD	$127.98\pm 88.30$	$123.23\pm 61.66$	$127.01\pm 83.36$	$37.75\pm 54.63$	$59.32\pm 39.29$	$44.02\pm 54.74$	$6.68\pm 2.69$
Sent. / Doc Median	105.5	110.0	107.0	22.0	42.5	33.0	6.5

Appendix

Table 7: Comprehensive corpus and annotation statistics for the three evaluation benchmarks as used in our evaluations.

System	Verification Mechanism	Conflicting			Cross-report			Semantic
		P	R	F1	P	R	F1	P	R	F1
Condition 1: No fine-tuning (Phase 1: 3 $\times$ GPT-4o-mini)
ARBITRA (QBAF)	QBAF Discussion	0.286	0.295	0.273	0.310	0.303	0.288	0.309	0.316	0.286
ARBITRA (Verifier LLM)	Verifier LLM (GPT-4o-mini)	0.274	0.315	0.274	0.298	0.337	0.297	0.279	0.309	0.269
	$\Delta$ QBAF $-$ Verifier	+1.2 pp	$-$ 2.0 pp	$-$ 0.1 pp	+1.2 pp	$-$ 3.4 pp	$-$ 0.9 pp	+3.0 pp	+0.7 pp	+1.7 pp
Condition 2: Reasoning FT (Phase 1: $e_{\mathrm{anchor}}$ , $e_{\mathrm{contrast}}$ , $e_{\mathrm{inference}}$ ; verifier: GPT-4o-mini)

Appendix

Table 8: Adversarial robustness ablation on three adversarial TRAM2 variants: QBAF Discussion ( ARBITRA ) vs. a single Verifier LLM as the Phase 2 mechanism. Conditions are detailed in the section headers; ARBITRA (QBAF) values in conditions 2 and 3 are identical and repeated for direct comparison. Clean baseline performance is reported in Table 4 .

\Delta

rows show ARBITRA minus Verifier LLM in percentage points (pp); green / red = QBAF advantage/disadvantage. Bold : higher value within each condition block.

System	Expert Config	P	R	F1
Document-level LLM baselines
GPT-4o-mini		0.446	0.398	0.416
GPT-4.1-mini		0.603	0.609	0.597
Finetuned expert instances — trained on TRAM2, transferred to CTIBench-ATE
$e_{\mathrm{basic}}$ (finetuned GPT-4o-mini)		0.510	0.381	0.429
$e_{\mathrm{anchor}}$ (finetuned GPT-4.1-mini)		0.469	0.408	0.427

Appendix

Table 9: Zero-shot transfer results on CTIBench-ATE. Backbone models are indicated in parentheses. ARBITRA applies no CTIBench-ATE-specific tuning. Percentage-point deltas (pp) are relative to the corresponding standalone baseline: the single LLM for No-FT rows,

e_{\mathrm{basic}}

for Basic FT rows, and the best-F1 expert (

e_{\mathrm{anchor}}

for TRAM2;

e_{\mathrm{contrast}}

for AnnoCTR) for Reasoning FT rows. Bold : best per column.

Cyber Threat Intelligence

Cyber threat intelligence turns evidence about adversaries, campaigns, and vulnerabilities into knowledge that helps defenders anticipate and respond to attacks.

See all 43 papers

CTI2TTP: Evidence-Grounded TTP Extraction from Cyber Threat Intelligence via Diverge-then-Converge Verification
May 25, 2026Yutong Cheng, Changze Li, Raihan Sultan Pasha Basuki +3Cyber Threat Intelligence Document Information Extraction
Virginia Tech · Universitas Ary Ginanjar, Jakarta, Indonesia · Amazon
Retrieval-Constrained Policy Optimization for Attack Technique Extraction from Cyber Threat Intelligence
Aug 7, 2026Jiayun Zhang, Junshen Xu, Zejun Xie +1Cyber Threat Intelligence Cybersecurity
Amazon Web Services New York, New York, USA · Amazon Web Services Boston, Massachusetts, USA · Rutgers University Piscataway, New Jersey, USA
Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reports
Jun 16, 2026Ahmed Ryan, Saad Sakib Noor, Md Erfan +3LLM Evaluation Cyber Threat Intelligence
The University of Alabama · The University of Dhaka

May 25, 2026J/K move · Enter open · S save

CTI2TTP: Evidence-Grounded TTP Extraction from Cyber Threat Intelligence via Diverge-then-Converge Verification

Yutong Cheng, Changze Li, Raihan Sultan Pasha Basuki, Qian Cui+2

Virginia Tech · Universitas Ary Ginanjar, Jakarta, Indonesia · Amazon

Cyber Threat Intelligence Document Information Extraction Cybersecurity LLMs for Cybersecurity

Open paper AskPDF arXiv

Extracting MITRE ATT&CK techniques from cyber threat intelligence (CTI) reports is an open-set, multi-label problem requiring both high recall (not missing techniques) and high precision (not hallucinating unsupported ones). Existing rule-based, supervised, and LLM-based methods struggle to achieve both: rule-based and supervised approaches lack generalizability across diverse attack descriptions, while LLM-based approaches that couple candidate generation and validation within a single inference step suffer from limited recall and precision simultaneously. We propose CTI2TTP, which addresses this challenge through a diverge-then-converge design inspired by how human analysts work: first extracting broadly, then verifying rigorously. Reports are first decomposed into atomic behaviors, each anchored by a deterministic span localization stage to a specific evidence window in the source text. Candidate techniques are then proposed broadly for each behavior, and a convergent verification stage retains only candidates supported by both the localized evidence and the authoritative MITRE definition. We contribute two evaluation resources, a cleaned TRAM benchmark (TRAM-Clean) and a new annotated dataset (CTI2TTP-Bench), to address known annotation noise in existing benchmarks and elevate the task to document-level TTP extraction. On TRAM-Clean and CTI2TTP-Bench, CTI2TTP achieves 76.48% and 87.39% macro-F1 respectively, outperforming chain-of-thought prompting with the same backbone by 63.5% and 29.4%.

Abstract

Cyber Threat Intelligence

Explore similar work

CTI2TTP: Evidence-Grounded TTP Extraction from Cyber Threat Intelligence via Diverge-then-Converge Verification

Retrieval-Constrained Policy Optimization for Attack Technique Extraction from Cyber Threat Intelligence

Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reports