Label Unification for Cross-Dataset Generalization in Cybersecurity NER

Maciej Jalocha; Johan Hausted Schmidt; William Michelsen

cs.CLJul 18, 2025

Label Unification for Cross-Dataset Generalization in Cybersecurity NER

Authors: Maciej Jalocha, Johan Hausted Schmidt, William Michelsen

Abstract

The field of cybersecurity NER lacks standard-ized labels, making it challenging to combine datasets. We investigate label unification across four cybersecurity datasets to increase data resource usability. We perform a coarse-grained label unification and conduct pairwise cross-dataset evaluations using RoBERTa models. Qualitative analysis of predictions reveals errors, limitations, and dataset differences. To address unification limitations, we propose alternative architectures including a multi-head model and a graph-based transfer model. Under our selected mapping and training setup, unified training underperformed the corresponding single-dataset baselines on all four development sets. The multi-head model with weight sharing provides no consistent benefits over single-dataset baselines, and the graph-based transfer model shows no consistent benefits.

Figures & tables

Dataset	Original Labels	Unified Label
APTNER	APT, SECTEAM	Organization
	OS	System
	VULNAME	Vulnerability
	MAL	Malware
DNRTI	HackOrg, SecTeam, org	Organization
	Tool	System

Table 1: Mapping of original entity labels to unified schema used across all four datasets.

Sentence	APT32 actors continue to deliver the malicious attachments via spear-phishing emails.
labels A	B-Organization O O O O O B-Malware I-Malware O B-Vulnerability O O
labels B	B-Organization O O O O O O O O O

Table 2: Comparison of token-level annotation of the same sentence in two different datasets.

Component	Specification
Backbone	RoBERTa-base encoder
Token Classification Head	Linear layer on top of the encoder hidden states for token labels
Input Encoding	Tokenized word-level input with RoBERTa tokenizer; truncation to max length 512
Label Alignment	Only the first subword of each token receives the true label; remaining subwords are masked

Table 3: Model architecture for the fine-tuned RoBERTa cross-dataset model

Hyperparameter	Value
Base Model	RoBERTa-base
Loss Function	CE (first token only)
Optimizer	AdamW
Learning Rate	2e-5
Batch Size	2
Epochs	10

Table 4: Training hyperparameters for the fine-tuned RoBERTa cross-dataset model

Train \Dev	DNRTI	AttackER	APTNER	CyNER
DNRTI	0.63	0.30	0.21	0.16
AttackER	0.34	0.61	0.26	0.37
APTNER	0.39	0.32	0.59	0.55
CyNER	0.25	0.24	0.32	0.73

Table 5: RoBERTa span-F1 on each development dataset after training on each dataset.

Train \Dev	DNRTI	AttackER	APTNER	CyNER
DNRTI	0.63	0.29	0.15	0.13
AttackER	0.38	0.61	0.20	0.31
APTNER	0.48	0.40	0.51	0.58
CyNER	0.30	0.29	0.28	0.76

Table 6: RoBERTa precision on each development dataset after training on each dataset.

Train \Dev	DNRTI	AttackER	APTNER	CyNER
DNRTI	0.64	0.32	0.38	0.21
AttackER	0.32	0.62	0.40	0.45
APTNER	0.33	0.27	0.69	0.52
CyNER	0.22	0.21	0.37	0.71

Table 7: RoBERTa recall on each development dataset after training on each dataset.

Train \Dev	DNRTI	AttackER	APTNER	CyNER
DNRTI	0.71	0.43	0.37	0.44
AttackER	0.55	0.67	0.36	0.48
APTNER	0.55	0.45	0.66	0.60
CyNER	0.57	0.42	0.58	0.82

Table 8: RoBERTa unlabelled span-F1 on each development dataset after training on each dataset.

Train \Dev	DNRTI	AttackER	APTNER	CyNER
DNRTI	0.69	0.40	0.27	0.20
AttackER	0.44	0.69	0.39	0.48
APTNER	0.44	0.41	0.67	0.59
CyNER	0.29	0.38	0.35	0.78

Table 9: RoBERTa loose span-F1 on each development dataset after training on each dataset.

Datasets	DNRTI	AttackER	APTNER	CyNER
DNRTI	0	0.23	0.04	0.05
AttackER	0.23	0	0.24	0.19
APTNER	0.04	0.24	0	0.07
CyNER	0.05	0.19	0.07	0

Table 10: JS-div between training datasets for distributions over span length of entities.

Distribution	RoBERTa	p-value
Words	-0.02	0.95
POS labels	-0.23	0.48
Entity span lengths	0.00	0.99
Entity span-based counts	0.13	0.68

Table 11: Pearson correlations between JS divergence and cross-dataset span-F1 over off-diagonal dataset pairs.

Dataset	Unified	Train-Only
Unified	0.53	-
DNRTI	0.52	0.63
AttackER	0.51	0.61
APTNER	0.49	0.59
CyNER	0.67	0.73

Table 12: Span-F1 on development datasets for a model trained on unified dataset or a corresponding train dataset.

Dataset	Reference	Multi-head
DNRTI	0.62	0.61
AttackER	0.43	0.42
APTNER	0.59	0.60
CyNER	0.77	0.80

Table 13: Span-F1 per dataset for RoBERTa multi-head vs single-dataset fine-grained reference.

Dataset	Baseline (RoBERTa)	LST-NER (ours)
DNRTI	0.72	0.70 ∗
AttackER	0.44	0.43
APTNER	0.51	0.48 ∗
CyNER	0.76	0.81 ∗

Table 14: Span-F1 comparing our RoBERTa baseline and the graph-matching model (LST-NER). ∗ indicates a statistically significant difference (paired bootstrap test,

p<0.05

); AttackER shows no significant difference (

p=0.35

Hyperparameter	Value
Learning rate	5e-5
Batch Size	16
Epochs	3
Max Length	128
Temp	4
Edge Threshold	1.5

Table 15: Training hyperparameters for RoBERTa and LST-NER (only the first four applies to RoBERTa, whereas all hyperparameters apply LST-NER).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Training config
Cross-dataset	1.09 GCD-h
unified	1.15 GCD-h
Multi-head	1.18 GCD-h

Appendix

Table 16: Compute-only RoBERTa fine-tuning costs measured on LUMI. Each experiment run on one GCD (Graphic Compute Die) of one AMD MI250x node.

Dataset	GPU	GPU-hours
AttackER	RTX 2070	0.37
APTNER	A30	0.93
DNRTI	RTX 2070	0.68
CyNER	RTX 6000	0.24
Total	—	2.22

Appendix

Table 17: GPU time for the graph-model experiment. Runs used different GPU models; GPU-hours represent elapsed GPU time, not normalized compute.

Cross-Dataset Generalization

Cross-dataset generalization studies whether AI models retain accuracy, calibration, and usefulness on data from sources beyond their training set.

See all 142 papers

Cybersecurity is the True Frontier for Generative AI Success or Failure
Jun 27, 2026Edward Raff, Maor Ashkenazi, Sagar Samtani +2Cybersecurity LLMs for Cybersecurity
CrowdStrike · University of Maryland, Baltimore County · Edward Raff∗† +2
Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark
Apr 14, 2026Terra Blevins, Stephen Mayhew, Marek Šuppa +11Named Entity Recognition Multilingual Language Model Evaluation
Northeastern University, USA · Duolingo, USA · Comenius University Bratislava, Slovakia +10
Universal or Language-Family-Specific Script Unification for Cross-Lingual Transfer? A Case Study on Turkic Languages
Aug 10, 2026Zijie ZhangCross-Lingual Transfer
The Chinese University of Hong Kong, Shenzhen

Jun 27, 2026J/K move · Enter open · S save

Cybersecurity is the True Frontier for Generative AI Success or Failure

Edward Raff, Maor Ashkenazi, Sagar Samtani, David J. Elkind+1

CrowdStrike · University of Maryland, Baltimore County · Edward Raff∗†+2

Cybersecurity LLMs for Cybersecurity

Open paper AskPDF arXiv

Cybersecurity is a real-life test-bed for many machine learning problems at once, especially when considering modern strides in using Large Language Models (LLMs) to automate processes as ``agents.'' Cybersecurity workflows require orchestrating hundreds of standard and bespoke tools through various formats. The scale of cybersecurity data is enormous; for example, a single malware sample can be viewed as a sequence of billions of tokens. The cost of labeling any file by experts is enormous and labor-intensive, in part because an adversary (possibly a well-funded nation state actor) is attempting to subvert your detection methods. Even skilled experts may disagree on the correct label, creating ambiguity in what constitutes ground truth. When deployed, models must run quickly on billions of items a day, where low-latency is critical for operational success, in a continuously changing environment. In addition, explainability is not optional: analysts demand clear reasoning for model decisions to cope with the large number of false-positive alerts they face daily, and to quickly develop remediation and understand how something went wrong. In short, the amount of complexity cybersecurity is greater than that of natural language and computer vision, and thus we posit that cybersecurity is the better test-case for general AI progress than other, well-studied fields.

Abstract

Cross-Dataset Generalization

Explore similar work

Cybersecurity is the True Frontier for Generative AI Success or Failure

Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark

Universal or Language-Family-Specific Script Unification for Cross-Lingual Transfer? A Case Study on Turkic Languages