The field of cybersecurity NER lacks standard-ized labels, making it challenging to combine datasets. We investigate label unification across four cybersecurity datasets to increase data resource usability. We perform a coarse-grained label unification and conduct pairwise cross-dataset evaluations using RoBERTa models. Qualitative analysis of predictions reveals errors, limitations, and dataset differences. To address unification limitations, we propose alternative architectures including a multi-head model and a graph-based transfer model. Under our selected mapping and training setup, unified training underperformed the corresponding single-dataset baselines on all four development sets. The multi-head model with weight sharing provides no consistent benefits over single-dataset baselines, and the graph-based transfer model shows no consistent benefits.
Figures & tables
Dataset
Original Labels
Unified Label
APTNER
APT, SECTEAM
Organization
OS
System
VULNAME
Vulnerability
MAL
Malware
DNRTI
HackOrg, SecTeam, org
Organization
Tool
System
Table 1: Mapping of original entity labels to unified schema used across all four datasets.
Sentence
APT32 actors continue to deliver the malicious attachments via spear-phishing emails.
labels A
B-Organization O O O O O B-Malware I-Malware O B-Vulnerability O O
labels B
B-Organization O O O O O O O O O
Table 2: Comparison of token-level annotation of the same sentence in two different datasets.
Component
Specification
Backbone
RoBERTa-base encoder
Token Classification Head
Linear layer on top of the encoder hidden states for token labels
Input Encoding
Tokenized word-level input with RoBERTa tokenizer; truncation to max length 512
Label Alignment
Only the first subword of each token receives the true label; remaining subwords are masked
Table 3: Model architecture for the fine-tuned RoBERTa cross-dataset model
Hyperparameter
Value
Base Model
RoBERTa-base
Loss Function
CE (first token only)
Optimizer
AdamW
Learning Rate
2e-5
Batch Size
2
Epochs
10
Table 4: Training hyperparameters for the fine-tuned RoBERTa cross-dataset model
Train \Dev
DNRTI
AttackER
APTNER
CyNER
DNRTI
0.63
0.30
0.21
0.16
AttackER
0.34
0.61
0.26
0.37
APTNER
0.39
0.32
0.59
0.55
CyNER
0.25
0.24
0.32
0.73
Table 5: RoBERTa span-F1 on each development dataset after training on each dataset.
Train \Dev
DNRTI
AttackER
APTNER
CyNER
DNRTI
0.63
0.29
0.15
0.13
AttackER
0.38
0.61
0.20
0.31
APTNER
0.48
0.40
0.51
0.58
CyNER
0.30
0.29
0.28
0.76
Table 6: RoBERTa precision on each development dataset after training on each dataset.
Train \Dev
DNRTI
AttackER
APTNER
CyNER
DNRTI
0.64
0.32
0.38
0.21
AttackER
0.32
0.62
0.40
0.45
APTNER
0.33
0.27
0.69
0.52
CyNER
0.22
0.21
0.37
0.71
Table 7: RoBERTa recall on each development dataset after training on each dataset.
Train \Dev
DNRTI
AttackER
APTNER
CyNER
DNRTI
0.71
0.43
0.37
0.44
AttackER
0.55
0.67
0.36
0.48
APTNER
0.55
0.45
0.66
0.60
CyNER
0.57
0.42
0.58
0.82
Table 8: RoBERTa unlabelled span-F1 on each development dataset after training on each dataset.
Train \Dev
DNRTI
AttackER
APTNER
CyNER
DNRTI
0.69
0.40
0.27
0.20
AttackER
0.44
0.69
0.39
0.48
APTNER
0.44
0.41
0.67
0.59
CyNER
0.29
0.38
0.35
0.78
Table 9: RoBERTa loose span-F1 on each development dataset after training on each dataset.
Datasets
DNRTI
AttackER
APTNER
CyNER
DNRTI
0
0.23
0.04
0.05
AttackER
0.23
0
0.24
0.19
APTNER
0.04
0.24
0
0.07
CyNER
0.05
0.19
0.07
0
Table 10: JS-div between training datasets for distributions over span length of entities.
Distribution
RoBERTa
p-value
Words
-0.02
0.95
POS labels
-0.23
0.48
Entity span lengths
0.00
0.99
Entity span-based counts
0.13
0.68
Table 11: Pearson correlations between JS divergence and cross-dataset span-F1 over off-diagonal dataset pairs.
Dataset
Unified
Train-Only
Unified
0.53
-
DNRTI
0.52
0.63
AttackER
0.51
0.61
APTNER
0.49
0.59
CyNER
0.67
0.73
Table 12: Span-F1 on development datasets for a model trained on unified dataset or a corresponding train dataset.
Dataset
Reference
Multi-head
DNRTI
0.62
0.61
AttackER
0.43
0.42
APTNER
0.59
0.60
CyNER
0.77
0.80
Table 13: Span-F1 per dataset for RoBERTa multi-head vs single-dataset fine-grained reference.
Dataset
Baseline (RoBERTa)
LST-NER (ours)
DNRTI
0.72
0.70 ∗
AttackER
0.44
0.43
APTNER
0.51
0.48 ∗
CyNER
0.76
0.81 ∗
Table 14: Span-F1 comparing our RoBERTa baseline and the graph-matching model (LST-NER). ∗ indicates a statistically significant difference (paired bootstrap test, p<0.05 ); AttackER shows no significant difference ( p=0.35 ).
Hyperparameter
Value
Learning rate
5e-5
Batch Size
16
Epochs
3
Max Length
128
Temp
4
Edge Threshold
1.5
Table 15: Training hyperparameters for RoBERTa and LST-NER (only the first four applies to RoBERTa, whereas all hyperparameters apply LST-NER).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Experiment
Training config
Cross-dataset
1.09 GCD-h
unified
1.15 GCD-h
Multi-head
1.18 GCD-h
Appendix
Table 16: Compute-only RoBERTa fine-tuning costs measured on LUMI. Each experiment run on one GCD (Graphic Compute Die) of one AMD MI250x node.
Dataset
GPU
GPU-hours
AttackER
RTX 2070
0.37
APTNER
A30
0.93
DNRTI
RTX 2070
0.68
CyNER
RTX 6000
0.24
Total
—
2.22
Appendix
Table 17: GPU time for the graph-model experiment. Runs used different GPU models; GPU-hours represent elapsed GPU time, not normalized compute.
Cybersecurity is a real-life test-bed for many machine learning problems at once, especially when considering modern strides in using Large Language Models (LLMs) to automate processes as ``agents.'' Cybersecurity workflows require orchestrating hundreds of standard and bespoke tools through various formats. The scale of cybersecurity data is enormous; for example, a single malware sample can be viewed as a sequence of billions of tokens. The cost of labeling any file by experts is enormous and labor-intensive, in part because an adversary (possibly a well-funded nation state actor) is attempting to subvert your detection methods. Even skilled experts may disagree on the correct label, creating ambiguity in what constitutes ground truth. When deployed, models must run quickly on billions of items a day, where low-latency is critical for operational success, in a continuously changing environment. In addition, explainability is not optional: analysts demand clear reasoning for model decisions to cope with the large number of false-positive alerts they face daily, and to quickly develop remediation and understand how something went wrong. In short, the amount of complexity cybersecurity is greater than that of natural language and computer vision, and thus we posit that cybersecurity is the better test-case for general AI progress than other, well-studied fields.
Edward Raff, Maor Ashkenazi, Sagar Samtani +2
CrowdStrike · University of Maryland, Baltimore County · Edward Raff∗† +2
While multilingual language models promise to bring the benefits of LLMs to speakers of many languages, gold-standard evaluation benchmarks in most languages to interrogate these assumptions remain scarce. The Universal NER project, now entering its fourth year, is dedicated to building gold-standard multilingual Named Entity Recognition (NER) benchmark datasets. Inspired by existing massively multilingual efforts for other core NLP tasks (e.g., Universal Dependencies), the project uses a general tagset and thorough annotation guidelines to collect standardized, cross-lingual annotations of named entity spans. The first installment (UNER v1) was released in 2024, and the project has continued and expanded since then, with various organizers, annotators, and collaborators in an active community.
Terra Blevins, Stephen Mayhew, Marek Šuppa +11
Northeastern University, USA · Duolingo, USA · Comenius University Bratislava, Slovakia +10
Closely related languages written in different scripts expose little surface overlap to multilingual models, limiting cross-lingual transfer. We compare two approaches to script unification: the general-purpose uroman romanizer and the family-specific Common Turkic Script (CTS). We train matched fastText models on transliterated Wikipedia corpora from 11 Turkic languages and evaluate them on WikiANN named entity recognition and Universal Dependencies part-of-speech tagging. CTS and uroman show no significant difference on NER, while both substantially outperform the official monolingual fastText baselines. POS results reveal no universal winner: language-specific differences are associated with the cross-lingual character n-gram coverage induced by each representation, while within-language coverage becomes more important when target-language supervision is available. Although CANINE-c achieves higher overall POS averages, the substantially simpler fastText-based systems remain competitive on several treebanks. Overall, the effectiveness of script unification depends on the language, the induced subword overlap, and the available supervision.