Federated Lightweight Transformer for Network-Layer Cyberattack Detection in O-RAN
Authors: Hong Nhung Nguyen, Abdulahi Badrudeen, Yekaterina Kim, Truong Thu Huong, Sunwoo Kim
Organizations: Department of Electronic Engineering, Hanyang University, Seoul, South Korea · School of Electrical and Electronic Engineering, Hanoi University of Science and Technology, Hanoi, Vietnam
Open Radio Access Network (O-RAN) improves radio access network flexibility through disaggregation, open interfaces, and programmable RAN Intelligent Controllers (RICs). However, the same openness and distribution increase the attack surface and make centralized intrusion detection difficult to scale across heterogeneous RAN sites. This paper presents a federated lightweight Transformer framework for privacy-preserving network-layer cyberattack detection in O-RAN. Each participating site trains a compact Transformer locally on temporally ordered network-flow records, while only model parameters are shared with a federated coordinator. To model realistic site heterogeneity, the client datasets are partitioned according to a Dirichlet distribution, resulting in non-independent and identically distributed (non-IID) local data. The local detector uses feature projection, positional encoding, multi-head self-attention, feed-forward layers, and temporal pooling to classify five network-layer attack categories: ICMP, Slowloris, SYN, TCP-ACK, and UDP. Experiments on the NetsLab 5G O-RAN intrusion-detection dataset show an overall accuracy of 94.89 % and a macro-F1 score of 81.09 %.
Figures & tables
Fig. 1: Overview of the federated lightweight Transformer architecture for distributed O-RAN network-layer cyberattack detection.
Parameter
Value
Number of clients
5
Dirichlet α
0.5
Embedding dimension
64
Transformer encoder layers
2
Attention heads
4
Feed-forward dimension
128
TABLE I: Experimental configuration
Class
Precision (%)
Recall (%)
F1 (%)
ICMP
86.96
32.79
47.62
Slowloris
65.94
77.84
71.40
SYN
99.71
97.99
98.84
TCP-ACK
98.24
98.04
98.14
UDP
85.39
93.95
89.47
Macro average
87.25
80.12
81.09
TABLE II: Class-Wise Detection Performance
Fig. 2: Confusion matrix of the proposed method’s output on the test dataset.
Network intrusion detection identifies malicious or abnormal network activity to help protect computers, connected devices, and critical infrastructure.
O-RAN enables a disaggregated baseband stack with programmable functions that communicate over standardized open interfaces. The same openness that enables multi-vendor composition also expands the attack surface across logically decoupled tiers that make up the compute continuum. Among these threats, Denial-of-Service and performance-degradation attacks, which account for the majority of catalogued O-RAN threats, are particularly difficult to detect. Traditional Time-Series Anomaly Detection (TSAD) methods fail in this new regime where labelled baselines are scarce, threats evolve faster than detectors can be retrained, and the high-dimensional multivariate telemetry overwhelms monolithic inference models. To address these challenges, we present DAST, a zero-shot multi-agent framework for cross-interface anomaly detection in O-RAN that chains a three-stage VLM → LLM → VLM pipeline. DAST converts multivariate KPI streams into visual representations, scores textual per-interface descriptions against O-RAN domain knowledge, and verifies suspects on high-resolution heatmaps to output the problematic interfaces, the anomalous time intervals, an indicative O-RAN WG11-aligned operational impact rating and the decision rationale. We evaluate DAST on real network traces collected from an O-RAN testbed under representative performance degradation scenarios, achieving 0.910 F1-Score and 0.843 Accuracy, outperforming state-of-the-art TSAD baselines.
Francesco Spinelli, Esteban Municio, Pau Baguer +2
i2CAT Foundation · NEC Laboratories Europe · ICREA
Federated learning enables privacy-conscious collaboration for network intrusion detection without centralizing sensitive traffic data, yet its deployment in operational environments must simultaneously satisfy three competing requirements: formal differential privacy guaranties, tolerance to Byzantine-adversarial participants, and reliable detection coverage across severely imbalanced attack categories. Existing literature treats these properties as independently composable, an assumption that this paper challenges both theoretically and empirically. In this paper, we study how these requirements interact in class-imbalanced federated NIDS and introduce geometric indistinguishability as a conceptual lens for a regime in which privacy-induced dispersion in client updates can make minority-class signals harder for robust aggregation to preserve. Using UNSW-NB15 as a case study, we evaluate DP-SGD combined with coordinate-wise median under label-flip and model-poisoning attacks, with threat coverage assessed across attack categories. Our results provide initial evidence that the joint use of privacy noise and robust aggregation can disproportionately degrade detection of rare attacks relative to majority classes. We also show that part of the observed collapse under strong privacy can arise from training miscalibration, while a residual performance floor may remain for ultra-rare categories even after epsilon-dependent tuning. These findings motivate studying privacy, robustness, and rare-attack coverage jointly rather than as independently composable properties, and suggest that aggregation-aware modeling and sample-aware evaluation are promising directions for trustworthy federated NIDS.
Adrita Rahman Tory, ABM Shawkat Ali, Md Abu Layek +1
Bangladesh University of Business and Technology (BUBT), Mirpur-2, Dhaka-1216, Bangladesh · Jagannath University, Dhaka, Bangladesh · University of New South Wales (UNSW), ACT 2601, Australia
This paper presents AutoSec-RAN, a specification-grounded large language model (LLM) framework for automated security triage of Open Radio Access Network (O-RAN) testing. Unlike approaches relying on static rule sets or privileged access to raw packet captures, AutoSec-RAN fine-tunes Mistral-7B-Instruct through two Low-Rank Adaptation (LoRA) stages over a curated corpus of 69 O-RAN, 3GPP, and adjacent standards documents, then selectively augments inference with retrieval-augmented generation (RAG) over the same specification index. At inference, the model receives only an investigator-style question and a scenario metadata block, simulating an automated analyst operating without raw packet evidence. AutoSec-RAN is validated on a purpose-built 5G/O-RAN testbed across labeled fronthaul distributed denial of service replay scenarios. A Gemma-4-31B judge LLM scores each candidate's answer on faithfulness, correctness, and actionability relative to a reference authored from full packet capture and log evidence. AutoSec-RAN achieves a mean judge overall score of 0.713 across all 313 evaluation items (0.755 restricted to the five supervised taxonomy classes), clearing the 0.70 reporting threshold, and a blended score of 0.778 (0.808 supervised-only) when cosine similarity over Beijing Academy of Artificial Intelligence (BAAI) General Embedding (BGE) is incorporated as a secondary semantic signal. The framework performs strongest on mitigation and threat-model reasoning (with a mean overall of 0.849 and 0.848, respectively) and identifies a measurable gap on the held-out attack-direction class (0.515), a controlled result that exposes a specific training data requirement. These results establish specification-grounded LoRA fine-tuning combined with RAG retrieval as a viable foundation for automated security validation of AI-native O-RAN.
Anoop Mishra, Joshua Moore, Aly Sabri Abdalla +1
Dept. of Electrical and Computer Engineering, Mississippi State University, USA