Multimodal Reasoning with LLM for Encrypted Traffic Interpretation: A Benchmark
Authors: Longgang Zhang, Xiaowei Fu, Fuxiang Huang, Lei Zhang
Organizations: School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China. · School of Data Science, Lingnan University, Hong Kong, China.
Network traffic, as a key media format, is crucial for ensuring security and communications in modern internet infrastructure. While existing methods offer excellent performance, they face two key bottlenecks: (1) They fail to capture multidimensional semantics beyond unimodal sequence patterns. (2) Their black box property, i.e., providing only category labels, lacks an auditable reasoning process. We identify a key factor that existing network traffic datasets are primarily designed for classification and inherently lack rich semantic annotations, failing to generate human-readable evidence report. To address data scarcity, this paper proposes a Byte-Grounded Traffic Description (BGTD) benchmark for the first time, combining raw bytes with structured expert annotations. BGTD provides necessary behavioral features and verifiable chains of evidence for multimodal reasoning towards explainable encrypted traffic interpretation. Built upon BGTD, this paper proposes an end-to-end traffic-language representation framework (mmTraffic), a multimodal reasoning architecture bridging physical traffic encoding and semantic interpretation. In order to alleviate modality interference and generative hallucinations, mmTraffic adopts a jointly-optimized perception-cognition architecture. By incorporating a perception-centered traffic encoder and a cognition-centered LLM generator, mmTraffic achieves refined traffic interpretation with guaranteed category prediction. Extensive experiments demonstrate that mmTraffic autonomously generates high-fidelity, human-readable, and evidence-grounded traffic interpretation reports, while maintaining highly competitive classification accuracy comparing to specialized unimodal model (e.g., NetMamba). The source code is available at https://github.com/lgzhangzlg/Multimodal-Reasoning-with-LLM-for-Encrypted-Traffic-Interpretation-A-Benchmark
Figures & tables
Fig. 1: Comparison of traffic analysis paradigms. (a) Traditional classification methods that act as a “black box”, providing only a label and low-level feature weights that lack operational value. (b) Our proposed multimodal reasoning framework, composed of a Traffic Perception Encoder and a Cognitive LLM, instructed by Byte-Grounded Knowledge, generating an evidence-grounded report with human-understandable reasoning and executable insights.
Model
Traffic Representation
Core Structure
Pre-training
Limitations
ET-BERT [ 19 ]
4-hex Bigram / Burst Segmentation
Transformer Encoder
MBM / SBP
Ignores protocol hierarchy; uses natural language subword tokenization
YaTC [ 38 ]
Multi-level Flow Representation (MFR) Matrix
Dual-Attention Transformer
MAE (Matrix Masking)
Fixed matrix dimensions; truncates long-range session features
NetMamba [ 34 ]
Stride-based Byte Sequence
Unidirectional Mamba (SSM)
Masked Stride Reconstruction
Purely numerical mapping; lacks interpretability
FlowletFormer [ 22 ]
Flowlet Behavioral Unit / Field Tokenization
Transformer Encoder
MFM / FPT
Black-box classifier; unable to output forensic reasoning
TABLE I: Comparisons of different paradigms for network traffic analysis. MBM, SBP, MAE, MFM, and FPT represent Masked Byte Model, Segment Burst Prediction, Masked Autoencoder, Masked Flow Model, and Flow Prediction Task, respectively.
Fig. 2: Pipeline of developing BGTD dataset: (a) session extraction and class balancing from raw PCAP files, (b) fixed-length 10×160 NPY array generation via priority-based packet sampling, and (c) LLM-assisted ground-truth synthesis using Claude Opus-4.6 prompted as a senior network security expert.
Fig. 3: Statistical overview of the BGTD dataset.
Fig. 4: Overview of the mmTraffic framework. (a) The frozen traffic encoder Tθ extracts high-dimensional features from raw traffic data. (b) The linear connector Cω projects traffic features into the LLM token space, with the CGHF mechanism injecting a class-aware anchor token into the input sequence. (c) The LLM Gϕ autoregressively generates a structured forensic report containing behavioral traits, evidence chain, and diagnostic description.
Dataset
Classes
Nmin
Nmax
Train
Test
CrossPlatform-Android
212
50
2,000
31,029
7,644
CrossPlatform-iOS
196
50
3,000
29,302
7,233
ISCXVPN2016
7
200
6,000
33,600
8,400
ISCX-Tor-2016
8
3,000
10,000
64,000
16,000
CSTNet-TLS1.3
120
0
6,000
37,148
9,224
USTC-TFC-2016
12
3,000
6,000
53,112
13,276
TABLE II: Statistics of the six benchmark datasets after preprocessing.
Dataset
Method
Classification
Generation
Evidence
Description
Acc / JClsAcc%
JSON Valid%
ROUGE-L
BERTScore
ROUGE-L
BERTScore
ISCX-Tor-2016
NetMamba
0.9961
–
–
–
–
–
Zero-shot LLM
0.0003
100.00%
0.1247
0.8322
0.1164
0.8469
Vanilla
0.7092
100.00%
0.6002
0.9217
0.5831
0.9266
mmTraffic (Ours)
0.9331
100.00%
0.8192
0.9641
0.7751
0.9481
ISCXVPN2016
NetMamba
0.9917
–
–
–
–
–
TABLE III: Results on ISCX-Tor-2016, ISCXVPN2016, and CSTNet-TLS1.3. NetMamba is the unimodal classifier (“–” means no generation); Zero-shot LLM feeds features directly to LLM without tuning; Vanilla freezes the encoder without auxiliary constraints. Bold indicates the best per column within each dataset group.
Dataset
Method
Classification
Generation
Evidence
Description
Acc / JClsAcc%
JSON Valid%
ROUGE-L
BERTScore
ROUGE-L
BERTScore
CrossPlatform-iOS
NetMamba
0.9060
–
–
–
–
–
Zero-shot LLM
0.0000
100.00%
0.1962
0.8509
0.1268
0.8503
Vanilla
0.0058
100.00%
0.2218
0.8591
0.1255
0.8535
mmTraffic (Ours)
0.8865
100.00%
0.6880
0.9387
0.5972
0.9283
CrossPlatform-Android
NetMamba
0.9104
–
–
–
–
–
TABLE IV: Results on CrossPlatform-iOS, CrossPlatform-Android, and USTC-TFC-2016. NetMamba is the unimodal classifier (“–” means no generation); Zero-shot LLM feeds features directly to LLM without tuning; Vanilla freezes the encoder without auxiliary constraints. Bold indicates the best per column within each dataset group.
Fig. 5: Analysis on Structural Consistency Metrics. The semantic-priority constraints in mmTraffic ensure high logical rigor.
(1) TLS record header detected , indicating an encrypted TLS session. (2) TCP dominant (100%) , consistent with Tor CHAT profile. (3) Unlike Tor VOIP (codec-driven timing), Tor CHAT has irregular human-conversation-driven timing . …
(1) TLS record header detected, indicating an encrypted TLS session. (2) TLS dominant (100%), consistent with Tor CHAT profile. (3) Unlike Tor VOIP (codec-driven timing), Tor CHAT has irregular human-conversation-driven timing. …
Description
TLS record headers present, indicating encrypted transport. Instant messaging and chat protocols (AIM, ICQ, Facebook Chat, Hangouts, Skype Chat) tunneled through Tor onion routing. Dominant protocol: TCP (100%). …
TLS record headers present, indicating encrypted transport. Instant messaging and chat protocols (AIM, ICQ, Facebook Chat, Hangouts, Skype Chat) tunneled through Tor onion routing. Dominant protocol: TLS (100%).
Notes
Chat over Tor hides who is communicating with whom; message content may still be visible at exit nodes if not end-to-end encrypted.
Chat over Tor provides metadata anonymity; message content may still be visible at exit nodes if not end-to-end encrypted.
TABLE V: ISCX-Tor-2016 for sample #6227: mmTraffic correctly identifies the class. Evidence and description fields are abbreviated. Green means correctly interpreted key content.
Field
mmTraffic (Ours)
Ground Truth
Class
Steam ✓
Steam
Traits
has_tls: true has_http: false ascii: mid entropy: high zero_pad: low
has_tls: true has_http: false ascii: low entropy: high zero_pad: low
Evidence
(1) TLS record header detected , indicating an encrypted TLS session. (2) High Shannon entropy : encrypted or compressed payload. (3) Low zero-padding : sustained bulk data transfer. …
(1) TLS record header detected, indicating an encrypted TLS session. (2) Low ASCII with high entropy: encrypted payload. (3) Low zero-padding: sustained bulk data transfer. (4) TCP dominant (84.2%). …
Description
TLS record headers alongside high-entropy payload confirm encrypted communication. Valve Steam gaming platform for game downloads, updates, and multiplayer services over TLS 1.3. Large avg. packet (858 bytes). …
TLS record headers alongside high-entropy payload confirm encrypted communication. Valve Steam gaming platform for game downloads, updates, and multiplayer services over TLS 1.3. TCP dominant (84.2%). …
Notes
Game distribution platform; monitor for Steam account phishing; Workshop mods can contain malicious code.
Game distribution platform; game downloads are very large; monitor for Steam account phishing.
TABLE VI: CSTNet-TLS1.3 for sample #350: mmTraffic correctly identifies the class. Evidence and description fields are abbreviated. Green means correctly interpreted key content.
has_tls: true has_http: false ascii: low entropy: high zero_pad: low
Evidence
(1) TLS record header detected , indicating an encrypted TLS session. (2) TLS dominant (100%) , consistent with Outlook profile. (3) Unlike Gmail (HTTP/2 and QUIC), Outlook uses MAPI protocol . …
(1) TLS record header detected, indicating an encrypted TLS session. (2) Low ASCII with high entropy: encrypted payload. (3) Low zero-padding: sustained bulk data transfer. (4) TLS dominant (100%). …
Description
TLS record headers present, indicating encrypted transport. Microsoft Outlook email client using HTTPS and MAPI . Dominant protocol: TLS (100%). …
TLS record headers alongside high-entropy payload confirm encrypted communication. Microsoft Outlook email client using HTTPS and MAPI . Dominant protocol: TLS (100%).
Notes
Generally benign; verify connections go to Microsoft ASN; anomalous sync volumes may indicate data exfiltration.
Generally benign; verify connections go to Microsoft ASN; anomalous sync volumes or off-hours activity may indicate data exfiltration via email.
TABLE VII: USTC-TFC-2016 for sample #11786: mmTraffic correctly identifies the class. Evidence and description fields are abbreviated. Green means correctly interpreted key content.
Field
mmTraffic (Ours)
Vanilla VLM
Ground Truth
Class
VIDEO ✓
BROWSING ✗
VIDEO
Evidence
(1) Video streaming services (YouTube, Vimeo) tunneled via Tor onion routing. (2) High throughput (49.8 MB/s) indicates active data exchange. (3) Sustained download dominant over upload , consistent with video consumption. …
(1) TLS record header detected, indicating encrypted session. (2) High zero-padding: short flow with small payload . (3) TCP dominant (100%), consistent with HTTPS browsing profile . …
(1) TCP dominant (100%), consistent with VIDEO profile. (2) Unlike Tor AUDIO (moderate throughput), VIDEO requires significantly higher bandwidth. (3) Unlike Tor BROWSING (bursty), VIDEO maintains sustained high-throughput download. …
Description
Mixed entropy and ASCII across packet slices. Video streaming services (youtube.com, vimeo.com) tunneled via Tor onion routing. TCP dominant. High throughput (498 KB/s). …
TLS record headers indicate encrypted transport. HTTPS/SSL web browsing tunneled through Tor hides both location and activities. High zero-padding suggests background activity. …
Mixed entropy and ASCII across packet slices. Video streaming services (YouTube HTML5, YouTube Flash, Vimeo) tunneled through Tor. Dominant protocol: TCP (100%).
TABLE VIII: ISCX-Tor-2016 for sample #15750: mmTraffic correctly identifies the VIDEO class while Vanilla VLM misclassifies it as BROWSING. Evidence and description fields are abbreviated. Green and Red mean correct and wrong interpretations.
Field
mmTraffic (Ours)
Vanilla VLM
Ground Truth
Class
Adobe ✓
baidu.com ✗
Adobe
Evidence
(1) TLS record header detected . (2) High Shannon entropy : encrypted or binary payload. (3) Low zero-padding : sustained bulk data transfer. …
(1) TLS record header detected. (2) High zero-padding: short flow with small payload. (3) TCP dominant (68.5%), consistent with Baidu’s search and CDN infrastructure . …
(1) TLS record header detected. (2) Low ASCII with high entropy: encrypted payload. (3) TLS dominant (55.6%), consistent with Adobe profile. …
Description
TLS record headers alongside high-entropy payload confirm encrypted communication. Adobe creative software, cloud services, and document management platform over TLS 1.3. …
TLS record headers indicate active encrypted communication. Large volume of bulk control-plane packets over TCP. Baidu search and CDN infrastructure. …
TLS record headers alongside high-entropy payload confirm encrypted communication. Adobe creative software, cloud services, and document management platform over TLS 1.3. …
TABLE IX: CSTNet-TLS1.3 for sample #4728: mmTraffic correctly identifies the Adobe while Vanilla VLM misclassifies it as baidu.com . Under TLS 1.3, byte-level features are nearly identical across all websites. Green and Red mean correct and wrong interpretations.
Field
mmTraffic (Ours)
Vanilla VLM
Ground Truth
Class
Geodo ✓
Htbot ✗
Geodo
Evidence
(1) High Shannon entropy : encrypted or binary payload. (2) Low zero-padding : sustained bulk data transfer. (3) TCP dominant (100%). (4) Unlike Zeus (banking only), Geodo also performs spam relay .
(1) High zero-padding: short flow, small payload. (2) TCP dominant (100%). (3) Unlike Zeus (banking only), Geodo also performs spam relay. …
Description
Mixed entropy and ASCII characteristics. Geodo/Emotet banking trojan and email spam botnet . Dominant protocol: TCP (100%). …
HTTP method tokens and readable ASCII confirm plaintext botnet-driven web scraping or C&C command execution (Htbot) . Low throughput suggests sparse traffic. …
Mixed entropy and ASCII characteristics. Geodo/Emotet banking trojan and email spam botnet . Dominant protocol: TCP (100%).
TABLE X: USTC-TFC-2016 for sample #10195: mmTraffic correctly identifies the Geodo malware flow while Vanilla VLM misclassifies it as Htbot. Both are HTTP-based botnets with similar byte signatures. Green and Red mean correct and wrong interpretations.
Fig. 6: Ablation analysis on ISCX-Tor-2016 and ISCXVPN2016, with respect to the classification and generation metrics for four variants from V1 to V4.
(1) TCP dominant (100%), consistent with profile. (2) Unlike direct HTTPS browsing, Tor browsing shows only Tor relay IPs . (3) More bursty than Tor AUDIO or VOIP . …
(1) TCP dominant (100%), consistent with FILE-TRANSFER profile. (2) Unlike Tor P2P (bidirectional), FILE-TRANSFER is predominantly unidirectional. (3) FILE-TRANSFER shows sustained throughput vs Tor BROWSING (bursty with idle gaps). …
Description
HTTPS/SSL web browsing tunneled through Tor onion routing , including both direct Tor Browser usage and gateway-proxied SSL browsing. Dominant protocol: TCP. …
Mixed entropy and ASCII characteristics. File transfer protocols (FTP, SFTP, Skype file transfer) tunneled through Tor onion routing. Dominant protocol: TCP (100%).
Notes
Tor browsing provides strong anonymity; exit node traffic is unencrypted unless HTTPS is used end-to-end.
File transfer over Tor hides source and destination; commonly used to transfer sensitive documents.
TABLE XI: ISCX-Tor-2016 for sample #6622: mmTraffic fails to classify. Red means wrong interpretations in key semantics.
Field
mmTraffic (Ours)
Ground Truth
Class
arXiv ✗
Semantic Scholar
Traits
has_tls: true has_http: false ascii: low entropy: high zero_pad: mid
has_tls: true has_http: false ascii: mid entropy: high zero_pad: mid
Evidence
(1) TLS record header detected. (2) Low ASCII with high entropy: encrypted payload. (3) TCP dominant (96.8%), consistent with arXiv profile . (4) Similar PDF download pattern to ACM and IEEE . …
(1) TLS record header detected. (2) High Shannon entropy: encrypted payload. (3) TCP dominant (97.9%), consistent with Semantic Scholar profile. (4) More API-driven than traditional academic publishers. …
Description
TLS record headers alongside high-entropy payload confirm encrypted communication. arXiv open-access academic preprint repository for physics, math, and CS over TLS 1.3. Large avg. packet (925 bytes). …
TLS record headers alongside high-entropy payload confirm encrypted communication. Semantic Scholar AI-powered academic paper search and knowledge graph over TLS 1.3. Large avg. packet (964 bytes). …
Notes
Open academic repository; monitor for automated bulk downloading indicating research scraping bots.
Allen Institute AI-operated academic search; open access focus means fewer paywall authentication flows.
TABLE XII: CSTNet-TLS1.3 for sample #6404: mmTraffic fails to classify. Red means wrong interpretations in key semantics.
Field
mmTraffic (Ours)
Ground Truth
Class
Geodo ✗
Htbot
Traits
has_tls: false has_http: false ascii: high entropy: low zero_pad: mid
has_tls: false has_http: false ascii: high entropy: low zero_pad: mid
Evidence
(1) High ASCII with low entropy: repetitive plaintext content. (2) DNS dominant (100%), consistent with profile. (3) Unlike Zeus (banking only), Geodo also performs spam relay . …
(1) High ASCII with low entropy: repetitive plaintext content. (2) DNS dominant (100%), consistent with Htbot profile. (3) Unlike Miuref (click fraud), Htbot focuses on C&C command execution.
Description
Substantial readable ASCII content present. Geodo/Emotet banking trojan and email spam botnet . Dominant protocol: DNS (89.1%). Uses HTTP-based C&C more frequently than Zeus.
Substantial readable ASCII content present. HTTP-based botnet using web proxies for C&C (Htbot) . Dominant protocol: DNS (100%).
Notes
High-risk banking trojan; block known Geodo C&C IPs; inspect SMTP traffic for spam relay.
Monitor for HTTP requests with unusual headers; correlate with known Htbot infrastructure.
TABLE XIII: USTC-TFC-2016 for sample #4090: mmTraffic fails to classify. Red means wrong interpretations in key semantics.
Encrypted traffic classification has achieved strong performance, but its decision process remains difficult to interpret. Existing methods usually combine flow statistics, packet sequences, and byte-level representations into opaque latent features, making it unclear which type of evidence actually drives the prediction. In this paper, we propose Traffic-CBM, a structurally interpretable multimodal framework for encrypted traffic classification. Instead of directly fusing heterogeneous traffic signals into a black-box representation, Traffic-CBM organizes them into a unified hierarchical concept space. These concepts are not manually annotated semantic attributes; rather, they are scalar evidence summaries constrained by predefined traffic evidence groups. More specifically, grouped flow statistics are mapped to statistical concepts, dedicated temporal encoders learn temporal concepts from disjoint feature subspaces, and byte-level evidence is further organized into packet-level and cross-packet concepts. This design turns heterogeneous traffic evidence into an explicit concept representation and makes different levels of traffic evidence easier to analyze. We evaluate Traffic-CBM on multiple encrypted traffic benchmarks. Results show that it achieves competitive and balanced classification performance while providing a clearer structural interpretation interface than conventional end-to-end fusion models. Further analyses suggest that the learned concept space is actively used in the prediction process and provides a clearer structural explanation of multimodal traffic evidence.
Honglei Jin, Wenshuo Chen, Shaofeng Liang +6
HKUST (GZ) Guangzhou, China · Shandong University Qingdao, China
Encrypted traffic classification is a critical task for network security. While deep learning has advanced this field, the occlusion of payload semantics by encryption severely challenges standard modeling approaches. Most existing frameworks rely on static and homogeneous pipelines that apply uniform parameter sharing and static fusion strategies across all inputs. This one-size-fits-all static design is inherently flawed: by forcing structured headers and randomized payloads into a unified processing pipeline, it inevitably entangles the raw protocol signals with stochastic encryption noise, thereby degrading the fine-grained discriminative features. In this paper, we propose TrafficMoE, a framework that breaks through the bottleneck of static modeling by establishing a Disentangle-Filter-Aggregate (DFA) paradigm. Specifically, to resolve the structural between-components conflict, the architecture disentangles headers and payloads using dual-branch sparse Mixture-of-Experts (MoE), enabling modality-specific modeling. To mitigate the impact of stochastic noise, an uncertainty-aware filtering mechanism is introduced to quantify reliability and selectively suppress high-variance representations. Finally, to overcome the limitations of static fusion, a routing-guided strategy aggregates cross-modality features dynamically, that adaptively weighs contributions based on traffic context. With this DFA paradigm, TrafficMoE maximizes representational efficiency by focusing solely on the most discriminative traffic features. Extensive experiments on six datasets demonstrate TrafficMoE consistently outperforms state-of-the-art methods, validating the necessity of heterogeneity-aware modeling in encrypted traffic analysis. The source code is publicly available at https://github.com/Posuly/TrafficMoE_main.
Qing He, Xiaowei Fu, Lei Zhang
School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China.
Traditional traffic analysis is being fundamentally challenged by the rapid adoption of encryption, tunnelling, and privacy-preserving protocols, which increasingly obscure packet payloads and limit the usefulness of Deep Packet Inspection (DPI). Although machine learning has advanced encrypted traffic analysis, existing approaches often remain tied to protocol-specific header features, depend on large labelled datasets, and degrade when deployed across heterogeneous network environments. We present GETA, a protocol-agnostic framework for encrypted traffic analysis that models network flows as multivariate time series using only traffic metadata, thereby avoiding reliance on packet payloads or header semantics. GETA combines meta-learning, embedding refinement, and self-attention to support few-shot adaptation to previously unseen domains with minimal labelled data. Across nine public datasets spanning application identification, VPN traffic classification, IoT device fingerprinting, and attack detection, GETA consistently outperforms state-of-the-art baselines. These results show that GETA offers a practical and generalisable foundation for robust traffic analysis in modern encrypted networks.