cs.CRApr 9, 2026
SaveMultimodal Reasoning with LLM for Encrypted Traffic Interpretation: A Benchmark
Organizations: School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China. · School of Data Science, Lingnan University, Hong Kong, China.
Abstract
Network traffic, as a key media format, is crucial for ensuring security and communications in modern internet infrastructure. While existing methods offer excellent performance, they face two key bottlenecks: (1) They fail to capture multidimensional semantics beyond unimodal sequence patterns. (2) Their black box property, i.e., providing only category labels, lacks an auditable reasoning process. We identify a key factor that existing network traffic datasets are primarily designed for classification and inherently lack rich semantic annotations, failing to generate human-readable evidence report. To address data scarcity, this paper proposes a Byte-Grounded Traffic Description (BGTD) benchmark for the first time, combining raw bytes with structured expert annotations. BGTD provides necessary behavioral features and verifiable chains of evidence for multimodal reasoning towards explainable encrypted traffic interpretation. Built upon BGTD, this paper proposes an end-to-end traffic-language representation framework (mmTraffic), a multimodal reasoning architecture bridging physical traffic encoding and semantic interpretation. In order to alleviate modality interference and generative hallucinations, mmTraffic adopts a jointly-optimized perception-cognition architecture. By incorporating a perception-centered traffic encoder and a cognition-centered LLM generator, mmTraffic achieves refined traffic interpretation with guaranteed category prediction. Extensive experiments demonstrate that mmTraffic autonomously generates high-fidelity, human-readable, and evidence-grounded traffic interpretation reports, while maintaining highly competitive classification accuracy comparing to specialized unimodal model (e.g., NetMamba). The source code is available at https://github.com/lgzhangzlg/Multimodal-Reasoning-with-LLM-for-Encrypted-Traffic-Interpretation-A-Benchmark
Figures & tables
Fig. 1: Comparison of traffic analysis paradigms. (a) Traditional classification methods that act as a “black box”, providing only a label and low-level feature weights that lack operational value. (b) Our proposed multimodal reasoning framework, composed of a Traffic Perception Encoder and a Cognitive LLM, instructed by Byte-Grounded Knowledge, generating an evidence-grounded report with human-understandable reasoning and executable insights.
| Model | Traffic Representation | Core Structure | Pre-training | Limitations |
| ET-BERT [ 19 ] | 4-hex Bigram / Burst Segmentation | Transformer Encoder | MBM / SBP | Ignores protocol hierarchy; uses natural language subword tokenization |
| YaTC [ 38 ] | Multi-level Flow Representation (MFR) Matrix | Dual-Attention Transformer | MAE (Matrix Masking) | Fixed matrix dimensions; truncates long-range session features |
| NetMamba [ 34 ] | Stride-based Byte Sequence | Unidirectional Mamba (SSM) | Masked Stride Reconstruction | Purely numerical mapping; lacks interpretability |
| FlowletFormer [ 22 ] | Flowlet Behavioral Unit / Field Tokenization | Transformer Encoder | MFM / FPT | Black-box classifier; unable to output forensic reasoning |
TABLE I: Comparisons of different paradigms for network traffic analysis. MBM, SBP, MAE, MFM, and FPT represent Masked Byte Model, Segment Burst Prediction, Masked Autoencoder, Masked Flow Model, and Flow Prediction Task, respectively.
Fig. 2: Pipeline of developing BGTD dataset: (a) session extraction and class balancing from raw PCAP files, (b) fixed-length NPY array generation via priority-based packet sampling, and (c) LLM-assisted ground-truth synthesis using Claude Opus-4.6 prompted as a senior network security expert.
Fig. 3: Statistical overview of the BGTD dataset.
Fig. 4: Overview of the mmTraffic framework. (a) The frozen traffic encoder extracts high-dimensional features from raw traffic data. (b) The linear connector projects traffic features into the LLM token space, with the CGHF mechanism injecting a class-aware anchor token into the input sequence. (c) The LLM autoregressively generates a structured forensic report containing behavioral traits, evidence chain, and diagnostic description.
| Dataset | Classes | Train | Test | ||
| CrossPlatform-Android | 212 | 50 | 2,000 | 31,029 | 7,644 |
| CrossPlatform-iOS | 196 | 50 | 3,000 | 29,302 | 7,233 |
| ISCXVPN2016 | 7 | 200 | 6,000 | 33,600 | 8,400 |
| ISCX-Tor-2016 | 8 | 3,000 | 10,000 | 64,000 | 16,000 |
| CSTNet-TLS1.3 | 120 | 0 | 6,000 | 37,148 | 9,224 |
| USTC-TFC-2016 | 12 | 3,000 | 6,000 | 53,112 | 13,276 |
TABLE II: Statistics of the six benchmark datasets after preprocessing.
| Dataset | Method | Classification | Generation | Evidence | Description | ||
| Acc / JClsAcc% | JSON Valid% | ROUGE-L | BERTScore | ROUGE-L | BERTScore | ||
| ISCX-Tor-2016 | NetMamba | 0.9961 | – | – | – | – | – |
| Zero-shot LLM | 0.0003 | 100.00% | 0.1247 | 0.8322 | 0.1164 | 0.8469 | |
| Vanilla | 0.7092 | 100.00% | 0.6002 | 0.9217 | 0.5831 | 0.9266 | |
| mmTraffic (Ours) | 0.9331 | 100.00% | 0.8192 | 0.9641 | 0.7751 | 0.9481 | |
| ISCXVPN2016 | NetMamba | 0.9917 | – | – | – | – | – |
TABLE III: Results on ISCX-Tor-2016, ISCXVPN2016, and CSTNet-TLS1.3. NetMamba is the unimodal classifier (“–” means no generation); Zero-shot LLM feeds features directly to LLM without tuning; Vanilla freezes the encoder without auxiliary constraints. Bold indicates the best per column within each dataset group.
| Dataset | Method | Classification | Generation | Evidence | Description | ||
| Acc / JClsAcc% | JSON Valid% | ROUGE-L | BERTScore | ROUGE-L | BERTScore | ||
| CrossPlatform-iOS | NetMamba | 0.9060 | – | – | – | – | – |
| Zero-shot LLM | 0.0000 | 100.00% | 0.1962 | 0.8509 | 0.1268 | 0.8503 | |
| Vanilla | 0.0058 | 100.00% | 0.2218 | 0.8591 | 0.1255 | 0.8535 | |
| mmTraffic (Ours) | 0.8865 | 100.00% | 0.6880 | 0.9387 | 0.5972 | 0.9283 | |
| CrossPlatform-Android | NetMamba | 0.9104 | – | – | – | – | – |
TABLE IV: Results on CrossPlatform-iOS, CrossPlatform-Android, and USTC-TFC-2016. NetMamba is the unimodal classifier (“–” means no generation); Zero-shot LLM feeds features directly to LLM without tuning; Vanilla freezes the encoder without auxiliary constraints. Bold indicates the best per column within each dataset group.
Fig. 5: Analysis on Structural Consistency Metrics. The semantic-priority constraints in mmTraffic ensure high logical rigor.
| Field | mmTraffic (Ours) | Ground Truth |
| Class | CHAT ✓ | CHAT |
| Traits | has_tls: true has_http: false ascii: low entropy: mid zero_pad: mid | has_tls: true has_http: false ascii: mid entropy: mid zero_pad: mid |
| Evidence | (1) TLS record header detected , indicating an encrypted TLS session. (2) TCP dominant (100%) , consistent with Tor CHAT profile. (3) Unlike Tor VOIP (codec-driven timing), Tor CHAT has irregular human-conversation-driven timing . … | (1) TLS record header detected, indicating an encrypted TLS session. (2) TLS dominant (100%), consistent with Tor CHAT profile. (3) Unlike Tor VOIP (codec-driven timing), Tor CHAT has irregular human-conversation-driven timing. … |
| Description | TLS record headers present, indicating encrypted transport. Instant messaging and chat protocols (AIM, ICQ, Facebook Chat, Hangouts, Skype Chat) tunneled through Tor onion routing. Dominant protocol: TCP (100%). … | TLS record headers present, indicating encrypted transport. Instant messaging and chat protocols (AIM, ICQ, Facebook Chat, Hangouts, Skype Chat) tunneled through Tor onion routing. Dominant protocol: TLS (100%). |
| Notes | Chat over Tor hides who is communicating with whom; message content may still be visible at exit nodes if not end-to-end encrypted. | Chat over Tor provides metadata anonymity; message content may still be visible at exit nodes if not end-to-end encrypted. |
TABLE V: ISCX-Tor-2016 for sample #6227: mmTraffic correctly identifies the class. Evidence and description fields are abbreviated. Green means correctly interpreted key content.
| Field | mmTraffic (Ours) | Ground Truth |
| Class | Steam ✓ | Steam |
| Traits | has_tls: true has_http: false ascii: mid entropy: high zero_pad: low | has_tls: true has_http: false ascii: low entropy: high zero_pad: low |
| Evidence | (1) TLS record header detected , indicating an encrypted TLS session. (2) High Shannon entropy : encrypted or compressed payload. (3) Low zero-padding : sustained bulk data transfer. … | (1) TLS record header detected, indicating an encrypted TLS session. (2) Low ASCII with high entropy: encrypted payload. (3) Low zero-padding: sustained bulk data transfer. (4) TCP dominant (84.2%). … |
| Description | TLS record headers alongside high-entropy payload confirm encrypted communication. Valve Steam gaming platform for game downloads, updates, and multiplayer services over TLS 1.3. Large avg. packet (858 bytes). … | TLS record headers alongside high-entropy payload confirm encrypted communication. Valve Steam gaming platform for game downloads, updates, and multiplayer services over TLS 1.3. TCP dominant (84.2%). … |
| Notes | Game distribution platform; monitor for Steam account phishing; Workshop mods can contain malicious code. | Game distribution platform; game downloads are very large; monitor for Steam account phishing. |
TABLE VI: CSTNet-TLS1.3 for sample #350: mmTraffic correctly identifies the class. Evidence and description fields are abbreviated. Green means correctly interpreted key content.
| Field | mmTraffic(Ours) | Ground Truth |
| Class | Outlook ✓ | Outlook |
| Traits | has_tls: true has_http: false ascii: low entropy: mid zero_pad: mid | has_tls: true has_http: false ascii: low entropy: high zero_pad: low |
| Evidence | (1) TLS record header detected , indicating an encrypted TLS session. (2) TLS dominant (100%) , consistent with Outlook profile. (3) Unlike Gmail (HTTP/2 and QUIC), Outlook uses MAPI protocol . … | (1) TLS record header detected, indicating an encrypted TLS session. (2) Low ASCII with high entropy: encrypted payload. (3) Low zero-padding: sustained bulk data transfer. (4) TLS dominant (100%). … |
| Description | TLS record headers present, indicating encrypted transport. Microsoft Outlook email client using HTTPS and MAPI . Dominant protocol: TLS (100%). … | TLS record headers alongside high-entropy payload confirm encrypted communication. Microsoft Outlook email client using HTTPS and MAPI . Dominant protocol: TLS (100%). |
| Notes | Generally benign; verify connections go to Microsoft ASN; anomalous sync volumes may indicate data exfiltration. | Generally benign; verify connections go to Microsoft ASN; anomalous sync volumes or off-hours activity may indicate data exfiltration via email. |
TABLE VII: USTC-TFC-2016 for sample #11786: mmTraffic correctly identifies the class. Evidence and description fields are abbreviated. Green means correctly interpreted key content.
| Field | mmTraffic (Ours) | Vanilla VLM | Ground Truth |
| Class | VIDEO ✓ | BROWSING ✗ | VIDEO |
| Evidence | (1) Video streaming services (YouTube, Vimeo) tunneled via Tor onion routing. (2) High throughput (49.8 MB/s) indicates active data exchange. (3) Sustained download dominant over upload , consistent with video consumption. … | (1) TLS record header detected, indicating encrypted session. (2) High zero-padding: short flow with small payload . (3) TCP dominant (100%), consistent with HTTPS browsing profile . … | (1) TCP dominant (100%), consistent with VIDEO profile. (2) Unlike Tor AUDIO (moderate throughput), VIDEO requires significantly higher bandwidth. (3) Unlike Tor BROWSING (bursty), VIDEO maintains sustained high-throughput download. … |
| Description | Mixed entropy and ASCII across packet slices. Video streaming services (youtube.com, vimeo.com) tunneled via Tor onion routing. TCP dominant. High throughput (498 KB/s). … | TLS record headers indicate encrypted transport. HTTPS/SSL web browsing tunneled through Tor hides both location and activities. High zero-padding suggests background activity. … | Mixed entropy and ASCII across packet slices. Video streaming services (YouTube HTML5, YouTube Flash, Vimeo) tunneled through Tor. Dominant protocol: TCP (100%). |
TABLE VIII: ISCX-Tor-2016 for sample #15750: mmTraffic correctly identifies the VIDEO class while Vanilla VLM misclassifies it as BROWSING. Evidence and description fields are abbreviated. Green and Red mean correct and wrong interpretations.
| Field | mmTraffic (Ours) | Vanilla VLM | Ground Truth |
| Class | Adobe ✓ | baidu.com ✗ | Adobe |
| Evidence | (1) TLS record header detected . (2) High Shannon entropy : encrypted or binary payload. (3) Low zero-padding : sustained bulk data transfer. … | (1) TLS record header detected. (2) High zero-padding: short flow with small payload. (3) TCP dominant (68.5%), consistent with Baidu’s search and CDN infrastructure . … | (1) TLS record header detected. (2) Low ASCII with high entropy: encrypted payload. (3) TLS dominant (55.6%), consistent with Adobe profile. … |
| Description | TLS record headers alongside high-entropy payload confirm encrypted communication. Adobe creative software, cloud services, and document management platform over TLS 1.3. … | TLS record headers indicate active encrypted communication. Large volume of bulk control-plane packets over TCP. Baidu search and CDN infrastructure. … | TLS record headers alongside high-entropy payload confirm encrypted communication. Adobe creative software, cloud services, and document management platform over TLS 1.3. … |
TABLE IX: CSTNet-TLS1.3 for sample #4728: mmTraffic correctly identifies the Adobe while Vanilla VLM misclassifies it as baidu.com . Under TLS 1.3, byte-level features are nearly identical across all websites. Green and Red mean correct and wrong interpretations.
| Field | mmTraffic (Ours) | Vanilla VLM | Ground Truth |
| Class | Geodo ✓ | Htbot ✗ | Geodo |
| Evidence | (1) High Shannon entropy : encrypted or binary payload. (2) Low zero-padding : sustained bulk data transfer. (3) TCP dominant (100%). (4) Unlike Zeus (banking only), Geodo also performs spam relay . | (1) HTTP tokens (GET/POST) indicate plaintext HTTP requests. (2) High ASCII: plaintext application-layer content. (3) Low zero-padding: sustained bulk data transfer. … | (1) High zero-padding: short flow, small payload. (2) TCP dominant (100%). (3) Unlike Zeus (banking only), Geodo also performs spam relay. … |
| Description | Mixed entropy and ASCII characteristics. Geodo/Emotet banking trojan and email spam botnet . Dominant protocol: TCP (100%). … | HTTP method tokens and readable ASCII confirm plaintext botnet-driven web scraping or C&C command execution (Htbot) . Low throughput suggests sparse traffic. … | Mixed entropy and ASCII characteristics. Geodo/Emotet banking trojan and email spam botnet . Dominant protocol: TCP (100%). |
TABLE X: USTC-TFC-2016 for sample #10195: mmTraffic correctly identifies the Geodo malware flow while Vanilla VLM misclassifies it as Htbot. Both are HTTP-based botnets with similar byte signatures. Green and Red mean correct and wrong interpretations.
Fig. 6: Ablation analysis on ISCX-Tor-2016 and ISCXVPN2016, with respect to the classification and generation metrics for four variants from V1 to V4.
| Field | mmTraffic (Ours) | Ground Truth |
| Class | BROWSING ✗ | FILE-TRANSFER |
| Traits | has_tls: false has_http: false ascii: low entropy: mid zero_pad: mid | has_tls: false has_http: false ascii: mid entropy: low zero_pad: mid |
| Evidence | (1) TCP dominant (100%), consistent with profile. (2) Unlike direct HTTPS browsing, Tor browsing shows only Tor relay IPs . (3) More bursty than Tor AUDIO or VOIP . … | (1) TCP dominant (100%), consistent with FILE-TRANSFER profile. (2) Unlike Tor P2P (bidirectional), FILE-TRANSFER is predominantly unidirectional. (3) FILE-TRANSFER shows sustained throughput vs Tor BROWSING (bursty with idle gaps). … |
| Description | HTTPS/SSL web browsing tunneled through Tor onion routing , including both direct Tor Browser usage and gateway-proxied SSL browsing. Dominant protocol: TCP. … | Mixed entropy and ASCII characteristics. File transfer protocols (FTP, SFTP, Skype file transfer) tunneled through Tor onion routing. Dominant protocol: TCP (100%). |
| Notes | Tor browsing provides strong anonymity; exit node traffic is unencrypted unless HTTPS is used end-to-end. | File transfer over Tor hides source and destination; commonly used to transfer sensitive documents. |
TABLE XI: ISCX-Tor-2016 for sample #6622: mmTraffic fails to classify. Red means wrong interpretations in key semantics.
| Field | mmTraffic (Ours) | Ground Truth |
| Class | arXiv ✗ | Semantic Scholar |
| Traits | has_tls: true has_http: false ascii: low entropy: high zero_pad: mid | has_tls: true has_http: false ascii: mid entropy: high zero_pad: mid |
| Evidence | (1) TLS record header detected. (2) Low ASCII with high entropy: encrypted payload. (3) TCP dominant (96.8%), consistent with arXiv profile . (4) Similar PDF download pattern to ACM and IEEE . … | (1) TLS record header detected. (2) High Shannon entropy: encrypted payload. (3) TCP dominant (97.9%), consistent with Semantic Scholar profile. (4) More API-driven than traditional academic publishers. … |
| Description | TLS record headers alongside high-entropy payload confirm encrypted communication. arXiv open-access academic preprint repository for physics, math, and CS over TLS 1.3. Large avg. packet (925 bytes). … | TLS record headers alongside high-entropy payload confirm encrypted communication. Semantic Scholar AI-powered academic paper search and knowledge graph over TLS 1.3. Large avg. packet (964 bytes). … |
| Notes | Open academic repository; monitor for automated bulk downloading indicating research scraping bots. | Allen Institute AI-operated academic search; open access focus means fewer paywall authentication flows. |
TABLE XII: CSTNet-TLS1.3 for sample #6404: mmTraffic fails to classify. Red means wrong interpretations in key semantics.
| Field | mmTraffic (Ours) | Ground Truth |
| Class | Geodo ✗ | Htbot |
| Traits | has_tls: false has_http: false ascii: high entropy: low zero_pad: mid | has_tls: false has_http: false ascii: high entropy: low zero_pad: mid |
| Evidence | (1) High ASCII with low entropy: repetitive plaintext content. (2) DNS dominant (100%), consistent with profile. (3) Unlike Zeus (banking only), Geodo also performs spam relay . … | (1) High ASCII with low entropy: repetitive plaintext content. (2) DNS dominant (100%), consistent with Htbot profile. (3) Unlike Miuref (click fraud), Htbot focuses on C&C command execution. |
| Description | Substantial readable ASCII content present. Geodo/Emotet banking trojan and email spam botnet . Dominant protocol: DNS (89.1%). Uses HTTP-based C&C more frequently than Zeus. | Substantial readable ASCII content present. HTTP-based botnet using web proxies for C&C (Htbot) . Dominant protocol: DNS (100%). |
| Notes | High-risk banking trojan; block known Geodo C&C IPs; inspect SMTP traffic for spam relay. | Monitor for HTTP requests with unusual headers; correlate with known Htbot infrastructure. |
TABLE XIII: USTC-TFC-2016 for sample #4090: mmTraffic fails to classify. Red means wrong interpretations in key semantics.
More in this topic
Encrypted Network Traffic Analysis
Encrypted network traffic analysis infers applications, threats, and network conditions from traffic patterns without decrypting payloads.
See all 29 papers