Pruned Traffic Trees: Native Semantic Compression with a Protocol-Structured Model Family for Encrypted Traffic Classification

Network traffic classification underpins network security and management but faces growing challenges from pervasive encryption and evolving protocols. Existing representations involve clear trade-offs: hand-crafted flow statistics are efficient but lossy, raw-bit encodings can be accurate but costly, and pre-trained embeddings support transfer but often flatten the protocol stack and entangle cross-layer signals. We observe substantial redundancy both across and within network layers, which existing paradigms do not explicitly address, wasting capacity and encouraging shortcut learning that degrades generalization. To address this, we propose PACC, a redundancy-aware cross-layer representation framework. PACC models protocol layers as multiview inputs and learns faithful, compact projections factorized into shared (cross-layer) and private (layer-specific) components. It incorporates an information-preserving sparse segment representation to reduce structural repetition and supervised intra-layer filtering to suppress task-irrelevant semantic redundancy. A behavior-guided shared/private decomposition aligns cross-layer consensus while retaining layer-specific information, supported by information-theoretic analysis. Across encrypted application classification, IoT device identification, benign/malicious traffic classification, and Tor traffic detection, PACC achieves the best overall performance. On encrypted traffic, it achieves an F1 score of 0.9685, substantially outperforming existing baselines while using approximately 15x fewer computes and less GPU memory than the latest pre-trained traffic model.

Graph-based deep learning methods have been widely employed in encrypted traffic analysis to exploit latent correlations across different granularities. However, while complex preprocessing pipelines and sophisticated model structures often achieve strong performance, they may obscure inherent protocol semantics during representation learning. Moreover, the hierarchical structure of protocol layers and their corresponding fields, defined by protocol specifications and routinely utilized in manual traffic analysis, remains underexplored in existing learning frameworks. In this paper, we propose Protocol Tree Graph Attention with Mixture of Experts (PTGAMoE), a semantic-preserving hierarchical graph-based expert framework for encrypted traffic analysis. The field-based graph construction and expert committee design enable PTGAMoE to quantify the model's preferences for specific fields and protocols. Extensive experimental results on representative benchmark datasets under strict no-data-leakage settings demonstrate that PTGAMoE significantly outperforms state-of-the-art (SOTA) models. Furthermore, the semantic-preserving design provides interpretable insights into protocol-level feature importance and expert-level contributions, reflecting the model's decision-making logic in encrypted traffic classification tasks.

Encrypted traffic classification is a critical task for network security. While deep learning has advanced this field, the occlusion of payload semantics by encryption severely challenges standard modeling approaches. Most existing frameworks rely on static and homogeneous pipelines that apply uniform parameter sharing and static fusion strategies across all inputs. This one-size-fits-all static design is inherently flawed: by forcing structured headers and randomized payloads into a unified processing pipeline, it inevitably entangles the raw protocol signals with stochastic encryption noise, thereby degrading the fine-grained discriminative features. In this paper, we propose TrafficMoE, a framework that breaks through the bottleneck of static modeling by establishing a Disentangle-Filter-Aggregate (DFA) paradigm. Specifically, to resolve the structural between-components conflict, the architecture disentangles headers and payloads using dual-branch sparse Mixture-of-Experts (MoE), enabling modality-specific modeling. To mitigate the impact of stochastic noise, an uncertainty-aware filtering mechanism is introduced to quantify reliability and selectively suppress high-variance representations. Finally, to overcome the limitations of static fusion, a routing-guided strategy aggregates cross-modality features dynamically, that adaptively weighs contributions based on traffic context. With this DFA paradigm, TrafficMoE maximizes representational efficiency by focusing solely on the most discriminative traffic features. Extensive experiments on six datasets demonstrate TrafficMoE consistently outperforms state-of-the-art methods, validating the necessity of heterogeneity-aware modeling in encrypted traffic analysis. The source code is publicly available at https://github.com/Posuly/TrafficMoE_main.