Organizations: Jiajun Zhou, Yadong Li, Xuanze Chen, Chen Ma, Shanqing Yu and Qi Xuan are with the Institute of Cyberspace Security, Zhejiang University of Technology, Hangzhou 310023, China, with the Binjiang Cyberspace Security Institute of ZJUT, Hangzhou, 310056, China, and with the Soovar Technologies Co., Ltd., Hangzhou 310056, China. · Chuang Zhao is with the Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, China.
Mixture-of-Experts (MoE) architectures offer a scalable path for Graph Neural Networks (GNNs) in node classification tasks but typically rely on static and rigid routing strategies that enforce a uniform expert budget or coarse-grained expert toggles on all nodes. This limitation overlooks the varying discriminative difficulty of nodes and leads to under-fitting for hard nodes and redundant computation for easy ones. To resolve this issue, we propose D2MoE, a novel framework that shifts the focus from static expert selection to node-wise expert resource allocation. By using predictive entropy as a real-time proxy for difficulty, D2MoE employs a difficulty-driven top-p routing mechanism to adaptively concentrate expert resources on hard nodes while reducing overhead for easy ones, achieving continuous and fine-grained expert budget scaling for node classification. Experiments on 13 benchmarks demonstrate that D2MoE achieves consistent state-of-the-art performance, surpassing leading baselines by up to 7.92% in accuracy on heterophilous graphs. Notably, on large-scale graphs, it reduces memory consumption by up to 73.07% and training time by 46.53% compared to the best-performing Graph MoE, thereby validating its superior efficiency.
Figures & tables
Fig. 1 : Impact of static expert budgets (top- k ) on node classification performance across predictive entropy intervals.
Fig. 2 : The architecture of D 2 MoE. The complete workflow proceeds as follows: (1) D 2 MoE first quantifies node-wise difficulty via predictive entropy; (2) Top- p router adaptively scales the expert resources, activating a sparse set of experts for easy nodes while mobilizing a comprehensive ensemble for hard nodes; (3) Selected expert outputs are fused to update node representations, which generate predictions and initiate the next difficulty quantification.
TABLE I : Node classification results: average test accuracy (%) ± standard deviation. Boldface letters mark the best performance while underlined letters indicate the second best.
Fig. 3 : Fine-grained performance comparison across different node difficulty levels. We stratify test nodes into 10 equal-sized intervals (deciles) based on their predictive entropy (calculated by a fixed proxy model). The x-axis represents entropy percentiles (from Easy to Hard), and the y-axis denotes classification accuracy. D 2 MoE demonstrates superior robustness, particularly in high-entropy (hard) intervals.
Fig. 4 : Ablation analysis on D 2 MoE.
Fig. 5 : Node density distribution and average number of activated experts across varying levels of discriminative difficulty.
Fig. 6 : Visualization of learned node representations via t-SNE across 12 datasets. For each dataset, nodes are colored by Ground-Truth Labels (Top) and Activated Expert Counts (Bottom). The results highlight a clear boundary-focused resource allocation pattern.
Fig. 7 : Visualization of expert activation patterns across 12 datasets. The heatmaps illustrate the average routing weights assigned to different experts (Y-axis) across four quartiles of node predictive entropy (X-axis, ranging from low to high). Brighter colors indicate higher activation weights.
Fig. 8 : Efficiency analysis on large-scale graphs.
Fig. 9 : All results for Ogbn-arxiv: (a) Visualization of average routing weights; (b) Label distribution vs. expert activation count distribution; (c) Performance comparison across node difficulty levels; (d) Ablation analysis on D 2 MoE; (e) Node density distribution and average number of acti- vated experts across varying levels of discriminative difficulty.
Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling neural networks while maintaining computational efficiency. However, standard MoE implementations rely on two rigid design assumptions: (1) fixed Top-K routing where exactly K experts are activated per token, and (2) uniform expert allocation across all layers. This paper introduces DynaMoE, a novel MoE framework that relaxes both constraints through dynamic token-level expert activation and layer-wise adaptive capacity allocation. DynaMoE introduces a principled routing mechanism where the number of active experts per token varies based on input complexity. Concurrently, the framework implements six distinct scheduling strategies for distributing expert capacity across network depth, including descending, ascending, pyramid, and wave patterns. We theoretically analyze the expressivity gains of dynamic routing and derive bounds on computational efficiency. Through extensive experiments on MNIST, Fashion-MNIST, CIFAR-10 (image classification), and Recycling-the-Web (language modeling) across multiple model scales, we demonstrate that DynaMoE achieves superior parameter efficiency compared to static baselines. Our key finding is that optimal expert schedules are task- and scale-dependent: descending schedules (concentrating capacity in early layers) outperform uniform baselines on image classification. For language modeling, optimal schedules vary by model size, descending for Tiny, ascending for Small, and uniform for Medium. Furthermore, dynamic routing reduces gradient variance during training, leading to improved convergence stability. DynaMoE establishes a new framework for adaptive computation in neural networks, providing principled guidance for MoE architecture design.
Mixture-of-Experts (MoE) offers flexible graph reasoning by combining multiple views of a graph through a learned router. We investigate routing-aware explanations for MoE graph models in malware detection using control flow graphs (CFGs). Our architecture builds diversity at two levels. At the node level, each layer computes multiple neighborhood statistics and fuses them with an MLP, guided by a degree reweighting factor rho and a pooling choice lambda in {mean, std, max}, producing distinct node representations that capture complementary structural cues in CFGs. At the readout level, six experts, each tied to a specific (rho, lambda) view, output graph-level logits that the router weights into a final prediction. Post-hoc explanations are generated with edge-level attributions per expert and aggregated using the router gates so the rationale reflects both what each expert highlights and how strongly it is selected. Evaluated against single-expert GNN baselines such as GCN, GIN, and GAT on the same CFG dataset, the proposed MoE achieves strong detection accuracy while yielding stable, faithful attributions under sparsity-based perturbations. The results indicate that making the router explicit and combining multi-statistic node encoding with expert-level diversity can improve the transparency of MoE decisions for malware analysis.
Graph Neural Networks (GNNs) excel on graph-structured data but suffer catastrophic forgetting when classes arrive sequentially. In class-incremental graph learning, predictions are often dominated by recently learned classes, causing severe recency bias. Dynamic expansion can isolate knowledge across sessions, but many methods assume task-incremental settings and require task identities at test time to select branches or heads---information unavailable in class-incremental learning. Learnable routers offer an alternative, yet continual fine-tuning the router on new data can bias routing toward recent components. We propose MAEM, a replay-free, capacity-controlled dynamic mixture-of-experts framework for class-incremental graph learning. MAEM builds representations with Simplified Graph Convolution (SGC) and trains a dedicated expert for each new session, then freezes it to protect prior knowledge. For routing without task IDs or a shared trainable router, each expert uses a masked autoencoder (MAE); test nodes are assigned to the expert with the lowest reconstruction error, reducing routing-level recency bias. When the expert pool reaches a fixed capacity, MAEM merges the least-used experts via Git Re-Basin alignment and further distills them into a fused expert using topology-aware pseudo data synthesized from MAE distributions (DeepInversion-style), without storing or revisiting historical graph data. On four class-incremental graph benchmarks, MAEM mitigates recency bias, outperforms strong continual graph learning baselines, and keeps model capacity bounded under strict replay-free learning.
Junlin Chen, Wenhao Wang, Junchao Xu +5
Beihang University, Beijing, China · Renmin University of China, Beijing, China · Trine University, Indiana, USA +1