Mixture-of-Experts (MoE) offers flexible graph reasoning by combining multiple views of a graph through a learned router. We investigate routing-aware explanations for MoE graph models in malware detection using control flow graphs (CFGs). Our architecture builds diversity at two levels. At the node level, each layer computes multiple neighborhood statistics and fuses them with an MLP, guided by a degree reweighting factor rho and a pooling choice lambda in {mean, std, max}, producing distinct node representations that capture complementary structural cues in CFGs. At the readout level, six experts, each tied to a specific (rho, lambda) view, output graph-level logits that the router weights into a final prediction. Post-hoc explanations are generated with edge-level attributions per expert and aggregated using the router gates so the rationale reflects both what each expert highlights and how strongly it is selected. Evaluated against single-expert GNN baselines such as GCN, GIN, and GAT on the same CFG dataset, the proposed MoE achieves strong detection accuracy while yielding stable, faithful attributions under sparsity-based perturbations. The results indicate that making the router explicit and combining multi-statistic node encoding with expert-level diversity can improve the transparency of MoE decisions for malware analysis.
Figures & tables
Figure 1: From CFG to prediction and explanation: diverse node representation, six experts, Top-k routing, and gate-weighted explanation.
Figure 2: The two-step node feature embedding process, including rule-based instruction encoding and autoencoder-based dimensionality reduction.
Dataset
#Samples
Avg. Nodes
Avg. Edges
Label
BODMAS
122
63,226.44
66,033.72
Malware
DikeDataset
319
9,059.07
15,171.37
Benign
PMML
390
14,246.54
23,977.81
Malware
Table 1: Characteristics of the evaluated datasets.
Model
Benign
Malicious
Accuracy
Precision
Recall
F1
Precision
Recall
F1
MalGNE framework ( Peng et al., 2024 )
GCN backbone
81.97
84.75
83.33
90.53
88.66
89.58
87.18
GAT backbone
86.21
84.75
85.47
90.82
91.75
91.28
89.10
Consistency framework ( Shokouhinejad et al., 2025a )
GCN backbone
81.67
83.05
82.35
89.58
88.66
89.12
86.54
Table 2: Classwise performance of different models.
Figure 3: Expert co-selection analysis in the Top-2 MoE configuration, comparing the diversity of expert activation with and without the load balancing term.
Figure 4: Distribution of gating weights across experts in the Top-2 MoE configuration with load balancing.
Figure 5: Average gate weights assigned to each expert in the temperature softmax configuration with 95% confidence intervals.
Figure 6: ECDF of normalized router entropy H~(α) in the temperature softmax configuration.
Figure 7: Comparison of Fidelity+ and Fidelity− across graph sparsity levels for five MoE configurations.
Figure 8: Characterization Score across sparsity levels for the evaluated MoE configurations.
Malware detection is a critical task in cybersecurity, and graph neural networks over control flow graphs have shown promising results for it. However, detectors are usually evaluated on a random split of a corpus collected over a single period, which cannot show how well a model generalizes to later samples. This study addresses that limitation with a strict temporal split: every model is trained on one period and scored once on a later one. Two corpora of control flow graphs, each node carrying 37 features, were extracted statically from 1,989 Windows portable executables: 459 graphs from 2024-2025 for training and 223 from 2026 for evaluation. Twelve variants and a flat-feature control were trained on the earlier corpus. The choice of message-passing operator changes robustness to the shift significantly, and every pairwise gap that survives correction separates an aggregating architecture from one built around a learned attentional readout. The ranking also reverses: the flat control, which sees node features but no topology, is the best in-distribution model and among the worst across the boundary, so a conventional benchmark would have rejected message passing. Neither recalibration nor ensembling substitutes for the operator choice. Attributions do not shift, but explanation validity is architecture-specific, and the most accurate operator on the later corpus is the hardest to explain. An architecture derived from the finding matches the best searched operator without search. The shift affects both malware and benign classes alike, so these are results about robustness to distribution shift, not malware evolution.
Sparse Mixture of Experts (MoE) models scale more efficiently than dense models by routing tokens to modular expert networks that are only active for processing a fraction of tokens. A leading hypothesis for the performance of MoE models is that each expert specialises in a single, coherent domain. However, interpretability efforts that assume this hypothesis have generally been unsuccessful. We propose and present evidence for an alternative account that we call the Superposed Specialisation Hypothesis (SSH): experts specialise in a disjoint union of fine-grained features rather than one broad domain. Leveraging the SSH, we introduce RouterInterp, a method for interpreting expert routing that identifies Sparse Autoencoder features most predictive of routing decisions and produces unified natural language explanations. On gpt-oss-20b, RouterInterp explains expert routing with ∼65% higher detection accuracy than prior token statistics based methods. This work provides a scalable method for generating more accurate explanations of expert routing and increases our understanding of a previously uninterpretable component of foundation models.
Ilya Lasy, Nora Yinuo Cai, Kola Ayonrinde
Faculty of Informatics, TU Wien · Independent · UK AI Security Institute
Sparse Mixture-of-Experts (MoE) models scale parameters while fixing active computation per token, but the specialization of individual experts remains opaque. In a companion paper we showed that routing topology is quality-neutral: five structurally different configurations converge to statistically equivalent language modeling quality. Here we show that expert identity is nonetheless causally meaningful: individual rank-1 experts are monosemantic by construction, and cosine-similarity routing in a low-dimensional metric space makes their specialization directly inspectable. We present four lines of evidence. First, projecting expert output vectors through the unembedding matrix yields a Semantic Dictionary: 15% of experts are monosemantic specialists spanning 10 categories (temporal, geographic, cardinal, discourse, emotional, financial, military, scientific). Second, routing exhibits a frequency-to-syntax gradient: early layers separate tokens by word frequency, deeper layers by syntactic class (Zipf-confound controls, all p<0.001). Third, causal interventions confirm these labels: steering toward a temporal expert's centroid increases P(temporal) by +321% (median across 44 prompts); suppressing a geographic expert drops P(geographic) by -23%; rewriting an expert's output vector halves target-category probability, and effects compose additively across layers. Fourth, the interventions are not unique to cosine routing: linear routers support comparable steering, but only cosine routing provides geometric transparency -- expert specialization is readable directly from the centroid matrix. MoE expert-level specialization is a first-class interpretability primitive: architecturally monosemantic, causally validated, and controllable at inference with zero overhead.
Ivan Ternovtsii, Yurii Bilak
Department of Software Systems, Uzhhorod National University, Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes