Bridging Local and Global Knowledge: Cascaded Mixture-of-Experts Learning for Near-Shortest Path Routing

Yung-Fu Chen; Anish Arora

cs.LGMar 16, 2026

Bridging Local and Global Knowledge: Cascaded Mixture-of-Experts Learning for Near-Shortest Path Routing

Authors: Yung-Fu Chen, Anish Arora

Organizations: The Ohio State University, Columbus, OH, USA

Abstract

While deep learning models that leverage local features have demonstrated significant potential for near-optimal routing in dense Euclidean graphs, they struggle to generalize well in sparse networks where topological irregularities require broader structural awareness. To address this limitation, we train a Cascaded Mixture of Experts (Ca-MoE) to solve the all-pairs near-shortest path (APNSP) routing problem. Our Ca-MoE is a modular two-tier architecture that supports the decision-making for forwarder selection with lower-tier experts relying on local features and upper-tier experts relying on global features. It performs adaptive inference wherein the upper-tier experts are triggered only when the lower-tier ones do not suffice to achieve adequate decision quality. Computational efficiency is thus achieved by escalating model capacity only when necessitated by topological complexity, and parameter redundancy is avoided. Furthermore, we incorporate an online meta-learning strategy that facilitates independent expert fine-tuning and utilizes a stability-focused update mechanism to prevent catastrophic forgetting as new graph environments are encountered. Experimental evaluations demonstrate that Ca-MoE routing improves accuracy by up to 29.1% in sparse networks compared to single-expert baselines and maintains performance within 1%-6% of the theoretical upper bound across diverse graph densities.

Figures & tables

Fig. 1: The Ca-MoE framework for forwarder selection. The Router directs the graph sample $\langle O,D,v,nbr(v)\rangle$ to a selected lower-tier expert that predicts a $Q$ -value for each neighbor node $u$ in $nbr(v)$ , where $nbr(v)$ denotes the set of $v$ ’s neighbors. The candidate forwarder $\hat{u}$ for a near-shortest path from node $v$ to destination node $D$ is identified as a node in $nbr(v)$ with the highest $Q$ -value. This candidate is validated by the Deferral rule : the sample is cascaded to the upper-tier expert only if the confidence is insufficient. Dashed lines denote conditional execution paths, illustrating selective activation based on the graph sample (in orange) and the adaptive activation based on the candidate forwarder for the graph sample (in purple).

Panel of Figure 2 — Fig. 2: Comparative performance of the Greedy-Tensile , Greedy-Lax , and Greedy-Spectral routing policies across different densities of Euclidean graphs. Each sub-figure illustrates a set of graphs where a different routing policy dominates: different graphs are enumerated on the x-axis; for each graph, the y-axis denotes the frequency with which each expert selects the best next forwarder for nodes $v$ given samples $\langle O,D,v,nbr(v)\rangle$ from that graph. In sub-figures (b) and (c), the choices for $v$ are constrained relative to the shortest paths from origin $O$ to $D$ . Recall that a forwarder is determined to be the “best” if it yields the maximal optimal $Q$ -value, $Q^{*}(O,D,v,u)$ .

Fig. 3: The online meta-learning pipeline. Upon the arrival of a new graph $G_{t}$ , experts are selectively fine-tuned via a fine-tuning sampler. To prevent catastrophic forgetting, updated models ( $H^{\prime}$ ) are validated against a persistent Evaluation Dataset ( $ED$ ) before replacing the current experts ( $H^{t-1}$ ).

Symbol	Meaning	Value
$N_{train}$	size of seed graphs for training experts	50
$\rho_{GT}$	density of seed graph	5
	for training Greedy-Tensile
$\rho_{GL/GS}$	density of seed graphs	2
	for training Greedy-Lax and Greedy-Spectral
$R$	communication radius	1000

TABLE I: Simulation Parameters

Panel of Figure 5 — Fig. 4: Comparative APNSP prediction accuracy on individual graph instances ( $G_{t}$ ) for Ca-MoE against single-expert baselines. The x-axis enumerates specific graph instances, while the y-axis reports the routing accuracy.

Panel of Figure 6 — Fig. 5: Evolution of average APNSP prediction accuracy across the sequence of seen graphs $G_{1}..G_{t}$ shown in Figure 4 . The figure illustrates the continual generalization capability of the framework. The x-axis represents the chronological sequence of arriving graphs, and the y-axis denotes the average accuracy on all graphs encountered up to time $t$ .

Fig. 6: Comparative APNSP prediction accuracy across the sequence of seen graphs $G_{1}..G_{t}$ shown in Figure 5(a) . This comparison highlights the framework’s robustness to initialization by tracking the convergence of Ca-MoE(Random) (arbitrarily initialized experts) against the standard Ca-MoE framework and other baselines.

Panel of Figure 8 — Fig. 7: Ca-MoE’s expert utilization on individual graph instances. The x-axis enumerates specific graph instances, while the y-axis reports the the frequency with which each expert is activated for contributing the final selection of forwarder.

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 454 papers

Apr 23, 2026

Teacher-Guided Routing for Sparse Vision Mixture-of-Experts

Recent progress in deep learning has been driven by increasingly large-scale models, but the resulting computational cost has become a critical bottleneck. Sparse Mixture of Experts (MoE) offers an effective solution by activating only a small subset of experts for each input, achieving high scalability without sacrificing inference speed. Although effective, sparse MoE training exhibits characteristic optimization difficulties. Because the router receives informative gradients only through the experts selected in the forward pass, it suffers from gradient blocking and obtains little information from unselected routes. This limited, highly localized feedback makes it difficult for the router to learn appropriate expert-selection scores and often leads to unstable routing dynamics, such as fluctuating expert assignments during training. To address this issue, we propose TGR-MoE: Teacher-Guided Routing for Sparse Vision Mixture-of-Experts, a simple yet effective method that stabilizes router learning using supervision derived from a pretrained dense teacher model. TGR-MoE constructs a teacher router from the teacher's intermediate representations and uses its routing outputs as pseudo-supervision for the student router, suppressing frequent routing fluctuations during training and enabling knowledge-guided expert selection from the early stages of training. Extensive experiments on ImageNet-1K and CIFAR-100 demonstrate that TGR consistently improves both accuracy and routing consistency, while maintaining stable training even under highly sparse configurations.

Masahiro Kada, Ryota Yoshihashi, Satoshi Ikehata +2

Institute of Science Tokyo · Tokyo, Japan · DENSO IT Laboratory

Adaptive Model Routing Sparse Mixture-of-Experts Expert Routing

Ask PDF

Apr 1, 2026

Routing-Free Mixture-of-Experts

Standard Mixture-of-Experts (MoE) models rely on centralized routing mechanisms that introduce rigid inductive biases. We propose Routing-Free MoE which eliminates any hard-coded centralized designs including external routers, Softmax, Top-K and load balancing, instead encapsulating all activation functionalities within individual experts and directly optimized through continuous gradient flow, enabling each expert to determine its activation entirely on its own. We introduce a unified adaptive load-balancing framework to simultaneously optimize both expert-balancing and token-balancing objectives through a configurable interpolation, allowing flexible and customizable resource allocation. Extensive experiments show that Routing-Free MoE can consistently outperform baselines with better scalability and robustness. We analyze its behavior in detail and offer insights that may facilitate future MoE design ad optimization.

Yilun Liu, Jinru Han, Sikuan Yan +2

Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles

Mixture-of-Experts Models Expert Load Balancing

Ask PDF

Oct 5, 2026

How Sparse Probability Maps Shape Mixture-of-Experts Routing

Mixture-of-experts (MoE) routers typically apply softmax to the router scores and keep the top-K experts, making every token use exactly K experts. Sparsity-inducing probability maps such as sparsemax, alpha-entmax and normmax can adaptively assign exact zeros to selected experts, and therefore appear to offer token-dependent expert participation, even when using the same top-K machinery. In this work, we study whether and how this sparsity survives training. We train matched 300M and 1B top-2 MoE language models with softmax, 1.5-entmax, sparsemax and 2-normmax, and find that the maps behave very differently once trained: at 1B, entmax discards 30% less probability mass than softmax while almost never dropping a selected expert, sparsemax retains the most mass, and normmax routes 21% of tokens to a single expert. These outcomes are not properties of the maps alone. Each map drops a selected expert only when the gap between the two largest scores reaches a fixed threshold, and the trained routers differ in the score distribution they learn: the entmax router learns scores with roughly half the spread of softmax's, which keeps its top-2 gaps below its threshold, while sparsemax and normmax, which share the same threshold, learn different gap distributions and hence different participation. Routers thus co-adapt their scores to the map, and a map's capacity to produce zeros does not by itself determine expert participation. While none of the sparse maps improves validation loss over softmax, they make the trained models far less sensitive to selecting more experts at inference: sparsemax trained with K=2 loses 0.02 nats when run with K=8, where softmax loses 0.58. Our results indicate that adaptive MoE routing has to be designed around the joint behavior of the probability map and the learned scores, rather than around the map alone.

Tomás Brogueira, Marcos Treviso, Miguel Couceiro

Técnico, Universidade de Lisboa · INESC-ID · Instituto de Telecomunicações +2

Mixture-of-Experts Language Models Mixture-of-Experts Inference Expert Routing

Ask PDF

Abstract

Mixture-of-Experts Models

Explore similar work

Teacher-Guided Routing for Sparse Vision Mixture-of-Experts

Routing-Free Mixture-of-Experts

How Sparse Probability Maps Shape Mixture-of-Experts Routing