Conventional linear routers in sparse mixture-of-experts models select experts directly from token-expert matching scores, coupling input matching with expert allocation. We propose Prototype-Mediated Routing (PMR), a two-stage router with shared prototypes. PMR first computes each token's distribution over prototypes, then uses it to mix prototype-conditioned expert distributions according to the law of total probability. This formulation models input matching and expert allocation separately, explicitly represents many-to-many associations between prototypes and experts, and provides separate interfaces for context adaptation and load regulation. Within this framework, short-term states summarize preceding tokens to adapt the prototypes, while congestion prices updated from dispatch feedback achieve loss-free load balancing by adjusting the conditional expert distributions. We use a two-pass blockwise approximation to accelerate short-term state processing, achieving over 20 times the training throughput of the eager tokenwise PMR implementation while maintaining high agreement in routing decisions. Across three pretraining configurations with 0.43B, 1.18B, and 0.48B total parameters, PMR improves mean zero-shot accuracy on seven benchmarks over linear routing baselines with matched expert architectures by 1.54, 2.65, and 1.45 percentage points (pp), respectively. The corresponding gains on ARC-Easy are 6.71, 5.80, and 3.92 pp. In parameter-efficient fine-tuning of 7B-8B backbones, PMR's improvements over MixLoRA vary across backbones, with gains of up to 1.76 pp in mean accuracy across four tasks.
Figures & tables
Fig. 1: (a) Linear routing computes expert-indexed scores from the token representation. (b) PMR separately represents the prototype response distribution ( qt ) and the prototype–expert association table ( B ). Token fragments illustrate matching responses. The selected experts process the full token representation.
Fig. 2: Overview of PMR. Context-adaptive prototype matching produces the response distribution qt , which is combined with the shared prototype–expert association table B to obtain expert probabilities πt for sparse dispatch. Short-term states adapt the matching references, while congestion prices regulate the association table using observed expert loads.
Benchmark
Dataset Size (# Samples)
HellaSwag
59,950
PIQA
21,035
WinoGrande-XL
43,432
ARC-Easy
5,197
ARC-Challenge
2,590
OpenBookQA
5,957
TABLE I: Dataset sizes of the seven zero-shot evaluation benchmarks.
Configuration
V1
V2
V3
Total parameters (M)
432.39
1182.17
477.47
Active parameters (M)
175.53
351.70
276.14
Layers
10
16
16
Hidden dimensionality
896
1024
1024
Q/KV heads
14 / 2
16 / 4
16 / 4
Expert dimensionality
2389
2816
2048
TABLE II: Architectural configurations of the three TinyMixtral models and key PMR routing configurations with their paired TinyMixtral architectures in pretraining experiments.
Method
Total Params
Active Params
HellaS.
PIQA
WinoG.
ARC-E
ARC-C
OBQA
LAMBADA
Avg.
4B token pre-training (TinyMixtral-V1)
TinyMixtral-V1
432.39M
175.53M
31.00
61.30
50.80
42.20
24.70
30.80
24.00
37.83
TinyMixtral-V1 + PMR
433.50M
176.64M
31.57
61.70
51.62
48.91
25.43
31.80
24.57
39.37
8B token pre-training (TinyMixtral-V2)
TinyMixtral-V2
1182.17M
351.70M
32.90
63.00
52.30
47.90
27.90
30.60
23.40
39.71
TABLE III: Zero-shot evaluation accuracy (%) of three TinyMixtral configurations and their PMR-enhanced counterparts on seven benchmarks. Avg. denotes the unweighted mean accuracy across the seven benchmarks. Best results are in bold.
Method
ARC-C
ARC-E
OBQA
PIQA
Avg.
Llama-2-7B
Base
32.51
40.24
34.00
56.91
40.91
MixLoRA
54.78
77.86
77.20
81.77
72.90
MixLoRA + PMR
59.39
77.99
79.80
81.45
74.66
Llama-3-8B
TABLE IV: Fine-tuning accuracy (%) on four reasoning benchmarks across five pretrained backbones. Base represents the unadapted backbone. Best results are in bold.
Configuration
HellaS.
PIQA
WinoG.
ARC-E
ARC-C
OBQA
LAMBADA
Avg.
Linear + auxiliary
31.00
61.30
50.80
42.20
24.70
30.80
24.00
37.83
PMR + auxiliary
31.67
62.62
50.91
46.00
25.09
29.20
25.95
38.78
PMR + prices
31.57
61.70
51.62
48.91
25.43
31.80
24.57
39.37
TABLE V: Zero-shot evaluation accuracy (%) on seven benchmarks for TinyMixtral-V1 trained on 4B tokens under different routing and load-balancing designs. Best results are in bold.
Method
ARC-C
ARC-E
OBQA
PIQA
Avg.
Llama-2-7B
MixLoRA + PMR
59.39
77.99
79.80
81.45
74.66
w/o Short-term
58.02
77.31
78.20
81.77
73.83
Llama-3-8B
MixLoRA + PMR
76.96
89.06
85.40
85.64
84.26
TABLE VI: Fine-tuning accuracy (%) with and without short-term sequence state across four reasoning benchmarks and five pretrained backbones. Best results are in bold.
Fig. 3: Prototype-response distances and Top-2 routing agreement in PMR-V2 (a,b) and PMR-V3 (c,d). (a,c): Density of token-pair TV-distance pairs (dq,dπ) , where dq and dπ measure differences in prototype-response distributions and expert probability distributions, respectively. The color scale indicates token-pair counts on a logarithmic scale, and dashed lines mark dπ=dq . (b,d): Proportions of token pairs selecting the same Top-2 experts (blue) or sharing at least one expert (green), plotted against dq . Dashed curves show matched baselines.
Fig. 4: Input-dependent allocation diversity and load regulation in TinyMixtral-V2 and TinyMixtral-V3 with PMR routing. (a) Input-dependent distribution divergence Iπ during training and at the final checkpoints, comparing saved congestion prices with zero prices. (b) Training dispatch coefficient of variation (CV), averaged across layers. (c) Final dispatch CV by layer on the two fixed probe sequences, comparing saved and zero prices. Shaded bands indicate interquartile ranges across layers for saved-price measurements.
Fig. 5: Heatmap of expert contributions in TinyMixtral-V3 with PMR routing after 8B training tokens, measured by the accuracy difference between the full model and the model with the corresponding expert removed. Results are shown across the seven pretraining benchmarks and 64 expert interventions, covering all four experts in each of the 16 layers. Layers are zero-indexed, with experts E0–E3 ordered from left to right within each layer. Positive values indicate accuracy degradation after expert removal.
Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.
Standard Mixture-of-Experts (MoE) models rely on centralized routing mechanisms that introduce rigid inductive biases. We propose Routing-Free MoE which eliminates any hard-coded centralized designs including external routers, Softmax, Top-K and load balancing, instead encapsulating all activation functionalities within individual experts and directly optimized through continuous gradient flow, enabling each expert to determine its activation entirely on its own. We introduce a unified adaptive load-balancing framework to simultaneously optimize both expert-balancing and token-balancing objectives through a configurable interpolation, allowing flexible and customizable resource allocation. Extensive experiments show that Routing-Free MoE can consistently outperform baselines with better scalability and robustness. We analyze its behavior in detail and offer insights that may facilitate future MoE design ad optimization.
Yilun Liu, Jinru Han, Sikuan Yan +2
Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles
We model Mixture-of-Experts (MoE) token routing as a congestion game with a single effective parameter, the congestion coefficient gamma_eff, that quantifies the balance-quality tradeoff. Tracking gamma_eff across training checkpoints of two open-source MoE models, OLMoE-1B-7B (20 checkpoints, with dense sampling in the surge region) and OpenMoE-8B (6 checkpoints), reveals a three-phase trajectory: a surge phase where the router learns to balance load (gamma_eff: 14 to 36-39, peaking in the step 30K-40K region), a stabilization phase where experts specialize under steady balance (B_0: 2.4 to 2.3, steps 100K-400K), and a relaxation phase where the router trades balance for quality as experts differentiate (gamma_eff: 27 to 9, steps 400K-1.2M). This non-monotone trajectory, invisible to post-hoc analysis of converged models, reveals that early MoE training prioritizes balance while late training prioritizes quality. The theoretical framework is honest about its limits: the single-type equilibrium reduces to temperature-scaled softmax (held-out L1: MFG = 0.199 vs. softmax = 0.200). The game is not a better predictor; it reveals what the temperature means and, critically, how that temperature evolves. We complement the dynamics with an effective congestion decomposition, a multi-type extension that improves load prediction via token clustering on all 16 layers (mean: 30%), scope diagnostics (K/M, epsilon_l), and robustness verification across four independent quality estimators (r >= 0.89). All confidence intervals are from bootstrap resampling over 50 independent text batches.
Charafeddine Mouzouni
OPIT – Open Institute of Technology, and Cohorte AI, Paris, France.
In current Mixture-of-Experts architectures, routing is performed based on representations dominated by structure shared across all tokens, limiting expert specialization. We show that contrasting each token against an Exponential Moving Average of the layer's hidden states, rather than routing on absolute magnitude, concentrates the routing signal onto a low-dimensional, highly separable subspace. Building on this, we propose the Contrastive Routing Mechanism (CoRM), which scores each expert by the gap between its affinity for the incoming token and its affinity for this shared reference state, interpreted through a distinct per-expert projection. The resulting experts have routing boundaries that align with linguistic structure significantly more than the Top-k baseline. Our experiments show that CoRM improves average zero-shot accuracy by +0.67 to +1.69 points (Top-1) and +1.38 to +1.77 points (Top-2) over standard Top-k MoE baselines on nine zero-shot reasoning benchmarks, at the minimal cost of 2.9% added parameters and 2.6% added FLOPs per token.