Conventional linear routers in sparse mixture-of-experts models select experts directly from token-expert matching scores, coupling input matching with expert allocation. We propose Prototype-Mediated Routing (PMR), a two-stage router with shared prototypes. PMR first computes each token's distribution over prototypes, then uses it to mix prototype-conditioned expert distributions according to the law of total probability. This formulation models input matching and expert allocation separately, explicitly represents many-to-many associations between prototypes and experts, and provides separate interfaces for context adaptation and load regulation. Within this framework, short-term states summarize preceding tokens to adapt the prototypes, while congestion prices updated from dispatch feedback achieve loss-free load balancing by adjusting the conditional expert distributions. We use a two-pass blockwise approximation to accelerate short-term state processing, achieving over 20 times the training throughput of the eager tokenwise PMR implementation while maintaining high agreement in routing decisions. Across three pretraining configurations with 0.43B, 1.18B, and 0.48B total parameters, PMR improves mean zero-shot accuracy on seven benchmarks over linear routing baselines with matched expert architectures by 1.54, 2.65, and 1.45 percentage points (pp), respectively. The corresponding gains on ARC-Easy are 6.71, 5.80, and 3.92 pp. In parameter-efficient fine-tuning of 7B-8B backbones, PMR's improvements over MixLoRA vary across backbones, with gains of up to 1.76 pp in mean accuracy across four tasks.
Figures & tables
Fig. 1: (a) Linear routing computes expert-indexed scores from the token representation. (b) PMR separately represents the prototype response distribution ( qt ) and the prototype–expert association table ( B ). Token fragments illustrate matching responses. The selected experts process the full token representation.
Fig. 2: Overview of PMR. Context-adaptive prototype matching produces the response distribution qt , which is combined with the shared prototype–expert association table B to obtain expert probabilities πt for sparse dispatch. Short-term states adapt the matching references, while congestion prices regulate the association table using observed expert loads.
Benchmark
Dataset Size (# Samples)
HellaSwag
59,950
PIQA
21,035
WinoGrande-XL
43,432
ARC-Easy
5,197
ARC-Challenge
2,590
OpenBookQA
5,957
TABLE I: Dataset sizes of the seven zero-shot evaluation benchmarks.
Configuration
V1
V2
V3
Total parameters (M)
432.39
1182.17
477.47
Active parameters (M)
175.53
351.70
276.14
Layers
10
16
16
Hidden dimensionality
896
1024
1024
Q/KV heads
14 / 2
16 / 4
16 / 4
Expert dimensionality
2389
2816
2048
TABLE II: Architectural configurations of the three TinyMixtral models and key PMR routing configurations with their paired TinyMixtral architectures in pretraining experiments.
Method
Total Params
Active Params
HellaS.
PIQA
WinoG.
ARC-E
ARC-C
OBQA
LAMBADA
Avg.
4B token pre-training (TinyMixtral-V1)
TinyMixtral-V1
432.39M
175.53M
31.00
61.30
50.80
42.20
24.70
30.80
24.00
37.83
TinyMixtral-V1 + PMR
433.50M
176.64M
31.57
61.70
51.62
48.91
25.43
31.80
24.57
39.37
8B token pre-training (TinyMixtral-V2)
TinyMixtral-V2
1182.17M
351.70M
32.90
63.00
52.30
47.90
27.90
30.60
23.40
39.71
TABLE III: Zero-shot evaluation accuracy (%) of three TinyMixtral configurations and their PMR-enhanced counterparts on seven benchmarks. Avg. denotes the unweighted mean accuracy across the seven benchmarks. Best results are in bold.
Method
ARC-C
ARC-E
OBQA
PIQA
Avg.
Llama-2-7B
Base
32.51
40.24
34.00
56.91
40.91
MixLoRA
54.78
77.86
77.20
81.77
72.90
MixLoRA + PMR
59.39
77.99
79.80
81.45
74.66
Llama-3-8B
TABLE IV: Fine-tuning accuracy (%) on four reasoning benchmarks across five pretrained backbones. Base represents the unadapted backbone. Best results are in bold.
Configuration
HellaS.
PIQA
WinoG.
ARC-E
ARC-C
OBQA
LAMBADA
Avg.
Linear + auxiliary
31.00
61.30
50.80
42.20
24.70
30.80
24.00
37.83
PMR + auxiliary
31.67
62.62
50.91
46.00
25.09
29.20
25.95
38.78
PMR + prices
31.57
61.70
51.62
48.91
25.43
31.80
24.57
39.37
TABLE V: Zero-shot evaluation accuracy (%) on seven benchmarks for TinyMixtral-V1 trained on 4B tokens under different routing and load-balancing designs. Best results are in bold.
Method
ARC-C
ARC-E
OBQA
PIQA
Avg.
Llama-2-7B
MixLoRA + PMR
59.39
77.99
79.80
81.45
74.66
w/o Short-term
58.02
77.31
78.20
81.77
73.83
Llama-3-8B
MixLoRA + PMR
76.96
89.06
85.40
85.64
84.26
TABLE VI: Fine-tuning accuracy (%) with and without short-term sequence state across four reasoning benchmarks and five pretrained backbones. Best results are in bold.
Fig. 3: Prototype-response distances and Top-2 routing agreement in PMR-V2 (a,b) and PMR-V3 (c,d). (a,c): Density of token-pair TV-distance pairs (dq,dπ) , where dq and dπ measure differences in prototype-response distributions and expert probability distributions, respectively. The color scale indicates token-pair counts on a logarithmic scale, and dashed lines mark dπ=dq . (b,d): Proportions of token pairs selecting the same Top-2 experts (blue) or sharing at least one expert (green), plotted against dq . Dashed curves show matched baselines.
Fig. 4: Input-dependent allocation diversity and load regulation in TinyMixtral-V2 and TinyMixtral-V3 with PMR routing. (a) Input-dependent distribution divergence Iπ during training and at the final checkpoints, comparing saved congestion prices with zero prices. (b) Training dispatch coefficient of variation (CV), averaged across layers. (c) Final dispatch CV by layer on the two fixed probe sequences, comparing saved and zero prices. Shaded bands indicate interquartile ranges across layers for saved-price measurements.
Fig. 5: Heatmap of expert contributions in TinyMixtral-V3 with PMR routing after 8B training tokens, measured by the accuracy difference between the full model and the model with the corresponding expert removed. Results are shown across the seven pretraining benchmarks and 64 expert interventions, covering all four experts in each of the 16 layers. Layers are zero-indexed, with experts E0–E3 ordered from left to right within each layer. Positive values indicate accuracy degradation after expert removal.
Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.