cs.AINov 14, 2026

PMR: MoE Routing with Shared Prototypes and Congestion Pricing

Authors: Wending Wang, Binqi Huang, Ju'an Li, Yu Guo, Lei Duan, Yuqi Ouyang, Dengchun Li

Organizations: Sichuan University, Chengdu, China · University of Science and Technology of China, Hefei, China

Abstract

Conventional linear routers in sparse mixture-of-experts models select experts directly from token-expert matching scores, coupling input matching with expert allocation. We propose Prototype-Mediated Routing (PMR), a two-stage router with shared prototypes. PMR first computes each token's distribution over prototypes, then uses it to mix prototype-conditioned expert distributions according to the law of total probability. This formulation models input matching and expert allocation separately, explicitly represents many-to-many associations between prototypes and experts, and provides separate interfaces for context adaptation and load regulation. Within this framework, short-term states summarize preceding tokens to adapt the prototypes, while congestion prices updated from dispatch feedback achieve loss-free load balancing by adjusting the conditional expert distributions. We use a two-pass blockwise approximation to accelerate short-term state processing, achieving over 20 times the training throughput of the eager tokenwise PMR implementation while maintaining high agreement in routing decisions. Across three pretraining configurations with 0.43B, 1.18B, and 0.48B total parameters, PMR improves mean zero-shot accuracy on seven benchmarks over linear routing baselines with matched expert architectures by 1.54, 2.65, and 1.45 percentage points (pp), respectively. The corresponding gains on ARC-Easy are 6.71, 5.80, and 3.92 pp. In parameter-efficient fine-tuning of 7B-8B backbones, PMR's improvements over MixLoRA vary across backbones, with gains of up to 1.76 pp in mean accuracy across four tasks.

Figures & tables

Explore similar work

CardsList
  1. Routing-Free Mixture-of-Experts

    Apr 1, 2026Yilun Liu, Jinru Han, Sikuan Yan +2Mixture-of-Experts ModelsExpert Load Balancing

  2. Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training

    Apr 5, 2026Charafeddine MouzouniCongestion GamesExpert Load Balancing

  3. Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts

    Sep 1, 2026Nikolaos Xiros, Dimitrios Damianos, Maria-Eleni Zoumpoulidi +3Mixture-of-Experts Language ModelsExpert Routing