cs.CLSep 26, 2026

HERO-MoE: Historical Expert Routing with Scale-Preserving Fusion

Authors: Junxiang Qiu, Zhengsu Chen, Xinting Hu, Shuo Wang, Hengheng Zhang, Shaofeng Zhang, Changcheng Li, Boyu Shi, 

Organizations: University of Science and Technology of China · Huawei Inc. · Southeast University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Mixture-of-Experts (MoE) architectures have become a standard way to scale model capacity while keeping computation sparse, yet routing remains a key determinant of MoE quality and training behavior. Prior empirical studies suggest that MoE routing reflects input semantics and upstream computation across depth, but standard routers do not explicitly use the routing distributions produced by preceding layers. We propose HERO-MoE, Historical Expert ROuting with Scale-Preserving Fusion, a routing framework that injects historical routing priors into MoE routers by reusing detached, dense routing distributions collected from preceding MoE layers. The key idea is simple: HERO-MoE preserves the original token-conditioned routing branch and adds a residual historical routing contribution before the standard softmax and top-kk dispatch. To stabilize this historical signal, HERO-MoE introduces a scale-preserving fusion mechanism that matches the magnitude of historical routing memory to the current hidden representation and accounts for the number of visible historical layers, without introducing an auxiliary routing loss or a fusion-specific tuning parameter. By reusing routing distributions already computed by preceding MoE layers, HERO-MoE improves training-loss reduction with modest end-to-end overhead. The resulting router remains compatible with standard sparse dispatch, including top-kk and group-limited routing, and can be inserted into existing MoE backbones with minimal architectural changes. Experiments on an approximately 8B-parameter MoE model with 0.5B active parameters, trained from scratch on 100B tokens, show that HERO-MoE reduces the final loss from 1.6393 to 1.6184, while peak memory and FLOPs increase by only 0.44% and 0.64%, respectively.

Figures & tables

Figure 1: Pipeline of HERO-MoE, including the original router branch, historical residual branch, sparse dispatch, and history update.
Figure 2: Training loss of the Baseline, RCP, HERO-MoE, historical-window ablations, and the no-scaling ablation.
ModelGSM8KMathQAMATHARC-CARC-C-ZHARC-EHellaSwagBBHMMLUCMMLUHumanEvalLAMBADAPIPAAvg.
Baseline10.9930.998.1435.3230.9770.9640.6229.342.143.215.2445.5572.0336.57
HERO-MoE12.2130.959.637.6331.8370.9242.0129.0942.2242.3818.946.5272.237.42
Δ\Delta1.22−- 0.041.462.310.86−- 0.041.39−- 0.210.12−- 0.823.660.970.170.85
Table 1: General benchmark comparison between the baseline MoE and HERO-MoE.
Figure 3: Layer-wise historical dependencies, distance-wise strong coupling ratios, and a representative expert-level coupling matrix learned by HERO-MoE.
MethodPeak MemoryFLOPs (Router only)FLOPs (Total)Forward TimeRouter Step TimeTotal Train Time
Baseline5440.8 MiB1.55 T211.83 T32.579 ms0.14 s / iter71.88 h
HERO-MoE5464.9 MiB2.90 T213.18 T37.984 ms0.54 s / iter74.65 h
Δ\Delta+0.44%+87.50%+0.64%+16.59%+281.14%+3.85%
Table 2: Time, memory and FLOPs comparison between Baseline and HERO-MoE.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Expert-level coupling matrices for all adjacent MoE-layer pairs learned by HERO-MoE.
Figure 5: Expert utilization frequency across 30 layers and 128 experts.
Figure 6: Training loss and loss advantage of Qwen3 with HERO-MoE.
More in this topic

Mixture-of-Experts Language Models

Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.

See all 225 papers

Explore similar work

CardsList
Apr 1, 2026

Self-Routing: Parameter-Free Expert Routing from Hidden States

Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token, and typically rely on a learned router to map hidden states to expert assignments. In this work, we ask whether a dedicated learned router is strictly necessary for MoE routing. We propose Self-Routing, a parameter-free routing mechanism that uses a designated subspace of the token hidden state directly as expert logits, eliminating the router projection entirely while leaving the rest of the MoE layer unchanged. We evaluate Self-Routing on language modeling across different expert counts and model scales, and on ImageNet-1K classification by comparing it against a standard learned router, random-routing baselines, and dense non-MoE baselines. Our results show that Self-Routing remains competitive with the learned-router baseline while removing all dedicated routing parameters, and yields more balanced expert utilization, with about 17% higher average normalized routing entropy and no explicit load-balancing loss. On ImageNet-1K with DeiT-S/16, Self-Routing also slightly improves over the corresponding learned-router MoE. These findings suggest that effective MoE routing can emerge from the hidden representation itself without requiring a separate learned router module.

Jama Hussein Mohamud, Drew Wagner, Mirco Ravanelli

Mila – Quebec AI Institute · Universite de Montreal · Concordia University

Parameter-Free Mixture-of-Experts RoutingLLM RoutingMixture-of-Experts Models
AskPDF
May 8, 2026

Hierarchical Mixture-of-Experts with Two-Stage Optimization

Sparse Mixture-of-Experts (MoE) models scale capacity by routing each token to a small subset of experts. However, their routers exhibit a fundamental trade-off: strong load balancing can suppress expert specialization, while aggressive diversity often causes routing collapse. We propose Hi-MoE, a grouped MoE framework that decomposes routing control into two coupled levels: (i) inter-group balancing that enforces fair traffic across expert groups, and (ii) intra-group specialization that promotes complementary expert behaviors while preventing within-group collapse. Our analysis provides a principled explanation of how our hierarchical objectives reshape the router, thereby promoting stable specialization and mitigating collapse. We observe consistent improvements over recent sparse-routing and grouped-MoE baselines across NLP and vision benchmarks, and confirm robustness via scaling studies (model size, expert count) and targeted ablations. In large-scale pre-training on 58B tokens, Hi-MoE-7B achieves a 5.6% perplexity reduction and a 40% improvement in expert balance over OLMoE-7B across diverse evaluation domains.

Gleb Molodtsov, Alexander Miasnikov, Aleksandr Beznosikov

BRAIn Lab, Moscow, Russia

Expert Load BalancingMixture of ExpertsSparse Mixture-of-ExpertsExpert RoutingMixture-of-Experts Models
AskPDF
Oct 5, 2026

Memory-Efficient Expert Routing for Distributed MoE Training

As Mixture-of-Experts (MoE) models scale toward hundreds of experts and higher top-kk routing, memory efficiency in distributed training becomes a critical bottleneck. Peak memory is dominated by the MoE block, not attention: every intermediate buffer in the MoE dispatch pipeline is individually scaled by top-k routing. The standard all-to-all dispatcher sends all routed tokens in a single collective step, requiring the full top-kk-expanded buffer to be constructed at once. In this work, we propose RelayMoE, a ring-based MoE execution model that computes locally as expert weights or tokens circulate, avoiding full top-kk-expanded dispatch buffers. RelayMoE selects between expert and token routing according to communication volume and overlaps transfers with computation. The ring structure naturally supports memory-efficient MoE recomputation during backward: each hop reconstructs expert intermediates, uses them to compute gradients, and releases them before the next hop. The saved memory supports longer sequences and larger batches, or retains more attention activations to reduce attention recomputation and improve training throughput. We evaluate RelayMoE on 30B−-57B production MoE models and varied expert configurations. In single-layer MoE experiments, RelayMoE achieves a 2×2\times average speedup over Megatron-LM. In full-model training under the same GPU memory budget, it improves throughput by up to 2.02×2.02\times and extends the largest tested trainable sequence length by up to 2.85×2.85\times.

Arnab Kanti Tarafder, Jaume Guasch-Martí, Gokcen Kestor +1

William & Mary · Barcelona Supercomputing Center

Expert RoutingMixture-of-Experts ModelsMemory-Efficient TrainingCommunication-Efficient Distributed Training
AskPDF