Organizations: School of Mathematical Sciences, Peking University, Beijing, China · Zhejiang Lab, Hangzhou, China · Yuanpei College, Peking University, Beijing, China · Center for Machine Learning Research, Peking University, Beijing, China
Traditional Mixture-of-Experts (MoE) training typically proceeds without any structural priors, effectively requiring the model to simultaneously train expert weights while searching for an optimal routing policy within a vast combinatorial space. This entanglement often leads to sluggish convergence and training instabilities. This paper introduces Grouter, a preemptive routing method that by distilling high-quality structures from fully-trained MoE models and serving as a fixed router for target models. By decoupling structural optimization from weight updates, Grouter significantly accelerates both the speed and quality of model convergence. To ensure the framework's versatility, we also introduce expert folding to adapt Grouter across varying model configurations and expert tuning to rebalance workloads across different data distributions. Furthermore, by leveraging the structural priors provided by preemptive routing, we can implement targeted optimizations to further enhance training throughput. Experiments demonstrate that Grouter achieves superior performance and efficiency which boosts pre-training data utilization by 4.28x and achieves up to 33.5% throughput acceleration, establishing preemptive routing as a fundamental paradigm for scalable MoE training. We publicly release our code and pretrained Grouter checkpoints at https://github.com/JimmyAwoe/Grouter.
Figures & tables
Figure 1: (a) The percentage of tokens that maintain an exactly identical set of k activated experts for the same input across adjacent checkpoints. We present the more detailed descriptions and figures in Appendix B . (b) Sensitivity analysis of expert specialization via random routing perturbations. Perturbations were applied at specific intervals with a fixed learning rate of 10−5 . The resulting loss trajectories (lines) and average gradient norms (bars) reveal a clear trend: while early-stage training (1k steps) is resilient to routing noise due to a lack of specialization, later stages exhibit severe instability and loss spikes under perturbation. This demonstrates that experts gradually develop deep specialization as the router stabilizes, making the model increasingly sensitive to routing errors. (c) The coefficient of variation of the gradient norm within a sliding window of length 100. The results demonstrate that the Grouter method achieves a significant improvement in stability, whereas both the state-of-the-art load balancing approach and pure static routing assignment lead to substantial gradient fluctuations.
Figure 2: (a) Overview of the GRouter Workflow. The GRouter first extracts a highly optimized structural prior from the Source Model, and then injects this prior into the Target Model in a frozen state. (b) Illustration of Our Expert Tuning and Expert Folding Techniques.
Figure 3: The workflow of our EP communication optimization strategy. Step 1: Sequences pass through GRouter to generate token-level assignments, which are aggregated into sequence-level routing affinity vectors. Step 2: Sequences are clustered based on affinity. Using cluster centroids as preference weights, we solve an optimization problem to assign experts to EP devices, maximizing the alignment between experts and sequence clusters. Step 3: With expert locations fixed, sequences are assigned to the EP device that minimizes the resulting communication volume.
Figure 4: (a) Pre-training Validation Loss Curves Across 30B Tokens. Grouter Raw denotes the distilled Grouter used without the subsequent Expert-Tuning. The shaded Gap region illustrates the loss difference between the Grouter curve and the best baseline. Overall, Grouter achieves a 4.28× acceleration or a maximum loss reduction of 0.85 at the same training data volume. (b) Comparison of Performance and Load Balance Trade-off. The third quadrant is highlighted in green to emphasize that models located within this region achieve an optimal balance between low load violation and superior model performance
Figure 5: (a) Downstream Task Results: GRouter achieves an average improvement of 2.80 across six benchmarks, with gains of up to 10 points on specific tasks. This demonstrates that the reduced validation loss achieved by GRouter translates into genuine enhancements in model capabilities, rather than merely overfitting the validation metric. (b) Throughput Scaling: We evaluate throughput on 1, 2, and 4 nodes with Expert Parallelism (EP) degrees set to 8, 16, and 32, respectively. For the multi-node setups (2 and 4 nodes), we apply node-granularity communication optimization, while for the single-node case, we utilize GPU-granularity optimization.
Mini-GPT-OSS
Mini-DS-V2-Lite
Mini-Qwen3
Small-Qwen3
Attn
GQA
MLA
GQA
GQA
Structure
MoE
Dense+MoE
MoE
MoE
Tokens
30B
30B
50B
50B
Grouter
2.940
2.469
1.763
1.648
ZL
3.346
3.057
2.740
2.633
Table 1: Performance Comparison Across Diverse Model Configurations
Figure 6: Routing fluctuations of a 550M model trained on 30B tokens. Given the high token-to-parameter ratio of 60, the model has largely converged. It is observed that the router exhibits intense volatility during the early stages of training; however, as training progresses, the routing patterns gradually converge and maintain stability.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Number of Active Experts During Training for Dynamic Activation Baselines. The number of active experts remains consistently higher than the expected value throughout the training process, confirming a fair comparison.
Figure 8: Training curves of Grouter and Z-loss for models with varying sizes, architectures, and expert counts. The results demonstrate that a single Grouter can be migrated to different configurations via expert folding and tuning, while retaining superior convergence speed and precision.
Multi Head Attn
Multi Latent Attn
Residual Linear
Decoder
Encoder
Decoder
Encoder
Linear
Use_Pos
No_Pos
Use_Pos
No_Pos
Use_Pos
No_Pos
Use_Pos
No_Pos
Use_Pos
No_Pos
3.342
3.365
3.318
3.367
3.335
3.387
3.356
3.360
3.375
3.359
Appendix
Table 2: Ablation Study on GRouter Architectures
Figure 9: Ablation Study on Expert Folding Strategies.
Model
Mini-GPT-OSS
Mini-DS-V2-Lite
Mini-Qwen3
Parameter
315M
958B
2.81B
Expert Parameter
199M
785M
2.41B
Expert Ratio
63.19%
81.99%
85.96%
Appendix
Table 3: Parameter statistics for the evaluated models
Model
Mini-GPT-OSS
Tiny-Qwen3
Mini-DS-V2-Lite
Mini-Qwen3
Parameter
350M
550M
1B
3B
Hidden_Size
720
384
512
1024
Num_Layer
8
8
12
16
Num_Experts
32
128
64
128
Top_k
4
8
6
8
FFN_Hidden_size
720
688
2736
3072
Appendix
Table 4: Model Configurations
Figure 10: Learning Curve of the Distillation Process
Figure 11: Analysis of training stability and gradient magnitudes. (a) The CV of the gradient norm, computed using a sliding window of size 50. This illustrates Grouter’s superior stability at short timescales, whereas baseline methods exhibit significant spikes. (b) The gradient norm CV with a larger sliding window size of 500, demonstrating Grouter’s sustained stability over longer timescales. (c) The trajectory of the gradient norm throughout training. An Moving Average curve (window size 50) is highlighted for visual clarity. The results indicate that Grouter consistently maintains a higher gradient norm compared to baselines.
Figure 12: Validation loss comparison for cross-scale transfer. GRouter matches or even outperforms the baseline throughout, demonstrating effective cross-scale structural transfer.
Stage
Cost
Relative Cost
Distillation
150,995
4.00%
Expert Folding
981
0.03%
Expert Tuning
2,944
0.08%
Preemptive Routing
113,246
3.00%
Pretraining w/ GRouter
452,983
12.00%
Total w/ GRouter
721,149
19.11%
Appendix
Table 5: Computational cost breakdown of the GRouter pipeline compared with standard pretraining. Percentages are relative to the baseline pretraining cost.
Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.