cs.CLOct 13, 2025

Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers

Authors: Wenhan Ma, Hailin Zhang, Liang Zhao, Yifan Song, Yudong Wang, Zhifang Sui, Fuli Luo

Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · LLM-Core Xiaomi · Independent Researcher

Abstract

Reinforcement learning (RL) has emerged as a crucial approach for enhancing the capabilities of large language models. However, in Mixture-of-Experts (MoE) models, the routing mechanism often introduces instability, even leading to catastrophic RL training collapse. We analyze the training-inference consistency of MoE models and identify a notable discrepancy in routing behaviors between the two phases. Moreover, even under identical conditions, the routing framework can yield divergent expert selections across repeated forward passes. To address this foundational inconsistency, we propose Rollout Routing Replay (R3), a method that records routing distributions from the inference engine and replays them during training. R3 significantly reduces training-inference policy KL divergence and mitigates extreme discrepancies without compromising training speed. Extensive experiments on various settings confirm that R3 succeeds in stabilizing RL training, preventing collapse and outperforming methods such as GSPO and TIS. We believe this work can offer a new solution for stabilizing RL in MoE models.

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning

    May 29, 2026Daize Dong, Junlin Chen, Haolong Jia +9RL for Language Model ReasoningExpert Routing

  2. Expert-Space Exploration in MoE Reinforcement Learning

    Sep 11, 2026Hongyi He, Zhenghao Lin, Xiao Liu +3Reinforcement LearningExpert Routing

  3. Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning

    Oct 15, 2026Liangyao Jin, Zhixin Wang, Liming Liu +8Reinforcement LearningOff-Policy RL