Reinforcement learning (RL) can become off-policy due to training-inference mismatch, mini-batch updates, and asynchronous training. The resulting mismatch is especially harmful in Mixture-of-Experts (MoE) models, where rollout-training discrepancies can change routing decisions and lead to router drift. Rollout Routing Replay (R3) mitigates this mismatch by reusing expert selections from the rollout stage during training. Our analysis shows that routing replay introduces gradient bias by losing contributions from experts selected by the training router but not the rollout router. To address this, we propose Router Drift Control (RDC), which uses routing disagreement between rollout and training as a signal of off-policy staleness and selectively rejects high-drift responses. The sequence-level filtering avoids gradient bias introduced by replay while preserving native expert selection and the associated learning signals for retained responses. Experiments show that RDC achieves larger gains as policy staleness increases and remains effective in asynchronous RL. In asynchronous training, RDC combined with Truncated Importance Sampling (TIS) improves the average peak validation accuracy across AIME 2024 and AIME 2025 by 5.5 points over TIS and 4.5 points over R3.
Figures & tables
Figure 1: Three common sources of off-policy mismatch in LLM reinforcement learning. Training and inference discrepancy, sequential mini-batch updates, and asynchronous policy lag can cause the policy used for generation to diverge from the policy used for optimization.
Figure 2: Router drift in MoE reinforcement learning. The same response can activate different experts during rollout and training, resulting in different computation paths and expert-level gradient attribution.
Figure 3: Overview of Router Drift Control (RDC). RDC aggregates layer-wise differences in expert usage between rollout and training into a response-level drift score. It filters high-drift responses before policy updates. Retained responses follow their native training routes.
(Brollout,Bupdate)
Method
ZebraLogic
AIME24
AIME25
MATH-500
(2048,2048)
GRPO
61.2
77.7
65.2
96.6
GRPO+R3
63.0
74.9
65.4
97.6
GRPO+TIS
66.1
80.0
71.3
98.2
GRPO+TIS+RDC
73.9
78.5
72.4
97.8
(2048,512)
GRPO
46.3
62.2
50.8
96.0
GRPO+R3
55.2
69.9
57.9
96.6
Table 1: Best validation accuracy under different update batch sizes in synchronous training.
Method
ZebraLogic
AIME24
AIME25
MATH-500
GRPO
46.9
63.4
50.7
96.2
GRPO+R3
53.9
68.7
53.1
96.2
GRPO+TIS
53.7
65.0
54.8
96.4
GRPO+TIS+RDC
53.0
71.2
59.6
97.2
Table 2: Best validation accuracy under asynchronous RL.
Figure 4: Validation accuracy during asynchronous reinforcement learning on AIME24, AIME25, MATH-500, and ZebraLogic.
Figure 5: Training dynamics under asynchronous reinforcement learning, showing training reward, gradient norm, response length, and training–inference KL for GRPO, TIS, R3, and RDC+TIS.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Learning rate (synchronous / asynchronous)
3×10−6 / 1×10−6
Weight decay / gradient clipping threshold
0.1 / 1.0
PPO epochs
1
Loss aggregation
token-mean
Sampling temperature (training / validation)
1.0 / 0.6
Top- p (training / validation)
1.0 / 0.95
Appendix
Table 3: Additional experimental hyperparameters. TP, PP, EP, and ETP denote tensor, pipeline, expert, and expert tensor parallelism, respectively.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Mixture of Experts (MoE) Large Language Models (LLMs) achieve strong performance at scale. However, reinforcement learning (RL) on MoE-based LLMs often suffers from training instability. A root cause is router drift, i.e., expert activations can change drastically across model updates and differ between disaggregated rollout and training phases, causing large rollout--training mismatch and unstable importance sampling weights in PPO-style RL algorithms. Routing replay mitigates this issue by freezing the replay route within each reasoning trajectory, but it ignores how the router evolves under off-policy updates and thus causes router staleness. To address this limitation, we propose Predictive Routing Replay (PR2), which augments each router with a lightweight evolution predictor that learns to anticipate short-horizon router evolution. During the rollout phase, we use the predictive routing distribution to apply top-k routing, enabling gradients to reach experts that are likely to become active after updates. During the training phase, we replay the resulting predicted route to retain consistency for stable importance estimation. Theoretical analysis and experiments support that PR2 reduces routing-induced mismatch, improves RL stability, and yields stronger performance across various reasoning benchmarks.
Reinforcement learning (RL) has become central to post-training of large language models. Recent advances in RL for Mixture-of-Experts (MoE) models have primarily focused on improving optimization stability and training efficiency, while treating the expert selection as a fixed component. Since routing determines the sparse computation paths that induce output distributions, expert selection offers an additional source of rollout diversity. Through empirical analysis, we find that perturbing expert routing effectively alters model output and increases rollout diversity, which is similar to increasing the decoding temperature. However, direct perturbation can activate unsuitable experts and substantially degrade rollout quality. Motivated by these observations, we introduce Expert-Space Exploration Reinforcement Learning (ESRL), an architecture-aware framework that explicitly explores the expert-routing space of MoE models. ESRL preserves high-confidence experts as anchors, and restricts stochastic routing to a plausible candidate pool, thereby retaining reliable computation paths. The perturbation strength is further adapted according to router entropy to avoid over-perturbation. To mitigate the routing mismatch introduced by perturbation, ESRL records the expert paths used during rollout and replays them during policy optimization. Experiments demonstrate that ESRL achieves the best performance across MoE backbones with top-K, top-1, and shared-expert routing, as well as across mathematics, science, and code tasks without additional sampling or computational cost. Specifically, ESRL on Qwen3-30B-A3B achieves the best among all compared methods, improving average Pass@1 and Pass@8 over GRPO by 3.2 and 4.5 percentage points, respectively. Further analyses of expert utilization and training dynamics provide insights into how exploiting MoE-specific routing structure benefits RL training.
Standard Mixture-of-Experts (MoE) models rely on centralized routing mechanisms that introduce rigid inductive biases. We propose Routing-Free MoE which eliminates any hard-coded centralized designs including external routers, Softmax, Top-K and load balancing, instead encapsulating all activation functionalities within individual experts and directly optimized through continuous gradient flow, enabling each expert to determine its activation entirely on its own. We introduce a unified adaptive load-balancing framework to simultaneously optimize both expert-balancing and token-balancing objectives through a configurable interpolation, allowing flexible and customizable resource allocation. Extensive experiments show that Routing-Free MoE can consistently outperform baselines with better scalability and robustness. We analyze its behavior in detail and offer insights that may facilitate future MoE design ad optimization.
Yilun Liu, Jinru Han, Sikuan Yan +2
Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles