Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · LLM-Core Xiaomi · Independent Researcher
Reinforcement learning (RL) has emerged as a crucial approach for enhancing the capabilities of large language models. However, in Mixture-of-Experts (MoE) models, the routing mechanism often introduces instability, even leading to catastrophic RL training collapse. We analyze the training-inference consistency of MoE models and identify a notable discrepancy in routing behaviors between the two phases. Moreover, even under identical conditions, the routing framework can yield divergent expert selections across repeated forward passes. To address this foundational inconsistency, we propose Rollout Routing Replay (R3), a method that records routing distributions from the inference engine and replays them during training. R3 significantly reduces training-inference policy KL divergence and mitigates extreme discrepancies without compromising training speed. Extensive experiments on various settings confirm that R3 succeeds in stabilizing RL training, preventing collapse and outperforming methods such as GSPO and TIS. We believe this work can offer a new solution for stabilizing RL in MoE models.
Figures & tables
Figure 1: Left: Illustration of the Rollout Routing Replay (R3). Top right: Training and inference discrepancies before and after applying R3. Bottom right: Reinforcement learning training performance before and after applying R3.
Figure 2: (a): Illustration of the training-inference discrepancy in the MoE model. (b): Illustration of the training-inference discrepancy in the MoE+R3 model. (c): Illustration of the training-inference discrepancy in the Dense model. (d): Extreme Token Distribution Function, calculated based on Equation 4 .
Figure 3: Illustration of how the router amplifies a numerical perturbation into a large policy discrepancy. For the same input token, the router scores in the inference engine (left) and the training engine (right) differ only by floating-point noise. Since experts 4 and 5 are nearly tied at the Top- K cutoff, this tiny difference is enough to flip which of the two is activated. Because a different expert contributes to the layer output, the resulting output token distribution can differ substantially between the two engines, which makes the collected rollouts off-policy.
Figure 4: Router discrepancy analysis
Best Metric(Best Global Step)
Method
AIME24 (↑)
AIME25 (↑)
AMC23 (↑)
MATH500 Lv5 (↑)
Avg (↑)
Crash Step
Qwen3-30B-A3B-SFT, mini_step=8
GRPO
32.81(65)
20.73(90)
74.84(60)
71.83(100)
48.84(100)
120
GSPO
55.52(165)
38.23 (160)
90.16 (125)
86.38(125)
66.76(125)
-
GRPO+R3
57.92 (180)
38.02(155)
90.16 (155)
88.62 (170)
68.05 (180)
-
GSPO+R3
58.44 (160)
39.17 (160)
92.50 (165)
87.87 (165)
69.00 (165)
-
Table 1: Main evaluation results. This table presents the best scores (with corresponding global step in parentheses) achieved by various methods on evaluation benchmarks under three training configurations. The table also reports the average score (Avg) and the step at which training collapse occurred (Crash Step).
Figure 5: Analysis of training–inference collapse. The plot shows the estimated training–inference KL divergence and the extreme token distribution function F(τ=2) (Eq. 4 ) at each training step.
Model: Qwen3-30B-A3B
Method
SWE-bench Verified (↑)
Crash Step
GRPO
31.80
90
GRPO+R3
38.60
-
Table 2: Evaluation results of RL training on SWE Task
Model: DeepSeek-V2-Lite-ReasoningSFT
Method
Math Avg (↑)
Crash Step
GRPO
23.55(55)
100
GRPO+R3
26.43 (235)
-
Table 3: Evaluation results of the RL training process of DeepSeek-V2-Lite-ReasoningSFT
Metric
w/o R3
w/ R3
# Global Step
10
10
# Token
203,989,395
203,201,992
Runtime (sec)
4820
4967
# Token/sec
42,321
40,910
Relative Time Cost
100%
103.45%
Table 4: Throughput analysis result of Rollout Routing Replay
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The detailed evaluation results of the experiment in Section 5 .
Figure 8: The detailed training metrics of the experiment in Section 5 .
Figure 9: Default setting of partial rollout. Orange and blue denote states computed with θold and θnew , respectively. After the parameter update, TA is re-prefilled with θnew , and the inference engine continues the rollout to obtain TB .
Method
Context of TB at Inference
Training Sequences (Replayed Masks)
TA
TB
Drawback
Re-prefill only
CAnew under IAnew
TA⊕TB ( IAnew⊕IBnew )
✗
✓
Inconsistent on TA
Concat old
CAnew under IAnew
TA⊕TB ( IAold⊕IBnew )
✓
✗
Inconsistent on TB
Split training
CAnew under IAnew
TA ( IAold ), TA⊕TB ( IAnew⊕IBnew )
✓
✓
Extra training computation
Replay in re-prefill
CAnew under IAold
TA⊕TB ( IAold⊕IBnew )
✓
✓
Complex implementation
Cache retention
CAold under IAold
TA⊕TB ( IAold⊕IBnew )
✓
✓
Stale KV cache
Appendix
Table 5: Methods for combining R3 with partial rollout. Parentheses give the routing masks replayed on each training sequence, and ✓/✗ indicate whether the tokens in TA and TB are routing-consistent.
Table 10: Evaluation results of the RL training process of Qwen3-30B-A3B-ReasoningSFT
Figure 11: The detailed training metrics and evaluation results of the RL training process of Qwen3-30B-A3B-ReasoningSFT
KV Cache
Routing Mask Cache
Each Token
DataType × #Layer × (KDim + VDim)
DataType × #Layer × TopK
Each Token in Qwen3-30B-A3B
2×48×(512+512)
2×48×8
437K Tokens per GPU
40 GB on GPU HBM
0.3 GB on CPU Memory
437K × 8 Tokens per Node (8 GPUs)
40 GB×8 on HBM
2.5 GB on CPU Memory
Appendix
Table 11: Comparison of KV Cache and Routing Mask Cache in the Inference Framework
Rank
Token ID
Token String
Mean Token-Level Diff Router
1
15
"0"
16.9834
2
37018
"frac"
16.6235
3
248
(unvisualizable)
16.5253
4
50853
"cdot"
15.9350
5
230
(unvisualizable)
15.8782
6
20
"5"
15.7010
Appendix
Table 12: The top 50 tokens ranked by routing discrepancy
Figure 12: Visualization of routing discrepancy for the response segment, where red indicates large routing discrepancy and green indicates small routing discrepancy
Figure 13: Per-layer routing divergence (Mean Diff) between training and inference for Qwen3-30B-A3B across all 48 layers.
Model
KL (Base)
KL (Thinking)
F(τ=2) (Base)
F(τ=2) (Thinking)
0.6B
6.631e-04
7.549e-04
4.294e-06
1.767e-05
1.7B
1.028e-03
9.389e-04
6.371e-06
6.185e-05
4B
6.037e-04
6.446e-04
1.868e-05
1.066e-05
8B
9.834e-04
6.471e-04
1.575e-05
8.651e-06
32B
–
8.709e-04
–
1.428e-05
30B-A3B
1.339e-03
1.548e-03
2.992e-04
3.377e-04
Appendix
Table 13: Training-inference discrepancy across different model sizes. KL denotes KL divergence and F(τ=2) denotes the F metric with threshold τ=2 .
Hyperparameter
Value
Batch Size
256
Group Size
8
Mini Step
8 or 1
Max Prompt Length
2048
Max Response Length
30720
Optimizer
AdamW
Appendix
Table 14: Default Training hyperparameters of Reinforcement Learning.
Mixture of Experts (MoE) Large Language Models (LLMs) achieve strong performance at scale. However, reinforcement learning (RL) on MoE-based LLMs often suffers from training instability. A root cause is router drift, i.e., expert activations can change drastically across model updates and differ between disaggregated rollout and training phases, causing large rollout--training mismatch and unstable importance sampling weights in PPO-style RL algorithms. Routing replay mitigates this issue by freezing the replay route within each reasoning trajectory, but it ignores how the router evolves under off-policy updates and thus causes router staleness. To address this limitation, we propose Predictive Routing Replay (PR2), which augments each router with a lightweight evolution predictor that learns to anticipate short-horizon router evolution. During the rollout phase, we use the predictive routing distribution to apply top-k routing, enabling gradients to reach experts that are likely to become active after updates. During the training phase, we replay the resulting predicted route to retain consistency for stable importance estimation. Theoretical analysis and experiments support that PR2 reduces routing-induced mismatch, improves RL stability, and yields stronger performance across various reasoning benchmarks.
Reinforcement learning (RL) has become central to post-training of large language models. Recent advances in RL for Mixture-of-Experts (MoE) models have primarily focused on improving optimization stability and training efficiency, while treating the expert selection as a fixed component. Since routing determines the sparse computation paths that induce output distributions, expert selection offers an additional source of rollout diversity. Through empirical analysis, we find that perturbing expert routing effectively alters model output and increases rollout diversity, which is similar to increasing the decoding temperature. However, direct perturbation can activate unsuitable experts and substantially degrade rollout quality. Motivated by these observations, we introduce Expert-Space Exploration Reinforcement Learning (ESRL), an architecture-aware framework that explicitly explores the expert-routing space of MoE models. ESRL preserves high-confidence experts as anchors, and restricts stochastic routing to a plausible candidate pool, thereby retaining reliable computation paths. The perturbation strength is further adapted according to router entropy to avoid over-perturbation. To mitigate the routing mismatch introduced by perturbation, ESRL records the expert paths used during rollout and replays them during policy optimization. Experiments demonstrate that ESRL achieves the best performance across MoE backbones with top-K, top-1, and shared-expert routing, as well as across mathematics, science, and code tasks without additional sampling or computational cost. Specifically, ESRL on Qwen3-30B-A3B achieves the best among all compared methods, improving average Pass@1 and Pass@8 over GRPO by 3.2 and 4.5 percentage points, respectively. Further analyses of expert utilization and training dynamics provide insights into how exploiting MoE-specific routing structure benefits RL training.
Reinforcement learning (RL) can become off-policy due to training-inference mismatch, mini-batch updates, and asynchronous training. The resulting mismatch is especially harmful in Mixture-of-Experts (MoE) models, where rollout-training discrepancies can change routing decisions and lead to router drift. Rollout Routing Replay (R3) mitigates this mismatch by reusing expert selections from the rollout stage during training. Our analysis shows that routing replay introduces gradient bias by losing contributions from experts selected by the training router but not the rollout router. To address this, we propose Router Drift Control (RDC), which uses routing disagreement between rollout and training as a signal of off-policy staleness and selectively rejects high-drift responses. The sequence-level filtering avoids gradient bias introduced by replay while preserving native expert selection and the associated learning signals for retained responses. Experiments show that RDC achieves larger gains as policy staleness increases and remains effective in asynchronous RL. In asynchronous training, RDC combined with Truncated Importance Sampling (TIS) improves the average peak validation accuracy across AIME 2024 and AIME 2025 by 5.5 points over TIS and 4.5 points over R3.
Liangyao Jin, Zhixin Wang, Liming Liu +8
Peking University · Infrawaves Inc · Shanghai Innovation Institute +2