Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · LLM-Core Xiaomi · Independent Researcher
Reinforcement learning (RL) has emerged as a crucial approach for enhancing the capabilities of large language models. However, in Mixture-of-Experts (MoE) models, the routing mechanism often introduces instability, even leading to catastrophic RL training collapse. We analyze the training-inference consistency of MoE models and identify a notable discrepancy in routing behaviors between the two phases. Moreover, even under identical conditions, the routing framework can yield divergent expert selections across repeated forward passes. To address this foundational inconsistency, we propose Rollout Routing Replay (R3), a method that records routing distributions from the inference engine and replays them during training. R3 significantly reduces training-inference policy KL divergence and mitigates extreme discrepancies without compromising training speed. Extensive experiments on various settings confirm that R3 succeeds in stabilizing RL training, preventing collapse and outperforming methods such as GSPO and TIS. We believe this work can offer a new solution for stabilizing RL in MoE models.
Figures & tables
Figure 1: Left: Illustration of the Rollout Routing Replay (R3). Top right: Training and inference discrepancies before and after applying R3. Bottom right: Reinforcement learning training performance before and after applying R3.
Figure 2: (a): Illustration of the training-inference discrepancy in the MoE model. (b): Illustration of the training-inference discrepancy in the MoE+R3 model. (c): Illustration of the training-inference discrepancy in the Dense model. (d): Extreme Token Distribution Function, calculated based on Equation 4 .
Figure 3: Illustration of how the router amplifies a numerical perturbation into a large policy discrepancy. For the same input token, the router scores in the inference engine (left) and the training engine (right) differ only by floating-point noise. Since experts 4 and 5 are nearly tied at the Top- K cutoff, this tiny difference is enough to flip which of the two is activated. Because a different expert contributes to the layer output, the resulting output token distribution can differ substantially between the two engines, which makes the collected rollouts off-policy.
Figure 4: Router discrepancy analysis
Best Metric(Best Global Step)
Method
AIME24 (↑)
AIME25 (↑)
AMC23 (↑)
MATH500 Lv5 (↑)
Avg (↑)
Crash Step
Qwen3-30B-A3B-SFT, mini_step=8
GRPO
32.81(65)
20.73(90)
74.84(60)
71.83(100)
48.84(100)
120
GSPO
55.52(165)
38.23 (160)
90.16 (125)
86.38(125)
66.76(125)
-
GRPO+R3
57.92 (180)
38.02(155)
90.16 (155)
88.62 (170)
68.05 (180)
-
GSPO+R3
58.44 (160)
39.17 (160)
92.50 (165)
87.87 (165)
69.00 (165)
-
Table 1: Main evaluation results. This table presents the best scores (with corresponding global step in parentheses) achieved by various methods on evaluation benchmarks under three training configurations. The table also reports the average score (Avg) and the step at which training collapse occurred (Crash Step).
Figure 5: Analysis of training–inference collapse. The plot shows the estimated training–inference KL divergence and the extreme token distribution function F(τ=2) (Eq. 4 ) at each training step.
Model: Qwen3-30B-A3B
Method
SWE-bench Verified (↑)
Crash Step
GRPO
31.80
90
GRPO+R3
38.60
-
Table 2: Evaluation results of RL training on SWE Task
Model: DeepSeek-V2-Lite-ReasoningSFT
Method
Math Avg (↑)
Crash Step
GRPO
23.55(55)
100
GRPO+R3
26.43 (235)
-
Table 3: Evaluation results of the RL training process of DeepSeek-V2-Lite-ReasoningSFT
Metric
w/o R3
w/ R3
# Global Step
10
10
# Token
203,989,395
203,201,992
Runtime (sec)
4820
4967
# Token/sec
42,321
40,910
Relative Time Cost
100%
103.45%
Table 4: Throughput analysis result of Rollout Routing Replay
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The detailed evaluation results of the experiment in Section 5 .
Figure 8: The detailed training metrics of the experiment in Section 5 .
Figure 9: Default setting of partial rollout. Orange and blue denote states computed with θold and θnew , respectively. After the parameter update, TA is re-prefilled with θnew , and the inference engine continues the rollout to obtain TB .
Method
Context of TB at Inference
Training Sequences (Replayed Masks)
TA
TB
Drawback
Re-prefill only
CAnew under IAnew
TA⊕TB ( IAnew⊕IBnew )
✗
✓
Inconsistent on TA
Concat old
CAnew under IAnew
TA⊕TB ( IAold⊕IBnew )
✓
✗
Inconsistent on TB
Split training
CAnew under IAnew
TA ( IAold ), TA⊕TB ( IAnew⊕IBnew )
✓
✓
Extra training computation
Replay in re-prefill
CAnew under IAold
TA⊕TB ( IAold⊕IBnew )
✓
✓
Complex implementation
Cache retention
CAold under IAold
TA⊕TB ( IAold⊕IBnew )
✓
✓
Stale KV cache
Appendix
Table 5: Methods for combining R3 with partial rollout. Parentheses give the routing masks replayed on each training sequence, and ✓/✗ indicate whether the tokens in TA and TB are routing-consistent.
Table 10: Evaluation results of the RL training process of Qwen3-30B-A3B-ReasoningSFT
Figure 11: The detailed training metrics and evaluation results of the RL training process of Qwen3-30B-A3B-ReasoningSFT
KV Cache
Routing Mask Cache
Each Token
DataType × #Layer × (KDim + VDim)
DataType × #Layer × TopK
Each Token in Qwen3-30B-A3B
2×48×(512+512)
2×48×8
437K Tokens per GPU
40 GB on GPU HBM
0.3 GB on CPU Memory
437K × 8 Tokens per Node (8 GPUs)
40 GB×8 on HBM
2.5 GB on CPU Memory
Appendix
Table 11: Comparison of KV Cache and Routing Mask Cache in the Inference Framework
Rank
Token ID
Token String
Mean Token-Level Diff Router
1
15
"0"
16.9834
2
37018
"frac"
16.6235
3
248
(unvisualizable)
16.5253
4
50853
"cdot"
15.9350
5
230
(unvisualizable)
15.8782
6
20
"5"
15.7010
Appendix
Table 12: The top 50 tokens ranked by routing discrepancy
Figure 12: Visualization of routing discrepancy for the response segment, where red indicates large routing discrepancy and green indicates small routing discrepancy
Figure 13: Per-layer routing divergence (Mean Diff) between training and inference for Qwen3-30B-A3B across all 48 layers.
Model
KL (Base)
KL (Thinking)
F(τ=2) (Base)
F(τ=2) (Thinking)
0.6B
6.631e-04
7.549e-04
4.294e-06
1.767e-05
1.7B
1.028e-03
9.389e-04
6.371e-06
6.185e-05
4B
6.037e-04
6.446e-04
1.868e-05
1.066e-05
8B
9.834e-04
6.471e-04
1.575e-05
8.651e-06
32B
–
8.709e-04
–
1.428e-05
30B-A3B
1.339e-03
1.548e-03
2.992e-04
3.377e-04
Appendix
Table 13: Training-inference discrepancy across different model sizes. KL denotes KL divergence and F(τ=2) denotes the F metric with threshold τ=2 .
Hyperparameter
Value
Batch Size
256
Group Size
8
Mini Step
8 or 1
Max Prompt Length
2048
Max Response Length
30720
Optimizer
AdamW
Appendix
Table 14: Default Training hyperparameters of Reinforcement Learning.