Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs
Organizations: Huawei
Abstract
In this work, we study end-to-end FP4 reinforcement learning (RL) post-training for Mixture-of-Experts (MoE) large language models. Recent work on FP4 RL has introduced effective recovery recipes that restore accuracy under aggressive low-precision quantization for dense models, but we find that the same approach yields no improvement over the naive FP4 baseline in the MoE setting. We conduct an in-depth study to understand why. Our analysis identifies the root cause in the routing mechanism unique to MoE: because the router makes discrete expert-selection decisions, training-inference policy alignment becomes acutely sensitive to whether the same recovery technique is applied symmetrically on the training and inference sides. When an asymmetric recipe is used-including the dense-model approach, which corrects only the rollout-training-inference alignment deteriorates sharply, as measured by the Pearson correlation coefficient between training and inference probabilities, and router selection matching rates. Building on this finding, we propose MoEResQ, a quantization-recovery recipe for MoE RL post-training that enforces symmetry between the training- and inference-side corrections. MoEResQ restores training-inference alignment and substantially closes the accuracy gap to full precision, establishing symmetric recovery as the key principle in FP4 RL for MoE models.
Figures & tables
| Dataset | BF16 | HiF4 | RolloutResQ | MoEResQ | RolloutResQ (experts only) | MoEResQ (experts only) |
|---|---|---|---|---|---|---|
| AMC23 | 85.10 | 65.23 (-19.87) | 62.34 (-22.76) | 84.45 (-0.65) | 78.75 (-6.35) | 84.61 (-0.49) |
| AIME24 | 48.23 | 23.96 (-24.27) | 25.00 (-23.23) | 44.48 (-3.75) | 37.39 (-10.84) | 46.98 (-1.25) |
| AIME25 | 34.38 | 20.83 (-13.55) | 19.06 (-15.32) | 29.69 (-4.69) | 29.06 (-5.32) | 32.92 (-1.46) |
| MATH500 | 86.71 | 79.21 (-7.50) | 74.00 (-12.71) | 85.42 (-1.29) | 82.46 (-4.25) | 86.17 (-0.54) |
| RC | RC (Experts Only) | |||||||
|---|---|---|---|---|---|---|---|---|
| Metric | BF16 | HiF4 | Inference | Training | Both | Inference | Training | Both |
| PCC | 0.9951 | 0.9410 | 0.9444 | 0.9580 | 0.9940 | 0.9702 | 0.9796 | 0.9952 |
| top_k_match_frac | ||||||||
| Layer 0 | 0.9869 | 0.9801 | 0.8100 | 0.8195 | 0.9709 | 0.9867 | 0.9875 | 0.9867 |
| Layer 12 | 0.8799 | 0.6172 | 0.6279 | 0.6590 | 0.8682 | 0.7373 | 0.7879 | 0.8730 |
| Layer 23 | 0.8562 | 0.5883 | 0.5990 | 0.6402 | 0.8418 | 0.7209 | 0.7674 | 0.8506 |
RL Post-Training
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
See all 460 papers