In this work, we study end-to-end FP4 reinforcement learning (RL) post-training for Mixture-of-Experts (MoE) large language models. Recent work on FP4 RL has introduced effective recovery recipes that restore accuracy under aggressive low-precision quantization for dense models, but we find that the same approach yields no improvement over the naive FP4 baseline in the MoE setting. We conduct an in-depth study to understand why. Our analysis identifies the root cause in the routing mechanism unique to MoE: because the router makes discrete expert-selection decisions, training-inference policy alignment becomes acutely sensitive to whether the same recovery technique is applied symmetrically on the training and inference sides. When an asymmetric recipe is used-including the dense-model approach, which corrects only the rollout-training-inference alignment deteriorates sharply, as measured by the Pearson correlation coefficient between training and inference probabilities, and router selection matching rates. Building on this finding, we propose MoEResQ, a quantization-recovery recipe for MoE RL post-training that enforces symmetry between the training- and inference-side corrections. MoEResQ restores training-inference alignment and substantially closes the accuracy gap to full precision, establishing symmetric recovery as the key principle in FP4 RL for MoE models.
Figures & tables
Dataset
BF16
HiF4
RolloutResQ
MoEResQ
RolloutResQ (experts only)
MoEResQ (experts only)
AMC23
85.10
65.23 (-19.87)
62.34 (-22.76)
84.45 (-0.65)
78.75 (-6.35)
84.61 (-0.49)
AIME24
48.23
23.96 (-24.27)
25.00 (-23.23)
44.48 (-3.75)
37.39 (-10.84)
46.98 (-1.25)
AIME25
34.38
20.83 (-13.55)
19.06 (-15.32)
29.69 (-4.69)
29.06 (-5.32)
32.92 (-1.46)
MATH500
86.71
79.21 (-7.50)
74.00 (-12.71)
85.42 (-1.29)
82.46 (-4.25)
86.17 (-0.54)
Table 1: Evaluation results for Qwen3-30B-A3B trained on DAPO-Math-17K with GRPO. For HiF4, RolloutResQ and MoEResQ, the attention projections and expert projections are in HiF4 precision. For RolloutResQ (experts only) and MoEResQ (experts only), only expert projections are in HiF4 precision. In all experiments, the routing layers, constituting only 0.041% of the total parameters, are kept in BF16 as outputs are highly sensitive to routing selection.
Figure 1: Visualization of the alignment between training and inference probabilities for BF16, HiF4, RC Inference, RC Training, and RC Both.
RC
RC (Experts Only)
Metric
BF16
HiF4
Inference
Training
Both
Inference
Training
Both
PCC
0.9951
0.9410
0.9444
0.9580
0.9940
0.9702
0.9796
0.9952
top_k_match_frac
Layer 0
0.9869
0.9801
0.8100
0.8195
0.9709
0.9867
0.9875
0.9867
Layer 12
0.8799
0.6172
0.6279
0.6590
0.8682
0.7373
0.7879
0.8730
Layer 23
0.8562
0.5883
0.5990
0.6402
0.8418
0.7209
0.7674
0.8506
Table 2: Training-inference alignment base on three metrics: the Pearson correlation coefficient (PCC) between training and inference probabilities, the fraction of tokens whose top- k expert set matches exactly ( top_k_match_frac ), and the mean pairwise overlap of the top- k sets ( topk_overlap_mean ). For top_k_match_frac , we measured the values for layer 0, 12, 23, 35, and 47. RC denotes applying RC when quantizing both attention linear projections and expert linear projections, and RC (Experts only) denotes applying RC when quantizing expert linear projection only.
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
Reinforcement learning (RL) for post-training large language models (LLMs) incurs substantial computation and memory overhead during rollout generation, which motivates low-precision rollout for efficient RL training. However, existing FP4 RL methods suffer from a key limitation: they primarily optimize quantization accuracy on the training and rollout paths independently rather than directly reducing the discrepancy between the two quantized execution paths. In this work, we propose TRACE (Train-Rollout Quantization Alignment via Compact GuidancE), an FP4 quantization framework for RL training of Mixture-of-Experts (MoE) language models that addresses the limitation of existing FP4 RL methods. TRACE incorporates rollout-guided quantization-aware training that uses rollout-side quantization outcomes to guide training-side FP4 rounding decisions, directly reducing train-rollout discrepancy. Moreover, TRACE adopts an efficient quantization-information caching scheme that selectively retains mantissa and scale information from deeper layers to reduce the storage and communication overhead introduced by rollout guidance. We evaluate TRACE on four large-scale MoE language models across reasoning, coding, and long-horizon RL tasks. Our results demonstrate that TRACE enables joint FP4 weight/activation and FP4 KV-cache rollout with RL performance comparable to BF16 rollout, while achieving up to 5.4xrollout speedup and strong final FP4 performance compared with post-hoc FP4 quantization of BF16-trained policies.
Xin Wang, Hao Yu, Zhengyang Zhuge +9
Alibaba Token Hub, Alibaba Group · Ohio State University
Rollout generation is a major bottleneck in Reinforcement Learning (RL) for Mixture-of-Experts (MoE) Large Language Models, motivating low-precision rollout acceleration such as FP8. As an emerging low-precision format, NVFP4 combines fine-grained scaling for accuracy preservation with native W4A4 FP4 GEMMs for higher throughput than FP8. However, we find that directly applying NVFP4 to MoE RL rollout is impractical. NVFP4 rollout with BF16 training collapses after roughly 150 steps, accompanied by rapidly growing rollout-trainer log-probability gaps. Through training-inference error analysis and controlled ablations, we identify activation error, rather than weight error, as the dominant source of FP4 RL instability: weights can be synchronized and aligned by a shared quantization-dequantization path, whereas activations are recomputed online and error is amplified by the coarse E2M1 grid. Therefore, to stabilize NVFP4 RL for MoE, we propose QUantization-error Alignment across Dual Sides (QUADS). On the trainer side, we introduce Asymmetric Quantization-Aware Training fake-quantizing weights while keeping activations unquantized for better alignment. On the rollout side, Residual Activation Compensation corrects high-error activation channels while preserving native W4A4 GEMMs. In our MoE RL experiments on several benchmarks, QUADS achieves BF16-level accuracy, improves average pass@1 by 21.49 points over naive NVFP4 RL, and delivers ~16% higher rollout throughput than FP8.
We present, to our knowledge, the first end-to-end FP4 RL post-training, in which both the rollout and training policies, including their forward and backward passes, operate at 4-bit precision. A systematic study reveals that the dominant source of degradation in FP4 RL is not training-side quantization error but rollout activation quantization: outliers stretch the dynamic range so far that a large number of activation values underflow to zero under FP4. Counterintuitively, restoring the training policy to higher precision while keeping the rollout in FP4 makes accuracy worse than full FP4 baseline, exposing rollout--training mismatch as the principal failure mode and ruling out standard pretraining-style fixes. We address this with Rollout Residual Quantization (Rollout-ResQ): a single residual correction term constrained to a hardware-friendly sparsity pattern, added only to the FP4 rollout matmul -- a lightweight correction that recovers most of the precision lost to outlier-driven underflow without inflating the rollout's compute footprint. On Qwen2.5-3B, Rollout-ResQ paired with the HiFloat4 (HiF4) format -- whose three-level hierarchical scaling preserves resolution under FP4's tight 4-bit budget -- closes the accuracy gap to BF16 from 3.59% to 1.39%, bringing fully quantized FP4 RL within close reach of full precision. Applied to the open-standard MXFP4, the same recipe narrows the gap from 13.6% to 5.3%, revealing that FP4 format choice is a key factor that determines the ceiling on recoverable accuracy. The experimental results on Qwen2.5-Math-7B further confirm the recovery benefits of Rollout-ResQ with HiF4. Together, these results establish HiF4 as the enabling format for end-to-end FP4 RL post-training, and Rollout-ResQ as the activation-side mechanism that makes the gap to BF16 closable.