Reinforcement learning (RL) post-training increasingly targets low-bit deployment, with NVFP4 weight-and activation quantization (W4A4) becoming the state-of-the-practice format. However, integrating NVFP4 into RL degrades rollout quality and amplifies cross-engine training-inference mismatch (TIM), destabilizing training. We show that these failures are concentrated rather than uniform - severely mismatched tail tokens concentrate in a few outlier sequences, while policy weights remain largely intact and quantization error concentrates in a few high-variance activation channels. Guided by this, we present FourSight, an algorithm-system co-design with two targeted solutions: TIM-aware sequence filtering that excludes drifted trajectories from policy updates, and sparse precision compensation that keeps only top-variance channels in BF16 while retaining native NVFP4 elsewhere. We further design a fused SpliceGEMM kernel with specialized prefill and decode datapaths to minimize overhead. FourSight sustains near-BF16 accuracy and strong FP4 deployment quality across models and engines, while improving step-to-accuracy by 1.76-2.60x and staying within 2.3% of FP4 step time up to 120B-scale models.
Figures & tables
Figure 1: Macroscopic training dynamics of Qwen3-8B and Qwen3-1.7B under BF16 and native FP4 (with default TIS) on GSM8K Cobbe et al. (2021) dataset: (a) mean training reward, illustrating rapid initial learning followed by mid-training collapse ( ∼ Step 280) and an incomplete, fluctuating rebound ( ∼ Step 330); (b) sampled train–inference KL divergence; and (c) policy entropy.
Figure 2: Token-level rollout probability ( π^roll ) versus training probability ( πtrain ) of Qwen3-1.7B at Step 50 and Step 350. While BF16 remains tightly aligned along y=x (a, b), FP4 evolves from a wider fusiform envelope (c) to severe dispersion with dense accumulation along the πtrain≈0 boundary (d).
Figure 3: Pre-softmax rollout logit ( zroll ) versus training logit ( ztrain ) of Qwen3-1.7B. Unwrapping Softmax reveals that at Step 420, these tail tokens form a distinct off-diagonal cluster, with moderate rollout logits but training logits ztrain≈0 .
Figure 4: Spatial distribution of tail tokens ( πtrain<10−4 , red) versus normal tokens (green) of Qwen3-1.7B across 1,024 batch trajectories sorted by tail-token fraction at Step 260, 320, and 350.
Figure 5: Diagnosis of Qwen3-8B rollout degradation: (a) switching rollout from FP4 to BF16 immediately recovers +20 pp GSM8K test accuracy; (b) layer-wise and channel-wise quantization error across QKV projections, showing concentration in a few outlier channels; and (c) evolution of relative policy output error versus relative weight change across training steps.
Figure 6: SpliceGEMM compute flow across the GPU and memory hierarchy. Operand preparation, Register Files, Tensor Cores, and Tensor Memory are denoted as Prep , RF , TC , and TMEM .
Method
vLLM + Megatron-LM
vLLM + FSDP
AIME24
AIME25
AMC23
MATH-500
OCW
AIME24
AIME25
AMC23
MATH-500
OCW
BF16
21.5 ± 6.6
19.0 ± 6.5
65.3 ± 5.9
82.4 ± 1.3
47.7 ± 2.5
23.1 ± 7.0
21.0 ± 6.9
65.2 ± 6.0
81.4 ± 1.4
47.0 ± 2.5
FP4
14.6 ± 5.6
12.1 ± 4.8
44.5 ± 6.5
80.4 ± 1.4
46.8 ± 2.5
14.4 ± 5.1
16.2 ± 5.7
59.5 ± 5.9
78.8 ± 1.5
44.2 ± 2.4
QaRL
23.1 ± 6.4
19.4 ± 6.3
67.8 ± 6.0
83.5 ± 1.3
49.3 ± 2.5
21.9 ± 5.9
21.3 ± 6.8
66.7 ± 5.5
81.8 ± 1.4
45.3 ± 2.5
QuRL
6.7 ± 3.4
2.7 ± 2.0
30.9 ± 5.9
68.1 ± 1.7
42.2 ± 2.4
11.7 ± 4.6
10.8 ± 4.3
54.1 ± 6.1
77.3 ± 1.6
47.1 ± 2.6
QUADS
21.9 ± 6.4
21.9 ± 6.5
70.6 ± 5.5
83.7 ± 1.3
47.7 ± 2.5
20.4 ± 5.7
17.9 ± 6.1
63.3 ± 6.0
78.0 ± 1.4
38.7 ± 2.3
Table 1: Qwen3-8B accuracy (%) under FP4 inference (%, Mean@16, temperature T=0.6 , 8k response limit; mean ± standard error). QUADS and FourSight retain their respective compensation at inference.
Figure 7: Qwen3-8B training with DAPO: AIME24 validation accuracy and training–inference mismatch on trained vLLM + Megatron-LM (a,b) and vLLM + FSDP (c,d). BF16 validation uses BF16 inference; other methods use FP4.
Figure 8: Qwen3-1.7B training with GRPO: FourSight sustains close-to-BF16 accuracy and controls TIM as pure FP4 degrades.
Method
GSM8K
SVAMP
MATH-500
GSM1K
BF16
67.30 ± 0.97
74.54 ± 0.96
37.82 ± 1.63
59.13 ± 1.07
FP4
45.86 ± 1.10
49.67 ± 1.28
16.25 ± 1.24
39.38 ± 1.13
FourSight
80.65 ± 0.83
87.26 ± 0.81
51.34 ± 1.77
75.53 ± 0.95
Table 2: Qwen3-1.7B FP4 inference accuracy (%, Mean@16, temperature T=0.6 , 4k response limit; mean ± standard error).
Figure 9: Step time and breakdown at matched response lengths. FourSight introduces less than 2.3% overhead to pure FP4.
Figure 10: Kernel performance measured by speedup over BF16 with 64-channel compensation. “Up” and “down” denote the two feed-forward projections; the blue line marks BF16 speed.
Figure 11: Ablation study of sequence filtering (SF), precision compensation (PC), and their combination across training engines.
Method
AIME24
AIME25
AMC23
MATH-500
OCW
vLLM + Megatron-LM
SF-only
20.8 ± 6.0
19.2 ± 6.0
69.7 ± 5.2
83.4 ± 1.3
48.6 ± 2.5
PC-only
10.0 ± 4.3
4.4 ± 2.6
40.6 ± 5.9
69.1 ± 1.7
41.2 ± 2.4
FourSight
26.3 ± 6.6
23.3 ± 6.6
73.3 ± 5.4
85.9 ± 1.3
48.4 ± 2.5
vLLM + FSDP
SF-only
12.5 ± 4.7
14.8 ± 5.4
50.5 ± 6.6
81.0 ± 1.4
47.4 ± 2.6
Table 3: Final FP4 accuracy (%) with sequence filtering (SF), precision compensation (PC), or both. Evaluation follows Table 1 .
Figure 12: Sensitivity of the low-probability token criterion. Top: retained and discarded tokens. Bottom: token retention versus probability threshold τ and count K ; gold marks the default. The footer reports TIM KL before and after filtering.
Figure 13: Frozen-checkpoint AIME24 accuracy with random (Rand), error-based (Err), or variance-based (Var) channel selection; right: gain in percentage points (pp) versus channel budget. Bars show 95% confidence intervals over 30 samples.
Parameter Name
GRPO (1.7B)
DAPO (8B)
trainer.nnodes
1
1
trainer.n_gpu_per_node
8
8
data.train_batch_size
256
32
data.max_prompt_length
1024
1024
data.max_response_length
512
8192
rollout.n
4
8
Table 4: Experimental settings for the GRPO and DAPO recipes.
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
We present, to our knowledge, the first end-to-end FP4 RL post-training, in which both the rollout and training policies, including their forward and backward passes, operate at 4-bit precision. A systematic study reveals that the dominant source of degradation in FP4 RL is not training-side quantization error but rollout activation quantization: outliers stretch the dynamic range so far that a large number of activation values underflow to zero under FP4. Counterintuitively, restoring the training policy to higher precision while keeping the rollout in FP4 makes accuracy worse than full FP4 baseline, exposing rollout--training mismatch as the principal failure mode and ruling out standard pretraining-style fixes. We address this with Rollout Residual Quantization (Rollout-ResQ): a single residual correction term constrained to a hardware-friendly sparsity pattern, added only to the FP4 rollout matmul -- a lightweight correction that recovers most of the precision lost to outlier-driven underflow without inflating the rollout's compute footprint. On Qwen2.5-3B, Rollout-ResQ paired with the HiFloat4 (HiF4) format -- whose three-level hierarchical scaling preserves resolution under FP4's tight 4-bit budget -- closes the accuracy gap to BF16 from 3.59% to 1.39%, bringing fully quantized FP4 RL within close reach of full precision. Applied to the open-standard MXFP4, the same recipe narrows the gap from 13.6% to 5.3%, revealing that FP4 format choice is a key factor that determines the ceiling on recoverable accuracy. The experimental results on Qwen2.5-Math-7B further confirm the recovery benefits of Rollout-ResQ with HiF4. Together, these results establish HiF4 as the enabling format for end-to-end FP4 RL post-training, and Rollout-ResQ as the activation-side mechanism that makes the gap to BF16 closable.
Rollout generation is a major bottleneck in Reinforcement Learning (RL) for Mixture-of-Experts (MoE) Large Language Models, motivating low-precision rollout acceleration such as FP8. As an emerging low-precision format, NVFP4 combines fine-grained scaling for accuracy preservation with native W4A4 FP4 GEMMs for higher throughput than FP8. However, we find that directly applying NVFP4 to MoE RL rollout is impractical. NVFP4 rollout with BF16 training collapses after roughly 150 steps, accompanied by rapidly growing rollout-trainer log-probability gaps. Through training-inference error analysis and controlled ablations, we identify activation error, rather than weight error, as the dominant source of FP4 RL instability: weights can be synchronized and aligned by a shared quantization-dequantization path, whereas activations are recomputed online and error is amplified by the coarse E2M1 grid. Therefore, to stabilize NVFP4 RL for MoE, we propose QUantization-error Alignment across Dual Sides (QUADS). On the trainer side, we introduce Asymmetric Quantization-Aware Training fake-quantizing weights while keeping activations unquantized for better alignment. On the rollout side, Residual Activation Compensation corrects high-error activation channels while preserving native W4A4 GEMMs. In our MoE RL experiments on several benchmarks, QUADS achieves BF16-level accuracy, improves average pass@1 by 21.49 points over naive NVFP4 RL, and delivers ~16% higher rollout throughput than FP8.
Training large language models directly in 4-bit floating-point (FP) formats promises substantial improvements in throughput and energy efficiency. While some recipe design choices have been validated at scale, many promising approaches remain untested beyond small models and short token horizons, leaving open the question of which trends will hold. We present a systematic comparison of recent NVFP4 recipes at medium model scale, evaluating dense 8B and 30B-A3B MoE models trained for up to 1T tokens, with a focus on which ingredients are necessary for accuracy recovery. We propose a final recipe grounded in the principles behind established stable NVFP4 training at scale, incorporating state-of-the-art techniques such as unbiased gradient estimation with lower quantization error than stochastic rounding. To the best of our knowledge, this is the strongest FP4 training result demonstrated at this scale to date in loss gap to BF16. Through ablation studies, we find that: (i) each technique in the optimized recipe measurably improves loss trajectory, (ii) selective high-precision layers are necessary for recovering accuracy at scale, (iii) not all tensors in the backward pass benefit equally from de-biasing, leaving room to apply complementary error-reduction techniques to the remaining tensors.
Anjulie Agrusa, Andrei Panferov, Elizabeth Wei +7
NVIDIA · Institute of Science and Technology Austria · Red Hat AI