Reinforcement learning (RL) post-training increasingly targets low-bit deployment, with NVFP4 weight-and activation quantization (W4A4) becoming the state-of-the-practice format. However, integrating NVFP4 into RL degrades rollout quality and amplifies cross-engine training-inference mismatch (TIM), destabilizing training. We show that these failures are concentrated rather than uniform - severely mismatched tail tokens concentrate in a few outlier sequences, while policy weights remain largely intact and quantization error concentrates in a few high-variance activation channels. Guided by this, we present FourSight, an algorithm-system co-design with two targeted solutions: TIM-aware sequence filtering that excludes drifted trajectories from policy updates, and sparse precision compensation that keeps only top-variance channels in BF16 while retaining native NVFP4 elsewhere. We further design a fused SpliceGEMM kernel with specialized prefill and decode datapaths to minimize overhead. FourSight sustains near-BF16 accuracy and strong FP4 deployment quality across models and engines, while improving step-to-accuracy by 1.76-2.60x and staying within 2.3% of FP4 step time up to 120B-scale models.
Figures & tables
Figure 1: Macroscopic training dynamics of Qwen3-8B and Qwen3-1.7B under BF16 and native FP4 (with default TIS) on GSM8K Cobbe et al. (2021) dataset: (a) mean training reward, illustrating rapid initial learning followed by mid-training collapse ( ∼ Step 280) and an incomplete, fluctuating rebound ( ∼ Step 330); (b) sampled train–inference KL divergence; and (c) policy entropy.
Figure 2: Token-level rollout probability ( π^roll ) versus training probability ( πtrain ) of Qwen3-1.7B at Step 50 and Step 350. While BF16 remains tightly aligned along y=x (a, b), FP4 evolves from a wider fusiform envelope (c) to severe dispersion with dense accumulation along the πtrain≈0 boundary (d).
Figure 3: Pre-softmax rollout logit ( zroll ) versus training logit ( ztrain ) of Qwen3-1.7B. Unwrapping Softmax reveals that at Step 420, these tail tokens form a distinct off-diagonal cluster, with moderate rollout logits but training logits ztrain≈0 .
Figure 4: Spatial distribution of tail tokens ( πtrain<10−4 , red) versus normal tokens (green) of Qwen3-1.7B across 1,024 batch trajectories sorted by tail-token fraction at Step 260, 320, and 350.
Figure 5: Diagnosis of Qwen3-8B rollout degradation: (a) switching rollout from FP4 to BF16 immediately recovers +20 pp GSM8K test accuracy; (b) layer-wise and channel-wise quantization error across QKV projections, showing concentration in a few outlier channels; and (c) evolution of relative policy output error versus relative weight change across training steps.
Figure 6: SpliceGEMM compute flow across the GPU and memory hierarchy. Operand preparation, Register Files, Tensor Cores, and Tensor Memory are denoted as Prep , RF , TC , and TMEM .
Method
vLLM + Megatron-LM
vLLM + FSDP
AIME24
AIME25
AMC23
MATH-500
OCW
AIME24
AIME25
AMC23
MATH-500
OCW
BF16
21.5 ± 6.6
19.0 ± 6.5
65.3 ± 5.9
82.4 ± 1.3
47.7 ± 2.5
23.1 ± 7.0
21.0 ± 6.9
65.2 ± 6.0
81.4 ± 1.4
47.0 ± 2.5
FP4
14.6 ± 5.6
12.1 ± 4.8
44.5 ± 6.5
80.4 ± 1.4
46.8 ± 2.5
14.4 ± 5.1
16.2 ± 5.7
59.5 ± 5.9
78.8 ± 1.5
44.2 ± 2.4
QaRL
23.1 ± 6.4
19.4 ± 6.3
67.8 ± 6.0
83.5 ± 1.3
49.3 ± 2.5
21.9 ± 5.9
21.3 ± 6.8
66.7 ± 5.5
81.8 ± 1.4
45.3 ± 2.5
QuRL
6.7 ± 3.4
2.7 ± 2.0
30.9 ± 5.9
68.1 ± 1.7
42.2 ± 2.4
11.7 ± 4.6
10.8 ± 4.3
54.1 ± 6.1
77.3 ± 1.6
47.1 ± 2.6
QUADS
21.9 ± 6.4
21.9 ± 6.5
70.6 ± 5.5
83.7 ± 1.3
47.7 ± 2.5
20.4 ± 5.7
17.9 ± 6.1
63.3 ± 6.0
78.0 ± 1.4
38.7 ± 2.3
Table 1: Qwen3-8B accuracy (%) under FP4 inference (%, Mean@16, temperature T=0.6 , 8k response limit; mean ± standard error). QUADS and FourSight retain their respective compensation at inference.
Figure 7: Qwen3-8B training with DAPO: AIME24 validation accuracy and training–inference mismatch on trained vLLM + Megatron-LM (a,b) and vLLM + FSDP (c,d). BF16 validation uses BF16 inference; other methods use FP4.
Figure 8: Qwen3-1.7B training with GRPO: FourSight sustains close-to-BF16 accuracy and controls TIM as pure FP4 degrades.
Method
GSM8K
SVAMP
MATH-500
GSM1K
BF16
67.30 ± 0.97
74.54 ± 0.96
37.82 ± 1.63
59.13 ± 1.07
FP4
45.86 ± 1.10
49.67 ± 1.28
16.25 ± 1.24
39.38 ± 1.13
FourSight
80.65 ± 0.83
87.26 ± 0.81
51.34 ± 1.77
75.53 ± 0.95
Table 2: Qwen3-1.7B FP4 inference accuracy (%, Mean@16, temperature T=0.6 , 4k response limit; mean ± standard error).
Figure 9: Step time and breakdown at matched response lengths. FourSight introduces less than 2.3% overhead to pure FP4.
Figure 10: Kernel performance measured by speedup over BF16 with 64-channel compensation. “Up” and “down” denote the two feed-forward projections; the blue line marks BF16 speed.
Figure 11: Ablation study of sequence filtering (SF), precision compensation (PC), and their combination across training engines.
Method
AIME24
AIME25
AMC23
MATH-500
OCW
vLLM + Megatron-LM
SF-only
20.8 ± 6.0
19.2 ± 6.0
69.7 ± 5.2
83.4 ± 1.3
48.6 ± 2.5
PC-only
10.0 ± 4.3
4.4 ± 2.6
40.6 ± 5.9
69.1 ± 1.7
41.2 ± 2.4
FourSight
26.3 ± 6.6
23.3 ± 6.6
73.3 ± 5.4
85.9 ± 1.3
48.4 ± 2.5
vLLM + FSDP
SF-only
12.5 ± 4.7
14.8 ± 5.4
50.5 ± 6.6
81.0 ± 1.4
47.4 ± 2.6
Table 3: Final FP4 accuracy (%) with sequence filtering (SF), precision compensation (PC), or both. Evaluation follows Table 1 .
Figure 12: Sensitivity of the low-probability token criterion. Top: retained and discarded tokens. Bottom: token retention versus probability threshold τ and count K ; gold marks the default. The footer reports TIM KL before and after filtering.
Figure 13: Frozen-checkpoint AIME24 accuracy with random (Rand), error-based (Err), or variance-based (Var) channel selection; right: gain in percentage points (pp) versus channel budget. Bars show 95% confidence intervals over 30 samples.
Parameter Name
GRPO (1.7B)
DAPO (8B)
trainer.nnodes
1
1
trainer.n_gpu_per_node
8
8
data.train_batch_size
256
32
data.max_prompt_length
1024
1024
data.max_response_length
512
8192
rollout.n
4
8
Table 4: Experimental settings for the GRPO and DAPO recipes.
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.