End-to-End MXFP8 Reinforcement Learning on Blackwell: What Matching Buys, When It Pays Off, and How It Fails

Wengee Zhang

cs.LGOct 26, 2026

End-to-End MXFP8 Reinforcement Learning on Blackwell: What Matching Buys, When It Pays Off, and How It Fails

Authors: Wengee Zhang

Abstract

Low-precision arithmetic can make reinforcement learning (RL) for large language models cheaper, but RL requires the sampler and the learner to agree. Several frameworks now run FP8 on both sides. We implement end-to-end MXFP8 RL in verl (Megatron/Transformer Engine training; vLLM and SGLang rollout; dense and MoE models) and measure on B200 GPUs, with Qwen3-8B and Qwen3-30B-A3B, what this design buys, when it is faster, and how it fails. (1) Matching. For an MXFP8 learner, a sampler on the learner's grid is more consistent than a higher-precision BF16 sampler: in one engine at fixed weights, train-rollout KL is 11% lower with half of the sampler's linear layers on the grid and 16% lower with all of them, and the synchronized weights we checked equal the learner's FP8 forward weights byte for byte. On the MoE model, an FP8 pipeline whose sampler uses another grid has twice the shared-grid pipeline's KL from the first step. Without a quantized KV cache, BF16 end to end remains 2.5-10x more consistent. (2) Speed. MXFP8 training adds a fixed cost per micro-batch and saves a per-token cost, so in our dense setting it pays off only above about 4.5k tokens per micro-batch; a micro-batch sweep on identical data confirms the per-micro-batch cost, and a trace shows the GPU waiting for the host for most of the added time. End to end, MXFP8 cuts MoE step time by 10.7% and generation time by 24%, and slows a short-sequence dense workload by 5%. (3) Correctness. We state seven conditions every weight synchronization must preserve and checks that test them against the live engine. Each of the seven failures we met on hardware broke one condition, five silently; the checks caught two at their first opportunity, each missed by the other checks, and CPU fault-injection tests reproduce five. All runs are single-seed and at most 60 steps long; we make no claim about final model quality.

Figures & tables

Learner GEMMs	Sampler GEMMs	Grid relation	Reported in	Here
BF16	BF16	same values, different kernels	standard RL; [ 23 , 5 ]	$\checkmark$
BF16	FP8 / INT8	post-training quantization of the learner	[ 14 , 24 , 31 ]
FP8 tensorwise	BF16	sampler not quantized	this work	$\checkmark$
MXFP8	BF16	sampler not quantized	this work (ablation)	$\checkmark$
MXFP8	FP8 blockwise	different formats	[ 1 ] (reward collapse)
FP8 blockwise	FP8 blockwise	different scales (SM100: power-of-two vs. FP32)	this work	$\checkmark$

Table 1: Learner/sampler precision pairings. “Grid” is the relation between the sampler’s weights and the weight operand of the learner’s forward GEMM. Rows marked

\checkmark

are measured in this paper.

Figure 1: End-to-end MXFP8 RL. The learner computes with MXFP8 GEMMs over BF16 master weights. At every synchronization the weights are quantized with the learner’s own TE quantizer, so the sampler receives the grid of the learner’s forward weight operand. The sampler re-derives its kernel layouts from those weights in place. Three guards turn silent failures into errors.

	A : dense, gsm8k	B : dense, DAPO	C : dense, gsm8k, SGLang	D : MoE, DAPO
Model	Qwen3-8B	Qwen3-8B	Qwen3-8B	Qwen3-30B-A3B (48 layers, 128 experts, top-8)
GPUs, parallelism	2 B200, TP 2	2 B200, TP 2	2 B200, TP 2	4 B200, TP 2 $\times$ EP 2
Prompt / response cap	1024 / 2048	1024 / 8192	1024 / 2048	1024 / 8192
Tokens per step	0.29–0.68 M	2.9–3.7 M	0.37–0.67 M	3.4 M
Steps	60	32	30	10
Micro-batching	4 sequences (update), 8 (log-prob)	dynamic, 12 288 tokens/GPU	as A	dynamic, 20 480 tokens/GPU

Table 2: Experimental settings. A, B and D ran all their arms on one fast host (AMD EPYC 9555); C and the MXFP8 / BF16 arm of A ran on a host with a slower CPU (Intel Xeon 6960P), so their timings are not compared with the others.

Figure 2: Train–rollout KL per step (log scale). In (a) and (b) the vLLM MXFP8 / MXFP8 sampler kept gate_up_proj in BF16 (F2); the SGLang sampler in (a) and the MoE sampler in (c) put every decoder linear on the learner’s grid.

Figure 3: Mean train–rollout KL by learner/sampler pairing. (a) For the same MXFP8 learner recipe, KL is lower when more of the sampler’s linear layers are on the learner’s grid (none

\rightarrow

qkv/o/down

\rightarrow

all); the fully matched sampler is SGLang, the other two are vLLM. The two control pairings, whose samplers are BF16, agree across the two engines within 2.5%. (b) On the MoE model, the shared-grid pipeline has half the KL of the FP8-blockwise pipeline, whose sampler quantizes with FP32 scales while its learner is restricted to power-of-two scales. (c) The controlled experiment in one engine, with the checkpoint’s weights at every step and KL pooled over steps 2–10, gives the same order as (a); Table 4 gives the paired reductions with their intervals.

Setting	Pairing	kl , step 1	kl	log_ppl_abs_diff	Pearson	chi2_token
A: dense, gsm8k (vLLM)	BF16 / BF16	0.00066	0.00069 (0.00071)	0.00143 (0.00129)	0.9954 (0.9965)	0.0015 (0.0014)
	FP8-TW / BF16	0.00859	0.01217 (0.01225)	0.01317 (0.01325)	0.9380 (0.9464)	0.0404 (0.0408)
	MXFP8 / BF16	0.00828	0.00864 (0.00865)	0.00953 (0.00940)	0.9678 (0.9724)	0.0220 (0.0230)
	MXFP8 / MXFP8, gate_up BF16	0.00723	0.00693 (0.00784)	0.00784 (0.00862)	0.9791 (0.9758)	0.0170 (0.0200)
C: dense, gsm8k (SGLang)	MXFP8 / MXFP8, all matched	0.00681	0.00670 (0.00703)	0.00730 (0.00759)	0.9806 (0.9792)	0.0158 (0.0183)
B: dense, DAPO (vLLM)	BF16 / BF16	0.00610	0.00882 (0.00764)	0.00864 (0.00749)	0.9828 (0.9870)	0.0224 (0.0184)

Table 3: Train–rollout mismatch. “Step 1” is kl at the first step, where all arms of a setting hold the same weights and see the same prompts; the other columns are last-10-step means (whole-run means in parentheses); lower is better except for the Pearson correlation.

\chi^{2}

is the median. Setting C’s two control arms are omitted; their step-1 KL is 0.00064 (BF16 / BF16) and 0.00881 (FP8-TW / BF16), and over the common steps 1–30 their KL agrees with A’s within 2.5%.

Pairing	kl , step 1	kl	log_ppl_abs_diff	Pearson	chi2_token	BF16 sampler	gate_up BF16
						KL reduction (95% interval) against
BF16 / BF16	0.00067	0.00072	0.00108	0.9979	0.00129	–	–
MXFP8 / BF16	0.00854	0.00812	0.00868	0.9778	0.02279	–	–
MXFP8 / MXFP8, gate_up BF16	0.00719	0.00726	0.00787	0.9794	0.02013	10.6% (8.6–12.7)	–
MXFP8 / MXFP8, all matched	0.00697	0.00682	0.00739	0.9809	0.01841	16.1% (14.1–18.1)	6.1% (3.9–8.3)

Table 4: Matching in one engine at fixed weights: vLLM, learning rate 0, setting A’s configuration. “Step 1” is kl at the first step; the other columns pool steps 2–10 (576 prompts

\times

8 samples per arm, the same prompts in every arm), so chi2_token and the Pearson correlation are taken over all tokens rather than as medians of steps. The KL reductions carry 95% intervals from 10 000 bootstrap resamples of prompts, paired across arms.

Figure 4: Step time (top) and tokens per step (bottom) of every arm; each column ran on one host. (a) Dense 8B, gsm8k: step time follows the tokens per step, which halve as responses shorten; MXFP8 is slower than BF16 at 57 of 60 steps. (b) Dense 8B, DAPO: MXFP8 is faster than BF16 at 31 of 32 steps at equal token counts; the per-tensor arm is faster still but generates fewer tokens. (c) MoE 30B-A3B: token counts agree with BF16’s within 0.2%, and MXFP8 is the fastest arm at every step. Per-stage curves are in Appendix B .

Micro-batch					Predicted MXFP8 $-$ BF16 (s)
sequences (tokens)	$N_{\text{mb}}$ per step	BF16 (s)	MXFP8 (s)	MXFP8 $-$ BF16 (s)	per micro-batch	per step
1 (1.3k)	512	69.12	84.49	$+15.37\pm 0.44$	$+18.9$	$-0.9$
2 (2.6k)	256	42.90	48.71	$+5.81\pm 0.97$	$+5.7$	$-0.9$
4 (5.2k)	128	32.72	33.36	$+0.64\pm 0.91$	$-0.9$	$-0.9$
8 (10.3k)	64	32.47	24.86	$-7.61\pm 1.68$	$-4.2$	$-0.9$
4, half batch (5.2k)	64	17.43	16.55	$-0.88\pm 0.75$	$-0.5$	$+2.8$

Table 5: Micro-batch sweep on identical replayed data: update-stage time per step (mean of steps 2–6) and the difference between the learners, against the predictions of a per-micro-batch and a per-step cost with setting A’s constants (

\Delta\delta=51.5

ms,

\Delta c=11.36\,\mu

s/token,

\Delta F=6.58

s); the two predictions coincide at 128 micro-batches, setting A’s own count, from which

\Delta\delta

was derived.

\pm

is the standard error over the five paired steps. Bottom: fits of

t=a+N_{\text{mb}}\,\delta+c\,T

to the 25 points of each learner, with standard errors.

Per micro-batch	BF16	MXFP8	Difference
Wall time under the profiler (ms)	159.3	202.6	$+43.4$
GPU kernel time, summed (ms)	72.0	75.5	$+3.5$
GEMM	20.4	13.3	$-7.1$
MXFP8 quantization	–	2.2	$+2.2$
scale rearrangement	–	2.6	$+2.6$
NCCL	13.3	18.3	$+5.1$

Table 6: Where the added cost goes: update stage with 1-sequence micro-batches, per micro-batch (Nsight Systems, step 3, rank 0, 512 micro-batches). Kernel times are summed over all streams. GPU busy time merges kernels and memory copies across streams; the rest of the wall time, with no GPU operation running, is split by when the host issued the operation that ends each gap. The profiler inflates host-side time. The last row is from the timed runs without the profiler.

Stage	Arm	$F$ (s)	$c$ ( $\mu$ s/token)	$R^{2}$	$\Delta\delta$ (ms per micro-batch)	$m^{\ast}$ (tokens per micro-batch)
update_actor (fwd+bwd)	BF16	$12.31\pm 0.30$	$26.74\pm 0.69$	0.964	–	–
	FP8-TW	$18.10\pm 0.28$	$14.60\pm 0.66$	0.899	45.3	$3\,726\pm 61$
	MXFP8	$18.89\pm 0.24$	$15.38\pm 0.56$	0.932	51.5	$4\,527\pm 108$
old_log_prob (fwd)	BF16	$3.40\pm 0.24$	$10.60\pm 0.55$	0.872	–	–
	FP8-TW	$4.08\pm 0.12$	$8.06\pm 0.27$	0.942	10.6	$4\,161\pm 666$
	MXFP8	$4.17\pm 0.23$	$8.03\pm 0.53$	0.809	12.0	$4\,668\pm 701$

Table 7: Stage cost model

t=F+c\,T

in setting A (fast host, steps 4–60), and the break-even point per micro-batch.

\Delta\delta=\Delta F/N_{\text{mb}}

with

N_{\text{mb}}=128

(update) and 64 (old log-prob);

m^{\ast}=\Delta\delta/\Delta c

\pm

are standard errors: of the fit for

F

and

c

, and for

m^{\ast}

from the delta method with the covariance of each fit’s intercept and slope (correlation

-0.98

). A bootstrap over steps gives 95% intervals for

m^{\ast}

of 4.3–4.8k (MXFP8) and 3.6–3.9k (FP8-TW) for update_actor , and 2.1–5.6k and 1.6–5.1k for the old log-prob, whose break-even point lies at the edge of the measured range.

Figure 5: (a) update_actor time per step against tokens per step in setting A, with least-squares fits. (b) Time saved per token by MXFP8 against tokens per micro-batch: Eq. ( 1 ) with constants from setting A (curves), measurements from A (filled) and from the held-out setting B (open), and the model’s prediction for B (

\times

Figure 6: Stage time relative to BF16 / BF16 on the same host. (a) Dense, setting B, per token, because the FP8-TW arm generated 3.5% fewer tokens. (b) MoE, setting D, per step (tokens per step within 0.2% across arms).

	BF16 / BF16	FP8-TW / BF16	FP8-BW / FP8-BW	MXFP8 / MXFP8
step (s)	506.0	487.2 ( $-3.7\%$ )	488.9 ( $-3.4\%$ )	451.8 ( $-10.7\%$ )
generation	133.6	133.1 ( $-0.4\%$ )	115.5 ( $-13.5\%$ )	101.7 ( $-23.8\%$ )
old log-prob	63.9	64.8 ( $+1.4\%$ )	66.1 ( $+3.4\%$ )	60.5 ( $-5.3\%$ )
reference (BF16)	57.3	57.3 ( $0.0\%$ )	56.5 ( $-1.3\%$ )	57.1 ( $-0.4\%$ )
update_actor	244.4	225.4 ( $-7.8\%$ )	240.8 ( $-1.5\%$ )	221.3 ( $-9.4\%$ )
weight sync	3.5	3.4	6.3	7.5

Table 8: Setting D (Qwen3-30B-A3B, 4 B200, one host): mean over 10 steps. Actor memory is the allocator high-water mark.

Setting		BF16 / BF16	FP8-TW / BF16	MXFP8 / BF16	MXFP8 / MXFP8
A	gsm8k accuracy, step 0 $\rightarrow$ 60	0.3169 $\rightarrow$ 0.9363	0.3169 $\rightarrow$ 0.9249	0.3154 $\rightarrow$ 0.9356	0.3359 $\rightarrow$ 0.9393
A	entropy change	$-49\%$	$-47\%$	$-23\%$	$-4\%$
C	gsm8k accuracy, step 0 $\rightarrow$ 30	0.3184 $\rightarrow$ 0.8870	0.3230 $\rightarrow$ 0.8946	–	0.3738 $\rightarrow$ 0.8795
C	entropy change	$-34\%$	$-32\%$	–	$+4\%$
B	score, last 10 steps	0.150	0.240	–	0.135
B	entropy change	$-31\%$	$-74\%$	–	$+4\%$

Table 9: Learning signals. gsm8k accuracy is on the full test set (1 319 problems) at step 0 and at the last step. Entropy is the mean of steps 1–3 against the last 10 steps (D: steps 8–10).

Figure 7: Policy entropy per step. Under MXFP8 training the entropy stays flat or rises; under BF16 and FP8-TW it decays, fastest under FP8-TW on DAPO-Math-17k (

-74\%

in 32 steps).

	Failure	Breaks	Symptom	Root cause	Caught now by; fix
F1	lm_head quantized in the sampler (silent)	C1	reward 0.000 for all 20 steps; responses at the length cap; NaN probability difference; untrained model scores 0.000 (healthy: 0.316)	not isolated: excluding lm_head removes the NaN logits; value range, scale handling in the vocabulary-parallel head and the kernel path remain candidates; with NaN logits EOS is never sampled	audit, sentinel; lm_head and embedding kept in BF16
F2	router exclusion silently removes dense MLP layers (silent)	C1	none in the metrics; 56 of 311 parameters flagged on Qwen3-1.7B	vLLM 0.24’s ModelOpt matcher falls back to substring matching, so mlp.gate also excludes mlp.gate_up_proj	audit (engine re-check); exact matching on module paths
F3	stale swizzled scales after sync, SGLang FlashInfer CUTLASS (silent)	C3	held-out accuracy 0.000 from step 0; KL 7.2; entropy 10.4 ( $\approx$ uniform over the vocabulary)	the backend derives an interleaved scale copy at load that the sync never rebuilds	self-check, sentinel; refit loader re-runs post-processing
F4	stale MoE expert layout after sync, vLLM TRT-LLM MXFP8 MoE (silent)	C3	expert probe relative error 1.74 at the first sync (healthy 0.05); KL 0.047 vs. 0.0022 on a small randomly initialized MoE	an in-place repack that keeps shape and dtype escaped shape-based detection; a “done” flag also blocked re-processing	self-check (MoE probe); record same-shape repacks, merged as verl #7986
F5	BF16 MoE sync crashes on SM100	C3	crash at the first sync	vLLM picks TRT-LLM’s unquantized MoE kernel, whose 4-D layout the BF16 per-expert sync cannot write	crash; in-place staging, merged as verl #7987
F6	FP32 block scales on SM100	C2	assertion at the first FP8 quantization	TE emulates block scaling with MXFP8 GEMMs, whose E8M0 scales keep only the exponent	assertion; FP32 scales documented as Hopper-only, verl #8062

Table 10: Failures found on B200 and the condition each breaks. “Silent” means the job exits 0 with metrics that are logged normally.

Check	Tests	Runs	Catches	Cannot see
quantized-layer audit	C1	once, at the first sync after a training step	any parameter whose precision differs between learner and engine, in either direction, read from the engine’s live state (vLLM)	values (C2–C6); the engine query counts only linear and expert layers, so a quantized lm_head shows only in the rule half; where no engine query exists (SGLang, FP8-blockwise path) it compares with the configured rule and inherits the rule’s errors
self-check	C3 (C6 for the probed layers)	every sync	stale derived state in every layer of a kind, whatever its shape and dtype, because it reads the kernel’s output	C2 (kernel and reference use the same synchronized weights); faults confined to layers or experts it does not probe; C4 (it calls the kernel eagerly, not through the CUDA graph)
step sentinel	C6	every step	non-finite probabilities, KL above 1, nearly every response truncated on two steps, whatever the cause	divergence below its thresholds (F2, F4)
refit-cycle assertions	C4, C5	every sync	an update that cannot be folded into the captured storage; staged buffers not folded back; scales a streamed update never wrote (SGLang)	storage the cycle does not track
byte comparison (offline)	C2, C3	at two synchronizations of the controlled runs	a sampler weight, scale or kernel layout that differs from TE’s quantization of the synchronized weights, or a learner forward weight that does (Section 9.3 )	layers it does not sample (3 of 36); it is not run during training

Table 11: Checks, the conditions they test, and their coverage by construction. All but the byte comparison run during training.

Fault	audit, rule half	audit, engine re-check	self-check	sentinel
F1 lm_head quantized	caught (t)	missed: the engine query counts only linear and expert layers (c)	missed: lm_head is not probed (c)	fires: NaN probability difference (m)
F2 gate_up_proj left in BF16	missed: 28 of 28 layers agree (m)	caught: 56 of 311 parameters (m)	missed: probed layers are healthy (c)	silent: KL 0.007–0.010, far below 1 (m)
F3 stale swizzled scales (SGLang)	missed: layer set correct (c)	no engine query (c)	caught (t)	fires: KL 7.2 (m)
F4 stale expert layout (vLLM)	missed: 2 of 2 layers agree (m)	missed: 69 of 69 parameters agree (m)	caught: 1.739 vs. 0.25 at the first sync (m)	silent: KL 0.047 for 4 steps (m)

Table 12: Which check catches each silent fault that reached a run. m: measured on B200; t: reproduced by a CPU test; c: follows from the check’s construction.

Injected violation	Breaks	Expected to catch it	Evidence now
a class of layers left in BF16 by the sampler	C1	audit, engine re-check	CPU tests; hardware (F2)
new weights with stale derived scales	C3	self-check	CPU tests; hardware (F4)
content permuted, shape and dtype kept	C3	self-check, which reads the output	CPU tests; hardware (F4)
corruption confined to one expert or a late layer	C3	none, by construction	open: size of the blind spot
update written outside the CUDA-graph storage	C4	the refit cycle’s fold; the eager self-check cannot see it	CPU tests of the fold; open: graph replay
update incomplete across buckets	C5	refit-cycle assertions	CPU tests (SGLang loader, BF16 MoE staging); open on hardware

Table 13: Fault-injection protocol: injected violation, the condition it breaks, the check expected to catch it, and the evidence available now.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Time per step of generation (top), the old-log-prob pass (middle) and update_actor (bottom), for the arms and hosts of Figure 4 .

RL Fine-Tuning

Reinforcement learning fine-tuning adapts pretrained models and policies through task rewards, preference feedback, or environment interaction.

See all 232 papers

Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floor
May 19, 2026Xiaocan Li, Shiliang Wu, Zheng ShenLLM Quantization 4-Bit Quantization
Huawei Canada
Towards Full Pipeline FP8 Reinforcement Learning for LLMs
Sep 19, 2026Fanchao Chen, Ziheng Jiang, Ziyun Wei +5RL for Language Models RL Fine-Tuning
University of Wisconsin–Madison · Work done during internship at ByteDance Seed · ByteDance Seed
Training an MoE LLM in MXFP4: How We Closed the Gap to BF16
Oct 23, 2026Han Wang, Zhitao Wang, Natalia Frumkin +4LLM Quantization Low-Precision Training
Advanced Micro Devices, Inc.

May 19, 2026J/K move · Enter open · S save

Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floor

Xiaocan Li, Shiliang Wu, Zheng Shen

Huawei Canada

LLM Quantization 4-Bit Quantization RL Post-Training Post-Training Quantization Neural Network Quantization Quantization Error Compensation

Open paper AskPDF arXiv

MXFP4 arithmetic can dramatically accelerate reinforcement learning (RL) post-training of large language models (LLMs), yet the quantization error introduces severe accuracy degradation. Existing work treats the quantization error as a monolithic noise term, missing the distinct mechanisms upon interpreting how quantization error damages training. We prove an exact three-way decomposition of quantization error and show how each component dominates a distinct RL training pathway. Our theoretical and empirical analysis decomposes the MXFP4 quantization error into three additive components: "scale bias" from power-of-two rounding, "deadzone truncation" from zeroing small values, and "grid noise" from rounding to the nearest 4-bit grid. Each component dominates a distinct RL failure mode: scale bias accumulates multiplicatively through the backward pass, affecting gradient accuracy; deadzone truncation degrades rollout quality; and grid noise raises the policy's entropy. We combine corrections that are RL failure mode-targeted but not component-exclusive: Macro-block scaling to reduce scale bias, Outlier Fallback recovers deadzone entries, but also partially reduces scale bias induced error, and Adaptive Quantization Noise (AQN) for controlling the policy entropy. On Qwen2.5-3B dense and Qwen3-30B-A3B-Base mixture-of-experts model, the targeted corrections recover BF16 accuracy to within 0.7% and exceed BF16 by +1.0% respectively.

Abstract

RL Fine-Tuning

Explore similar work

Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floor

Towards Full Pipeline FP8 Reinforcement Learning for LLMs

Training an MoE LLM in MXFP4: How We Closed the Gap to BF16