Training with MXFP4, a 4-bit microscaling format, is non-deterministic: identical inputs can produce different results across parallelism layouts and GPU vendors. These differences complicate debugging and training audits and can cause training and inference engines to disagree. We present CertiSum to make MXFP4 training bitwise invariant across runs, parallelism layouts, and GPU vendors. Its key insight is that native FP32 accumulation of MXFP4 products is often already exact for every summation order. CertiSum collects statistics during quantization to certify native matrix multiplication results, recomputes uncertified outputs exactly, and uses exact kernels directly where certification rarely succeeds. CertiSum keeps quantization blocks unchanged across layouts and merges partial sums without rounding, including in attention and weight-gradient computation. In 200-step Qwen3-8B-Base experiments, CertiSum preserves training states bit for bit across AMD MI350X and NVIDIA GB200 and across changes in data, tensor, and context parallelism on MI350X. Across three 8B--14B models on FineWeb-Edu using MI350X, it adds 11.8--60.5% to training-step latency relative to its non-exact MXFP4 counterpart at the larger evaluated input shapes, with speedups of up to 1.24× over BF16 and 1.70× over deterministic BF16.
Figures & tables
Figure 1: Qwen3-8B-Base Yang et al. (2025) on FineWeb-Edu Penedo et al. (2024) , trained on 4 GPUs instead of 2, all else identical. (a) Differing greedy next-token predictions before training. (b) Relative training-loss difference (centered 10-step mean). Repeating the run of each format on two GPUs reproduces the result bit for bit; CertiSum (§ 3 ) differs in neither predictions nor loss.
Run-to-
Parallelization
Hard-
Order-
Back-
Overhead
Precision
run
Batch
DP
TP
CP
ware
free
ward
(reported)
(evaluated)
Order- fixing
Megatron Core Aithal et al. (2026)
✓
✗
✗
✗
✗
✗
✗
✓
2%
FP8, FP4
Batch-inv. kernels He and Thinking Machines Lab (2025)
✓
✓
–
–
–
–
✗
✗
1.6–2.1 ×
BF16
TBIK Zhang et al. (2026)
✓
✓
\LEFTcircle
✓
–
–
✗
–
22–63%
BF16
IsoExec Jiang and SkyRL Team (2026)
\LEFTcircle
✓
\LEFTcircle
✓
✗
–
✗
–
25%
n/r
EasyScale Li et al. (2023)
✓
✗
✓
–
–
\LEFTcircle a
✗
✓
< 1% b
n/r
Table 1: Comparison with prior work. A ✓ means guaranteed or supported, \LEFTcircle partial or argued only, ✗ not provided or out of scope, – not addressed, and n/r not reported. Overhead is latency relative to non-deterministic execution. a NVIDIA GPUs only. b 236% across GPU types (CNNs). c Claimed for any device count; no replay on a different device count is reported. d CPU only. e Latency overhead of 1.2-1.6 × argued from previous work. f GEMM latency overhead Iakymchuk et al. (2016) . g Attention only. h Targets accuracy, not reproducibility.
Figure 2: Upper bound on the significand bits needed for exact MXFP4 accumulation in any order, for GEMM outputs over 200 steps of training Qwen3-8B-Base on FineWeb-Edu. The bound uses the sum of absolute products; the shaded panel magnifies the tail (log scale).
Figure 3: CertiSum GEMM execution; steps ❶–❻ described in § 4 .
Figure 4: Certificates from row (❶), block (❷), and absolute-product (❸) statistics, in increasing cost and tightness.
CertiSum; its native MXFP4 GEMM path uses AITER Advanced Micro Devices (2026a) on MI350X. At the smaller input shapes, the faster measured result of ordinary (eager) execution and a static-route variant that replays recorded GPU operations.
T
MXFP4 twin: CertiSum’s structure with non-exact MXFP4 GEMMs and non-exact versions of our attention kernels.
F
BF16: the faster measured result of our BF16 implementation and torchtitan Liang et al. (2025) .
D
F in PyTorch deterministic mode PyTorch Contributors (2026) ; deterministic torchtitan uses SDPA attention.
Vendor
Native MXFP4 GEMMs through hipBLASLt Advanced Micro Devices (2026b) with our pinned Tensile library on MI350X and cuBLASLt NVIDIA (2026a) on GB200, with BF16 FlashAttention on dequantized MXFP4 inputs (AITER on MI350X, FlashAttention-4 Zadouri et al. (2026) on GB200).
Table 3: Compared implementations.
Figure 5: Cross-Platform Invariance (MI350X vs. GB200). (a.1), (a.2): FineWeb-Edu training loss on one GPU for the vendor baseline (Table 3 ) and CertiSum. (a)–(d): relative cross-vendor loss difference on a log scale; exact zeros below the axis break; triangles mark differences below 0.001% .
Figure 6: Elastic Training. Relative loss difference from the uninterrupted one-GPU reference on (a) FineWeb-Edu and (b) The Stack v3. Dots are steps; lines are a centered nine-step geometric mean of positive values; zeros are below the axis break.
Figure 7: Training-step latency on FineWeb-Edu using AMD MI350X, normalized to the MXFP4 twin (dashed line); lower is better. BF16 bars use the faster measured implementation per mode (Table 3 ). Shapes are sequences × tokens, global except DP4 (per GPU). Superscript s : static-route CertiSum; OOM: all implementations ran out of memory.
Figure 8: Traced step time on four AMD MI350X GPUs (DP4, 4×4096 per GPU) on (a) FineWeb-Edu and (b) The Stack v3; bars are the MXFP4 twin, CertiSum, and BF16. Each bar is the slowest GPU, averaged over steps 3–28 of one run. Compute segments are scaled from GPU-averaged kernel sums; crosshatching marks certification, repair, and exact GEMMs; communication excludes overlap; unfilled segments are host/launch gaps and logging.
Method
Batch 4, 1 GPU
Batch 4, TP4
CertiSum
32 /32
32 /32
MXFP4 twin
31/32
0/32
Deterministic BF16
29/32
25/32
Table 4: Generated sequences (out of 32) identical to each method’s batch-1, one-GPU reference.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Main shapes
Qwen3-8B-Base
Qwen3-14B-Base
Llama-3.1-8B
Layout
B×S
E (ms)
E/T
E/F
D (ms)
E/D
E (ms)
E/T
E/F
D (ms)
E/D
E (ms)
E/T
E/F
D (ms)
E/D
1 GPU
4×4096
823.1
1.57
0.88
1112.5
0.74
OOM
763.7
1.49
0.83
1079.0
0.71
1 GPU
1×16384
1042.7
1.46
0.88
1715.7
0.61
OOM
931.9
1.41
0.81
1587.4
0.59
TP2
4×4096
816.4
1.20
0.90
976.9
0.84
1345.4
1.40
0.95
1512.5
0.89
749.2
1.23
0.90
895.9
0.84
TP4
1×16384
562.3
1.12
0.88
825.1
0.68
992.3
1.38
1.00
1204.5
0.82
495.7
1.12
0.86
744.4
0.67
CP2
1×16384
911.1
1.27
0.94
1137.5
0.80
1625.9
1.54
0.86
2058.1
0.79
823.9
1.24
0.90
1105.4
0.75
Appendix
Table 5: Training-step latency on AMD MI350X with FineWeb-Edu inputs. Implementations E , T , F , and D as in Table 3 . Ratios below one favor CertiSum. B×S : sequences × tokens, global except DP4 (per GPU). s Static-route E ; unmarked E and all T are eager. OOM: all compared implementations ran out of memory before completing the first step.
Assigned work (%)
Forward (%)
Repair
Dataset
Native
Exact
Int.
Native
Repair
Work (%)
Qwen3-8B-Base, 1 GPU, 4×4096
FineWeb-Edu
36.92
63.08
0.0000
88.60
0.0229
0.0069
The Stack v3
36.98
62.97
0.0451
88.89
0.0186
0.0072
Qwen3-8B-Base, TP2, 4×4096
FineWeb-Edu
37.19
62.81
0.0000
89.77
0.0558
0.0123
Appendix
Table 6: Routing and recorded repairs on AMD MI350X. Shapes are global B×S . Assigned work splits nominal step GEMM FLOPs among native, direct exact, and integer-engine (Int.) routes, the last including padding. Forward columns give native candidates as a share of transformer linear-layer outputs and recorded repairs as a share of those. Repair work is a share of nominal GEMM FLOPs. Values are rounded independently.
Why does full-pipeline FP4 training of large language models often diverge, even when forward activations and activation gradients remain stable? We address this question through a controlled study of MXFP4 quantization in transformer training, progressively enabling FP4 across forward propagation (Fprop), activation gradients (Dgrad), and weight gradients (Wgrad) while holding all other factors fixed. In full pretraining of Llama 3.1-8B on the C4 dataset, we observe that quantizing Wgrad is the primary driver of convergence degradation, whereas FP4 in Fprop and Dgrad alone introduces only modest additional token requirements. To interpret this behavior, we evaluate both structured and stochastic interventions under a controlled experimental setting. We find that stochastic rounding and randomized Hadamard rotations fail to stabilize training once Wgrad is quantized, whereas deterministic Hadamard rotations consistently restore stable optimization. These results suggest that FP4 training instability is driven by structured micro-scaling errors along sensitive gradient paths, rather than by insufficient stochasticity. We run experiments with native MXFP4 support on AMD Instinct MI355X GPUs, enabling controlled investigation of these effects without reliance on software emulation.
Musa Cim, Poovaiah Palangappa, Miro Hodak +3
1The Pennsylvania State University · 2Advanced Micro Devices, Inc.
On hardware whose fastest matrix datapath is INT8×INT8, such as the Neural Accelerators of Apple's M5 GPUs, 4-bit weights must become INT8 operands. Existing MXFP4 kernels already map FP4 (E2M1) values to integers by doubling them, but apply every 32-element block scale inside the reduction. We show when no per-block work is needed: a row of MXFP4 weights is exactly one INT8 vector times one power-of-two scale if its block exponents span at most three, so a linear layer becomes a single INT8 GEMM with one rescale per output. Enforcing the condition adds at most 1/16 of a row's largest rounding error to the blocks it changes, and real weights already satisfy it: 97.9-99.97% of rows across seven models, including 99.93% of the 95M rows of a 125B-parameter mixture-of-experts (MoE) model, Qwen3.8-Flash-Next. The resulting spread-constrained MXFP4 files stay standard MXFP4. With integer instead of power-of-two block multipliers, an integer-exact format (16-wide blocks, multipliers 1..10, 4.26 bits) runs as one exact INT8 GEMM and is more accurate than NVFP4 (4.5 bits) with its standard scale rule on five small models, within 0.06 perplexity of NVFP4 with scale search. On Qwen3.8-Flash-Next's routed experts with INT8 activations it is as close to the FP8 model as NVFP4 with scale search (KL divergence 0.0557 vs. 0.0553) and closer than MLX's MXFP4 and affine INT4; with every matrix quantized it solves 94.8% of GSM8K vs. 91.0% for MLX's MXFP4. Quantizing only the 74k routed expert matrices (96% of the parameters) to either of our formats with INT8 activations costs at most 0.2% perplexity. A complete W4A8 system on the M5 produces outputs bit-identical to the quantized reference; prefill is 1.34-1.74x faster than MLX on a dense model, 1.10-1.57x on two MoE models up to 35B parameters and 1.22-1.29x on Qwen3.8-Flash-Next, where shared non-GEMM work bounds the gain.
Shinyoung Kang, Junhyung Ahn, Yujun Kim +1
UT Southwestern Medical Center · Department of Data and Decision Sciences, Emory University · Fuqua School of Business, Duke University +1
Training large language models (LLMs) in sub-8-bit arithmetic promises substantial gains in memory bandwidth and compute throughput, yet FP4 formats introduce quantization error severe enough to destabilize optimization if left unaddressed. We present ALTO, an open-source training recipe for GPT-OSS-20B, a 20-billion-parameter mixture-of-experts (MoE) model, on the MLPerf Small MoE Training benchmark under the MXFP4 microscaling format. The recipe combines established techniques from Abecassis, Felix et al. [arXiv:2509.25149] (hybrid 1D/2D block quantization, randomized Hadamard transforms, and stochastic rounding) with a simplified weight de-oscillation scheme adapted from TetraJet-v2 [arXiv:2510.27527]. On C4 validation at global batch size 16, MXFP4 + RHT + SR + de-oscillation reaches a validation loss of 3.3350 at 16,128 steps, only 0.007 above the BF16 baseline (3.3283), closing roughly 50% of the gap left by MXFP4 without de-oscillation (3.3418). Measured by steps to reach the MLPerf quality target (validation loss 3.34), de-oscillation cuts the FP4 convergence overhead relative to BF16 from +20.0% to +5.0%. We also report negative end-to-end results from differential gradient estimation, outlier clipping, and macro-block scaling, despite each showing operator-level SNR gains. All experiments use simulated MXFP4 kernels on AMD MI300X hardware and wall-clock speed has not been optimized.