Training with MXFP4, a 4-bit microscaling format, is non-deterministic: identical inputs can produce different results across parallelism layouts and GPU vendors. These differences complicate debugging and training audits and can cause training and inference engines to disagree. We present CertiSum to make MXFP4 training bitwise invariant across runs, parallelism layouts, and GPU vendors. Its key insight is that native FP32 accumulation of MXFP4 products is often already exact for every summation order. CertiSum collects statistics during quantization to certify native matrix multiplication results, recomputes uncertified outputs exactly, and uses exact kernels directly where certification rarely succeeds. CertiSum keeps quantization blocks unchanged across layouts and merges partial sums without rounding, including in attention and weight-gradient computation. In 200-step Qwen3-8B-Base experiments, CertiSum preserves training states bit for bit across AMD MI350X and NVIDIA GB200 and across changes in data, tensor, and context parallelism on MI350X. Across three 8B--14B models on FineWeb-Edu using MI350X, it adds 11.8--60.5% to training-step latency relative to its non-exact MXFP4 counterpart at the larger evaluated input shapes, with speedups of up to 1.24× over BF16 and 1.70× over deterministic BF16.
Figures & tables
Figure 1: Qwen3-8B-Base Yang et al. (2025) on FineWeb-Edu Penedo et al. (2024) , trained on 4 GPUs instead of 2, all else identical. (a) Differing greedy next-token predictions before training. (b) Relative training-loss difference (centered 10-step mean). Repeating the run of each format on two GPUs reproduces the result bit for bit; CertiSum (§ 3 ) differs in neither predictions nor loss.
Run-to-
Parallelization
Hard-
Order-
Back-
Overhead
Precision
run
Batch
DP
TP
CP
ware
free
ward
(reported)
(evaluated)
Order- fixing
Megatron Core Aithal et al. (2026)
✓
✗
✗
✗
✗
✗
✗
✓
2%
FP8, FP4
Batch-inv. kernels He and Thinking Machines Lab (2025)
✓
✓
–
–
–
–
✗
✗
1.6–2.1 ×
BF16
TBIK Zhang et al. (2026)
✓
✓
\LEFTcircle
✓
–
–
✗
–
22–63%
BF16
IsoExec Jiang and SkyRL Team (2026)
\LEFTcircle
✓
\LEFTcircle
✓
✗
–
✗
–
25%
n/r
EasyScale Li et al. (2023)
✓
✗
✓
–
–
\LEFTcircle a
✗
✓
< 1% b
n/r
Table 1: Comparison with prior work. A ✓ means guaranteed or supported, \LEFTcircle partial or argued only, ✗ not provided or out of scope, – not addressed, and n/r not reported. Overhead is latency relative to non-deterministic execution. a NVIDIA GPUs only. b 236% across GPU types (CNNs). c Claimed for any device count; no replay on a different device count is reported. d CPU only. e Latency overhead of 1.2-1.6 × argued from previous work. f GEMM latency overhead Iakymchuk et al. (2016) . g Attention only. h Targets accuracy, not reproducibility.
Figure 2: Upper bound on the significand bits needed for exact MXFP4 accumulation in any order, for GEMM outputs over 200 steps of training Qwen3-8B-Base on FineWeb-Edu. The bound uses the sum of absolute products; the shaded panel magnifies the tail (log scale).
Figure 3: CertiSum GEMM execution; steps ❶–❻ described in § 4 .
Figure 4: Certificates from row (❶), block (❷), and absolute-product (❸) statistics, in increasing cost and tightness.
CertiSum; its native MXFP4 GEMM path uses AITER Advanced Micro Devices (2026a) on MI350X. At the smaller input shapes, the faster measured result of ordinary (eager) execution and a static-route variant that replays recorded GPU operations.
T
MXFP4 twin: CertiSum’s structure with non-exact MXFP4 GEMMs and non-exact versions of our attention kernels.
F
BF16: the faster measured result of our BF16 implementation and torchtitan Liang et al. (2025) .
D
F in PyTorch deterministic mode PyTorch Contributors (2026) ; deterministic torchtitan uses SDPA attention.
Vendor
Native MXFP4 GEMMs through hipBLASLt Advanced Micro Devices (2026b) with our pinned Tensile library on MI350X and cuBLASLt NVIDIA (2026a) on GB200, with BF16 FlashAttention on dequantized MXFP4 inputs (AITER on MI350X, FlashAttention-4 Zadouri et al. (2026) on GB200).
Table 3: Compared implementations.
Figure 5: Cross-Platform Invariance (MI350X vs. GB200). (a.1), (a.2): FineWeb-Edu training loss on one GPU for the vendor baseline (Table 3 ) and CertiSum. (a)–(d): relative cross-vendor loss difference on a log scale; exact zeros below the axis break; triangles mark differences below 0.001% .
Figure 6: Elastic Training. Relative loss difference from the uninterrupted one-GPU reference on (a) FineWeb-Edu and (b) The Stack v3. Dots are steps; lines are a centered nine-step geometric mean of positive values; zeros are below the axis break.
Figure 7: Training-step latency on FineWeb-Edu using AMD MI350X, normalized to the MXFP4 twin (dashed line); lower is better. BF16 bars use the faster measured implementation per mode (Table 3 ). Shapes are sequences × tokens, global except DP4 (per GPU). Superscript s : static-route CertiSum; OOM: all implementations ran out of memory.
Figure 8: Traced step time on four AMD MI350X GPUs (DP4, 4×4096 per GPU) on (a) FineWeb-Edu and (b) The Stack v3; bars are the MXFP4 twin, CertiSum, and BF16. Each bar is the slowest GPU, averaged over steps 3–28 of one run. Compute segments are scaled from GPU-averaged kernel sums; crosshatching marks certification, repair, and exact GEMMs; communication excludes overlap; unfilled segments are host/launch gaps and logging.
Method
Batch 4, 1 GPU
Batch 4, TP4
CertiSum
32 /32
32 /32
MXFP4 twin
31/32
0/32
Deterministic BF16
29/32
25/32
Table 4: Generated sequences (out of 32) identical to each method’s batch-1, one-GPU reference.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Main shapes
Qwen3-8B-Base
Qwen3-14B-Base
Llama-3.1-8B
Layout
B×S
E (ms)
E/T
E/F
D (ms)
E/D
E (ms)
E/T
E/F
D (ms)
E/D
E (ms)
E/T
E/F
D (ms)
E/D
1 GPU
4×4096
823.1
1.57
0.88
1112.5
0.74
OOM
763.7
1.49
0.83
1079.0
0.71
1 GPU
1×16384
1042.7
1.46
0.88
1715.7
0.61
OOM
931.9
1.41
0.81
1587.4
0.59
TP2
4×4096
816.4
1.20
0.90
976.9
0.84
1345.4
1.40
0.95
1512.5
0.89
749.2
1.23
0.90
895.9
0.84
TP4
1×16384
562.3
1.12
0.88
825.1
0.68
992.3
1.38
1.00
1204.5
0.82
495.7
1.12
0.86
744.4
0.67
CP2
1×16384
911.1
1.27
0.94
1137.5
0.80
1625.9
1.54
0.86
2058.1
0.79
823.9
1.24
0.90
1105.4
0.75
Appendix
Table 5: Training-step latency on AMD MI350X with FineWeb-Edu inputs. Implementations E , T , F , and D as in Table 3 . Ratios below one favor CertiSum. B×S : sequences × tokens, global except DP4 (per GPU). s Static-route E ; unmarked E and all T are eager. OOM: all compared implementations ran out of memory before completing the first step.
Assigned work (%)
Forward (%)
Repair
Dataset
Native
Exact
Int.
Native
Repair
Work (%)
Qwen3-8B-Base, 1 GPU, 4×4096
FineWeb-Edu
36.92
63.08
0.0000
88.60
0.0229
0.0069
The Stack v3
36.98
62.97
0.0451
88.89
0.0186
0.0072
Qwen3-8B-Base, TP2, 4×4096
FineWeb-Edu
37.19
62.81
0.0000
89.77
0.0558
0.0123
Appendix
Table 6: Routing and recorded repairs on AMD MI350X. Shapes are global B×S . Assigned work splits nominal step GEMM FLOPs among native, direct exact, and integer-engine (Int.) routes, the last including padding. Forward columns give native candidates as a share of transformer linear-layer outputs and recorded repairs as a share of those. Repair work is a share of nominal GEMM FLOPs. Values are rounded independently.