cs.LGNov 16, 2026

CertiSum: Bitwise-Identical MXFP4 Training Across Platforms and Parallelism Layouts

Authors: Steve Rhyner, Yubo Gao, Stefan de Lasa, Sankeerth Durvasula, Nandita Vijaykumar

Organizations: University of Toronto · Vector Institute · NVIDIA

Abstract

Training with MXFP4, a 4-bit microscaling format, is non-deterministic: identical inputs can produce different results across parallelism layouts and GPU vendors. These differences complicate debugging and training audits and can cause training and inference engines to disagree. We present CertiSum to make MXFP4 training bitwise invariant across runs, parallelism layouts, and GPU vendors. Its key insight is that native FP32 accumulation of MXFP4 products is often already exact for every summation order. CertiSum collects statistics during quantization to certify native matrix multiplication results, recomputes uncertified outputs exactly, and uses exact kernels directly where certification rarely succeeds. CertiSum keeps quantization blocks unchanged across layouts and merges partial sums without rounding, including in attention and weight-gradient computation. In 200-step Qwen3-8B-Base experiments, CertiSum preserves training states bit for bit across AMD MI350X and NVIDIA GB200 and across changes in data, tensor, and context parallelism on MI350X. Across three 8B--14B models on FineWeb-Edu using MI350X, it adds 11.8--60.5% to training-step latency relative to its non-exact MXFP4 counterpart at the larger evaluated input shapes, with speedups of up to 1.24×1.24\times over BF16 and 1.70×1.70\times over deterministic BF16.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Pretraining large language models with MXFP4 on Native FP4 Hardware

    May 11, 2026Musa Cim, Poovaiah Palangappa, Miro Hodak +3Language Model PretrainingLLM Quantization

  2. Integer-Exact Microscaling: Running MXFP4 Language Models on INT8 Matrix Units

    Oct 31, 2026Shinyoung Kang, Junhyung Ahn, Yujun Kim +1LLM QuantizationNeural Network Quantization

  3. Training an MoE LLM in MXFP4: How We Closed the Gap to BF16

    Oct 23, 2026Han Wang, Zhitao Wang, Natalia Frumkin +4LLM QuantizationLow-Precision Training