Bit-Accurate Modeling of GPU Matrix Multiply-Accumulate Units: Demystifying Numerical Discrepancy and Accuracy

Peichen Xie; Shuotao Xu; Yang Wang; Fan Yang; Mao Yang

cs.ARNov 14, 2025

Bit-Accurate Modeling of GPU Matrix Multiply-Accumulate Units: Demystifying Numerical Discrepancy and Accuracy

Authors: Peichen Xie, Shuotao Xu, Yang Wang, Fan Yang, Mao Yang

Abstract

Modern AI accelerators rely on matrix multiply-accumulate units (MMAUs), such as NVIDIA Tensor Cores and AMD Matrix Cores, to accelerate deep neural network workloads. MMAUs expose only instruction-level or API-level interfaces of matrix multiply-accumulate (MMA) operations, while leaving internal floating-point arithmetic behavior undocumented. Consequently, MMAUs across vendors and architectural generations often produce numerical discrepancies for identical inputs, and sometimes exhibit reduced numerical accuracy that can cause training instability. Diagnosing and understanding the root causes of these effects is challenging without white-box models of their arithmetic behavior. This paper proposes closed-loop feature probing (CLFP), a generic and systematic framework for constructing bit-accurate arithmetic behavior models of MMA operations. Based on this framework, we analyze all MMA instructions on ten GPU architectures spanning NVIDIA Volta through RTX Blackwell and AMD CDNA1 through CDNA3, and derive the first bit-accurate arithmetic models for these MMAUs. Our models explain previously observed cross-platform numerical discrepancies and accuracy issues, enable white-box numerical error analysis, reveal four types of precision bottlenecks and one type of numerical asymmetry, and inform software workarounds as well as design suggestions for future MMAUs. This work is open-source at https://github.com/microsoft/MMA-Sim

Figures & tables

Fig. 1: The closed-loop feature probing (CLFP) framework for modeling the arithmetic behavior of the MMA operation. The loop of Steps 4 (verification) and 5 (adding new tests and revising the model) ensures completeness and bit-accuracy.

Panel of Figure 2 — Fig. 2: Examples of summation trees (left) and corresponding values of $d^{(i,j)}/v$ (right), where $(i,j)$ denotes the input with $p_{i}=U$ , $p_{j}=-U$ , and all other summands set to $v$ .

	GPU	MMA-Sim	Human	Total
Step 1	5 minutes	0	0	5 minutes
Step 2	$<$ 1 minute	0	0	$<$ 1 minute
Step 3	$<$ 1 minute	0	$<$ 1 hour	$<$ 1 hour
Step 4	5 minutes	1 hour*	0	1 hour
Step 5	$<$ 1 minute	$<$ 1 minute	4 hours	4 hours
Total	10-20 minutes	1-3 hours	1-9 hours	2-12 hours

TABLE I: Approximate time of running CLFP with 0-2 revision loops (typical). *MMA-Sim execution time has been reduced from 10-20 hours to 1 hour through optimization.

Category	Models	Elementary Ops.
MulAdd-based	$\Phi_{\text{FTZ-MulAdd}}$	FTZ-Add, FTZ-Mul
FMA-based	$\Phi_{\text{FMA}}$	FMA
FDPA-based	$\Phi_{\text{E-FDPA}}$ , $\Phi_{\text{T-FDPA}}$ , $\Phi_{\text{ST-FDPA}}$ , $\Phi_{\text{GST-FDPA}}$ , $\Phi_{\text{TR-FDPA}}$ , $\Phi_{\text{GTR-FDPA}}$	E-FDPA, T-FDPA, ST-FDPA, GST-FDPA, TR-FDPA, GTR-FDPA

TABLE II: Categories of our bit-accurate models for GPU matrix multiply-accumulate units.

Architecture	Instruction	Model	Input Type	Output Type	$L_{\text{max}}$	$F$	$\rho$
Volta	HMMA	$\Phi_{\text{T-FDPA}}$	FP16	FP32	4	23	RZ FP32
Volta	HMMA	$\Phi_{\text{T-FDPA}}$	FP16	FP16	4	23	RNE FP16
Turing	HMMA	$\Phi_{\text{T-FDPA}}$	FP16	FP32	8	24	RZ FP32
Turing	HMMA	$\Phi_{\text{T-FDPA}}$	FP16	FP16	8	24	RNE FP16
Ampere	DMMA	$\Phi_{\text{FMA}}$	FP64	FP64	1	N/A	Configurable
	HMMA	$\Phi_{\text{T-FDPA}}$	TF32	FP32	4	24	RZ FP32

TABLE III: Models and parameters for all NVIDIA and AMD GPU MMA instructions. In addition,

G=16

for all instructions modeled by

\Phi_{\text{GST-FDPA}}

Architecture	TF32/BF16 Instr.	FP16 Instr.	FP8 Instr.
Volta	N/A	$0.0$	N/A
Turing	N/A	$-0.5$	N/A
Ampere	$-0.5$	$-0.5$	N/A
Ada Lovelace	$-0.5$	$-0.5$	$0.0$
Hopper	$-0.75$	$-0.75$	$0.0$
Blackwell	$-0.75$	$-0.75$	$-0.75$

TABLE IV: The divergent results of different MMA instructions for the same input in Equation 10 . In addition, all FP64/FP32 instructions produce

d_{0,0}=-0.875

Elementary Op.	Error Source	Error Bound
FTZ-Add/Mul	Input FTZ	$2^{-126}$ or $2^{-14}$
	Add/Mul	$0.5\text{ ulp}$
	Output FTZ	$2^{-126}$
FMA, E-FDPA	Output rounding	$0.5\text{ ulp}$ or $1\text{ ulp}$
T-FDPA, others	Fused summation	$(L+1)2^{e_{\text{max}}-F}$
T-FDPA, others	Output rounding	$0.5\text{ ulp}$ or $1\text{ ulp}$

TABLE V: Sources and upper bounds of numerical error.

Architectures and Instructions	Concerns
AMD CDNA2, FP16-input	Input FTZ
NVIDIA Ada Lovelace and Hopper, FP8-input	Small $F$
NVIDIA Ada Lovelace and Hopper, FP8-input	$\rho=\text{RZ}_{\text{E8M13}}$
All NVIDIA architectures, FP16-output	$\rho=\text{RNE}_{\text{FP16}}$
AMD CDNA3, TF32/BF16/FP16/FP8-input	Asymmetry

TABLE VI: Concerns regarding numerical precision (top) and accuracy (bottom).

Fig. 3: Distributions of $\delta_{\text{RD}}$ , the numerical deviation of the AMD CDNA3 FP16 MMA instruction, which uses the “round down” (RD) mode internally, and $\delta_{\text{RZ}}$ , the numerical deviation of a hypothetical variant that replaces the internal RD operations with “round toward zero” (RZ) operations.

Explore similar work

Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures
Sep 22, 2026Liam Cooper, Shinnung Jeong, Hyeran Jeon +2LLM Inference Optimization Floating-Point
Georgia Institute of Technology · University of California, Merced
Rigel: Reverse-Engineering the Metal 4.1 Tensor Compute Path on the Apple M4 Max GPU
Jun 11, 2026Ramchand KumaresanMatrix Multiplication Shape-Memory Alloy
Murai Labs
Measuring and Reducing Cross-Vendor Mismatch in Language Models
Oct 4, 2026Erland Hilman Fuadi, Chong Tian, Xiaosong Ma +1Cross-Architecture Generalization Training-Inference Mismatch
Mohamed bin Zayed University of Artificial Intelligence

Sep 22, 2026, cs.ARJ/K move · Enter open · S save

Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures

Liam Cooper, Shinnung Jeong, Hyeran Jeon, Jeffrey Young+1

Georgia Institute of Technology · University of California, Merced

LLM Inference Optimization Floating-Point Matrix Multiplication Graphics Processing Unit Architectures Reproducibility

Open paper AskPDF arXiv

Large language model (LLM) outputs are expected to be reproducible under greedy decoding, yet in practice the same model, prompt, and software stack produce different outputs on different GPUs. The root cause is floating-point non-associativity combined with hardware-dependent kernel selection. Inference frameworks select different matrix-multiplication kernels on each architecture, with different parallel reduction orders and unspecified tensor-core arithmetic, and the resulting rounding differences can flip output tokens. Existing solutions have imperfect cross-architecture reproducibility and incur a significant performance penalty. We present a solution employing a set of fixed-configuration fused-upcast GEMM kernels that load 16-bit weights from memory, upcast them to FP32 in registers, and accumulate with IEEE-754 arithmetic in a reduction order that is a pure function of the problem shape and is therefore independent of the device, its SM count, or kernel scheduling. By fixing the floating-point reduction order as a function of problem shape alone, every GPU runs the same operation sequence, so cross-architecture reproducibility of the linear layers reduces to correct IEEE-754 arithmetic rather than to rounding differences staying below a tie-flip threshold. We confirm our solution's linear-layer outputs are bitwise identical across NVIDIA Ampere, Ada, and Hopper GPUs, while running

1.17

3.1\times

faster end-to-end than the state-of-the-art solution and cutting weight-memory traffic in half.

Abstract

Explore similar work

Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures

Rigel: Reverse-Engineering the Metal 4.1 Tensor Compute Path on the Apple M4 Max GPU

Measuring and Reducing Cross-Vendor Mismatch in Language Models