Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs

Haozhan Tang; Hao Kang; Han Cai; Song Han; Chenyan Xiong

cs.LGSep 29, 2026

Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs

Authors: Haozhan Tang, Hao Kang, Han Cai, Song Han, Chenyan Xiong

Organizations: Carnegie Mellon University · NVIDIA · NVIDIA & Massachusetts Institute of Technology

Abstract

Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce stale delta and empirically show how it distorts training dynamics. Our stale-delta hybrid runs show a modest loss gap at 569M parameters but substantial loss increases and downstream degradation at 1.67B and 5.29B. QK normalization, NoPE (no positional encoding), and lower-learning-rate context extension mitigate or delay degradation without eliminating it. This pattern suggests accumulated optimization error that smaller models and short runs can conceal. We propose Delta-Matching, proving that it restores the softmax gradient's zero-row-sum invariant under the stated numerical assumptions. It enables native block-scaled FP8 in every forward and backward attention-core matmul without architectural changes, smaller global batches, or auxiliary forward outputs. Across tested architectures, scales, and training stages, Delta-Matching matches BF16/FP32 mixed-precision training loss and overall downstream performance. We will release our implementation, trained models, and data recipes.

Figures & tables

Figure 1: A 1.67B GDN/GQA hybrid trained on 30B Nemotron-CC tokens. Left. Delta-Matching tracks BF16/FP32 training loss; FP8 baselines degrade. Right. Delta-Matching matches BF16/FP32 validation loss and overall downstream performance. Section 4.1 details training and evaluation.

Panel of Figure 2 — Figure 2: (a) DeepSeek-V3 uses FP8 for projections and FFNs in both passes, retaining higher precision for the $O(N^{2})$ attention core ( DeepSeek-AI et al., 2025 ; Ling Team et al., 2025 ) . Sub-8-bit recipes (dashed) mainly target FFNs ( NVIDIA et al., 2025b ; Kimi Team et al., 2026a ) . (b) Operand precision for seven attention-core matmuls (two forward, five backward). Of the two compared methods quantizing all seven to 8 bits, only Delta-Matching matches BF16/FP32 training.

Panel of Figure 3 — Figure 3: Stale delta distorts attention updates and amplifies selected channels. Last-layer data from a 1.67B GDN/GQA hybrid; (a,b) at step 6,000. (a) From shared FP8 logits, stale delta sharpens (left) or flattens (right) attention; Delta-Matching closely follows BF16/FP32 despite different forward logits. (b) Only stale delta combines strong sharpened-row preference with consistent gain increases in highlighted channels. (c) Stale delta selectively amplifies key RMSNorm gains, which stay controlled with Delta-Matching and BF16/FP32. Thick lines track channel 7; thin lines track the others. Appendix B.4 defines quantities and row selection.

Table 1: Comparison with existing FP8 attention training. Mean

\pm

SE over two seeds; downstream scores are percentages. CS9 averages the nine individual benchmark accuracies in (b).

Panel of Figure 4 — Figure 4: Training cross-entropy across GDN/GQA attention variants.

			In-Context Recall $\uparrow$			General Capabilities $\uparrow$
Configuration	Method	Validation CE $\downarrow$	RULER-4K	RULER-8K	Extraction	TriviaQA	MMLU	CS9
NoPE	BF16/FP32 MP	$1.4210\,{\scriptstyle\pm\,0.0000}$	$58.0\,{\scriptstyle\pm\,3.0}$	$51.0\,{\scriptstyle\pm\,2.6}$	$65.4\,{\scriptstyle\pm\,0.8}$	$16.0\,{\scriptstyle\pm\,0.4}$	$35.2\,{\scriptstyle\pm\,0.4}$	$58.5\,{\scriptstyle\pm\,0.1}$
	Naive FP8 (Stale Delta)	$1.6572\,{\scriptstyle\pm\,0.0973}$	$38.2\,{\scriptstyle\pm\,2.7}$	$29.3\,{\scriptstyle\pm\,2.7}$	$53.3\,{\scriptstyle\pm\,2.7}$	$\phantom{0}6.3\,{\scriptstyle\pm\,1.8}$	$28.9\,{\scriptstyle\pm\,1.9}$	$51.8\,{\scriptstyle\pm\,2.7}$
	Delta-Matching ( ours )	$\mathbf{1.4216}\,{\scriptstyle\pm\,0.0021}$	$\mathbf{58.6}\,{\scriptstyle\pm\,1.1}$	$\mathbf{49.7}\,{\scriptstyle\pm\,1.4}$	$\mathbf{64.7}\,{\scriptstyle\pm\,0.1}$	$\mathbf{15.7}\,{\scriptstyle\pm\,0.1}$	$\mathbf{32.7}\,{\scriptstyle\pm\,1.2}$	$\mathbf{58.9}\,{\scriptstyle\pm\,0.2}$
No QK-norm	BF16/FP32 MP	$1.4214\,{\scriptstyle\pm\,0.0012}$	$58.5\,{\scriptstyle\pm\,1.9}$	$46.5\,{\scriptstyle\pm\,5.2}$	$65.7\,{\scriptstyle\pm\,0.0}$	$16.1\,{\scriptstyle\pm\,0.3}$	$35.0\,{\scriptstyle\pm\,0.7}$	$59.1\,{\scriptstyle\pm\,0.2}$
	Naive FP8 (Stale Delta)	Training diverged

Table 2: Attention variants. Mean

\pm

SE over two seeds; downstream scores are percentages. Individual commonsense benchmark accuracies appear in Appendix C.9 .

Panel of Figure 5 — Figure 5: Training cross-entropy for 1.67B GDN/MLA (left) and KDA/GQA (right) hybrids. Delta-Matching tracks BF16/FP32 in both; stale delta develops a substantial loss gap.

			In-Context Recall $\uparrow$			General Capabilities $\uparrow$
Architecture	Method	Validation CE $\downarrow$	RULER-4K	RULER-8K	Extraction	TriviaQA	MMLU	CS9
GDN/MLA	BF16/FP32 MP	$1.4178\,{\scriptstyle\pm\,0.0032}$	$54.9\,{\scriptstyle\pm\,2.2}$	$47.4\,{\scriptstyle\pm\,0.1}$	$66.7\,{\scriptstyle\pm\,0.1}$	$15.7\,{\scriptstyle\pm\,0.3}$	$36.3\,{\scriptstyle\pm\,1.6}$	$59.2\,{\scriptstyle\pm\,0.0}$
	Naive FP8 (Stale Delta)	$1.9268\,{\scriptstyle\pm\,0.0844}$	$25.9\,{\scriptstyle\pm\,2.0}$	$18.9\,{\scriptstyle\pm\,5.3}$	$42.9\,{\scriptstyle\pm\,4.4}$	$\phantom{0}2.0\,{\scriptstyle\pm\,0.7}$	$26.1\,{\scriptstyle\pm\,0.5}$	$46.0\,{\scriptstyle\pm\,1.3}$
	Delta-Matching ( ours )	$\mathbf{1.4181}\,{\scriptstyle\pm\,0.0024}$	$\mathbf{60.4}\,{\scriptstyle\pm\,0.6}$	$\mathbf{50.0}\,{\scriptstyle\pm\,1.7}$	$\mathbf{67.5}\,{\scriptstyle\pm\,0.1}$	$\mathbf{16.3}\,{\scriptstyle\pm\,0.3}$	$\mathbf{34.4}\,{\scriptstyle\pm\,0.5}$	$\mathbf{59.2}\,{\scriptstyle\pm\,0.3}$
KDA/GQA	BF16/FP32 MP	$1.3994\,{\scriptstyle\pm\,0.0004}$	$65.6\,{\scriptstyle\pm\,0.5}$	$50.0\,{\scriptstyle\pm\,2.3}$	$67.9\,{\scriptstyle\pm\,0.4}$	$17.7\,{\scriptstyle\pm\,0.1}$	$34.8\,{\scriptstyle\pm\,0.1}$	$60.2\,{\scriptstyle\pm\,0.3}$
	Naive FP8 (Stale Delta)	$1.8224\,{\scriptstyle\pm\,0.1100}$	$26.2\,{\scriptstyle\pm\,10.9}$	$22.9\,{\scriptstyle\pm\,9.2}$	$44.2\,{\scriptstyle\pm\,11.2}$	$\phantom{0}3.4\,{\scriptstyle\pm\,1.0}$	$26.2\,{\scriptstyle\pm\,0.3}$	$48.5\,{\scriptstyle\pm\,2.2}$

Table 3: Model architectures. Mean

\pm

SE over two seeds; downstream scores are percentages. Individual commonsense benchmark accuracies appear in Appendix C.9 .

Panel of Figure 6 — Figure 6: Training cross-entropy for (a) 569M and (b) 5.29B GDN/GQA hybrids on 30B Nemotron-CC tokens ( Su et al., 2025 ; NVIDIA et al., 2025c ) . Delta-Matching tracks BF16/FP32 at both scales; naive FP8 has a larger loss gap at 5.29B.

			In-Context Recall $\uparrow$			General Capabilities $\uparrow$
Parameters	Method	Validation CE $\downarrow$	RULER-4K	RULER-8K	Extraction	TriviaQA	MMLU	CS9
0.569B	BF16/FP32 MP	$1.5233\,{\scriptstyle\pm\,0.0005}$	$45.4\,{\scriptstyle\pm\,0.5}$	$37.8\,{\scriptstyle\pm\,0.4}$	$65.6\,{\scriptstyle\pm\,0.6}$	$10.2\,{\scriptstyle\pm\,0.4}$	$28.9\,{\scriptstyle\pm\,0.5}$	$54.7\,{\scriptstyle\pm\,0.1}$
	Naive FP8 (Stale Delta)	$1.5399\,{\scriptstyle\pm\,0.0144}$	$\mathbf{48.7}\,{\scriptstyle\pm\,1.2}$	$\mathbf{40.6}\,{\scriptstyle\pm\,1.1}$	$62.3\,{\scriptstyle\pm\,0.9}$	$\phantom{0}9.6\,{\scriptstyle\pm\,0.6}$	$28.5\,{\scriptstyle\pm\,1.3}$	$53.8\,{\scriptstyle\pm\,0.6}$
	Delta-Matching ( ours )	$\mathbf{1.5228}\,{\scriptstyle\pm\,0.0015}$	$46.3\,{\scriptstyle\pm\,1.0}$	$38.2\,{\scriptstyle\pm\,0.8}$	$\mathbf{62.9}\,{\scriptstyle\pm\,0.2}$	$\phantom{0}\mathbf{9.9}\,{\scriptstyle\pm\,0.3}$	$\mathbf{28.7}\,{\scriptstyle\pm\,1.8}$	$\mathbf{54.6}\,{\scriptstyle\pm\,0.0}$
5.289B	BF16/FP32 MP	$1.3349\,{\scriptstyle\pm\,0.0011}$	$65.6\,{\scriptstyle\pm\,0.2}$	$53.5\,{\scriptstyle\pm\,1.8}$	$71.0\,{\scriptstyle\pm\,0.9}$	$21.7\,{\scriptstyle\pm\,0.4}$	$42.0\,{\scriptstyle\pm\,0.5}$	$63.1\,{\scriptstyle\pm\,0.1}$
	Naive FP8 (Stale Delta)	$1.9327\,{\scriptstyle\pm\,0.0349}$	$23.4\,{\scriptstyle\pm\,2.4}$	$16.3\,{\scriptstyle\pm\,5.8}$	$37.1\,{\scriptstyle\pm\,2.3}$	$\phantom{0}2.0\,{\scriptstyle\pm\,0.1}$	$26.3\,{\scriptstyle\pm\,0.2}$	$45.4\,{\scriptstyle\pm\,0.3}$

Table 4: Parameter scales. Mean

\pm

SE for two seeds per model (0.569B, 5.289B); downstream scores in percent. Individual commonsense benchmark accuracies are in Appendix C.9 .

Figure 7: Training cross-entropy during context extension from 8K to 64K tokens for a 1.67B GDN/GQA hybrid. Both cuDNN/Transformer Engine FP8 seeds are shown.

Method	RULER-4K $\uparrow$	RULER-8K $\uparrow$	RULER-16K $\uparrow$	RULER-32K $\uparrow$	RULER-64K $\uparrow$	Extraction $\uparrow$
BF16/FP32 MP	$66.3\,{\scriptstyle\pm\,0.9}$	$59.2\,{\scriptstyle\pm\,3.3}$	$55.1\,{\scriptstyle\pm\,3.7}$	$50.0\,{\scriptstyle\pm\,5.1}$	$46.0\,{\scriptstyle\pm\,4.5}$	$69.0\,{\scriptstyle\pm\,1.6}$
cuDNN/TE FP8 (1st seed)	Training diverged
cuDNN/TE FP8 (2nd seed)	$67.1$	$57.3$	$54.5$	$48.1$	$40.3$	$\mathbf{68.7}$
Naive FP8 (Stale Delta)	$62.8\,{\scriptstyle\pm\,1.7}$	$56.2\,{\scriptstyle\pm\,1.5}$	$50.0\,{\scriptstyle\pm\,2.2}$	$45.8\,{\scriptstyle\pm\,0.8}$	$40.1\,{\scriptstyle\pm\,0.6}$	$64.5\,{\scriptstyle\pm\,0.4}$
Delta-Matching ( ours )	$\mathbf{67.6}\,{\scriptstyle\pm\,4.4}$	$\mathbf{62.2}\,{\scriptstyle\pm\,2.8}$	$\mathbf{55.4}\,{\scriptstyle\pm\,1.1}$	$\mathbf{51.6}\,{\scriptstyle\pm\,0.1}$	$\mathbf{45.6}\,{\scriptstyle\pm\,0.6}$	$67.9\,{\scriptstyle\pm\,0.6}$

Table 5: In-context recall after 8K-to-64K context extension. Mean

\pm

SE over two seeds; cuDNN/TE FP8 seeds appear separately because one run diverged. Scores are percentages.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Params (B)	$L_{r}/L_{a}$	Hidden	FFN	$h_{q}/h_{kv}$	$d$	RoPE fraction
GDN/GQA	0.5693	15/5	1280	3840	6/2	256	$1/4$
GDN/GQA	1.6654	18/6	2048	6144	16/4	128	$1/4$
GDN/GQA	1.6654	18/6	2048	6144	8/2	256	$1/4$
GDN/GQA	5.2886	24/8	3072	9216	16/4	256	$1/4$
GDN/MLA	1.6654	18/6	2048	6144	16/16	128	$1/2$
KDA/GQA	1.6648	18/6	2048	8064	16/4	128	$1/4$

Appendix

Table 6: Model architectures. Parameter counts are in billions, rounded to four decimal places.

L_{r}

and

L_{a}

count recurrent and softmax-attention layers;

h_{q}/h_{kv}

gives the query and key-value head counts, and

d

is the attention head dimension. For MLA, the head counts describe the expanded tensors entering the attention core.

ID	Configuration	Attention core	FFNs	Projections
P0	BF16 reference	BF16	BF16	BF16
P1	FP8 FFNs only	BF16	FP8	BF16
P2	Attention-core-only FP8	FP8	BF16	BF16
P3	FP8 attention and FFNs	FP8	FP8	BF16
P4	FP8 attention, FFNs, and projections	FP8	FP8	FP8

Appendix

Table 7: Training GEMM operand precision. Attention projections denote QKVO for GQA and include the latent and up-projections for MLA. Precision scope is independent of stale, consistent-

dO

, or matched delta construction.

Results	Model or variant	MP	FP8	GPUs
Figure 3	1.67B GDN/GQA, $d=128$	P1	P4	H100
Figure 9	1.67B GDN/GQA, Muon, $d=128$	P1	P4	H100
Table 21	1.67B GDN/GQA, Muon and AdamW, $d=128$	P1	P4	H100
Table 1(a)	1.67B GDN/GQA, $d=128$	P1	P4	H100
Table 2	NoPE	P1	P4	H100
Table 2	No QK-norm	P1	P4	H200

Appendix

Table 8: Configurations behind the result tables and diagnostic figures. P0–P4 are defined in Table 7 . Each run uses eight GPUs of the listed type. The FP8-linear configurations are detailed in Appendix C.3 .

Setting	$D=128$	$D=256$
$Q,dO$ quantization blocks	128 query rows	64 query rows
$K,V$ quantization blocks	64 keys	32 keys
Forward probability groups	64 keys	32 keys
$dS$ quantization tiles	$64\times 128$	$64\times 64$
Forward $PV$ promotion window	256 keys	128 keys
$dV$ promotion window	512 query rows	512 query rows

Appendix

Table 9: Quantization blocks and accumulation windows of our FP8 attention kernels. Input blocks span all channels. Score-gradient tiles are query rows by keys.

BF16/FP32 checkpoint activations
	Before FP8 cast		After FP8 cast
Correction	Layers 3–19	Layer 23	Layers 3–19	Layer 23
Stale delta	$(1.1045\,{\scriptstyle\pm\,0.0075})10^{-2}$	$(4.660\,{\scriptstyle\pm\,0.057})10^{-2}$	$(1.1705\,{\scriptstyle\pm\,0.0085})10^{-2}$	$(4.679\,{\scriptstyle\pm\,0.056})10^{-2}$
Consistent $dO$	$(2.466\,{\scriptstyle\pm\,0.031})10^{-3}$	$(6.103\,{\scriptstyle\pm\,0.047})10^{-3}$	$(3.898\,{\scriptstyle\pm\,0.052})10^{-3}$	$(8.62\,{\scriptstyle\pm\,0.11})10^{-3}$
Delta-Matching	$(1.425\,{\scriptstyle\pm\,0.010})10^{-7}$	$(7.102\,{\scriptstyle\pm\,0.049})10^{-7}$	$(2.700\,{\scriptstyle\pm\,0.041})10^{-3}$	$(3.394\,{\scriptstyle\pm\,0.058})10^{-3}$
Delta-Matching checkpoint activations

Appendix

Table 10: Normalized score-gradient row-sum residuals on the common row set. The post-cast columns use the pre-cast denominator. Values are mean

\pm

one SE over 32 windows of per-layer row medians, with layers 3–19 averaged within each window. Checkpoint headings identify the activation source, not the probe correction. All values are dimensionless fractions.

Head dimension 128, 16 query heads, 4 key-value heads
Kernel	Forward	Backward	Forward + backward
FA3 BF16	$386.9\,{\scriptstyle\pm\,2.9}$ ( $1.00\times$ )	$230.6\,{\scriptstyle\pm\,0.5}$ ( $1.00\times$ )	$260.7\,{\scriptstyle\pm\,0.6}$ ( $1.00\times$ )
cuDNN BF16	$601.6\,{\scriptstyle\pm\,0.7}$ ( $1.55\times$ )	$472.9\,{\scriptstyle\pm\,6.5}$ ( $2.05\times$ )	$503.6\,{\scriptstyle\pm\,5.2}$ ( $1.93\times$ )
cuDNN/TE FP8	$525.4\,{\scriptstyle\pm\,1.4}$ ( $1.36\times$ )	$423.9\,{\scriptstyle\pm\,1.2}$ ( $1.84\times$ )	$448.7\,{\scriptstyle\pm\,1.1}$ ( $1.72\times$ )
Ours, stale delta	$661.0\,{\scriptstyle\pm\,1.7}$ ( $1.71\times$ )	$537.5\,{\scriptstyle\pm\,1.7}$ ( $2.33\times$ )	$567.8\,{\scriptstyle\pm\,1.2}$ ( $2.18\times$ )
Ours, Delta-Matching	$661.0\,{\scriptstyle\pm\,1.7}$ ( $1.71\times$ )	$421.5\,{\scriptstyle\pm\,1.0}$ ( $1.83\times$ )	$470.2\,{\scriptstyle\pm\,0.8}$ ( $1.80\times$ )

Appendix

Table 11: Attention kernel throughput. Values are TFLOP/s, reported as mean

\pm

SE over five timing rounds. Parentheses give ratios to FA3 BF16 in the same column and shape. cuDNN/TE FP8 includes input casting and is unsupported at head dimension 256 in the measured training configuration.

Method	Validation CE $\downarrow$	RULER-4K $\uparrow$	RULER-8K $\uparrow$	Extraction $\uparrow$	TriviaQA $\uparrow$	MMLU $\uparrow$
BF16/FP32 MP	$1.4135\,{\scriptstyle\pm\,0.0003}$	$62.94\,{\scriptstyle\pm\,1.56}$	$52.84\,{\scriptstyle\pm\,2.14}$	$0.6625\,{\scriptstyle\pm\,0.0240}$	$16.41\,{\scriptstyle\pm\,0.21}$	$35.97\,{\scriptstyle\pm\,0.22}$
BF16/FP32 MP + FP8 FFN	$1.4156\,{\scriptstyle\pm\,0.0008}$	$58.22\,{\scriptstyle\pm\,1.64}$	$49.21\,{\scriptstyle\pm\,1.95}$	$0.6751\,{\scriptstyle\pm\,0.0024}$	$16.01\,{\scriptstyle\pm\,0.21}$	$33.90\,{\scriptstyle\pm\,0.11}$
FP8 Attn (Stale Delta) + FP8 FFN	$2.0429\,{\scriptstyle\pm\,0.0166}$	$18.85\,{\scriptstyle\pm\,5.67}$	$14.09\,{\scriptstyle\pm\,5.86}$	$0.3401\,{\scriptstyle\pm\,0.0402}$	$\phantom{0}1.54\,{\scriptstyle\pm\,0.15}$	$26.43\,{\scriptstyle\pm\,0.41}$
Delta-Matching + FP8 FFN	$1.4150\,{\scriptstyle\pm\,0.0001}$	$64.11\,{\scriptstyle\pm\,3.22}$	$51.48\,{\scriptstyle\pm\,2.79}$	$0.6881\,{\scriptstyle\pm\,0.0107}$	$15.45\,{\scriptstyle\pm\,0.21}$	$34.78\,{\scriptstyle\pm\,0.13}$
Delta-Matching + FP8 FFN + FP8 QKVO	$1.4164\,{\scriptstyle\pm\,0.0001}$	$59.54\,{\scriptstyle\pm\,2.12}$	$51.42\,{\scriptstyle\pm\,1.73}$	$0.6781\,{\scriptstyle\pm\,0.0011}$	$15.36\,{\scriptstyle\pm\,0.23}$	$35.60\,{\scriptstyle\pm\,0.03}$

Appendix

Table 12: Delta-Matching composes with FP8 linear layers. Mean

\pm

standard error over two initialization seeds. Extraction scores are fractions; other downstream scores are percentages.

Method	Correction	Validation CE $\downarrow$	$\Delta$ CE vs MP
BF16/FP32 MP	Reference	$1.4779$	$0$
Naive FP8 (Stale Delta)	$\langle dO_{i},O_{i}^{\mathrm{FP8}}\rangle$	$1.6346$	$+0.1567$
Auxiliary BF16-output delta	$\langle dO_{i},O_{i}^{\mathrm{ref}}\rangle$	$1.5755$	$+0.0976$
Delta-Matching	$\delta_{i}^{\mathrm{matched}}$	$1.4781$	$+0.0002$

Appendix

Table 13: Auxiliary BF16-output correction after 14,000 training steps. Validation CE is in nats per token on 32 shared 8,192-token windows, evaluated through the same BF16 attention path. Each method uses one initialization seed. Only means were recorded, so no standard errors are reported.

		In-Context Recall $\uparrow$			General Capabilities $\uparrow$
Method	Validation CE $\downarrow$	RULER-4K	RULER-8K	Extraction	TriviaQA	MMLU	CS9
BF16/FP32 MP	$1.4178\,{\scriptstyle\pm\,0.0004}$	$61.5\,{\scriptstyle\pm\,3.0}$	$52.3\,{\scriptstyle\pm\,2.3}$	$65.7\,{\scriptstyle\pm\,2.0}$	$15.5\,{\scriptstyle\pm\,0.2}$	$35.4\,{\scriptstyle\pm\,1.4}$	$58.8\,{\scriptstyle\pm\,0.2}$
Naive FP8 (Stale Delta)	$1.8970\,{\scriptstyle\pm\,0.0296}$	$24.8\,{\scriptstyle\pm\,0.3}$	$19.8\,{\scriptstyle\pm\,2.1}$	$44.7\,{\scriptstyle\pm\,2.8}$	$\phantom{0}2.7\,{\scriptstyle\pm\,0.1}$	$26.3\,{\scriptstyle\pm\,0.1}$	$47.0\,{\scriptstyle\pm\,0.9}$
SageBwd-style	$1.4175\,{\scriptstyle\pm\,0.0005}$	$57.8\,{\scriptstyle\pm\,3.3}$	$46.0\,{\scriptstyle\pm\,1.1}$	$66.8\,{\scriptstyle\pm\,1.1}$	$15.8\,{\scriptstyle\pm\,0.0}$	$35.0\,{\scriptstyle\pm\,0.9}$	$59.1\,{\scriptstyle\pm\,0.1}$
Delta-Matching	$1.4162\,{\scriptstyle\pm\,0.0006}$	$61.4\,{\scriptstyle\pm\,1.2}$	$47.8\,{\scriptstyle\pm\,0.5}$	$68.0\,{\scriptstyle\pm\,0.5}$	$15.6\,{\scriptstyle\pm\,0.2}$	$34.0\,{\scriptstyle\pm\,0.7}$	$59.0\,{\scriptstyle\pm\,0.2}$

Appendix

Table 14: Protecting

dP

in the base 1.67B GDN/GQA configuration. Results are mean

\pm

one SE over two initialization seeds. Validation CE is in nats per token; downstream scores are percentages. The reference, naive FP8, and Delta-Matching rows reproduce Table 1(a) .

Seed	Layer	Method	$\#(g_{k,c}>4)$	Entropy	$\Delta\mathrm{CE}_{\mathrm{mask}}$
First	19	BF16/FP32 MP	1	$4.08$	$0.1101\,{\scriptstyle\pm\,0.0022}$
		SageBwd-style	64	$4.21$	$0.0987\,{\scriptstyle\pm\,0.0021}$
Second	15	BF16/FP32 MP	0	$4.71$	$0.0647\,{\scriptstyle\pm\,0.0013}$
		SageBwd-style	29	$4.67$	$0.0652\,{\scriptstyle\pm\,0.0014}$

Appendix

Table 15: Gain imbalance without attention collapse in the SageBwd-style runs. Counts cover all 128 channels. Entropy is averaged over 16 held-out windows and query positions 1,024 through 8,191.

\Delta\mathrm{CE}_{\mathrm{mask}}

is the loss increase from zeroing the layer’s attention output relative to each model’s unmasked loss, with paired SE over 500 windows. Entropy is in nats and CE in nats per token. Layer indices are zero-based.

			In-Context Recall $\uparrow$			General Capabilities $\uparrow$
Method	$n$	Validation CE $\downarrow$	RULER-4K	RULER-8K	Extraction	TriviaQA	MMLU	CS9
BF16/FP32 MP	2	$1.41354\,{\scriptstyle\pm\,0.00026}$	$62.94\,{\scriptstyle\pm\,1.56}$	$52.84\,{\scriptstyle\pm\,2.14}$	$66.25\,{\scriptstyle\pm\,2.40}$	$16.41\,{\scriptstyle\pm\,0.21}$	$35.97\,{\scriptstyle\pm\,0.22}$	$59.41\,{\scriptstyle\pm\,0.81}$
FA3-MS (stale)	1	$1.4445$	$58.97$	$51.51$	$67.91$	$13.71$	$32.15$	$57.21$
FA3-MS (externally matched)	1	$1.4349$	$58.49$	$45.05$	$67.06$	$15.09$	N/E	$58.10$
Delta-Matching ( ours )	2	$1.41409\,{\scriptstyle\pm\,0.00019}$	$63.20\,{\scriptstyle\pm\,0.86}$	$52.28\,{\scriptstyle\pm\,1.17}$	$68.33\,{\scriptstyle\pm\,0.92}$	$16.30\,{\scriptstyle\pm\,0.16}$	$33.68\,{\scriptstyle\pm\,0.19}$	$59.35\,{\scriptstyle\pm\,0.07}$

Appendix

Table 16: FA3-style max-shift versus Delta-Matching at head dimension 256. Validation CE is in nats per token; all downstream scores are percentages. CS9 is the unweighted mean accuracy over nine individual commonsense benchmarks. MP and Delta-Matching use the same two initialization seeds and report mean

\pm

one SE. Each FA3-MS row uses one of those seeds. N/E denotes not evaluated.

Panel of Figure 8 — Figure 8: Training cross-entropy for (a) a 1.475B pure GQA LLM and (b) a 1.67B Mamba-2/GQA hybrid, comparing BF16/FP32 mixed precision, naive FP8 with stale delta, and Delta-Matching.

1.67B GDN/GQA
Method	Validation CE	Max. gain product	Top-six share (%)	$\Delta\mathrm{CE}_{\mathrm{mask}}$
BF16/FP32 MP	$1.4174$	$3.1$	$6.9$	$0.0585\,{\scriptstyle\pm\,0.0003}$
Naive FP8 (Stale Delta)	$1.8674$	$2244.4$	$99.9$	$0.1269\,{\scriptstyle\pm\,0.0010}$
Delta-Matching	$1.4169$	$3.7$	$7.8$	$0.0151\,{\scriptstyle\pm\,0.0001}$
1.67B Mamba-2/GQA
BF16/FP32 MP	$1.4144$	$4.4$	$9.6$	$0.0288\,{\scriptstyle\pm\,0.0002}$

Appendix

Table 17: Last-attention-layer probes after 30B-token training, using one initialization seed per method and configuration. Masking zeros the attention output while retaining the residual and FFN.

\Delta\mathrm{CE}_{\mathrm{mask}}

is the resulting CE increase, with paired SEs over 2,000 shared 8,192-token windows evaluated in BF16. SEs are rounded to four decimal places. The maximum gain product is

\max_{c}|g_{q,c}g_{k,c}|

; the top-six share is the percentage of

\sum_{c}(g_{q,c}g_{k,c})^{2}

in the six largest channels. The 1.475B pure-GQA rows use the same configuration as Table 18 .

Method	Validation CE $\downarrow$	RULER-4K $\uparrow$	RULER-8K $\uparrow$	Extraction $\uparrow$	TriviaQA $\uparrow$	MMLU $\uparrow$	CS9 $\uparrow$
BF16/FP32 MP	$1.5283\,{\scriptstyle\pm\,0.0041}$	$49.0\,{\scriptstyle\pm\,0.6}$	$37.2\,{\scriptstyle\pm\,3.1}$	$48.8\,{\scriptstyle\pm\,0.3}$	$10.7\,{\scriptstyle\pm\,0.0}$	$26.1\,{\scriptstyle\pm\,0.1}$	$54.6\,{\scriptstyle\pm\,0.3}$
Naive FP8 (Stale Delta)	$1.5249\,{\scriptstyle\pm\,0.0014}$	$52.1\,{\scriptstyle\pm\,2.7}$	$37.9\,{\scriptstyle\pm\,0.8}$	$48.2\,{\scriptstyle\pm\,0.2}$	$11.1\,{\scriptstyle\pm\,0.0}$	$26.3\,{\scriptstyle\pm\,0.4}$	$54.6\,{\scriptstyle\pm\,0.2}$
Delta-Matching ( ours )	$1.5283\,{\scriptstyle\pm\,0.0029}$	$45.4\,{\scriptstyle\pm\,3.1}$	$36.0\,{\scriptstyle\pm\,1.2}$	$49.4\,{\scriptstyle\pm\,1.6}$	$10.7\,{\scriptstyle\pm\,0.1}$	$27.4\,{\scriptstyle\pm\,1.4}$	$54.8\,{\scriptstyle\pm\,0.2}$

Appendix

Table 18: Evaluation of the 1.475B pure grouped-query attention LLM. Mean

\pm

standard error over two initialization seeds. All scores except validation cross-entropy are percentages.

Method	Validation CE $\downarrow$	RULER-4K $\uparrow$	RULER-8K $\uparrow$	Extraction $\uparrow$	TriviaQA $\uparrow$	MMLU $\uparrow$	CS9 $\uparrow$
BF16/FP32 MP	$1.4152\,{\scriptstyle\pm\,0.0008}$	$56.0\,{\scriptstyle\pm\,1.6}$	$46.5\,{\scriptstyle\pm\,1.2}$	$67.9\,{\scriptstyle\pm\,0.8}$	$15.5\,{\scriptstyle\pm\,0.5}$	$30.5\,{\scriptstyle\pm\,0.0}$	$58.6\,{\scriptstyle\pm\,0.5}$
Naive FP8 (Stale Delta)	$1.4191\,{\scriptstyle\pm\,0.0024}$	$61.8\,{\scriptstyle\pm\,0.1}$	$49.0\,{\scriptstyle\pm\,1.3}$	$68.4\,{\scriptstyle\pm\,0.6}$	$15.5\,{\scriptstyle\pm\,0.2}$	$31.7\,{\scriptstyle\pm\,2.0}$	$58.2\,{\scriptstyle\pm\,0.4}$
Delta-Matching ( ours )	$1.4152\,{\scriptstyle\pm\,0.0004}$	$58.0\,{\scriptstyle\pm\,1.4}$	$48.0\,{\scriptstyle\pm\,0.6}$	$68.8\,{\scriptstyle\pm\,0.6}$	$15.8\,{\scriptstyle\pm\,0.1}$	$31.2\,{\scriptstyle\pm\,1.1}$	$58.4\,{\scriptstyle\pm\,0.6}$

Appendix

Table 19: Evaluation of a Mamba-2/GQA hybrid LLM. Mean

\pm

standard error over two initialization seeds. All scores except validation cross-entropy are percentages.

1.05M tokens per step, 28,610 updates
Method	Validation CE $\downarrow$	RULER-4K $\uparrow$	RULER-8K $\uparrow$	Extraction $\uparrow$	MMLU $\uparrow$	CS9 $\uparrow$
BF16/FP32 MP	$1.41387\,{\scriptstyle\pm\,0.00036}$	$62.38\,{\scriptstyle\pm\,1.06}$	$52.16\,{\scriptstyle\pm\,1.41}$	$67.21\,{\scriptstyle\pm\,1.69}$	$35.86\,{\scriptstyle\pm\,0.17}$	$59.16\,{\scriptstyle\pm\,0.53}$
Naive FP8 (Stale Delta)	$2.04837\,{\scriptstyle\pm\,0.02806}$	$13.47\,{\scriptstyle\pm\,2.02}$	$10.12\,{\scriptstyle\pm\,2.45}$	$31.52\,{\scriptstyle\pm\,2.72}$	$26.04\,{\scriptstyle\pm\,0.14}$	$44.07\,{\scriptstyle\pm\,0.22}$
Delta-Matching	$1.41498\,{\scriptstyle\pm\,0.00088}$	$63.80\,{\scriptstyle\pm\,1.69}$	$55.14\,{\scriptstyle\pm\,2.10}$	$67.59\,{\scriptstyle\pm\,0.86}$	$34.69\,{\scriptstyle\pm\,1.00}$	$58.91\,{\scriptstyle\pm\,0.12}$
2.10M tokens per step, 14,305 updates
Naive FP8 (Stale Delta)	$1.6197$	$46.64$	$36.80$	$59.45$	$26.93$	$52.69$

Appendix

Table 20: Effect of tokens per optimizer step at a fixed 30B-token budget. The 1.05M-token rows report mean

\pm

one SE across three initialization seeds. The 2.10M-token rows use one of those seeds and have no across-seed error bars. Validation CE is in nats per token; other scores are percentages.

Figure 9: Training dynamics with Muon-based updates (solid) and AdamW (dashed) over the first 4,500 optimizer steps. The left panel shows the layer-23 mean query–key gain product

G_{23}

on a logarithmic axis; the right shows training cross-entropy relative to each optimizer’s BF16/FP32 reference on a linear axis. Stale delta produces excess gain growth under both optimizer recipes, while Delta-Matching closely tracks the corresponding reference.

			In-Context Recall $\uparrow$			General Capabilities $\uparrow$
Optimizer	Method	Validation CE $\downarrow$	RULER-4K	RULER-8K	Extraction	TriviaQA	MMLU	CS9
Muon	BF16/FP32 MP	$1.3988$	$62.1$	$53.0$	$68.2$	$16.5$	$36.2$	$59.2$
	Naive FP8 (Stale Delta)	$1.8077$	$32.5$	$26.2$	$47.5$	$\phantom{0}3.3$	$28.2$	$47.2$
	Delta-Matching ( ours )	$1.3985$	$61.2$	$58.7$	$70.6$	$16.8$	$38.1$	$60.4$
AdamW	BF16/FP32 MP	$1.4174$	$64.5$	$54.5$	$67.7$	$15.3$	$34.0$	$59.0$
	Naive FP8 (Stale Delta)	$1.8674$	$25.2$	$21.9$	$47.6$	$\phantom{0}2.9$	$26.4$	$47.8$

Appendix

Table 21: Validation loss and downstream performance across optimizers. Results after 28,610 updates with one shared initialization and data order. Validation CE is in nats per token; downstream scores are percentages. CS9 is the unweighted mean of the nine benchmark accuracies in Table 22 . Single-initialization results carry no standard errors.

Optimizer	Method	LAMBADA	HellaSwag	PIQA	ARC-Easy	SciQ	OpenBookQA	WinoGrande	COPA	ARC-Challenge	Avg
Muon	BF16/FP32 MP	$47.5$	$58.1$	$71.7$	$62.2$	$90.8$	$37.2$	$58.5$	$70.0$	$36.9$	$59.2$
	Naive FP8 (Stale Delta)	$25.1$	$37.8$	$64.6$	$47.7$	$77.7$	$29.8$	$50.0$	$64.0$	$27.8$	$47.2$
	Delta-Matching ( ours )	$48.1$	$58.1$	$73.2$	$65.0$	$90.5$	$37.8$	$60.5$	$72.0$	$38.7$	$60.4$
AdamW	BF16/FP32 MP	$46.1$	$57.3$	$72.1$	$64.6$	$90.1$	$36.4$	$56.7$	$71.0$	$36.8$	$59.0$
	Naive FP8 (Stale Delta)	$25.1$	$36.4$	$64.3$	$46.5$	$78.6$	$31.6$	$52.7$	$68.0$	$27.4$	$47.8$
	Delta-Matching ( ours )	$46.5$	$57.0$	$72.4$	$62.5$	$89.9$	$38.0$	$56.8$	$72.0$	$37.2$	$59.2$

Appendix

Table 22: Commonsense individual benchmark accuracy across optimizers. The same single-initialization runs as in Table 21 . Scores are percentages; Avg is the unweighted mean of the nine benchmarks.

Method	TriviaQA $\uparrow$	MMLU $\uparrow$	CS9 $\uparrow$
BF16/FP32 MP	$\phantom{0}4.7\,{\scriptstyle\pm\,4.6}$	$25.5\,{\scriptstyle\pm\,0.1}$	$58.9\,{\scriptstyle\pm\,0.4}$
cuDNN/TE FP8 (1st seed)	Training diverged
cuDNN/TE FP8 (2nd seed)	$\phantom{0}1.2$	$25.8$	$58.8$
Naive FP8 (Stale Delta)	$13.0\,{\scriptstyle\pm\,1.0}$	$25.7\,{\scriptstyle\pm\,0.3}$	$57.4\,{\scriptstyle\pm\,0.3}$
Delta-Matching ( ours )	$\phantom{0}2.9\,{\scriptstyle\pm\,1.6}$	$25.5\,{\scriptstyle\pm\,0.1}$	$59.9\,{\scriptstyle\pm\,0.0}$

Appendix

Table 23: General capabilities after context extension from 8K to 64K tokens. Results for the 1.67B GDN/GQA hybrid in Section 4.6 . cuDNN/TE FP8 is reported separately for the first and second seeds; other rows report mean

\pm

standard error over two initialization seeds. All scores are percentages. Individual commonsense benchmark accuracies appear in Table 27 .

Low-Precision Training

See all 23 papers

May 25, 2026

Max-Window Scale Estimation for Near-Lossless HiF8 W8A8 Quantization-Aware Training

Quantization-aware training (QAT) with low-bit floating-point formats enables efficient LLM deployment, yet introduces subtle failure modes invisible to standard training metrics. We present a systematic study of HiF8 W8A8 QAT for OpenPangu-Embedded-1B through the lens of Delayed Tensor Scaling (DTS). Across eight controlled experiments, we identify and disentangle two orthogonal failure modes: (i)amax saturation, where delayed scale estimates silently corrupt knowledge-sensitive representations via forward-pass clipping, and (ii)catastrophic forgetting, where an aggressive learning rate overwrites pretrained commonsense knowledge independently of quantization. Neither is detectable from training loss alone. We address amax saturation with a conservative max-algorithm DTS strategy over a 64-step history window, and mitigate forgetting via a 500-step BF16 warmup followed by QAT at lr=10^{-5}. Both fixes are necessary and sufficient: our final configuration achieves 0.43% MMLU drop, 0.58% HellaSwag drop, and 0.22% ARC-Challenge drop versus a matched BF16 baseline, with a training loss APE of only 0.11% over 10,000 steps.

Yingying Cheng, Jinquan Shi, Li Zhou +4

LLM Quantization Quantization-Aware Training Activation Quantization Catastrophic Forgetting

Ask PDF

Jul 27, 2026

Stable FP4 Training via Transposition-Invariant Block Quantization

Reducing training precision is a key lever for improving the e ciency of large language model (LLM) training, but pushing beyond FP8 to 4-bit oating point (FP4) remains challenging due to instability during optimization. We identify a fundamental source of this instability in existing microscaling approaches: scale inconsistency induced by tensor transposition. In conventional 1D block quantization, forward and backward passes assign di erent scaling factors to the same values after transposition, leading to biased and unstable gradient updates. To address this issue, we propose a low-precision training framework based on 2D block FP4 quantization, which enforces transposition-invariant scaling and preserves consistency between forward and backward computations. We further combine this with truncation-free scaling and stochastic rounding to control quantization error and maintain unbiased gradients. To handle the sensitivity of attention mechanisms, we adopt MXFP8 quantization for query and key projections, yielding a practical mixed-precision design. We evaluate our method on dense LLMs up to 7B parameters and a 30B Mixture-of-Experts model, trained on up to 100B tokens. Across all settings, our approach achieves stable end-to-end FP4 training and closely matches BF16 performance, with less than 1.3% degradation in perplexity and downstream accuracy. These results demonstrate that enforcing forwardbackward scaling consistency is su cient to enable practical FP4 training at scale, providing a simple and e ective pathway toward more e cient LLM training.

Mehdi Rahimifar, Amin Darabi, Mehran Taghian Jazi +6

Mixed-Precision Quantization 4-Bit Quantization LLM Training

Ask PDF

May 11, 2026

Pretraining large language models with MXFP4 on Native FP4 Hardware

Why does full-pipeline FP4 training of large language models often diverge, even when forward activations and activation gradients remain stable? We address this question through a controlled study of MXFP4 quantization in transformer training, progressively enabling FP4 across forward propagation (Fprop), activation gradients (Dgrad), and weight gradients (Wgrad) while holding all other factors fixed. In full pretraining of Llama 3.1-8B on the C4 dataset, we observe that quantizing Wgrad is the primary driver of convergence degradation, whereas FP4 in Fprop and Dgrad alone introduces only modest additional token requirements. To interpret this behavior, we evaluate both structured and stochastic interventions under a controlled experimental setting. We find that stochastic rounding and randomized Hadamard rotations fail to stabilize training once Wgrad is quantized, whereas deterministic Hadamard rotations consistently restore stable optimization. These results suggest that FP4 training instability is driven by structured micro-scaling errors along sensitive gradient paths, rather than by insufficient stochasticity. We run experiments with native MXFP4 support on AMD Instinct MI355X GPUs, enabling controlled investigation of these effects without reliance on software emulation.

Musa Cim, Poovaiah Palangappa, Miro Hodak +3

1The Pennsylvania State University · 2Advanced Micro Devices, Inc.

Language Model Pretraining LLM Quantization Walsh-Hadamard Transform

Ask PDF

Abstract

Low-Precision Training

Explore similar work

Max-Window Scale Estimation for Near-Lossless HiF8 W8A8 Quantization-Aware Training

Stable FP4 Training via Transposition-Invariant Block Quantization

Pretraining large language models with MXFP4 on Native FP4 Hardware