HiFloat4 Format for Language Model Pre-training on Ascend NPUs

Mehran Taghian; Yunke Peng; Xing Huang; Yao Wang; Yaoyuan Wang; Wei Guo; Yuanyong Luo; Tianchi Hu

cs.LGApr 9, 2026

HiFloat4 Format for Language Model Pre-training on Ascend NPUs

Authors: Mehran Taghian, Yunke Peng, Xing Huang, Yao Wang, Yaoyuan Wang, Wei Guo, Yuanyong Luo, Tianchi Hu,

Organizations: Huawei

Abstract

Training large foundation models at low numerical precision is one of the most promising directions for reducing the compute and memory cost of modern AI. Recent 4-bit floating-point formats such as MXFP4 and NVFP4 can be applied to linear GEMM operations in LLMs, but their limited dynamic range introduces numerical instability that prior work addresses by stacking stabilization mechanisms, typically executed at higher precision and partially eroding the efficiency gains that motivate FP4. In this work, we argue that numerical format design is itself a first-class lever for stable FP4 training, and present the first systematic study of FP4 LLM pretraining on energy-efficient Huawei Ascend NPUs. We compare the recently proposed HiFloat4 (HiF4) format against both MXFP4 and NVFP4, holding one recipe fixed across all three formats across dense (OpenPangu-1B, Llama3-8B) and Mixture-of-Experts (Qwen3-MoE-30B) architectures and executing all linear and expert GEMMs in FP4. At matched storage --- NVFP4 and HiF4 both spend 4.5 bits per value --- the three formats differ far more in what they require before they will train at all than in final accuracy. HiF4 reaches a relative loss of 1.55% with no stabilization, below fully stabilized MXFP4 (1.79%) and below NVFP4 carrying the per-tensor scaling it cannot train without (2.00%); NVFP4 diverges under every combination of stochastic rounding and Hadamard transform we tried. Our results suggest that stable, accurate FP4 training does not require an ever-growing stack of stabilization techniques; it requires the right numerical format.

Figures & tables

Panel of Figure 1 — Figure 1: FP4 format architectures. (a) MXFP4: 8 bits of shared metadata (E8M0) over 32 4-bit elements, 4.25 bits per value. (b) NVFP4: 8 bits (E4M3) over 16 elements, 4.5 bits per value, plus an FP32 per-tensor scale. (c) HiF4: 32 bits of three-level metadata over 64 elements, also 4.5 bits per value. NVFP4 and HiF4 are therefore storage-matched; MXFP4 is cheaper but coarser.

	Forward	Backward
Method	$\scriptstyle Y=XW^{\top}$	$\scriptstyle dX=DW$	$\scriptstyle dW=D^{\top}X$
NVIDIA ( Abecassis et al., 2025 )	–	–	$\scriptstyle(N+K)M\log r$
Quartet II ( Panferov et al., 2026 )	–	$\scriptstyle(M+K)N\log r$	$\scriptstyle(N+K)M\log r$
Metis ( Cao et al., 2025 )	$\scriptstyle Mk(K+N)$	$\scriptstyle MNj+jk(M+N+K)$	$\scriptstyle Kj(M+N)+jk(M+N+K)$

Table 1: Extra compute beyond FP4 GeMM and quantization for methods that apply FP4 in both forward/backward during pretraining.

X\in\mathbb{R}^{M\times K}

W\in\mathbb{R}^{N\times K}

, and

D\in\mathbb{R}^{M\times N}

r

is the RHT block size;

k,j

are Metis’s forward/backward randomized SVD ranks. Quartet II’s 4/6 forward scale selection adds a

\Theta(MK+NK)

constant-factor overhead to forward quantization, excluded here. “–” indicates no additional compute beyond baseline GeMM and quantization. The table isolates stabilization cost and excludes the baseline FP4 GEMM and quantization, which every method pays; Quartet II’s 4/6 selection is a constant-factor change to double quantization, which belongs to that baseline path, so it falls inside the exclusion as a scoping decision. See Appendix D .

Figure 2: Overview of the GEMM operation within a linear or expert layer. Activations and weights are quantized to HiF4, MXFP4 or NVFP4 using NR; gradients use SR for MXFP4 and NVFP4 and NR for HiF4. An RHT is applied to the weight gradients ( $\mathrm{d}W$ ) prior to quantization. The flow is identical for all three formats; only the quantizer and its stabilizers change.

	Attention		FFN		Experts
Model	QKV	Output	FC1	FC2	Up	Down	Total
OpenPangu-1B	10.61%	5.30%	42.45%	21.22%	–	–	79.58%
Llama3-8B	8.86%	5.91%	41.37%	20.68%	–	–	76.82 %
Qwen3-MoE-30B	3.13%	2.50%	–	–	60.18%	30.09%	95.90%

Table 2: Percentage of compute executed in FP4, broken down by component

Figure 3: Training loss curves for OpenPangu-1B (a), Llama3-8B (b) and Qwen3-MoE-30B (c) over 50B tokens, each FP4 scheme against BF16. Top: HiF4+RHT. Middle: MXFP4+TF+SR+RHT. Bottom: NVFP4+TS+SR+RHT. OpenPangu-1B is averaged across 4 seeds; insets show the late-training region where the gap is most visible. HiF4 tracks BF16 at least as closely as MXFP4 and NVFP4 on all three models despite using fewer stabilizers.

Model	Activate Parameters	MXFP4	NVFP4	HiFloat4
OpenPangu-1B	1B	1.75%	1.26%	1.25 %
Llama3-8B	8B	1.44%	0.99%	0.85 %
Qwen3-MoE-30B	4.6B	1.55%	0.99%	0.88 %

Table 3: Relative loss error with respect to the BF16 baseline across three models trained on 50B tokens, ordered by active parameter count. Each format uses its own best configuration (RHT for HiF4; TF+SR+RHT for MXFP4; PTS+SR+RHT for NVFP4, see Section 5.3 ). OpenPangu-1B is averaged across 4 seeds; the larger models are single runs without error bars.

Model	Quant	ARC-E	ARC-C	HellaS	PIQA	SciQ	BoolQ	WinoG	Mean	PPL
OpenPangu-1B	BF16	47.47	26.42	35.12	65.23	80.97	60.94	51.70	52.55	28.87
	MXFP4	-1.53	-0.66	-1.11	-0.56	-2.13	-3.39	-1.79	-1.59	+1.03
	NVFP4	-1.65	-0.40	-0.75	+0.11	-0.43	-6.19	-0.58	-1.41	+0.69
	HiF4	+0.28	-0.52	-0.88	-1.18	-2.47	-0.75	+0.53	-0.71	+0.71
Llama3-8B	BF16	67.32	36.89	59.03	74.43	93.47	57.80	58.01	63.85	15.19
	MXFP4	-1.08	-2.78	-1.79	-0.47	+0.07	+4.04	-2.97	-0.71	+0.60

Table 4: Comparison of HiF4, NVFP4, and MXFP4 against the BF16 baseline across three pretrained models. BF16 rows report absolute 5-shot accuracy (%), averaged over three seeds, and WikiText perplexity (PPL; lower is better, single run). FP4 rows report the change relative to the BF16 row of the same model, with red indicating degradation and blue improvement; absolute FP4 values are recovered by adding the two. Mean is computed over the seven accuracy benchmarks only. The best FP4 result per model is shown in bold. HiF4 yields a smaller mean accuracy gap and lower perplexity increase than MXFP4 across all three models; NVFP4 attains the lowest perplexity of the three FP4 formats but degrades sharply on BoolQ.

Figure 4: Ablation of FP4 stabilizers on (a) OpenPangu-1B and (b) Qwen-MoE-30B, both trained for 50B tokens. Stabilizers are SR, RHT, TF, and PTS; TF applies only to MXFP4 and PTS only to NVFP4. Bars show loss relative to the BF16 baseline; lower is better. NVFP4 diverges without PTS. The Qwen-MoE-30B runs cover HiF4 and NVFP4 only, testing whether the 1B ordering holds at scale. Best configuration per format is outlined and in bold.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Configuration	OpenPangu-1B	Llama3-8B	Qwen3-MoE-30B
Model & Data
Training Tokens	50B	50B	50B
Sequence Length	4K	4K	4K
Optimization
Optimizer	AdamW	AdamW	AdamW
Start Learning Rate	$10^{-4}$	$10^{-4}$	$10^{-4}$

Appendix

Table 5: Training configurations for all experiments.

Low-Precision Training

See all 24 papers

HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models
Jul 29, 2026Hei Yi Mak, Shadan Golestan, Hoang Le +10LLM Quantization 4-Bit Quantization
Huawei
What Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes
Oct 17, 2026Anjulie Agrusa, Andrei Panferov, Elizabeth Wei +7Low-Precision Training Language Model Pretraining
NVIDIA · Institute of Science and Technology Austria · Red Hat AI
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Jul 5, 2026Siyu Ding, Mingchuan Ma, Jiabo Tong +3Language Model Pretraining Mixed-Precision Quantization
Institute of Automation, Chinese Academy of Sciences · Sichuan University · Beijing Zhongguancun Academy +3

Jul 29, 2026J/K move · Enter open · S save

HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models

Hei Yi Mak, Shadan Golestan, Hoang Le, Mehran Taghian Jazi+9

Huawei

LLM Quantization 4-Bit Quantization Activation Quantization RL Post-Training

Open paper AskPDF arXiv

We present, to our knowledge, the first end-to-end FP4 RL post-training, in which both the rollout and training policies, including their forward and backward passes, operate at 4-bit precision. A systematic study reveals that the dominant source of degradation in FP4 RL is not training-side quantization error but rollout activation quantization: outliers stretch the dynamic range so far that a large number of activation values underflow to zero under FP4. Counterintuitively, restoring the training policy to higher precision while keeping the rollout in FP4 makes accuracy worse than full FP4 baseline, exposing rollout-training mismatch as the principal failure mode and ruling out standard pretraining-style fixes. We address this with Rollout Residual Quantization (Rollout-ResQ): a single residual correction term constrained to a hardware-friendly sparsity pattern, added only to the FP4 rollout matmul -- a lightweight correction that recovers most of the precision lost to outlier-driven underflow without inflating the rollout's compute footprint. On Qwen2.5-3B and Qwen2.5-Math-7B, Rollout-ResQ paired with the HiFloat4 (HiF4) format -- whose three-level hierarchical scaling preserves resolution under FP4's tight 4-bit budget -- closes the accuracy gap to BF16 from 4.9% to 1.1%, bringing fully quantized FP4 RL within striking distance of full precision. Applied to the open-standard MXFP4, the same recipe narrows the gap from 13.6% to 5.3%, revealing that FP4 format choice is a key factor that determines the ceiling on recoverable accuracy. Together, these results establish HiF4 as the enabling format for end-to-end FP4 RL post-training, and Rollout-ResQ as the activation-side mechanism that makes the gap to BF16 closable.

Abstract

Low-Precision Training

Explore similar work

HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models

What Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes

Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention