Quantization of Large Language Models (LLMs) has become increasingly important for reducing the computational costs and resource requirements for practical deployment. Growing interest in this area has motivated the community to move beyond 8-bit quantization towards 4-bit precision, leading to the development of customized 4-bit formats such as MXFP4 and NVFP4. Recently, HiFloat4 (HiF4) has emerged as a competitive alternative which adopts a Block Floating-Point (BFP) format with a three-level scaling hierarchy, thereby achieving a significantly larger dynamic range. While quantization can substantially reduce memory usage and accelerate inference, it frequently introduces accuracy degradation. Calibration is therefore essential for narrowing the performance gap between low-precision quantized models and their full-precision counterparts. Nevertheless, existing calibration methods are mainly designed and validated using integer-precision formats, such as INT4, lacking support for HiF4 due to its hierarchical BFP structure. In this paper, we investigate three calibration approaches, namely HiGPTQ, HiAutoRound, and HiFlatQuant, to further enhance model quantization based on the HiF4 format. Extensive experiment s demonstrate that all three proposed approaches improve inference accuracy compared with direct HiF4 conversion, reducing the gap between HiF4 and BF16 to less than 1% and outperforming the state-of-the-art NVFP4 format across various models and downstream tasks.
Figures & tables
Figure 1: HiFloat4 Block Floating-Point (BFP) data structure.
Figure 2: Illustration of the proposed calibration algorithms. (a) HiGPTQ : columns in white represent the group currently being quantized; (b) HiAutoRound : learnable parameters are introduced to automatically adjust the micro-exponent and mantissa components, which are optimized via gradient descent; (c) HiFlatQuant : learnable transformation matrices P1 , P2 and per-channel scaling c are introduced to flatten outliers in weights and activations, which are optimized via gradient descent.
Model
Quantization
Datasets
Average
ARC-C
ARC-E
BoolQ
HellaS
LamOpai
Piqa
WinoG
Llama2-7B
BF16
46.25
74.62
77.92
75.99
73.63
78.84
68.82
70.87
MXFP4
40.36
66.12
72.14
71.15
69.34
77.31
63.54
65.71 (-5.16)
NVFP4
42.75
72.90
75.35
74.59
73.47
78.07
68.75
69.41 (-1.46)
HiF4
44.03
73.44
76.76
74.94
72.91
78.35
67.56
69.71 (-1.16)
HiGPTQ
44.88
73.74
76.97
75.31
73.18
78.35
68.59
70.15 (-0.72)
Table 1: Experimental results on diverse models and tasks.
Metric
BF16
MXFP4
NVFP4
HiF4
HiGPTQ
HiAutoRound
HiFlatQuant
Accuracy
73.970
69.452
72.220
72.804
73.708
73.420
73.522
Accuracy gap
–
-4.518
-1.750
-1.166
-0.262
-0.550
-0.448
Table 2: Average performances across models and tasks.
Dimension
HiGPTQ
HiAutoRound
HiFlatQuant
Mechanism
Compensate quantization error based on Hessian inverse
Learn offset parameters in micro-exponent and mantissa components
Learn equivalent transformations to flatten outliers
Optimization
Second-order approximation
Gradient-based parameter tuning
Gradient-based transformation learning
Quantization target
Weight
Weight
Weight and activation
Inference overhead
None
None
Depends on transformation folding
Table 3: Comparison of three calibration approaches
We present, to our knowledge, the first end-to-end FP4 RL post-training, in which both the rollout and training policies, including their forward and backward passes, operate at 4-bit precision. A systematic study reveals that the dominant source of degradation in FP4 RL is not training-side quantization error but rollout activation quantization: outliers stretch the dynamic range so far that a large number of activation values underflow to zero under FP4. Counterintuitively, restoring the training policy to higher precision while keeping the rollout in FP4 makes accuracy worse than full FP4 baseline, exposing rollout--training mismatch as the principal failure mode and ruling out standard pretraining-style fixes. We address this with Rollout Residual Quantization (Rollout-ResQ): a single residual correction term constrained to a hardware-friendly sparsity pattern, added only to the FP4 rollout matmul -- a lightweight correction that recovers most of the precision lost to outlier-driven underflow without inflating the rollout's compute footprint. On Qwen2.5-3B, Rollout-ResQ paired with the HiFloat4 (HiF4) format -- whose three-level hierarchical scaling preserves resolution under FP4's tight 4-bit budget -- closes the accuracy gap to BF16 from 3.59% to 1.39%, bringing fully quantized FP4 RL within close reach of full precision. Applied to the open-standard MXFP4, the same recipe narrows the gap from 13.6% to 5.3%, revealing that FP4 format choice is a key factor that determines the ceiling on recoverable accuracy. The experimental results on Qwen2.5-Math-7B further confirm the recovery benefits of Rollout-ResQ with HiF4. Together, these results establish HiF4 as the enabling format for end-to-end FP4 RL post-training, and Rollout-ResQ as the activation-side mechanism that makes the gap to BF16 closable.
Training large foundation models at low numerical precision is one of the most promising directions for reducing the compute and memory cost of modern AI. Recent 4-bit floating-point formats such as MXFP4 and NVFP4 can be applied to linear GEMM operations in LLMs, but their limited dynamic range introduces numerical instability that prior work addresses by stacking stabilization mechanisms, typically executed at higher precision and partially eroding the efficiency gains that motivate FP4. In this work, we argue that numerical format design is itself a first-class lever for stable FP4 training, and present the first systematic study of FP4 LLM pretraining on energy-efficient Huawei Ascend NPUs. We compare the recently proposed HiFloat4 (HiF4) format against both MXFP4 and NVFP4, holding one recipe fixed across all three formats across dense (OpenPangu-1B, Llama3-8B) and Mixture-of-Experts (Qwen3-MoE-30B) architectures and executing all linear and expert GEMMs in FP4. At matched storage --- NVFP4 and HiF4 both spend 4.5 bits per value --- the three formats differ far more in what they require before they will train at all than in final accuracy. HiF4 reaches a relative loss of 1.55% with no stabilization, below fully stabilized MXFP4 (1.79%) and below NVFP4 carrying the per-tensor scaling it cannot train without (2.00%); NVFP4 diverges under every combination of stochastic rounding and Hadamard transform we tried. Our results suggest that stable, accurate FP4 training does not require an ever-growing stack of stabilization techniques; it requires the right numerical format.
Quantization addresses the high resource demand for large language models (LLMs) by alleviating memory pressure and bandwidth congestion and providing significantly scaled compute power with a tolerable impact on accuracy. Four-bit floating point (FP4), the lowest-precision format that preserves essential numerical properties such as exponent and sign, has begun to be adopted in cutting-edge architectures, including Blackwell and AMD CDNA, to support LLM quantization and reduce deployment costs. Although aggressive quantization can yield efficiency gains, the quantization sensitivity of within-transformer layers and whether these sensitivities generalize across existing FP4 formats and model scales remain underexplored. To elucidate quantization sensitivity, this study conducts a systematic analysis of two FP4 formats, MXFP4 and NVFP4, across three Qwen2.5 model scales (0.5B, 7B, and 14B), using controlled component-wise and block-wise isolation methodologies. We observe that MLP up- and down-projection layers consistently dominate in terms of sensitivity, while gate and attention projections are moderately and substantially less sensitive to FP4 quantization, respectively. We further find that sensitivity does not universally localize to the final blocks, but early blocks can be highly sensitive, particularly under MXFP4. Our results provide a diagnostic characterization of the inference behavior of FP4 across components, depths, and FP4 formats.
Musa Cim, Burak Topcu, Mahmut Taylan Kandemir
Department of Computer Science and Engineering The Pennsylvania State University University Park, PA, USA