Calibration of Large Model Quantization based on HiFloat4 Format
Organizations: Huawei
Abstract
Quantization of Large Language Models (LLMs) has become increasingly important for reducing the computational costs and resource requirements for practical deployment. Growing interest in this area has motivated the community to move beyond 8-bit quantization towards 4-bit precision, leading to the development of customized 4-bit formats such as MXFP4 and NVFP4. Recently, HiFloat4 (HiF4) has emerged as a competitive alternative which adopts a Block Floating-Point (BFP) format with a three-level scaling hierarchy, thereby achieving a significantly larger dynamic range. While quantization can substantially reduce memory usage and accelerate inference, it frequently introduces accuracy degradation. Calibration is therefore essential for narrowing the performance gap between low-precision quantized models and their full-precision counterparts. Nevertheless, existing calibration methods are mainly designed and validated using integer-precision formats, such as INT4, lacking support for HiF4 due to its hierarchical BFP structure. In this paper, we investigate three calibration approaches, namely HiGPTQ, HiAutoRound, and HiFlatQuant, to further enhance model quantization based on the HiF4 format. Extensive experiment s demonstrate that all three proposed approaches improve inference accuracy compared with direct HiF4 conversion, reducing the gap between HiF4 and BF16 to less than 1% and outperforming the state-of-the-art NVFP4 format across various models and downstream tasks.
Figures & tables
| Model | Quantization | Datasets | Average | ||||||
|---|---|---|---|---|---|---|---|---|---|
| ARC-C | ARC-E | BoolQ | HellaS | LamOpai | Piqa | WinoG | |||
| Llama2-7B | BF16 | 46.25 | 74.62 | 77.92 | 75.99 | 73.63 | 78.84 | 68.82 | 70.87 |
| MXFP4 | 40.36 | 66.12 | 72.14 | 71.15 | 69.34 | 77.31 | 63.54 | 65.71 (-5.16) | |
| NVFP4 | 42.75 | 72.90 | 75.35 | 74.59 | 73.47 | 78.07 | 68.75 | 69.41 (-1.46) | |
| HiF4 | 44.03 | 73.44 | 76.76 | 74.94 | 72.91 | 78.35 | 67.56 | 69.71 (-1.16) | |
| HiGPTQ | 44.88 | 73.74 | 76.97 | 75.31 | 73.18 | 78.35 | 68.59 | 70.15 (-0.72) | |
| Metric | BF16 | MXFP4 | NVFP4 | HiF4 | HiGPTQ | HiAutoRound | HiFlatQuant |
|---|---|---|---|---|---|---|---|
| Accuracy | 73.970 | 69.452 | 72.220 | 72.804 | 73.708 | 73.420 | 73.522 |
| Accuracy gap | – | -4.518 | -1.750 | -1.166 | -0.262 | -0.550 | -0.448 |
| Dimension | HiGPTQ | HiAutoRound | HiFlatQuant |
|---|---|---|---|
| Mechanism | Compensate quantization error based on Hessian inverse | Learn offset parameters in micro-exponent and mantissa components | Learn equivalent transformations to flatten outliers |
| Optimization | Second-order approximation | Gradient-based parameter tuning | Gradient-based transformation learning |
| Quantization target | Weight | Weight | Weight and activation |
| Inference overhead | None | None | Depends on transformation folding |
LLM Quantization
Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.
See all 434 papers