SoScale: Second-Order Scale Search for FP4 Quantization
Organizations: Huawei Technologies Co., Ltd.
Abstract
Low-bit quantization reduces the memory footprint and inference cost of large language models (LLMs), yet accurate FP4 quantization remains challenging. Practical formats such as MXFP4, NVFP4, and HiF4 pair a 4-bit floating-point payload with block-wise or hierarchical scale codes, making accuracy highly sensitive to how each local distribution is scaled. Default maximum-absolute-value scaling preserves dynamic range, but can spend scarce FP4 levels on rarely occupied outlier regions and leave dense tensor values poorly resolved. Existing scale-search methods minimize local element-wise reconstruction error, optimizing tensor fidelity rather than layer-output fidelity. We propose SoScale, a second-order weight scale-code search method for hierarchical FP4 quantization. SoScale searches hardware-valid scale codes under a Hessian-aware layer reconstruction objective and couples this search with GPTQ-style error recovery, selecting scales by their impact on layer outputs. Across Qwen3 models, SoScale consistently reduces downstream quantization degradation under MXFP4, NVFP4, and HiF4.
Figures & tables
| MXFP4 | NVFP4 | HiF4 | |
|---|---|---|---|
| Element format | E2M1 | E2M1 | E1M2 |
| Scale hierarchy | E8M0 | FP32 + E4M3 | E6M2 + E1 + E1 |
| Granularity | 32 | 16 + tensor | 64 / 8 / 4 |
| Bits / element | 4.25 | 4.50 | 4.50 |
| Model | Bit | Format | Method | GSM8K | HellaSwag | MMLU | ARC-C | ARC-E | BoolQ | PIQA | Avg. | Rel. |
| Qwen3-8B | – | BF16 | – | 0.8810 | 0.5713 | 0.7292 | 0.5538 | 0.8371 | 0.8663 | 0.7628 | 0.7431 | 1.0000 |
| W4A4 | MXFP4 | RTN | 0.8211 | 0.5292 | 0.6718 | 0.4923 | 0.7735 | 0.8425 | 0.7492 | 0.6971 | 0.9381 | |
| GPTQ | 0.8362 | 0.5329 | 0.6862 | 0.5068 | 0.7971 | 0.8520 | 0.7514 | 0.7089 | 0.9540 | |||
| BlockRotation | 0.7839 | 0.5329 | 0.6772 | 0.5077 | 0.7921 | 0.8440 | 0.7421 | 0.6971 | 0.9381 | |||
| ScaleSearch | 0.8317 | 0.5303 | 0.6802 | 0.5171 | 0.7837 | 0.8505 | 0.7459 | 0.7056 | 0.9496 | |||
| SoScale | 0.8575 | 0.5334 | 0.6924 | 0.5000 | 0.7875 | 0.8517 | 0.7421 | 0.7092 | 0.9545 |
| Model | Bit | Format | Method | GSM8K | HellaSwag | MMLU | ARC-C | ARC-E | BoolQ | PIQA | Avg. | Rel. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | W4A4 | MXFP4 | ScaleSearch-w | 0.8241 | 0.5273 | 0.6738 | 0.5009 | 0.7774 | 0.8492 | 0.7492 | 0.7003 | 0.9424 |
| SoScale-w | 0.8317 | 0.5292 | 0.6856 | 0.5000 | 0.7997 | 0.8508 | 0.7405 | 0.7054 | 0.9492 | |||
| SoScale-w w/ Had. | 0.8461 | 0.5438 | 0.6966 | 0.5205 | 0.8064 | 0.8635 | 0.7601 | 0.7195 | 0.9683 | |||
| NVFP4 | ScaleSearch-w | 0.8605 | 0.5533 | 0.7124 | 0.5341 | 0.8114 | 0.8575 | 0.7655 | 0.7278 | 0.9795 | ||
| SoScale-w | 0.8726 | 0.5542 | 0.7082 | 0.5384 | 0.8157 | 0.8602 | 0.7459 | 0.7279 | 0.9796 | |||
| SoScale-w w/ Had. | 0.8560 | 0.5541 | 0.7121 | 0.5410 | 0.8136 | 0.8657 | 0.7579 | 0.7286 | 0.9806 |
| Format | Variant | Loss | Rec. | Avg. | Rel. |
|---|---|---|---|---|---|
| MXFP4 | RTN | N/A | No | 0.6971 | 0.9381 |
| ScaleSearch | MSE | No | 0.7056 | 0.9496 | |
| MSE + recovery | MSE | ✓ | 0.7061 | 0.9503 | |
| SoScale | Hessian | ✓ | 0.7092 | 0.9545 | |
| NVFP4 | RTN | N/A | No | 0.7262 | 0.9773 |
| ScaleSearch | MSE | No | 0.7281 | 0.9798 |
LLM Quantization
Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.
See all 434 papers