Low-bit quantization reduces the memory footprint and inference cost of large language models (LLMs), yet accurate FP4 quantization remains challenging. Practical formats such as MXFP4, NVFP4, and HiF4 pair a 4-bit floating-point payload with block-wise or hierarchical scale codes, making accuracy highly sensitive to how each local distribution is scaled. Default maximum-absolute-value scaling preserves dynamic range, but can spend scarce FP4 levels on rarely occupied outlier regions and leave dense tensor values poorly resolved. Existing scale-search methods minimize local element-wise reconstruction error, optimizing tensor fidelity rather than layer-output fidelity. We propose SoScale, a second-order weight scale-code search method for hierarchical FP4 quantization. SoScale searches hardware-valid scale codes under a Hessian-aware layer reconstruction objective and couples this search with GPTQ-style error recovery, selecting scales by their impact on layer outputs. Across Qwen3 models, SoScale consistently reduces downstream quantization degradation under MXFP4, NVFP4, and HiF4.
Figures & tables
Figure 1: Scale-code alignment and model-level impact. On a real Qwen3-8B weight block, scale search changes the MXFP4 scale from 0.125 to 0.0625 . (a) Default scaling preserves a wider range but leaves extreme levels unused, while the searched scale concentrates levels near dense tensor values. (b) The searched scale reduces element-wise reconstruction error for most elements. (c) Local element-wise scale search does not consistently reduce perplexity, whereas SoScale does.
MXFP4
NVFP4
HiF4
Element format
E2M1
E2M1
E1M2
Scale hierarchy
E8M0
FP32 + E4M3
E6M2 + E1 + E1
Granularity
32
16 + tensor
64 / 8 / 4
Bits / element
4.25
≳ 4.50
4.50
Table 1: Compact comparison of hierarchical FP4 formats considered in this work.
Figure 2: Workflow of SoScale for one linear layer. Calibration builds the empirical Hessian and damped inverse-Hessian factor. In 2.1, SoScale searches legal FP4 scale codes with a Hessian-aware layer loss, whereas scale-only baselines such as ScaleSearch and Four Over Six use local MSE-style objectives without Hessian information; in 2.2, it propagates the selected block’s error to future blocks. Inference uses only stored FP4 payloads, legal scale codes, and lightweight per-token activation scales.
Model
Bit
Format
Method
GSM8K
HellaSwag
MMLU
ARC-C
ARC-E
BoolQ
PIQA
Avg.
Rel.
Qwen3-8B
–
BF16
–
0.8810
0.5713
0.7292
0.5538
0.8371
0.8663
0.7628
0.7431
1.0000
W4A4
MXFP4
RTN
0.8211
0.5292
0.6718
0.4923
0.7735
0.8425
0.7492
0.6971
0.9381
GPTQ
0.8362
0.5329
0.6862
0.5068
0.7971
0.8520
0.7514
0.7089
0.9540
BlockRotation
0.7839
0.5329
0.6772
0.5077
0.7921
0.8440
0.7421
0.6971
0.9381
ScaleSearch
0.8317
0.5303
0.6802
0.5171
0.7837
0.8505
0.7459
0.7056
0.9496
SoScale
0.8575
0.5334
0.6924
0.5000
0.7875
0.8517
0.7421
0.7092
0.9545
Table 2: Downstream accuracy under W4A4 hierarchical FP4 quantization; Avg. averages tasks and Rel. normalizes by the BF16 average for the same model. Highlighted rows are SoScale variants, w/ Had. denotes block Hadamard preprocessing, bold marks the best Avg. in each model-format group, and underlining marks the second best.
Model
Bit
Format
Method
GSM8K
HellaSwag
MMLU
ARC-C
ARC-E
BoolQ
PIQA
Avg.
Rel.
Qwen3-8B
W4A4
MXFP4
ScaleSearch-w
0.8241
0.5273
0.6738
0.5009
0.7774
0.8492
0.7492
0.7003
0.9424
SoScale-w
0.8317
0.5292
0.6856
0.5000
0.7997
0.8508
0.7405
0.7054
0.9492
SoScale-w w/ Had.
0.8461
0.5438
0.6966
0.5205
0.8064
0.8635
0.7601
0.7195
0.9683
NVFP4
ScaleSearch-w
0.8605
0.5533
0.7124
0.5341
0.8114
0.8575
0.7655
0.7278
0.9795
SoScale-w
0.8726
0.5542
0.7082
0.5384
0.8157
0.8602
0.7459
0.7279
0.9796
SoScale-w w/ Had.
0.8560
0.5541
0.7121
0.5410
0.8136
0.8657
0.7579
0.7286
0.9806
Table 3: W4A4 comparison with direct RTN activation quantization. The suffix “-w” denotes variants that use weight-side scale search while keeping activations on a direct RTN path; w/ Had. denotes block Hadamard preprocessing before weight quantization. Highlighted cells denote SoScale variants, and bold values mark the best average accuracy within each model-format group.
Format
Variant
Loss
Rec.
Avg.
Rel.
MXFP4
RTN
N/A
No
0.6971
0.9381
ScaleSearch
MSE
No
0.7056
0.9496
MSE + recovery
MSE
✓
0.7061
0.9503
SoScale
Hessian
✓
0.7092
0.9545
NVFP4
RTN
N/A
No
0.7262
0.9773
ScaleSearch
MSE
No
0.7281
0.9798
Table 4: Qwen3-8B ablation under W4A4 quantization; Rec. denotes the 2.2 error-propagation stage in Figure 2 .
Large language models (LLMs) achieve remarkable performance but are expensive to deploy due to their enormous size. FP4 quantization, with formats such as MXFP4 and NVFP4, offers an appealing solution with native hardware support on modern accelerators. However, maintaining accuracy under FP4 precision remains difficult. A key bottleneck lies in scale optimization: existing methods tightly couple the quantization and dequantization scales, forcing both to conform to the discrete low-precision format required by hardware, such as E8M0 in MXFP4. Yet the quantization scale is never stored and need not obey this constraint, suggesting a significant untapped optimization space. In this work, we propose FOCUS, a post-training quantization framework with end-to-end scale learning for FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling. Coupled-Relaxation Scaling (CRS) relaxes the tight coupling between quantization and dequantization scales with a learnable full-precision coefficient, enabling more effective optimization without breaking hardware compliance. Dual-Granularity Scaling (DGS) further refines the quantization scale at a finer sub-block granularity, allowing more precise adaptation to local weight distributions. Experiments across multiple LLM families and benchmarks show that FOCUS achieves state-of-the-art FP4 accuracy under both MXFP4 and NVFP4 formats, while introducing no additional inference overhead. Code and quantized models will be released at https://github.com/tencent/AngelSlim.
NVFP4 has recently emerged as an efficient 4-bit microscaling format for large language models (LLMs), offering superior numerical fidelity with native hardware support. However, existing methods often yield suboptimal performance due to inflexible scale selection and the coupled treatment of quantization and dequantization scales. To address these issues, we propose Scale Optimization for Accurate Reconstruction (SOAR), a novel post-training quantization framework that improves the accuracy of NVFP4 quantization. At its core, SOAR features Closed-form Joint Scale Optimization (CJSO), which jointly optimizes global and block-wise scales via analytical solutions derived from reconstruction error minimization. Furthermore, it incorporates Decoupled Scale Search (DSS). DSS decouples the high-precision quantization scale from its constrained dequantization counterpart, and performs discrete search to mitigate precision loss from scale quantization. Extensive experiments across multiple LLMs show that our method consistently outperforms existing NVFP4 quantization baselines, achieving superior accuracy under the same memory footprint with no additional hardware overhead. The code and models will be available at https://github.com/steven-bao1/SOAR.
Quantization has emerged as a standard technique for accelerating inference for generative models by enabling faster low-precision computations and reduced memory transfers. Recently, GPU accelerators have added first-class support for microscaling Block Floating Point (BFP) formats. Standard BFP algorithms use a fixed scale based on the maximum magnitude of the block. We observe that this scale choice can be suboptimal with respect to quantization errors. In this work, we propose ScaleSearch, an alternative strategy for selecting these scale factors: using a fine-grained search leveraging the mantissa bits in microscaling formats to minimize the quantization error for the given distribution. ScaleSearch can be integrated with existing quantization methods such as Post Training Quantization and low-precision attention, and is shown to improve their performance. Additionally, we introduce ScaleSearchAttention, an accelerated NVFP4-based attention algorithm, which uses ScaleSearch and adapted prior techniques to ensure near-0 performance loss for causal language modeling. Experiments show that ScaleSearch reduces quantization error by 27% for NVFP4 and improves language model PTQ by up to 15 points for MATH500 (Qwen3-8B), while ScaleSearchAttention improves Wikitext-2 PPL by upto 0.77 points for Llama 3.1 70B. The proposed methods closely match baseline performance while providing quantization accuracy improvements.
Tanmaey Gupta, Hayden Prairie, Xiaoxia Wu +10
Cornell University, Ithaca, NY, USA · Together AI, USA · University of California San Diego, La Jolla, CA, USA +1