Deploying large language models (LLMs) on edge devices requires extremely low-bit quantization. Ultra-low precision formats such as NVFP4 offer a promising solution for reducing memory footprint and accelerating computation. However, existing quantization methods typically rely on conventional rounding strategies and fail to account for the non-uniformity of the NVFP4 numerical grid, resulting in suboptimal rounding decisions and amplified quantization errors. To address this, we propose Format-Aware Adaptive Rounding (FAAR), a learnable rounding strategy tailored for the NVFP4 format. Unlike conventional quantization paradigms, FAAR explicitly incorporates the non-uniform NVFP4 grid into the optimization process. By adaptively adjusting rounding decisions guided by loss gradients, our method effectively approximates the theoretically optimal quantization. To complement FAAR, we introduce a 2-stages Format Alignment (2FA) fine-tuning scheme that aligns LLM parameters layer-by-layer to the NVFP4 numerical space, further narrowing the performance gap. Remarkably, this learnable optimization incurs a minimal training overhead of only 4 GPU hours on Llama3-1B. Extensive experiments demonstrate the effectiveness of our approach. Compared with Round-to-Nearest (RTN), our method reduces perplexity on WikiText-2 from 14.28 to 12.60 on Llama3-1B and from 23.06 to 21.27 on Qwen3-1.7B. Additionally, our method consistently outperforms state-of-the-art approaches across various zero-shot downstream tasks.
Figures & tables
Figure 1: The overall pipeline of our proposed quantization framework combining FAAR and 2FA. (1) FAAR + Stage 1: The continuous learnable rounding variables V are carefully initialized and incorporated into the layer-wise NVFP4 optimization. During this stage, we perform layer-wise rounding optimization against the frozen BF16 model. For each optimization step, only the current layer is updated while the rest of the network remains frozen. The rounding decisions are optimized via learnable variables parameterized by a differentiable sigmoid function and guided by Round Loss. (2) FAAR + Stage 2: The quantized layers are assembled into a full NVFP4 model and jointly optimized. Model-wise alignment is performed using KL divergence and last-hidden-state MSE losses to mitigate full-model error accumulation. (3) Hardening and Inference: After the two-stage optimization, the continuous variables V are deterministically hardened into discrete binary decisions, which are then deployed for efficient inference.
Figure 2: The non-uniform NVFP4 grid introduces magnitude-dependent rounding errors. (a) The mapping function relative to the original weight w , showing numerical nodes that are densely concentrated near zero but become increasingly sparse for larger magnitudes. (b) The absolute quantization error, highlighting the low-error region (shaded green) optimized for the core weight distribution and the amplified distortion for weights with larger magnitudes.
Rounding scheme
PPL
baseline
13.04
lower
13.05
upper
13.08
Stochastic
13.056±0.034
Stochastic (best)
13.02
Table 1: RTN is Suboptimal for NVFP4 Quantization. We compare WikiText-2 perplexity across different rounding schemes for Llama3-1B.
Step
Pseudocode
1
for each layer l do
2
W~=∣W∣/(sg⋅sglobal)
3
(Wlower,Wupper)=FindInterval(W~,N)
4
Vinit=(W~−Wlower)/(Wupper−Wlower)
5
Initialize learnable variable V=Vinit
6
hβ(V)=σ(β(V−0.5))
Table 2: Procedure for the FAAR and 2FA quantization framework.
WikiText-2 Word PPL ( ↓ )
C4 Word PPL ( ↓ )
Llama3
Qwen3
Llama3
Qwen3
Method
1B
8B
1.7B
8B
1B
8B
1.7B
8B
BF16
11.98
7.54
21.04
12.21
28.55
18.08
58.75
36.33
RTN
14.28
8.44
23.06
12.67
36.19
20.82
65.54
37.91
GPTQ
13.74
8.35
21.48
12.49
35.62
20.96
63.34
37.14
MR-GPTQ
13.73
8.32
21.42
12.44
35.59
20.92
63.31
37.19
Table 3: FAAR Framework Consistently Outperforms Existing PTQ approaches. Our method achieves the lowest word perplexity ( ↓ ) across Llama3 and Qwen3 models on both WikiText-2 and C4 datasets.
WikiText-2 Cosine Similarity ( ↑ )
C4 Cosine Similarity ( ↑ )
Llama3
Qwen3
Llama3
Qwen3
Method
1B
8B
1.7B
8B
1B
8B
1.7B
8B
BF16
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
RTN
96.08
97.23
97.97
97.92
94.27
98.03
97.11
97.11
GPTQ
97.78
98.45
98.75
99.01
96.01
96.88
97.95
98.63
MR-GPTQ
97.90
97.80
98.76
99.08
97.02
97.41
98.02
98.47
Table 4: FAAR Preserves the Internal Feature Space of BF16. Our method yields higher cosine similarity ( ↑ ) between quantized and BF16 last hidden states, indicating improved preservation of internal representations. Results are reported as percentages (%).
BoolQ( ↑ )
Arc-E( ↑ )
Arc-C( ↑ )
HellaSwag( ↑ )
Average( ↑ )
Method
1B
8B
1B
8B
1B
8B
1B
8B
1B
8B
BF16
63.61
83.12
62.08
82.71
36.86
54.96
64.24
79.27
56.70
75.02
RTN
58.55
80.24
57.31
77.51
34.28
52.49
59.97
77.66
52.53
71.98
MR-GPTQ
61.27
80.89
57.38
78.55
33.47
53.70
60.65
77.20
53.19
72.59
GPTQ
61.06
80.28
57.41
78.30
33.07
53.81
60.58
77.02
53.03
72.35
GPTQ+4/6
60.17
81.40
57.54
78.72
33.81
53.16
60.73
77.52
53.06
72.70
Table 5: Downstream task performance of Llama3 models across various PTQ methods. Results are reported as percentages (%).
Method
Llama3-1B
Qwen3-1.7B
BF16
11.98
21.04
RTN
14.28
23.06
FAAR
13.01
21.86
FAAR + 2FA
12.60
21.27
Table 6: Effect of Algorithmic Components. FAAR significantly reduces perplexity compared to RTN, and the addition of 2FA further mitigates errors and yields the best results (WikiText-2 Word PPL ↓ ).
The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference. However, it is difficult to adapt existing Post-Training Quantization (PTQ) strategies to these formats: rotation-based methods compromise fine-grained block isolation; smoothing techniques struggle with significant 4-bit quantization errors; and mixed-precision approaches often conflict with hardware constraints on unified-precision computation. To address these challenges, we propose ARCQuant, a framework that boosts NVFP4 performance via Augmented Residual Channels. Distinct from methods that compromise block isolation or hardware uniformity, ARCQuant maintains a strictly unified NVFP4 format by augmenting the activation matrix with quantized residual channels. This design integrates the error compensation process directly into the matrix reduction dimension, enabling the use of standard, highly optimized GEMM kernels with minimal overhead. Theoretical analysis confirms that the worst-case error bound of our dual-stage NVFP4 quantization is comparable to that of standard 8-bit formats such as MXFP8. Extensive experiments on LLaMA and Qwen models demonstrate that ARCQuant achieves state-of-the-art accuracy, comparable to full-precision baselines in perplexity and downstream tasks. Furthermore, deployment on RTX 5090 and RTX PRO 6000 GPUs confirms practical benefits, achieving up to 3x speedup over FP16. Our code is available at https://github.com/actypedef/ARCQuant.
Haoqian Meng, Yilun Luo, Yafei Zhao +3
School of Computer Science and Technology, Tianjin University
NVFP4 has recently emerged as an efficient 4-bit microscaling format for large language models (LLMs), offering superior numerical fidelity with native hardware support. However, existing methods often yield suboptimal performance due to inflexible scale selection and the coupled treatment of quantization and dequantization scales. To address these issues, we propose Scale Optimization for Accurate Reconstruction (SOAR), a novel post-training quantization framework that improves the accuracy of NVFP4 quantization. At its core, SOAR features Closed-form Joint Scale Optimization (CJSO), which jointly optimizes global and block-wise scales via analytical solutions derived from reconstruction error minimization. Furthermore, it incorporates Decoupled Scale Search (DSS). DSS decouples the high-precision quantization scale from its constrained dequantization counterpart, and performs discrete search to mitigate precision loss from scale quantization. Extensive experiments across multiple LLMs show that our method consistently outperforms existing NVFP4 quantization baselines, achieving superior accuracy under the same memory footprint with no additional hardware overhead. The code and models will be available at https://github.com/steven-bao1/SOAR.
Large language models (LLMs) achieve remarkable performance but are expensive to deploy due to their enormous size. FP4 quantization, with formats such as MXFP4 and NVFP4, offers an appealing solution with native hardware support on modern accelerators. However, maintaining accuracy under FP4 precision remains difficult. A key bottleneck lies in scale optimization: existing methods tightly couple the quantization and dequantization scales, forcing both to conform to the discrete low-precision format required by hardware, such as E8M0 in MXFP4. Yet the quantization scale is never stored and need not obey this constraint, suggesting a significant untapped optimization space. In this work, we propose FOCUS, a post-training quantization framework with end-to-end scale learning for FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling. Coupled-Relaxation Scaling (CRS) relaxes the tight coupling between quantization and dequantization scales with a learnable full-precision coefficient, enabling more effective optimization without breaking hardware compliance. Dual-Granularity Scaling (DGS) further refines the quantization scale at a finer sub-block granularity, allowing more precise adaptation to local weight distributions. Experiments across multiple LLM families and benchmarks show that FOCUS achieves state-of-the-art FP4 accuracy under both MXFP4 and NVFP4 formats, while introducing no additional inference overhead. Code and quantized models will be released at https://github.com/tencent/AngelSlim.