Deploying large language models (LLMs) on edge devices requires extremely low-bit quantization. Ultra-low precision formats such as NVFP4 offer a promising solution for reducing memory footprint and accelerating computation. However, existing quantization methods typically rely on conventional rounding strategies and fail to account for the non-uniformity of the NVFP4 numerical grid, resulting in suboptimal rounding decisions and amplified quantization errors. To address this, we propose Format-Aware Adaptive Rounding (FAAR), a learnable rounding strategy tailored for the NVFP4 format. Unlike conventional quantization paradigms, FAAR explicitly incorporates the non-uniform NVFP4 grid into the optimization process. By adaptively adjusting rounding decisions guided by loss gradients, our method effectively approximates the theoretically optimal quantization. To complement FAAR, we introduce a 2-stages Format Alignment (2FA) fine-tuning scheme that aligns LLM parameters layer-by-layer to the NVFP4 numerical space, further narrowing the performance gap. Remarkably, this learnable optimization incurs a minimal training overhead of only 4 GPU hours on Llama3-1B. Extensive experiments demonstrate the effectiveness of our approach. Compared with Round-to-Nearest (RTN), our method reduces perplexity on WikiText-2 from 14.28 to 12.60 on Llama3-1B and from 23.06 to 21.27 on Qwen3-1.7B. Additionally, our method consistently outperforms state-of-the-art approaches across various zero-shot downstream tasks.
Figures & tables
Figure 1: The overall pipeline of our proposed quantization framework combining FAAR and 2FA. (1) FAAR + Stage 1: The continuous learnable rounding variables V are carefully initialized and incorporated into the layer-wise NVFP4 optimization. During this stage, we perform layer-wise rounding optimization against the frozen BF16 model. For each optimization step, only the current layer is updated while the rest of the network remains frozen. The rounding decisions are optimized via learnable variables parameterized by a differentiable sigmoid function and guided by Round Loss. (2) FAAR + Stage 2: The quantized layers are assembled into a full NVFP4 model and jointly optimized. Model-wise alignment is performed using KL divergence and last-hidden-state MSE losses to mitigate full-model error accumulation. (3) Hardening and Inference: After the two-stage optimization, the continuous variables V are deterministically hardened into discrete binary decisions, which are then deployed for efficient inference.
Figure 2: The non-uniform NVFP4 grid introduces magnitude-dependent rounding errors. (a) The mapping function relative to the original weight w , showing numerical nodes that are densely concentrated near zero but become increasingly sparse for larger magnitudes. (b) The absolute quantization error, highlighting the low-error region (shaded green) optimized for the core weight distribution and the amplified distortion for weights with larger magnitudes.
Rounding scheme
PPL
baseline
13.04
lower
13.05
upper
13.08
Stochastic
13.056±0.034
Stochastic (best)
13.02
Table 1: RTN is Suboptimal for NVFP4 Quantization. We compare WikiText-2 perplexity across different rounding schemes for Llama3-1B.
Step
Pseudocode
1
for each layer l do
2
W~=∣W∣/(sg⋅sglobal)
3
(Wlower,Wupper)=FindInterval(W~,N)
4
Vinit=(W~−Wlower)/(Wupper−Wlower)
5
Initialize learnable variable V=Vinit
6
hβ(V)=σ(β(V−0.5))
Table 2: Procedure for the FAAR and 2FA quantization framework.
WikiText-2 Word PPL ( ↓ )
C4 Word PPL ( ↓ )
Llama3
Qwen3
Llama3
Qwen3
Method
1B
8B
1.7B
8B
1B
8B
1.7B
8B
BF16
11.98
7.54
21.04
12.21
28.55
18.08
58.75
36.33
RTN
14.28
8.44
23.06
12.67
36.19
20.82
65.54
37.91
GPTQ
13.74
8.35
21.48
12.49
35.62
20.96
63.34
37.14
MR-GPTQ
13.73
8.32
21.42
12.44
35.59
20.92
63.31
37.19
Table 3: FAAR Framework Consistently Outperforms Existing PTQ approaches. Our method achieves the lowest word perplexity ( ↓ ) across Llama3 and Qwen3 models on both WikiText-2 and C4 datasets.
WikiText-2 Cosine Similarity ( ↑ )
C4 Cosine Similarity ( ↑ )
Llama3
Qwen3
Llama3
Qwen3
Method
1B
8B
1.7B
8B
1B
8B
1.7B
8B
BF16
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
RTN
96.08
97.23
97.97
97.92
94.27
98.03
97.11
97.11
GPTQ
97.78
98.45
98.75
99.01
96.01
96.88
97.95
98.63
MR-GPTQ
97.90
97.80
98.76
99.08
97.02
97.41
98.02
98.47
Table 4: FAAR Preserves the Internal Feature Space of BF16. Our method yields higher cosine similarity ( ↑ ) between quantized and BF16 last hidden states, indicating improved preservation of internal representations. Results are reported as percentages (%).
BoolQ( ↑ )
Arc-E( ↑ )
Arc-C( ↑ )
HellaSwag( ↑ )
Average( ↑ )
Method
1B
8B
1B
8B
1B
8B
1B
8B
1B
8B
BF16
63.61
83.12
62.08
82.71
36.86
54.96
64.24
79.27
56.70
75.02
RTN
58.55
80.24
57.31
77.51
34.28
52.49
59.97
77.66
52.53
71.98
MR-GPTQ
61.27
80.89
57.38
78.55
33.47
53.70
60.65
77.20
53.19
72.59
GPTQ
61.06
80.28
57.41
78.30
33.07
53.81
60.58
77.02
53.03
72.35
GPTQ+4/6
60.17
81.40
57.54
78.72
33.81
53.16
60.73
77.52
53.06
72.70
Table 5: Downstream task performance of Llama3 models across various PTQ methods. Results are reported as percentages (%).
Method
Llama3-1B
Qwen3-1.7B
BF16
11.98
21.04
RTN
14.28
23.06
FAAR
13.01
21.86
FAAR + 2FA
12.60
21.27
Table 6: Effect of Algorithmic Components. FAAR significantly reduces perplexity compared to RTN, and the addition of 2FA further mitigates errors and yields the best results (WikiText-2 Word PPL ↓ ).