NVFP4 has grown increasingly popular as a 4-bit format for quantizing large language models due to its hardware support and its ability to retain useful information with relatively few bits per parameter. However, the format is not without limitations: recent work has shown that NVFP4 suffers from its error distribution, resulting in large amounts of quantization error on near-maximal values in each group of 16 values. In this work, we leverage this insight to design new Adaptive Block-Scaled Data Types that can adapt to the distribution of their input values. For four-bit quantization, our proposed IF4 (Int/Float 4) data type selects between FP4 and INT4 representations for each group of 16 values, which are then scaled by an E4M3 scale factor as is done with NVFP4. The selected data type is denoted using the scale factor's sign bit, which is currently unused in NVFP4, and we apply the same insight to design formats for other bit-widths, including IF3 and IF6. When used to quantize language models, we find that IF4 outperforms existing 4-bit block-scaled formats, achieving lower loss during quantized training and achieving higher accuracy on many tasks in post-training quantization. We additionally design and evaluate an IF4 Multiply-Accumulate (MAC) unit to demonstrate that IF4 can be implemented efficiently in next-generation hardware accelerators. Our code is available at https://github.com/mit-han-lab/fouroversix.
Figures & tables
Figure 1: A visual explanation of the IF4 data type. Like NVFP4, individual values are scaled by an FP8 scale factor, each of which is shared by 16 values. However, since each value has its own sign bit, in NVFP4 all scale factors are positive and the scale factor’s sign bit is never used. We repurpose this bit into an indicator which reflects whether the group’s values are stored as FP4 or scaled INT4.
Figure 2: Adaptive Block-Scaled Data Types (IF) provide better model performance for most low-precision memory constraints ( Section 5.1 ).
Format
Group Size
Scale Format
Maximum Value †
Minimum Value †
Relative Dynamic Range
MSE ‡ × 10 -3
MXFP4
32
UE8M0
6 × 2 127
0.5 × 2 -127
∞
13.2
NVFP4
16
E4M3
6 × 448
0.5 × 2 -9
100%
9.0
NVFP4 (4/6)
16
E4M3
6 × 256
0.5 × 2 -9
57.1%
7.5
NVINT4
16
E4M3
7 × 448
1 × 2 -9
58.3%
7.4
IF4
16
UE4M3
6 × 448
0.5 × 2 -9
100%
6.2
Table 1: Summary of Block-Scaled 4-Bit Quantization Schemes
Figure 3: IF4 quantization is done by quantizing each group to FP4 and INT4 and selecting the option with less mean squared error, resulting in less quantization error across many weight channels in Qwen3.5-35B-A3B with no storage overhead compared to NVFP4.
Figure 4: IF4 outperforms NVFP4 during W4A4G4 quantized training. When MS-EDEN Panferov et al. (2026) is used in the backward pass, IF4 results in an even larger improvement.
Figure 5: In W4A4G4 training and W4A4 PTQ, it is common to reduce the dynamic range of the inputs being quantized, such as through a Hadamard transformation. We find that NVINT4 deals with these transformed inputs better, explaining part of IF4’s success.
Nemotron 3
Qwen3.5
4B
30B-A3B
35B-A3B
122B-A10B
WikiText-2
C4
WikiText-2
C4
WikiText-2
C4
WikiText-2
C4
BF16
16.92
43.82
7.33
21.75
7.70
22.17
5.72
21.53
MXFP4
19.77
51.71
10.51
33.17
8.83
25.00
6.78
23.64
NVINT4
18.35
47.82
8.35
26.32
8.31
23.63
6.28
22.83
NVFP4
18.03
47.05
9.02
26.86
8.14
23.18
6.20
22.52
Table 2: Perplexity results using 4-bit block-scaled quantization schemes with round-to-nearest W4A4 PTQ.
Nemotron 3
Qwen3.5
Avg. ( ↑ )
4B
30B-A3B
9B
27B
35B-A3B
122B-A10B
397B-A17B
BF16
65.1
69.4
70.5
75.9
75.5
76.4
81.5
73.5
MXFP4
61.5
63.8
66.3
74.2
72.8
75.5
79.5
70.5
NVINT4
63.1
67.5
68.7
74.6
74.0
76.0
80.8
72.1
NVFP4
64.1
67.6
69.3
75.2
73.9
76.0
80.9
72.4
4/6
63.9
67.0
69.2
74.9
74.6
75.7
80.8
72.3
Table 3: W4A4 round-to-nearest post-training quantization accuracy averaged across five tasks: ARC-Easy, ARC-Challenge, HellaSwag, LAMBADA, and PIQA. Results for individual tasks can be found in Appendix D .
Figure 6: Hardware architecture of an IF4 multiply-accumulate (MAC) block, demonstrating the feasibility of IF4 in next-generation hardware accelerators.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Hyper-parameter
Value
Activation
Swish
AdamW β1
0.9
AdamW β2
0.95
Attention Heads
16
Context Length
8192
Gradient Clipping
1.0
Appendix
Table 4: Hyper-parameters used during training.
Figure 7: Average change in high-precision values after being quantized to each data type across 1000 trials with stochastic rounding. While large values change the most when quantized with IF4, we find this only affects a very small number of gradients, as most gradients (shown in gray) are significantly smaller than outlier gradient values.
Figure 8: Hardware architecture of NVFP4 multiply-accumulate (MAC) block, providing the baseline for IF4 MAC
NVFP4 MAC
IF4 MAC
Technology (nm)
28
28
Clock (MHz)
524
500
Latency (ns)
3.82
4.00
Throughput (GFLOPS)
16.77
16.00
Power (mW)
11.27
14.40
Area (mm 2 )
0.024
0.040
Appendix
Table 5: Post-synthesis performance comparison between NVFP4 MAC and IF4 MAC
Figure 9: We find WikiText-2 Perplexity to be a fairly strong indicator of average downstream task accuracy across a variety of Qwen3.5 model sizes. This helps us simulate future data types in Section 5.1 while using fewer resources.
Name
Scale Factor
Value
Block
Bits per
Avg. WikiText-2
Type
Type
Size
Parameter
Perplexity
BF16
–
–
–
16.0
7.87
MXFP3
UE8M0
E2M0
32
3.25
70.04
NVINT3
E4M3
INT3
16
3.5
12.78
NVFP3
E4M3
E2M0
16
3.5
11.12
IF3
UE4M3
INT3 / E2M0
16
3.5
10.67
Appendix
Table 6: Average WikiText-2 perplexity across Qwen3.5-9B, Qwen3.5-27B, Qwen3.5-35B-A3B, and Qwen3.5-122B-A10B when weights and activations in all linear and MoE layers but the LM head are quantized using the specified quantization scheme.
Model
Scheme
ARC-E
ARC-C
HellaSwag
LAMBADA
PIQA
Average ( ↑ )
Nemotron 3 Nano 4B
BF16
79.00
53.36
67.33
49.02
76.66
65.07
MXFP4
75.35
49.32
63.97
43.94
75.06
61.53
NVINT4
76.67
50.09
65.65
46.78
76.35
63.11
NVFP4
77.01
53.73
66.36
48.13
75.41
64.13
4/6
77.82
52.62
66.32
46.99
75.97
63.94
IF4
77.64
52.36
66.33
48.66
76.39
64.27
Appendix
Table 7: Full downstream task results for each combination of model and task from Table 3 . Each experiment is repeated three times and reported as an average across those three repetitions. Bold denotes the best quantized result.
Quantization has emerged as a standard technique for accelerating inference for generative models by enabling faster low-precision computations and reduced memory transfers. Recently, GPU accelerators have added first-class support for microscaling Block Floating Point (BFP) formats. Standard BFP algorithms use a fixed scale based on the maximum magnitude of the block. We observe that this scale choice can be suboptimal with respect to quantization errors. In this work, we propose ScaleSearch, an alternative strategy for selecting these scale factors: using a fine-grained search leveraging the mantissa bits in microscaling formats to minimize the quantization error for the given distribution. ScaleSearch can be integrated with existing quantization methods such as Post Training Quantization and low-precision attention, and is shown to improve their performance. Additionally, we introduce ScaleSearchAttention, an accelerated NVFP4-based attention algorithm, which uses ScaleSearch and adapted prior techniques to ensure near-0 performance loss for causal language modeling. Experiments show that ScaleSearch reduces quantization error by 27% for NVFP4 and improves language model PTQ by up to 15 points for MATH500 (Qwen3-8B), while ScaleSearchAttention improves Wikitext-2 PPL by upto 0.77 points for Llama 3.1 70B. The proposed methods closely match baseline performance while providing quantization accuracy improvements.
Tanmaey Gupta, Hayden Prairie, Xiaoxia Wu +10
Cornell University, Ithaca, NY, USA · Together AI, USA · University of California San Diego, La Jolla, CA, USA +1
NVFP4 has recently emerged as an efficient 4-bit microscaling format for large language models (LLMs), offering superior numerical fidelity with native hardware support. However, existing methods often yield suboptimal performance due to inflexible scale selection and the coupled treatment of quantization and dequantization scales. To address these issues, we propose Scale Optimization for Accurate Reconstruction (SOAR), a novel post-training quantization framework that improves the accuracy of NVFP4 quantization. At its core, SOAR features Closed-form Joint Scale Optimization (CJSO), which jointly optimizes global and block-wise scales via analytical solutions derived from reconstruction error minimization. Furthermore, it incorporates Decoupled Scale Search (DSS). DSS decouples the high-precision quantization scale from its constrained dequantization counterpart, and performs discrete search to mitigate precision loss from scale quantization. Extensive experiments across multiple LLMs show that our method consistently outperforms existing NVFP4 quantization baselines, achieving superior accuracy under the same memory footprint with no additional hardware overhead. The code and models will be available at https://github.com/steven-bao1/SOAR.
Deploying large language models (LLMs) on edge devices requires extremely low-bit quantization. Ultra-low precision formats such as NVFP4 offer a promising solution for reducing memory footprint and accelerating computation. However, existing quantization methods typically rely on conventional rounding strategies and fail to account for the non-uniformity of the NVFP4 numerical grid, resulting in suboptimal rounding decisions and amplified quantization errors. To address this, we propose Format-Aware Adaptive Rounding (FAAR), a learnable rounding strategy tailored for the NVFP4 format. Unlike conventional quantization paradigms, FAAR explicitly incorporates the non-uniform NVFP4 grid into the optimization process. By adaptively adjusting rounding decisions guided by loss gradients, our method effectively approximates the theoretically optimal quantization. To complement FAAR, we introduce a 2-stages Format Alignment (2FA) fine-tuning scheme that aligns LLM parameters layer-by-layer to the NVFP4 numerical space, further narrowing the performance gap. Remarkably, this learnable optimization incurs a minimal training overhead of only 4 GPU hours on Llama3-1B. Extensive experiments demonstrate the effectiveness of our approach. Compared with Round-to-Nearest (RTN), our method reduces perplexity on WikiText-2 from 14.28 to 12.60 on Llama3-1B and from 23.06 to 21.27 on Qwen3-1.7B. Additionally, our method consistently outperforms state-of-the-art approaches across various zero-shot downstream tasks.