Adaptive Block-Scaled Data Types

Jack Cook; Hyemin S. Lee; Kathryn Le; Junxian Guo; Giovanni Traverso; Anantha P. Chandrakasan; Song Han

cs.CLMar 30, 2026

Adaptive Block-Scaled Data Types

Authors: Jack Cook, Hyemin S. Lee, Kathryn Le, Junxian Guo, Giovanni Traverso, Anantha P. Chandrakasan, Song Han

Organizations: Massachusetts Institute of Technology · NVIDIA

Abstract

NVFP4 has grown increasingly popular as a 4-bit format for quantizing large language models due to its hardware support and its ability to retain useful information with relatively few bits per parameter. However, the format is not without limitations: recent work has shown that NVFP4 suffers from its error distribution, resulting in large amounts of quantization error on near-maximal values in each group of 16 values. In this work, we leverage this insight to design new Adaptive Block-Scaled Data Types that can adapt to the distribution of their input values. For four-bit quantization, our proposed IF4 (Int/Float 4) data type selects between FP4 and INT4 representations for each group of 16 values, which are then scaled by an E4M3 scale factor as is done with NVFP4. The selected data type is denoted using the scale factor's sign bit, which is currently unused in NVFP4, and we apply the same insight to design formats for other bit-widths, including IF3 and IF6. When used to quantize language models, we find that IF4 outperforms existing 4-bit block-scaled formats, achieving lower loss during quantized training and achieving higher accuracy on many tasks in post-training quantization. We additionally design and evaluate an IF4 Multiply-Accumulate (MAC) unit to demonstrate that IF4 can be implemented efficiently in next-generation hardware accelerators. Our code is available at https://github.com/mit-han-lab/fouroversix.

Figures & tables

Figure 1: A visual explanation of the IF4 data type. Like NVFP4, individual values are scaled by an FP8 scale factor, each of which is shared by 16 values. However, since each value has its own sign bit, in NVFP4 all scale factors are positive and the scale factor’s sign bit is never used. We repurpose this bit into an indicator which reflects whether the group’s values are stored as FP4 or scaled INT4.

Figure 2: Adaptive Block-Scaled Data Types (IF) provide better model performance for most low-precision memory constraints ( Section 5.1 ).

Format	Group Size	Scale Format	Maximum Value †	Minimum Value †	Relative Dynamic Range	MSE ‡ $\times$ 10 -3
MXFP4	32	UE8M0	6 $\times$ 2 127	0.5 $\times$ 2 -127	$\infty$	13.2
NVFP4	16	E4M3	6 $\times$ 448	0.5 $\times$ 2 -9	100%	9.0
NVFP4 (4/6)	16	E4M3	6 $\times$ 256	0.5 $\times$ 2 -9	57.1%	7.5
NVINT4	16	E4M3	7 $\times$ 448	1 $\times$ 2 -9	58.3%	7.4
IF4	16	UE4M3	6 $\times$ 448	0.5 $\times$ 2 -9	100%	6.2

Table 1: Summary of Block-Scaled 4-Bit Quantization Schemes

Panel of Figure 3 — Figure 3: IF4 quantization is done by quantizing each group to FP4 and INT4 and selecting the option with less mean squared error, resulting in less quantization error across many weight channels in Qwen3.5-35B-A3B with no storage overhead compared to NVFP4.

Figure 4: IF4 outperforms NVFP4 during W4A4G4 quantized training. When MS-EDEN Panferov et al. (2026) is used in the backward pass, IF4 results in an even larger improvement.

Panel of Figure 5 — Figure 5: In W4A4G4 training and W4A4 PTQ, it is common to reduce the dynamic range of the inputs being quantized, such as through a Hadamard transformation. We find that NVINT4 deals with these transformed inputs better, explaining part of IF4’s success.

	Nemotron 3				Qwen3.5
	4B		30B-A3B		35B-A3B		122B-A10B
	WikiText-2	C4	WikiText-2	C4	WikiText-2	C4	WikiText-2	C4
BF16	16.92	43.82	7.33	21.75	7.70	22.17	5.72	21.53
MXFP4	19.77	51.71	10.51	33.17	8.83	25.00	6.78	23.64
NVINT4	18.35	47.82	8.35	26.32	8.31	23.63	6.28	22.83
NVFP4	18.03	47.05	9.02	26.86	8.14	23.18	6.20	22.52

Table 2: Perplexity results using 4-bit block-scaled quantization schemes with round-to-nearest W4A4 PTQ.

	Nemotron 3		Qwen3.5					Avg. ( $\uparrow$ )
	4B	30B-A3B	9B	27B	35B-A3B	122B-A10B	397B-A17B	Avg. ( $\uparrow$ )
BF16	65.1	69.4	70.5	75.9	75.5	76.4	81.5	73.5
MXFP4	61.5	63.8	66.3	74.2	72.8	75.5	79.5	70.5
NVINT4	63.1	67.5	68.7	74.6	74.0	76.0	80.8	72.1
NVFP4	64.1	67.6	69.3	75.2	73.9	76.0	80.9	72.4
4/6	63.9	67.0	69.2	74.9	74.6	75.7	80.8	72.3

Table 3: W4A4 round-to-nearest post-training quantization accuracy averaged across five tasks: ARC-Easy, ARC-Challenge, HellaSwag, LAMBADA, and PIQA. Results for individual tasks can be found in Appendix D .

Figure 6: Hardware architecture of an IF4 multiply-accumulate (MAC) block, demonstrating the feasibility of IF4 in next-generation hardware accelerators.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Hyper-parameter	Value
Activation	Swish
AdamW $\beta_{1}$	0.9
AdamW $\beta_{2}$	0.95
Attention Heads	16
Context Length	8192
Gradient Clipping	1.0

Appendix

Table 4: Hyper-parameters used during training.

Figure 7: Average change in high-precision values after being quantized to each data type across 1000 trials with stochastic rounding. While large values change the most when quantized with IF4, we find this only affects a very small number of gradients, as most gradients (shown in gray) are significantly smaller than outlier gradient values.

Figure 8: Hardware architecture of NVFP4 multiply-accumulate (MAC) block, providing the baseline for IF4 MAC

	NVFP4 MAC	IF4 MAC
Technology (nm)	28	28
Clock (MHz)	524	500
Latency (ns)	3.82	4.00
Throughput (GFLOPS)	16.77	16.00
Power (mW)	11.27	14.40
Area (mm 2 )	0.024	0.040

Appendix

Table 5: Post-synthesis performance comparison between NVFP4 MAC and IF4 MAC

Figure 9: We find WikiText-2 Perplexity to be a fairly strong indicator of average downstream task accuracy across a variety of Qwen3.5 model sizes. This helps us simulate future data types in Section 5.1 while using fewer resources.

Name	Scale Factor	Value	Block	Bits per	Avg. WikiText-2
Name	Type	Type	Size	Parameter	Perplexity
BF16	–	–	–	16.0	7.87
MXFP3	UE8M0	E2M0	32	3.25	70.04
NVINT3	E4M3	INT3	16	3.5	12.78
NVFP3	E4M3	E2M0	16	3.5	11.12
IF3	UE4M3	INT3 / E2M0	16	3.5	10.67

Appendix

Table 6: Average WikiText-2 perplexity across Qwen3.5-9B, Qwen3.5-27B, Qwen3.5-35B-A3B, and Qwen3.5-122B-A10B when weights and activations in all linear and MoE layers but the LM head are quantized using the specified quantization scheme.

Model	Scheme	ARC-E	ARC-C	HellaSwag	LAMBADA	PIQA	Average ( $\uparrow$ )
Nemotron 3 Nano 4B	BF16	79.00	53.36	67.33	49.02	76.66	65.07
	MXFP4	75.35	49.32	63.97	43.94	75.06	61.53
	NVINT4	76.67	50.09	65.65	46.78	76.35	63.11
	NVFP4	77.01	53.73	66.36	48.13	75.41	64.13
	4/6	77.82	52.62	66.32	46.99	75.97	63.94
	IF4	77.64	52.36	66.33	48.66	76.39	64.27

Appendix

Table 7: Full downstream task results for each combination of model and task from Table 3 . Each experiment is repeated three times and reported as an average across those three repetitions. Bold denotes the best quantized result.

Neural Network Quantization

Neural network quantization represents weights, activations, and states with fewer bits to reduce memory use, computation, and deployment costs.

See all 404 papers

Search Your Block Floating Point Scales!
May 12, 2026Tanmaey Gupta, Hayden Prairie, Xiaoxia Wu +104-Bit Quantization LLM Inference Acceleration
Cornell University, Ithaca, NY, USA · Together AI, USA · University of California San Diego, La Jolla, CA, USA +1
SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization
May 12, 2026Chengzhu Bao, Xianglong Yan, Zhiteng Li +3LLM Quantization 4-Bit Quantization
Shanghai Jiao Tong University · Tencent Hunyuan
FAAR: Format-Aware Adaptive Rounding for NVFP4
Mar 23, 2026Hanglin Li, Shuchang Tian, Chen Lin +2LLM Quantization 4-Bit Quantization
Li Auto Inc.

May 12, 2026J/K move · Enter open · S save

Search Your Block Floating Point Scales!

Tanmaey Gupta, Hayden Prairie, Xiaoxia Wu, Reyna Abhyankar+9

Cornell University, Ithaca, NY, USA · Together AI, USA · University of California San Diego, La Jolla, CA, USA+1

4-Bit Quantization LLM Inference Acceleration Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

Quantization has emerged as a standard technique for accelerating inference for generative models by enabling faster low-precision computations and reduced memory transfers. Recently, GPU accelerators have added first-class support for microscaling Block Floating Point (BFP) formats. Standard BFP algorithms use a fixed scale based on the maximum magnitude of the block. We observe that this scale choice can be suboptimal with respect to quantization errors. In this work, we propose ScaleSearch, an alternative strategy for selecting these scale factors: using a fine-grained search leveraging the mantissa bits in microscaling formats to minimize the quantization error for the given distribution. ScaleSearch can be integrated with existing quantization methods such as Post Training Quantization and low-precision attention, and is shown to improve their performance. Additionally, we introduce ScaleSearchAttention, an accelerated NVFP4-based attention algorithm, which uses ScaleSearch and adapted prior techniques to ensure near-0 performance loss for causal language modeling. Experiments show that ScaleSearch reduces quantization error by 27% for NVFP4 and improves language model PTQ by up to 15 points for MATH500 (Qwen3-8B), while ScaleSearchAttention improves Wikitext-2 PPL by upto 0.77 points for Llama 3.1 70B. The proposed methods closely match baseline performance while providing quantization accuracy improvements.

Abstract

Neural Network Quantization

Explore similar work

Search Your Block Floating Point Scales!

SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization

FAAR: Format-Aware Adaptive Rounding for NVFP4