BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimization

Ji-Fu Li; Manyi Zhang; Xiaobo Xia; Han Bao; Haoli Bai; Zhenhua Dong; Xianzhi Yu

cs.CLMar 17, 2026

BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimization

Authors: Ji-Fu Li, Manyi Zhang, Xiaobo Xia, Han Bao, Haoli Bai, Zhenhua Dong, Xianzhi Yu

Organizations: Huawei Technologies · University of Science and Technology of China

Abstract

Microscaling floating-point (MXFP) formats have emerged as a promising standard for deploying Multi-modal Large Language Models (MLLMs) and Large Language Models (LLMs) on modern accelerator architectures. However, existing Post-Training Quantization (PTQ) methods, particularly rotation-based techniques designed for integer formats, suffer from severe performance collapse when applied to MXFP4. Recent studies attribute this failure to a fundamental format mismatch: global orthogonal rotations inadvertently transfer outlier energy across quantization blocks, inducing new outliers that disrupt local block-wise scaling, while often creating bimodal activation distributions that underutilize the limited quantization range. To address these issues, we propose BATQuant (Block-wise Affine Transformation), which restricts transformations to align with MXFP granularity to prevent cross-block outlier propagation, while relaxing orthogonality constraints to optimize distribution shaping. To ensure parameter efficiency, we introduce Global and Private Kronecker (GPK) decomposition to effectively reduces storage and runtime overhead and incorporate Block-wise Learnable Clipping to suppress residual outliers. Extensive experiments on both MLLMs and LLMs demonstrate that BATQuant establishes new state-of-the-art results under aggressive W4A4KV16 configurations, recovering up to 96.43% of full-precision performance on multimodal benchmarks and clearly outperforming existing methods across diverse tasks.

Figures & tables

Figure 1: Quantization performance on Qwen3-VL-8B-Instruct across various methods. Our method yields superior results compared to baselines across all bit-width settings. The advantage is particularly substantial in the W4A4 setting, where our method clearly outperforms existing methods.

Panel of Figure 2 — Figure 2: Activation distributions for the down_proj module in layer 35 of Qwen3-8B. The central 3D plots illustrate the activations after transformation. We specifically extract Block 5 (without outliers) and Block 295 (with extreme outliers), and visualize the values after scaling factor division but prior to rounding. (a) After applying the block Hadamard transform, block 295 exhibits a bimodal distribution, leading to inefficient utilization of the bit width. (b) After the block affine transformation, block 295 shows reduced magnitude compared to subplot (a) while effectively leveraging the floating-point quantization grids.

Figure 3: The overall framework of BATQuant. Bottom: Integration of BATQuant into the Transformer architecture. Weight-side transformations are fused offline into the linear layers, while activation-side transformations are applied online. Top: Exemplary view of the Block-wise Affine Transformation , where inputs are partitioned into MXFP-aligned blocks. Each block transformation is decomposed via the Global and Private Kronecker .

Method	Decomposition	MatMul Complexity	# Params of $\mathbf{P}$	Example Count
FlatQuant	Kronecker	$\mathcal{O}(SN^{\frac{3}{2}})$	$2N$	8,192
Ours	w/o	$\mathcal{O}(SNg)$	$N\cdot g$	131,072
	Naive Kronecker	$\mathcal{O}(SN(g_{1}+g_{2}))$	$k\cdot(g_{1}^{2}+g_{2}^{2})$	10,240
	GPK	$\mathcal{O}(SN(g_{1}+g_{2}))$	$g_{1}^{2}+k\cdot g_{2}^{2}$	2,112

Table 1: Comparison of decomposition methods on parameter counts and computational cost. For the example parameter count, we set the hidden dim

N=4096

and the MXFP quantization granularity

g=32

. The size of decomposed matrix

\mathbf{A}_{i}

and

\mathbf{B}_{i}

are set to

g_{1}=8

and

g_{2}=4

. The reported MatMul Complexity refers to the computational cost of the activation transformation

\mathbf{X}\mathbf{P}

Bits	Method	MME	OCRBench	DocVQA	RealWorldQA	VLMBlind	Recovery(%)
BF16	–	2377	906	95.81	70.98	73.98	100.00
W4A8KV16	RTN	2294	883	94.72	69.80	70.99	97.43
	QuaRot	2327	870	95.07	69.80	71.12	97.53
	SpinQuant	2321	872	94.79	70.46	69.82	97.29
	BRQ	2329	865	94.72	70.19	67.18	96.40
	FlatQuant	2351	886	95.31	69.02	73.90	98.66

Table 2: Performance comparison of various quantization methods on multimodal benchmarks across different bit-width configurations (e.g., W4A8KV16, W4A4KV16, W4A8KV8 and W4A8KV4).The recovery rate relative to the BF16 baseline is also provided and the best result in each case is marked in bold.

Panel of Figure 4 — Figure 4: Performance comparison of different methods on Qwen3-8B across LLM benchmarks under various quantization configurations. The results are categorized into Non-Reasoning (left) and Reasoning (right) tasks.

Bits	Method	GSM8K	MATH-500	AIME24	AIME25	GPQA-D	Avg.	Recovery(%)
BF16	–	95.15	96.87	71.46	63.12	58.13	76.95	100.00
W4A8KV16	RTN	93.71	95.53	64.58	55.00	54.39	72.64	93.64
	QuaRot	94.47	95.67	64.17	55.63	54.39	72.87	93.91
	SpinQuant	94.69	95.53	60.42	51.46	54.58	71.34	91.62
	BRQ	93.71	95.80	63.96	53.33	55.40	72.39	93.26
	FlatQuant	94.62	95.93	69.17	57.08	54.80	74.32	95.99

Table 3: Performance comparison of various quantization methods on reasoning benchmarks across different bit-width configurations (e.g., W4A8KV16, W4A4KV16, W4A8KV8 and W4A8KV4).The recovery rate relative to the BF16 baseline is also provided and the best result in each case is marked in bold.

Panel of Figure 5 — Figure 5: Activation distributions of the q_proj module in layer 6 of Qwen3-8B with different quantization methods.

Model	Components		Non-Reasoning Benchmarks					Avg.
Model	Block Trans	Block Clip	ARC-C	ARC-E	HellaSwag	PIQA	Winogrande	Avg.
Qwen3-8B	✓		53.16	76.36	71.02	74.27	67.72	68.51
		✓	52.35	77.44	71.71	76.01	63.69	68.24
	✓	✓	53.33	77.53	71.12	75.30	66.22	68.70
Model	Components		Multimodal Benchmarks					Recovery
Model	Block Trans	Block Clip	MME	OCRBench	DocVQA	RealWorldQA	VLMBlind	(%)

Table 4: Ablation study of block-wise affine transformation and block-wise learnable clipping. We conduct the experiments under W4A4KV16 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Bits	Method	ARC-C	ARC-E	HellaSwag	PIQA	Winogrande	Avg.	Recovery(%)
BF16	–	56.48	81.06	74.96	77.69	68.03	71.64	100.00
W4A8KV16	RTN	55.72	80.81	73.29	77.09	66.93	70.77	98.75
	QuaRot	55.20	78.70	72.77	76.88	65.11	69.73	97.31
	SpinQuant	54.69	76.98	72.76	78.07	66.85	69.87	97.52
	BRQ	53.67	78.87	73.27	76.66	66.93	69.88	97.43
	FlatQuant	55.72	79.63	72.66	76.82	66.22	70.21	98.01

Appendix

Table 5: Performance comparison of various quantization methods on non-reasoning benchmarks across different bit-width configurations (e.g., W4A8KV16, W4A4KV16, W4A8KV8 and W4A8KV4).The recovery rate relative to the BF16 baseline is also provided and the best result in each case is marked in bold.

Bits	Method	Quantizer	Multimodal Benchmark					Recovery
Bits	Method	Quantizer	MME	OCRBench	DocVQA	RealWorldQA	VLMBlind	(%)
W4A8KV16	QuaRot	RTN	2201	814	93.11	65.36	63.21	91.43
	BRQ		2272	831	93.66	69.80	62.11	93.47
	FlatQuant		2311	880	94.65	66.14	67.96	95.64
	BATQuant		2312	877	94.58	66.80	69.27	96.11
	QuaRot	GPTQ	2327	870	95.07	69.80	71.12	97.53

Appendix

Table 6: Performance comparison of different quantization methods on multimodal benchmarks using RTN and GPTQ as weight quantizers. Bold indicates the best result within each quantizer setting (RTN or GPTQ) for a specific bit configuration.

Bits	Method	Quantizer	Non-Reasoning Benchmark					Avg.
Bits	Method	Quantizer	ARC-C	ARC-E	HellaSwag	PIQA	Winogrande	Avg.
W4A8KV16	QuaRot	RTN	51.37	75.76	70.04	76.61	65.67	67.89
	BRQ		47.44	72.87	71.37	75.84	65.19	66.54
	FlatQuant		55.63	78.83	72.46	76.22	66.85	70.00
	BATQuant		54.33	77.48	72.23	76.61	68.25	69.78
	QuaRot	GPTQ	55.20	78.70	72.77	76.88	65.11	69.73

Appendix

Table 7: Performance comparison of different quantization methods on LLM non-reasoning benchmarks using RTN and GPTQ as weight quantizers. Bold indicates the best result within each quantizer setting (RTN or GPTQ) for a specific bit configuration.

Figure 8: Activation distributions within different quantization blocks of the down_proj module in layer 35 of Qwen3-8B with RTN.

Figure 9: Activation distributions within different quantization blocks of the down_proj module in layer 35 of Qwen3-8B with BRQ.

Figure 10: Activation distributions within different quantization blocks of the down_proj module in layer 35 of Qwen3-8B with QuaRot.

Figure 11: Activation distributions within different quantization blocks of the down_proj module in layer 35 of Qwen3-8B with BATQuant (Ours).

Figure 12: Case study of Qwen3-VL-8B-Instruct on VLMBlind . The input includes a real image (shown above) and a text question asking to count intersection points. Compared with the BRQ method which fails by hallucinating an intersection (1), BATQuant correctly identifies that there are no intersections (0), matching the BF16 baseline.

Figure 13: Case study of Qwen3-VL-8B-Instruct on OCRBench . The input includes a real image of a train and a question asking for the train number. Compared with the BRQ method which fails by only recognizing partial information ("055"), BATQuant correctly identifies the full train number ("055 05995"), matching the BF16 baseline.

Explore similar work

DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantization
Apr 20, 2026Haokun Lin, Xinle Jia, Haobo Xu +7LLM Quantization 4-Bit Quantization
CASIA · CityU · NJU +3
TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization
May 19, 2026Zukang Xu, Xing Hu, Dawei YangLLM Quantization Rotation-Based Quantization
1Houmo AI.
FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling
Aug 3, 2026Xianglong Yan, Hong Liu, Chengzhu Bao +4LLM Quantization 4-Bit Quantization
Shanghai Jiao Tong University · Tencent

Apr 20, 2026J/K move · Enter open · S save

DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantization

Haokun Lin, Xinle Jia, Haobo Xu, Bingchen Yao+6

CASIA · CityU · NJU+3

LLM Quantization 4-Bit Quantization Rotation-Based Quantization Activation Quantization Neural Network Quantization

Open paper AskPDF arXiv

The MXFP4 microscaling format, which partitions tensors into blocks of 32 elements sharing an E8M0 scaling factor, has emerged as a promising substrate for efficient LLM inference, backed by native hardware support on NVIDIA Blackwell Tensor Cores. However, activation outliers pose a unique challenge under this format: a single outlier inflates the shared block scale, compressing the effective dynamic range of the remaining elements and causing significant quantization error. Existing rotation-based remedies, including randomized Hadamard and learnable rotations, are data-agnostic and therefore unable to specifically target the channels where outliers concentrate. We propose DuQuant++, which adapts the outlier-aware fine-grained rotation of DuQuant to the MXFP4 format by aligning the rotation block size with the microscaling group size (B{=}32). Because each MXFP4 group possesses an independent scaling factor, the cross-block variance issue that necessitates dual rotations and a zigzag permutation in the original DuQuant becomes irrelevant, enabling DuQuant++ to replace the entire pipeline with a single outlier-aware rotation, which halves the online rotation cost while simultaneously smoothing the weight distribution. Extensive experiments on the LLaMA-3 family under MXFP4 W4A4 quantization show that DuQuant++ consistently achieves state-of-the-art performance. Our code is available at https://github.com/Hsu1023/DuQuant-v2.

Abstract

Explore similar work

DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantization

TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization

FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling