MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Models

Lulu Hu; Wenhu Xiao; Xin Chen; Xinhua Xu; Bowen Xu; Kun Li; Yongliang Tao

cs.CVMar 5, 2026

MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Models

Authors: Lulu Hu, Wenhu Xiao, Xin Chen, Xinhua Xu, Bowen Xu, Kun Li, Yongliang Tao

Organizations: Alibaba Cloud Computing, Alibaba Group

Abstract

Post-training quantization (PTQ) with computational invariance for Large Language Models~(LLMs) have demonstrated remarkable advances, however, their application to Multimodal Large Language Models~(MLLMs) presents substantial challenges. In this paper, we analyze SmoothQuant as a case study and identify two critical issues: Smoothing Misalignment and Cross-Modal Computational Invariance. To address these issues, we propose Modality-Aware Smoothing Quantization (MASQuant), a novel framework that introduces (1) Modality-Aware Smoothing (MAS), which learns separate, modality-specific smoothing factors to prevent Smoothing Misalignment, and (2) Cross-Modal Compensation (CMC), which addresses Cross-modal Computational Invariance by using SVD whitening to transform multi-modal activation differences into low-rank forms, enabling unified quantization across modalities. MASQuant demonstrates stable quantization performance across both dual-modal and tri-modal MLLMs. Experimental results show that MASQuant is competitive among the state-of-the-art PTQ algorithms. Source code: https://github.com/alibaba/EfficientAI.

Figures & tables

Figure 1: (a). Activation distributions during multimodal reasoning in MLLMs. Different dominant modalities emerge across MLLM components, leading to failure of general PTQ methods that diminish vision importance. (b). Impact of SmoothQuant’s uniform smoothing factors $S$ computation on MLLM quantization performance (low SQNR, high PPL). (c). MASQuant addresses smoothing misalignment through the combination of MAS and CMC, thereby significantly enhancing PTQ performance in MLLMs. MBR Loss indicates Modality Balanced Reconstruction Loss.

Figure 2: Comparative analysis of SQNR degradation of Qwen2.5-Omni-3B under multimodal input condition. We selected 32 samples from OmniBench and computed the average SQNR for each layer.

Figure 3: The illustrated case demonstrates a text-vision dual-modal setting. (a) Schematic workflow of MAS and CMC with calibration data, (b) Illustration of how low-rank matrices L1 and L2 in CMC are utilized in MASQuant, exemplified with an MLP block.

Method		Qwen2.5-VL-3B						Qwen2.5-VL-7B
	Bits	MMMU	OCR	Viz	S-QA	T-VQA	Avg	MMMU	OCR	Viz	S-QA	T-VQA	Avg
		Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$
Dense	FP16	42.2	79.3	69.1	81.9	77.9	70.1	46.7	83.8	70.8	88.4	82.9	74.5
RTN	W4A16	40.9	77.9	63.0	81.3	75.8	67.8	43.3	83.7	67.8	81.3	82.1	71.6
AWQ		41.9	78.6	68.1	81.7	77.2	69.5	43.3	83.7	70.6	87.8	82.2	73.5
MBQ		41.9	78.5	68.0	81.5	76.8	69.3	44.4	82.8	70.6	87.8	82.9	73.7

Table 1: Comparison of MASQuant with existing quantization methods on multimodal benchmarks. OCR: OCRBench. Viz: Vizwiz. S-QA: ScienceQA. T-VQA: TextVQA. SQ: SmoothQuant. The best results are highlighted in bold .

		Qwen2.5-Omni-3B				Qwen2.5-Omni-7B
Method	Bits	Audio-Text		Vision-Text	Vision-Audio-Text	Audio-Text		Vision-Text	Vision-Audio-Text
		Libri	Wen	MMMU	Omnibench	Libri	Wen	MMMU	Omnibench
		WER $\downarrow$	WER $\downarrow$	Acc $\uparrow$	Acc $\uparrow$	WER $\downarrow$	WER $\downarrow$	Acc $\uparrow$	Acc $\uparrow$
Dense	FP16	3.9	7.5	43.3	43.8	2.9	7.1	50.0	45.3
RTN	W4A16	4.4	9.2	32.2	39.6	3.4	6.7	44.4	43.8
AWQ		9.4	8.0	36.7	43.8	8.7	7.1	47.8	45.3

Table 2: Comparison of MASQuant with existing quantization methods on omni-modal MLLMs (vision, audio and text). SQ: SmoothQuant. Libri: Librispeech. Wen: Wenetspeech. The best results are highlighted in bold .

Panel of Figure 4 — Figure 4: Percentage of unified smoothing factors from different modalities using SmoothQuant on Omni and VL MLLMs.

Panel of Figure 5 — Figure 5: Effective ranks of $\Delta\mathbf{W}$ is reduced across layers after SVD-based Whitening b SQNR improves as the rank ratio increases on both Qwen2.5-VL-3B and Qwen2.5-Omni-3B.

		Omni-3B		VL-3B
Method	Opt.	Libri	OmniB.	T-VQA.	MMMU
Uniform		77.4	27.3	51.3	25.6
MAS		3.8	36.7	54.7	28.9
Uniform	✓	6.0	33.3	65.0	33.3
MAS	✓	3.6	47.7	68.2	46.7

Table 3: Effects of Modality-Aware Smoothing (W4A8).

		Qwen2.5-VL-3B
$\lambda_{t}$	$\lambda_{v}$	PPL	MMMU	T-VQA.	S-QA	Avg
1.0	1.0	17.2	37.8	55.2	77.8	56.9
1.0	0.5	17.8	30.0	57.8	80.0	55.9
1.0	0.1	33.7	32.2	65.4	76.7	58.1
0.5	1.0	17.3	33.3	53.9	68.9	52.0

Table 4: Effects of Modality Loss Weight (W4A8).

	Qwen2.5-VL-3B
Epoch	PPL	MMLU	T-VQA.	S-QA	Avg
1	23.9	38.9	52.2	79.5	56.9
2	18.6	37.8	67.5	78.4	61.2
5	17.6	34.4	65.9	77.4	59.2
10	17.0	37.8	68.2	76.5	60.8

Table 5: Effects of Modality-Aware Smoothing (W4A8).

Base modality	Computation	Memory
Text	$d$	0
Others	$d$ + $2rd$	$2mrd$

Table 6: Computation cost and memory at decoding phase, where

d

is hidden size,

r

is rank, and

m

is the number of extra modalities.

Figure 6: SQNR as a function of rank ratio for CMC on Qwen2.5-Omni-3B under W4A6 quantization. MAS applies independent modality-specific smoothing. MBQ employs a uniform factor optimized by modality balance reconstruction.

BS	Method	Rank	Prefill	Speedup	Mem	Mem
		Ratio	(ms)		(GB)	Saving
	FP16	-	191.82	/	13.73	/
1	MBQ	-	68.65	2.79 $\times$	4.85	2.83 $\times$
	MAS	0.01	71.62	2.67 $\times$	4.97	2.76 $\times$
	MAS	0.02	72.89	2.63 $\times$	5.04	2.73 $\times$
	MAS	0.05	77.10	2.49 $\times$	5.37	2.56 $\times$

Table 7: End-to-end prefill-stage performance of Qwen2.5-VL-7B on Desktop RTX 4090 with fused GPU kernels (sequence length = 2048) under W4A4 setting. MAS: MASQuant. BS: Batch Size.

Post-Training Quantization

Post-Training Quantization compresses trained neural networks into low-bit weights, activations, or caches to reduce memory use and inference cost.

See all 360 papers

MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Models
Jun 3, 2026Yue Wu, Changyuan Wang, Zixuan Wang +2LLM Quantization 4-Bit Quantization
LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
Sep 28, 2025Shubhang Bhatnagar, Andy Xu, Kar-Han Tan +1Mixed-Precision Quantization LLM Quantization
University of Illinois Urbana-Champaign · Work done as an intern at HP Inc. · University of California, Los Angeles +1
QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Models
Apr 3, 2026Xinhao Wang, Zhonyu Xia, Zhiwei Lin +2Visual Token Pruning LVLM Pruning
Wangxuan Institute of Computer Technology, Peking University

Jun 3, 2026J/K move · Enter open · S save

MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Models

Yue Wu, Changyuan Wang, Zixuan Wang, Shilin Ma+1

LLM Quantization 4-Bit Quantization Multimodal Large Language Models Activation Quantization Post-Training Quantization

Open paper AskPDF arXiv

Conventional Post-Training Quantization (PTQ) methods struggle with 4-bit Omni-modal Large Language Models (OLLMs) due to the extreme distribution heterogeneity and disparate outlier patterns across modalities. To address this, we propose MorphoQuant, a modality-aware PTQ framework engineered to preserve cross-modal morphology and mitigate outlier loss. Specifically, we introduce Distribution-Aware Bias Compensation (DABC), which selectively absorbs long-tailed outliers into channel-wise biases. This mechanism safeguards outlier magnitudes while maintaining high-precision discretization for dense inliers, thereby preserving accurate discretization across diverse modal distribution. Complementing this, we propose Morphology-Directed Quantization Function Optimization (MDQFO) to co-optimize the quantization grid with the bias mask, ensuring fine-grained alignment across modalities. Extensive evaluations on Qwen2.5-Omni across benchmarks like MMMU and Video-MME demonstrate our approach's superiority. Notably, our W4A4 model achieves 76.63% on ScienceQA, significantly outperforming SOTA W4A4 methods and surprisingly surpassing the W4A16 baseline, which fully demonstrates the exceptional accuracy-efficiency trade-off of our framework.

Abstract

Post-Training Quantization

Explore similar work

MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Models

LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models

QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Models