Low-bit quantization offers a promising avenue for reducing the computational and memory demands of Multimodal Large Language Models (MLLMs). Recent hardware support for low-precision formats, ranging from MXFP8 to ultra-low-bit formats such as MXFP4 and HiF4, has accelerated research into efficient MLLM training and deployment. In this work, we present a systematic study of these quantization schemes in representative MLLMs that span both video generation and reasoning tasks. Our analysis shows that MXFP8 achieves near-lossless performance, whereas aggressive 4-bit quantization leads to significant degradation. Through extensive ablations, we identify activation quantization as the primary source of this performance loss, contributing substantially more than weight quantization. Motivated by this observation, we propose Residual Fallback Quantization (RFQ), a lightweight activation reconstruction framework that supplements the primary ulta-low-bit activation representation with an auxiliary quantized residual pathway. By explicitly modeling and compensating for quantization errors, RFQ improves activation fidelity while preserving the efficiency advantages of ultra-low-bit computation. RFQ requires no architectural modifications and incurs negligible computational overhead. Extensive experiments on Wan2.2 and Qwen3-VL demonstrate that RFQ consistently recovers a substantial portion of the performance lost under the quantization of MXFP4 and HiF4, significantly narrowing the gap to BF16 baselines across both generation and 4 reasoning benchmarks. Our findings establish activation quantization as the dominant bottleneck in ultra-low-bit MLLMs and highlight residual-based activation reconstruction as an effective and practical strategy for robust 4-bit deployment.
Figures & tables
Figure 1: Overview of the impact of activation outliers and the proposed RFQ framework. Activation outliers degrade MLLM performance under FP4 quantization. RFQ reduces outlier-induced quantization errors through residual fallback correction, recovering generation quality close to the BF16 baseline.
Model
MXFP8
W4A8
MXFP4
HiF4
Wan2.2
0.30
0.80
7.10
2.70
Qwen3-VL
0.20
0.70
7.23
3.80
Table 1: Relative training loss increase (%) for the BF16 baseline after QAT on models trained with over 5B tokens. MXFP8 and W4A8 show minimal degradation, while MXFP4 and HiF4 incur larger errors.
Figure 2: Cross-modal sensitivity analysis under ultra-low-bit quantization.
Figure 3: Comparative analysis of activation tail tracking and fallback interception across model modalities. Top Row highlights the unmitigated quantization noise and overflow elements in standard MXFP4 compared against the BF16 baseline. Bottom Row demonstrates how Residual Fallback Quantization (RFQ) restores fidelity by snapping extreme outliers.
Model
Module
Parameter Distribution (%)
Quant.(%)
Atten. Proj.
FFN/MLP
MoE
Other
Wan2.2
Visual
39.86
46.51
–
11.62
97.99
Textual
28.35
53.16
–
–
81.50
Qwen3-VL
Visual
26.61
49.71
–
22.77
99.09
Textual
2.96
–
94.95
–
97.91
Table 2: Parameter breakdown (%) by functional component with respect to the total paramter size. Quant. (%) for the visual and textual modules denotes the proportion of parameters quantized within each module. Projection layers and mergers are included in the “Other” category.
Method
Subject Consistency
Background Consistency
Imaging Quality
Temporal Flickering
Motion Smoothness
Dynamic Degree
Overall Consistency
Aesthetic Quality
Baseline (BF16)
95.74
96.77
65.32
98.42
99.25
45.00
7.03
59.43
MXFP4
95.65
96.60
66.69
98.10
99.15
43.00
7.24
58.93
(-0.09)
(-0.17)
(+1.37)
(-0.32)
(-0.10)
(-2.00)
(+0.21)
(-0.5)
MXFP4 + RFQ (ours)
95.43
96.54
66.48
98.29
99.23
50.00
6.90
59.26
(-0.31)
(-0.23)
(+1.16)
(-0.13)
(-0.02)
(+5.00)
(-0.13)
(-0.16)
HiF4
95.23
96.67
67.36
98.33
99.24
53.00
6.98
59.44
Table 3: For Wan2.2, quantitative comparison on VBench. Values in parentheses indicate the relative change (%) with respect to the BF16 baseline. Blue denotes improvement and red denotes degradation.
Method
RealWorldQA
MMStar
MMBenchEN
SimpleVQA
BF16
72.68
70.80
90.77
16.83
MXFP4
70.98
69.67
90.72
15.16
(-1.70)
(-1.13)
(-0.05)
(-1.67)
MXFP4 + RFQ (ours)
72.16
70.73
90.40
16.44
(-0.52)
(-0.07)
(-0.37)
(-0.39)
HiF4
72.42
71.27
90.50
15.35
Table 4: Accuracy comparison across different datasets and quantization formats on Qwen3-VL. Values in parentheses indicate the absolute change with respect to the BF16 baseline. Blue denotes improvement, while red indicates degradation.
Figure 4: Qualitative results for video generation. While 4-bit quantization methods ( i.e. , MXFP4 and HiF4) degrade generation quality, our proposed RFQ recovers visual fidelity and improves temporal consistency.
Figure 5: Training loss curves comparing low-bit FP4 strategies against the BF16 baseline across Qwen3-VL-30B and Wan2.2-5B.
Model
MXFP4
MXFP4 + RFQ (ours)
HiF4
HiF4 + RFQ (ours)
Wan2.2
7.50
1.14
2.89
0.61
Qwen3VL-30B
7.23
1.43
2.32
0.66
Table 5: Relative training loss increase (%) over the BF16 baseline after QAT with 4-bit quantization. RFQ substantially mitigates the degradation caused by MXFP4 and HiF4.
Large Language Models (LLMs) with multimodal capabilities have revolutionized vision-language tasks, but their deployment often requires huge memory and computational resources. Post-training quantization (PTQ) has successfully compressed language models to as low as 1-bit precision, its effectiveness for multimodal LLMs (MLLMs) remains unexplored. In this paper, we present the first method for ultra-low-bit (<4-bit) quantization of MLLMs. Our analysis reveals that multimodal tokens and intermediate layer activations produced by them exhibit significantly higher entropy compared to text tokens, indicating greater functional complexity that makes MLLMs less tolerant to ultra-low bit quantization. However, this entropy varies significantly across layers, with some layers producing lower-entropy activation distributions that we empirically show can better tolerate ultra-low bit quantization. Existing PTQ methods optimize weight quantization within each layer but apply the same target precision uniformly, ignoring this variation in complexity across layers. Building on this insight, we propose LUQ: Layerwise Ultra-Low Bit Quantization, which characterizes each transformer layer's functional complexity via its output activation entropy and selectively applies ultra-low bit quantization to layers encoding simpler, more compressible functions. We also show that multimodal calibration (image and text tokens) boosts VQA performance in the ultra-low bit regime. Evaluated on LLaVA-1.5 and Qwen-2.5-VL across 9 VQA benchmarks, LUQ models use 40% and 31% less memory than their 4-bit counterparts while exhibiting less than 10% degradation on MME.
Shubhang Bhatnagar, Andy Xu, Kar-Han Tan +1
University of Illinois Urbana-Champaign · Work done as an intern at HP Inc. · University of California, Los Angeles +1
Serving large language models (LLMs) under diverse deployment constraints requires flexible trade-offs between accuracy, memory footprint, and throughput. However, conventional quantization methods typically require a separate checkpoint for each target bit-width. We introduce Recurrent Residual Quantization (RRQ), a post-training quantization (PTQ) framework that represents weights as a low-bit quantized base together with a sequence of quantized residual corrections, enabling multiple effective precisions from a single checkpoint. Starting from a 2-bit model obtained via post-training quantization (PTQ) or round-to-nearest (RTN), RRQ progressively adds lightweight 2-bit residuals generated via RTN to construct 4-, 6-, and 8-bit representations. The method is calibration-free and avoids joint multi-bit optimization. In our Qwen3-8B setup, the full all-RTN 2-/4-/6-/8-bit package is constructed in 1,293 seconds, 3.3 times faster than the measured MatGPTQ construction. Experiments on six recent LLMs show competitive accuracy at 6 and 8 bits, with model-dependent behavior at 4 bits. The code will be made publicly available upon publication.
Low-bit activation quantization remains a major bottleneck in efficient large language model (LLM) deployment. The difficulty is not only that activations contain outliers, but that their distributions are often poorly matched to a low-bit uniform quantizer. Existing post-training quantization (PTQ) methods suppress peaks, balance channels, or minimize reconstruction error, yet they rarely specify what activation distribution is actually easy to discretize. As a result, activations may appear numerically smoother while still incurring large quantization error because the quantization range remains wide or most values collapse into a few levels near the mean. We recast activation transformation as quantizer-facing distribution design and analyze quantization error from an information-theoretic perspective. Our analysis shows that quantization-friendly activations should jointly have a smaller numerical range and sufficient dispersion within that range. Guided by this analysis, we propose InfoQuant, a train-free method that employs Peak Suppression Orthogonal Transformation (PSOT) to shape activations into more quantization-friendly distributions. We further introduce adaptive outlier-token selection to improve the robustness of PSOT during optimization. Across multiple LLM families, InfoQuant consistently outperforms prior PTQ and end-to-end training baselines. Under W4A4KV4, it preserves 97% of floating-point accuracy on average and reduces the LLaMA-2 13B performance gap by 42% over the previous state of the art. Code is available at https://github.com/LLIKKE/InfoQuant
Ke Li, Dong An, Xiaoling Zang +6
School of Software Technology, Zhejiang University · Ant Group · College of Computer Science and Technology, Zhejiang University of Technology +3