MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM
Authors: Dongwei Wang, Jinhee Kim, Seokho Han, Denis Gudovskiy, Yohei Nakata, Tomoyuki Okuno, KhayTze Peong, Kang Eun Jeon, Jong Hwan Ko, Yiran Chen, Huanrui Yang
Organizations: University of Arizona · Panasonic AI Lab · Duke University · Sungkyunkwan University · Korea Advanced Institute of Science and Technology
Dynamic runtime latency and memory constraints necessitate flexible large language model (LLM) deployment, where an LLM can be inferred with various quantization precisions based on available computational resources. Recent work on such any-precision quantization either relies on hardware-inefficient vector quantization or induces additional scaling factors when switching between bit-widths. Meanwhile, existing post-training quantization (PTQ) methods calibrated for a fixed low precision show poor generalizability under runtime precision change. In this work, we attribute the source of poor generalization across bit-widths to a precision-dependent outlier migration phenomenon where the distribution of PTQ-sensitive tokens changes across precisions. Motivated by this observation, we propose MoBiQuant, a novel any-precision Mixture-of-Bits quantization framework that adjusts weight precision for flexible LLM inference based on token sensitivity. Specifically, we propose a many-in-one recursive residual quantization that can iteratively reconstruct higher-precision weights at runtime and mitigates outlier migration with a token-aware router to dynamically select the optimal inference precision of each token.Extensive experiments show that MoBiQuant matches or surpasses frontier single-precision PTQ while exhibiting strong elasticity, achieving significant memory savings and throughput gains of up to 1.34× over state-of-the-art any-precision methods.
Figures & tables
Figure 1: (Left) Existing PTQ methods poorly generalize under calibration–inference precision mismatch, e.g., inference at 4-bit with 3-bit calibrated parameters ( green vs. red ). Incorporating token-aware bit adjustment partially mitigates this degradation ( pink ). Our MoBiQuant improves generalization across precisions ( orange ). (Right) Per-token quantization error distribution at layer 5 in LLaMA3-8B: quantization outliers are highly non-uniform at each bit-width. Tokens well-fitted at 4-bit can become outliers at 3-bit, while 4-bit outliers may not be the primary error source at 3-bit.
Figure 2: MoBiQuant enables token-adaptive any-precision inference for linear blocks in LLMs (a). It consists of two main components: MoBiSlice that supports multiple precisions by decomposing FP16 weights into bit slices via recursive quantization (b); and MoBiRoute with token-adaptive precision adjustment that is trained to route each token to the optimal bit slice under a budget (c).
Figure 3: (a) AnyPrecisionLLM [ 22 ] with the inefficient bit-plane transposes and table lookups, (b) AnyBCQ [ 21 ] with the multi-scale storage and dequantization overhead, and (c) our MoBiQuant that performs direct bit-plane operations with the shared scales for efficient any-precision dequantization.
Figure 4: Any-precision inference performance comparison. MoBiQuant outperforms the PTQ baseline across multiple unseen target bit-widths on various LLaMA models, maintaining smooth and fine-grained precision scaling even at extremely low 2–3-bit precision range.
LLaMA
Quant.
WikiText2, PPL ↓
Throughput, tokens/sec. ↑
Static VQ
Any-precision
Static VQ
Any-precision
bits
QUIP#
QTIP
AP
ABCQ
MoBiQ
QUIP#
QTIP
AP
ABCQ
MoBiQ
2-7B
2
12.3
5.86
2e3
15.38
10.91
191
218
279
312
404
3
6.19
5.28
6.13
6.25
6.07
177
196
244
268
311
4
5.66
5.17
6.05
5.91
5.82
155
189
211
225
256
3-8B
2
15.20
7.25
1e3
19.01
58.12
157
180
228
247
352
Table 1: End-to-end comparison with AnyPrecisionLLM (AP), AnyBCQ (ABCQ), QUIP# and QTIP. MoBiQuant achieves higher decoding throughput while maintaining a favorable performance.
Method
Type
LLaMA2-7B
LLaMA2-13B
LLaMA3.2-1B
LLaMA3.2-3B
LLaMA3-8B
3-bit
4-bit
3-bit
4-bit
3-bit
4-bit
3-bit
4-bit
3-bit
4-bit
FP16
—
5.5
5.0
13.4
9.2
6.1
SmoothQuant
Static SQ
8.62
7.50
5.43
6.10
2e2
2e2
3e2
1e2
12.50
10.70
AWQ
6.24
5.83
5.32
5.07
20.69
15.02
15.51
12.76
8.22
7.36
GPTQ
6.29
5.90
5.42
5.60
20.81
15.19
15.77
13.14
8.28
7.43
SpinQuant
6.13
5.60
5.24
5.00
19.62
14.45
14.73
11.17
8.14
6.40
Table 2: The proposed any-precision MoBiQuant matches or surpasses the static scalar quantization (SQ) PTQ baselines with the same number of average bits on WikiText2, PPL.
Figure 5: Visualization of outlier migration mitigation. MoBiQuant router scores correlate with the actual token-wise error increments under precision switching (left), and it produces more consistent token-wise error distributions across bit-widths with reduced outlier migration (right).
Figure 6: (Left) Average precision assignments of different linear blocks by MoBiQuant , showing block-level variation and higher deviations in initial layers. (Right) Precision distributions for any-precision inference, showing heterogeneous token assignments across target bit-widths.
Figure 7: Kernel evaluation of MoBiQuant . (Left) End-to-end decoding latency comparison. (Middle) Latency breakdown of MoBiQuant . (Right) Memory savings under multi-precision deployment.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Method
Perplexity (PPL ↓ )
Downstream Tasks (Acc ↑ )
Wiki
C4
PTB
ARC E
ARC E(n)
ARC C
ARC C(n)
BoolQ
Hella
Hella(n)
Wino
Wiki
Omni.
21.597
32.860
37.467
0.510
0.472
0.245
0.284
0.593
0.378
0.488
0.551
MoBiQ
18.947
27.924
34.915
0.521
0.481
0.261
0.294
0.613
0.382
0.497
0.533
C4
Omni.
22.611
33.198
38.096
0.503
0.470
0.245
0.282
0.583
0.378
0.488
0.548
MoBiQ
26.360
33.382
43.888
0.522
0.452
0.282
0.293
0.619
0.369
0.477
0.538
PTB
Omni.
22.689
33.519
35.353
0.503
0.471
0.241
0.281
0.586
0.376
0.488
0.556
Appendix
Table 3: Ablation study on different calibration datasets for LLaMA3.2-1B-3bit.
Figure 8: Comparison of PPL ( ↓ ) sweep between different scheduling methods across WikiText2, C4, PTB datasets.
Figure 9: Comparison of PPL ( ↓ ) sweep between different target bit-widths for training across WikiText2, C4, PTB datasets.
Calibration Bits
Infer @ 3-bit
Infer @ 4-bit
3-bit
8.22
7.85
4-bit
8.45
7.36
Appendix
Table 4: Generalization gap (PPL) of AWQ on LLaMA2-7B.
Method
Calib 3 → Infer 4
Calib 4 → Infer 3
OmniQuant
5.61
23.19
MoBiQuant
5.42
19.38
Appendix
Table 5: Mistral-7B performance gap (PPL) under calibration–inference mismatch.
Method
Calib 4 → Infer 4
Calib 4 → Infer 3
OmniQ
5.74
32.80
OmniQ + QuaRot
5.65
27.55
MoBiQuant + QuaRot
5.65
22.24
Appendix
Table 6: Generalization gap (PPL) of QuaRot on LLaMA2-7B.
LLaMA2-7B
LLaMA3-8B
W-A
W3A4
W4A4
W5A4
W3A4
W4A4
W5A4
DuQuant
6.77
6.51
6.49
17.12
15.31
12.12
MoBiQuant + DuQuant
6.81
6.40
6.30
13.71
12.25
11.69
Appendix
Table 7: Generalization performance of DuQuant and MoBiQuant on unseen W-A configurations.
Figure 10: PPL ( ↓ ) tradeoff under 4 bit activation quantization on LLaMA2-13B. We compare with SmoothQuant [ 32 ] and OmniQuant [ 25 ] W4A4 baselines.
Method
Avg. Bits
Type
LLaMA-2-7B
LLaMA-2-13B
LLaMA-3.2-1B
LLaMA-3.2-3B
LLaMA-3-8B
0-shot
0-shot
0-shot
0-shot
0-shot
Avg.( ↑ )
Avg.( ↑ )
Avg.( ↑ )
Avg.( ↑ )
Avg.( ↑ )
Floating-point
16
-
69.68
73.19
59.42
66.71
74.19
RTN
4
Static SQ
64.72
64.19
-
-
70.40
SmoothQuant
63.87
67.46
-
-
67.47
AWQ
69.40
72.64
-
-
72.64
Appendix
Table 8: Zero-shot comparison with static scalar quantization (SQ) methods. We report average accuracy over six zero-shot commonsense reasoning tasks.MoBiQuant matches or surpasses static PTQ baselines. “-” denotes that no direct results are reported in the published papers.
Method
Flexible-Extract
Strict-Match
FP16
46.47%
42.15%
OmniQuant-4bit
30.86%
30.86%
Ours (Elastic)
32.83%
32.45%
Appendix
Table 9: GSM8K performance of LLaMA3.2-1B under 4-bit setting.
Mixed-precision quantization improves the accuracy of post-training quantization by allocating higher bitwidths to sensitive layers, but existing methods solve the allocation for a single fixed memory budget. In practice the budget varies across deployments and is unknown at calibration time. Adaptive quantization addresses this with one offline calibration that serves any budget, yet current methods score layer sensitivity in a manner that does not consider its dependency on quantization levels of other layers. We show that a layer's sensitivity depends strongly on the bitwidths of its upstream layers and that this dependence shifts the resulting preferred bit allocation. We propose MixQuant, a technique-agnostic adaptive framework that wraps any base quantizer. MixQuant marginalizes each layer's distortion over random quantized upstream configurations to obtain budget-agnostic scores, calibrates the quantizer's parameters on plans the allocator itself produces, and penalizes allocations that leave layers at the lowest bitwidths. A single greedy pass then serves any budget at deployment. Across Llama-3.2-3B, Llama-2-7B, and Mistral-7B under AWQ and GPTQ, MixQuant outperforms adaptive and mixed-precision baselines in every setting, improving average accuracy by up to 8 points and reducing perplexity from 12.43 to 10.70 at the tightest budget, while matching an ILP solver at negligible deployment cost.
Quantization-aware training (QAT) leverages lower-precision arithmetic to reduce the cost of LLM deployment, but aggressive quantization degrades final model performance. A common remedy is mixed-precision training, in which high precision is assigned to some of the layers to maintain performance while keeping the cost constrained. This approach then requires precision assignments for model layers during training. We provide a new approach, called Q-PACE, consisting of a second-order sensitivity model that predicts the loss increase as a sum of quantization noise MSE weighted by per-layer curvature coefficients. During training, we periodically re-compute these coefficients using perturbations across layers, and re-assign precision. Pretraining and supervised fine-tuning experiments on LLMs of up to 4B parameters show that Q-PACE consistently improves over existing mixed-precision training recipes, and achieves comparable loss at substantially lower total memory budgets. We further find that quantization sensitivity is highly predictable by depth and layer type, and its stability during training allows for infrequent, cheap recalibration.
Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2
Serving large language models (LLMs) under diverse deployment constraints requires flexible trade-offs between accuracy, memory footprint, and throughput. However, conventional quantization methods typically require a separate checkpoint for each target bit-width. We introduce Recurrent Residual Quantization (RRQ), a post-training quantization (PTQ) framework that represents weights as a low-bit quantized base together with a sequence of quantized residual corrections, enabling multiple effective precisions from a single checkpoint. Starting from a 2-bit model obtained via post-training quantization (PTQ) or round-to-nearest (RTN), RRQ progressively adds lightweight 2-bit residuals generated via RTN to construct 4-, 6-, and 8-bit representations. The method is calibration-free and avoids joint multi-bit optimization. In our Qwen3-8B setup, the full all-RTN 2-/4-/6-/8-bit package is constructed in 1,293 seconds, 3.3 times faster than the measured MatGPTQ construction. Experiments on six recent LLMs show competitive accuracy at 6 and 8 bits, with model-dependent behavior at 4 bits. The code will be made publicly available upon publication.