MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM

Dongwei Wang; Jinhee Kim; Seokho Han; Denis Gudovskiy; Yohei Nakata; Tomoyuki Okuno; KhayTze Peong; Kang Eun Jeon

cs.LGFeb 21, 2026

MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM

Authors: Dongwei Wang, Jinhee Kim, Seokho Han, Denis Gudovskiy, Yohei Nakata, Tomoyuki Okuno, KhayTze Peong, Kang Eun Jeon,

Organizations: University of Arizona · Panasonic AI Lab · Duke University · Sungkyunkwan University ·

Abstract

Dynamic runtime latency and memory constraints necessitate flexible large language model (LLM) deployment, where an LLM can be inferred with various quantization precisions based on available computational resources. Recent work on such any-precision quantization either relies on hardware-inefficient vector quantization or induces additional scaling factors when switching between bit-widths. Meanwhile, existing post-training quantization (PTQ) methods calibrated for a fixed low precision show poor generalizability under runtime precision change. In this work, we attribute the source of poor generalization across bit-widths to a precision-dependent outlier migration phenomenon where the distribution of PTQ-sensitive tokens changes across precisions. Motivated by this observation, we propose MoBiQuant, a novel any-precision Mixture-of-Bits quantization framework that adjusts weight precision for flexible LLM inference based on token sensitivity. Specifically, we propose a many-in-one recursive residual quantization that can iteratively reconstruct higher-precision weights at runtime and mitigates outlier migration with a token-aware router to dynamically select the optimal inference precision of each token.Extensive experiments show that MoBiQuant matches or surpasses frontier single-precision PTQ while exhibiting strong elasticity, achieving significant memory savings and throughput gains of up to $1.34\times$ over state-of-the-art any-precision methods.

Figures & tables

Figure 1: (Left) Existing PTQ methods poorly generalize under calibration–inference precision mismatch, e.g., inference at 4-bit with 3-bit calibrated parameters ( green vs. red ). Incorporating token-aware bit adjustment partially mitigates this degradation ( pink ). Our MoBiQuant improves generalization across precisions ( orange ). (Right) Per-token quantization error distribution at layer 5 in LLaMA3-8B: quantization outliers are highly non-uniform at each bit-width. Tokens well-fitted at 4-bit can become outliers at 3-bit, while 4-bit outliers may not be the primary error source at 3-bit.

Figure 2: MoBiQuant enables token-adaptive any-precision inference for linear blocks in LLMs (a). It consists of two main components: MoBiSlice that supports multiple precisions by decomposing FP16 weights into bit slices via recursive quantization (b); and MoBiRoute with token-adaptive precision adjustment that is trained to route each token to the optimal bit slice under a budget (c).

Figure 3: (a) AnyPrecisionLLM [ 22 ] with the inefficient bit-plane transposes and table lookups, (b) AnyBCQ [ 21 ] with the multi-scale storage and dequantization overhead, and (c) our MoBiQuant that performs direct bit-plane operations with the shared scales for efficient any-precision dequantization.

Figure 4: Any-precision inference performance comparison. MoBiQuant outperforms the PTQ baseline across multiple unseen target bit-widths on various LLaMA models, maintaining smooth and fine-grained precision scaling even at extremely low 2–3-bit precision range.

LLaMA	Quant.	WikiText2, PPL $\downarrow$					Throughput, tokens/sec. $\uparrow$
	Quant.	Static VQ		Any-precision			Static VQ		Any-precision
	bits	QUIP#	QTIP	AP	ABCQ	MoBiQ	QUIP#	QTIP	AP	ABCQ	MoBiQ
2-7B	2	12.3	5.86	2e3	15.38	10.91	191	218	279	312	404
	3	6.19	5.28	6.13	6.25	6.07	177	196	244	268	311
	4	5.66	5.17	6.05	5.91	5.82	155	189	211	225	256
3-8B	2	15.20	7.25	1e3	19.01	58.12	157	180	228	247	352

Table 1: End-to-end comparison with AnyPrecisionLLM (AP), AnyBCQ (ABCQ), QUIP# and QTIP. MoBiQuant achieves higher decoding throughput while maintaining a favorable performance.

Method	Type	LLaMA2-7B		LLaMA2-13B		LLaMA3.2-1B		LLaMA3.2-3B		LLaMA3-8B
Method	Type	3-bit	4-bit	3-bit	4-bit	3-bit	4-bit	3-bit	4-bit	3-bit	4-bit
FP16	—	5.5		5.0		13.4		9.2		6.1
SmoothQuant	Static SQ	8.62	7.50	5.43	6.10	2e2	2e2	3e2	1e2	12.50	10.70
AWQ		6.24	5.83	5.32	5.07	20.69	15.02	15.51	12.76	8.22	7.36
GPTQ		6.29	5.90	5.42	5.60	20.81	15.19	15.77	13.14	8.28	7.43
SpinQuant		6.13	5.60	5.24	5.00	19.62	14.45	14.73	11.17	8.14	6.40

Table 2: The proposed any-precision MoBiQuant matches or surpasses the static scalar quantization (SQ) PTQ baselines with the same number of average bits on WikiText2, PPL.

Figure 5: Visualization of outlier migration mitigation. MoBiQuant router scores correlate with the actual token-wise error increments under precision switching (left), and it produces more consistent token-wise error distributions across bit-widths with reduced outlier migration (right).

Figure 6: (Left) Average precision assignments of different linear blocks by MoBiQuant , showing block-level variation and higher deviations in initial layers. (Right) Precision distributions for any-precision inference, showing heterogeneous token assignments across target bit-widths.

Figure 7: Kernel evaluation of MoBiQuant . (Left) End-to-end decoding latency comparison. (Middle) Latency breakdown of MoBiQuant . (Right) Memory savings under multi-precision deployment.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Method	Perplexity (PPL $\downarrow$ )			Downstream Tasks (Acc $\uparrow$ )
Dataset	Method	Wiki	C4	PTB	ARC E	ARC E(n)	ARC C	ARC C(n)	BoolQ	Hella	Hella(n)	Wino
Wiki	Omni.	21.597	32.860	37.467	0.510	0.472	0.245	0.284	0.593	0.378	0.488	0.551
Wiki	MoBiQ	18.947	27.924	34.915	0.521	0.481	0.261	0.294	0.613	0.382	0.497	0.533
C4	Omni.	22.611	33.198	38.096	0.503	0.470	0.245	0.282	0.583	0.378	0.488	0.548
C4	MoBiQ	26.360	33.382	43.888	0.522	0.452	0.282	0.293	0.619	0.369	0.477	0.538
PTB	Omni.	22.689	33.519	35.353	0.503	0.471	0.241	0.281	0.586	0.376	0.488	0.556

Appendix

Table 3: Ablation study on different calibration datasets for LLaMA3.2-1B-3bit.

Figure 8: Comparison of PPL ( $\downarrow$ ) sweep between different scheduling methods across WikiText2, C4, PTB datasets.

Figure 9: Comparison of PPL ( $\downarrow$ ) sweep between different target bit-widths for training across WikiText2, C4, PTB datasets.

Calibration Bits	Infer @ 3-bit	Infer @ 4-bit
3-bit	8.22	7.85
4-bit	8.45	7.36

Appendix

Table 4: Generalization gap (PPL) of AWQ on LLaMA2-7B.

Method	Calib 3 $\rightarrow$ Infer 4	Calib 4 $\rightarrow$ Infer 3
OmniQuant	5.61	23.19
MoBiQuant	5.42	19.38

Appendix

Table 5: Mistral-7B performance gap (PPL) under calibration–inference mismatch.

Method	Calib 4 $\rightarrow$ Infer 4	Calib 4 $\rightarrow$ Infer 3
OmniQ	5.74	32.80
OmniQ + QuaRot	5.65	27.55
MoBiQuant + QuaRot	5.65	22.24

Appendix

Table 6: Generalization gap (PPL) of QuaRot on LLaMA2-7B.

	LLaMA2-7B			LLaMA3-8B
W-A	W3A4	W4A4	W5A4	W3A4	W4A4	W5A4
DuQuant	6.77	6.51	6.49	17.12	15.31	12.12
MoBiQuant + DuQuant	6.81	6.40	6.30	13.71	12.25	11.69

Appendix

Table 7: Generalization performance of DuQuant and MoBiQuant on unseen W-A configurations.

Figure 10: PPL ( $\downarrow$ ) tradeoff under 4 bit activation quantization on LLaMA2-13B. We compare with SmoothQuant [ 32 ] and OmniQuant [ 25 ] W4A4 baselines.

Method	Avg. Bits	Type	LLaMA-2-7B	LLaMA-2-13B	LLaMA-3.2-1B	LLaMA-3.2-3B	LLaMA-3-8B
			0-shot	0-shot	0-shot	0-shot	0-shot
			Avg.( $\uparrow$ )	Avg.( $\uparrow$ )	Avg.( $\uparrow$ )	Avg.( $\uparrow$ )	Avg.( $\uparrow$ )
Floating-point	16	-	69.68	73.19	59.42	66.71	74.19
RTN	4	Static SQ	64.72	64.19	-	-	70.40
SmoothQuant			63.87	67.46	-	-	67.47
AWQ			69.40	72.64	-	-	72.64

Appendix

Table 8: Zero-shot comparison with static scalar quantization (SQ) methods. We report average accuracy over six zero-shot commonsense reasoning tasks.MoBiQuant matches or surpasses static PTQ baselines. “-” denotes that no direct results are reported in the published papers.

Method	Flexible-Extract	Strict-Match
FP16	46.47%	42.15%
OmniQuant-4bit	30.86%	30.86%
Ours (Elastic)	32.83%	32.45%

Appendix

Table 9: GSM8K performance of LLaMA3.2-1B under 4-bit setting.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 381 papers

MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models
Jul 25, 2026Ashitabh Misra, Madhav Agrawal, Arham Jain +1Mixed-Precision Quantization LLM Quantization
University of Illinois at Urbana-Champaign
Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training
Oct 6, 2026Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2LLM Quantization Quantization-Aware Training
ISTA · EPFL, ISTA
Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
Aug 4, 2026Yu Luo, Bo Dong, Wenhua Cheng +1Mixed-Precision Quantization LLM Quantization
Intel

Jul 25, 2026J/K move · Enter open · S save

MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models

Ashitabh Misra, Madhav Agrawal, Arham Jain, Tarek Abdelzaher

University of Illinois at Urbana-Champaign

Mixed-Precision Quantization LLM Quantization Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

Mixed-precision quantization improves the accuracy of post-training quantization by allocating higher bitwidths to sensitive layers, but existing methods solve the allocation for a single fixed memory budget. In practice the budget varies across deployments and is unknown at calibration time. Adaptive quantization addresses this with one offline calibration that serves any budget, yet current methods score layer sensitivity in a manner that does not consider its dependency on quantization levels of other layers. We show that a layer's sensitivity depends strongly on the bitwidths of its upstream layers and that this dependence shifts the resulting preferred bit allocation. We propose MixQuant, a technique-agnostic adaptive framework that wraps any base quantizer. MixQuant marginalizes each layer's distortion over random quantized upstream configurations to obtain budget-agnostic scores, calibrates the quantizer's parameters on plans the allocator itself produces, and penalizes allocations that leave layers at the lowest bitwidths. A single greedy pass then serves any budget at deployment. Across Llama-3.2-3B, Llama-2-7B, and Mistral-7B under AWQ and GPTQ, MixQuant outperforms adaptive and mixed-precision baselines in every setting, improving average accuracy by up to 8 points and reducing perplexity from 12.43 to 10.70 at the tightest budget, while matching an ILP solver at negligible deployment cost.

Abstract

LLM Quantization

Explore similar work

MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models

Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs