CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Experts

Xiangyang Yin; Xingyu Liu; Tianhua Xia; Bo Bao; Vithursan Thangarasa; Valavan Manohararajah; Eric Sather; Sai Qian Zhang

cs.LGApr 12, 2026

CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Experts

Authors: Xiangyang Yin, Xingyu Liu, Tianhua Xia, Bo Bao, Vithursan Thangarasa, Valavan Manohararajah, Eric Sather, Sai Qian Zhang

Organizations: Courant Institute of Mathematical Sciences, New York University · Tandon School of Engineering, New York University · Cerebras Systems Inc.

Abstract

Outliers have emerged as a fundamental bottleneck in preserving accuracy for low-precision large models, particularly within Mixture-of-Experts (MoE) architectures that are increasingly central to large-scale language modeling. Under post-training quantization (PTQ), these outliers induce substantial quantization errors, leading to severe accuracy degradation. While recent rotation-based smoothing techniques alleviate the problem by redistributing outlier magnitudes, residual errors remain and continue to impede reliable low-precision deployment. In this work, we tackle this challenge by introducing CodeQuant, a unified quantization-and-clustering scheme that contains smoothing activation outliers via learnable rotation and absorbing weight outliers into fine-tuned cluster centroids for MoE. This design reduces the influence of extreme values by fitting them within cluster centroids, thereby lowering quantization error while maintaining expressive capacity. Coupled with a dedicated kernel design for GPU and CPU, CodeQuant achieves up to $4.15\times$ speedup while delivering significantly higher accuracy than state-of-the-art quantization approaches across diverse MoE models. Our results highlight CodeQuant as a promising direction for efficient and accurate deployment of MoE-based large language models under low-precision constraints. Our code is available at https://github.com/SAI-Lab-NYU/CodeQuant.

Figures & tables

Figure 1: Overview of the CodeQuant framework. The left panel illustrates the target architectures, including MoE FFN and Self-Attention blocks. The right panel depicts the four-stage calibration and deployment pipeline: Stage 1 applies learnable rotations to smooth activation outliers; Stage 2 permute weight for optimized distribution; Stage 3 introduces clustering fine-tune mechanism to align with objective; and Stage 4 deploys the quantized model using a specialized LUT kernel.

Figure 2: FFN layers within MoE is applied with rotational matrices for outlier smoothing.

Figure 3: The overview of the POG framework.

Figure 4: (a) One tile of the matrix multiplication. (b) The steps of CodeQuant kernel, including a one-time lookup table precomputation and table lookup. (c) The precomputed lookup table is stored in the shared memory in the Streaming Multiprocessors (SM) in GPU.

	Models	Methods	Wiki2 ( $\downarrow$ )	C4 ( $\downarrow$ )	A-c ( $\uparrow$ )	A-e ( $\uparrow$ )	HS ( $\uparrow$ )	ML ( $\uparrow$ )	PQ ( $\uparrow$ )	WG ( $\uparrow$ )	Avg ( $\uparrow$ )
A4W4 Embedding-wise	Phi-mini- MoE-Instruct	BF16	6.83	13.06	0.581	0.813	0.759	0.681	0.797	0.753	0.731
		RTN	9811.22	7431.27	0.287	0.268	0.261	0.232	0.501	0.516	0.344
		SqueezeLLM	8383.63	5619.01	0.279	0.281	0.263	0.236	0.515	0.500	0.346
		SmoothQuant	24071.25	16320.79	0.263	0.280	0.270	0.240	0.528	0.503	0.347
		QuaRot	7.93	14.44	0.545	0.784	0.725	0.633	0.775	0.702	0.694
		CodeQuant	7.63	13.94	0.538	0.790	0.728	0.644	0.784	0.716	0.700

Table 1: Performance in perplexity (PPL) on Wiki2 and C4 dataset, and accuracy on Arc-Challenge (A-c), Arc-easy (A-e), HellaSwag (HS), MMLU (ML), PIQA (PQ) and WinoGrande (WG). For each setting, we report the BF16 baseline in the first row. More results are shown in the Appendix.

	Models	Methods	Wiki2 ( $\downarrow$ )	C4 ( $\downarrow$ )	A-c ( $\uparrow$ )	A-e ( $\uparrow$ )	HS ( $\uparrow$ )	ML ( $\uparrow$ )	PQ ( $\uparrow$ )	WG ( $\uparrow$ )	Avg ( $\uparrow$ )
A4W4 Embedding-wise	DeepSeek- V2-Lite	DuQuant	8.43	11.94	0.455	0.708	0.623	0.400	0.775	0.693	0.624
		$\text{SpinQuant}_{had}$	9.24	12.71	0.427	0.692	0.706	0.425	0.774	0.638	0.610
		$\text{CodeQuant}_{had}$	8.16	11.38	0.445	0.723	0.727	0.454	0.782	0.644	0.629
	Qwen3- 30B-A3B	DuQuant	13.52	20.10	0.472	0.662	0.687	0.654	0.739	0.606	0.637
		$\text{SpinQuant}_{had}$	14.61	22.07	0.415	0.600	0.628	0.584	0.692	0.622	0.590
		$\text{CodeQuant}_{had}$	12.69	19.89	0.477	0.697	0.691	0.679	0.739	0.635	0.653

Table 2: Rotation-based method performance comparison.

\text{CodeQuant}_{had}

indicates that online Hadamard transforms are enabled during the quantization process.

Models	Methods	GSM8K ( $\uparrow$ )	MATH500 ( $\uparrow$ )
DeepSeek-V2-Lite	BF16	0.364	0.121
	QuaRot	0.231	0.093
	CodeQuant	0.330	0.108
Qwen3-30B-A3B	BF16	0.924	0.322
	QuaRot	0.508	0.128
	CodeQuant	0.867	0.241

Table 3: A4W4 Embedding-wise results on GSM8K (8-shot) and MATH500 (4-shot).

Figure 5: Normalized speedup on one A100 GPU.

Table 9

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	Models	Methods	Wiki2 ( $\downarrow$ )	C4 ( $\downarrow$ )	A-c ( $\uparrow$ )	A-e ( $\uparrow$ )	HS ( $\uparrow$ )	ML ( $\uparrow$ )	PQ ( $\uparrow$ )	WG ( $\uparrow$ )	Avg ( $\uparrow$ )
A8W4 Embedding-wise	Phi-mini- MoE-Instruct	BF16	6.83	13.06	0.581	0.813	0.759	0.681	0.797	0.753	0.731
		RTN	12.13	20.46	0.460	0.730	0.618	0.497	0.741	0.632	0.613
		SqueezeLLM	7.41	13.65	0.565	0.795	0.736	0.658	0.791	0.746	0.715
		SmoothQuant	9.50	16.23	0.481	0.741	0.653	0.569	0.756	0.638	0.634
		QuaRot	7.69	14.15	0.549	0.787	0.735	0.652	0.786	0.737	0.708
		CodeQuant	7.36	13.73	0.579	0.796	0.741	0.668	0.796	0.732	0.719

Appendix

Table 7: Performance in perplexity (PPL) on Wiki2 and C4 dataset, and accuracy on Arc-Challenage (A-c), Arc-easy (A-e), HellaSwag (HS), MMLU (ML), PIQA (PQ) and WinoGrande (WG). CodeQuant is set as A8W4 Embedding-wise. We report the BF16 baseline in the first row, and mark the methods as BF16.

	Models	Methods	Wiki2 ( $\downarrow$ )	C4 ( $\downarrow$ )	A-c ( $\uparrow$ )	A-e ( $\uparrow$ )	HS ( $\uparrow$ )	ML ( $\uparrow$ )	PQ ( $\uparrow$ )	WG ( $\uparrow$ )	Avg ( $\uparrow$ )
A8W4 Embedding-wise	DeepSeek- V2-Lite	DuQuant	8.43	11.94	0.455	0.723	0.728	0.406	0.815	0.665	0.632
		$\text{SpinQuant}_{had}$	8.62	11.95	0.431	0.696	0.726	0.432	0.781	0.650	0.619
		$\text{CodeQuant}_{had}$	7.80	10.85	0.457	0.726	0.742	0.472	0.787	0.676	0.643
	Qwen3- 30B-A3B	DuQuant	12.70	18.69	0.455	0.752	0.604	0.685	0.777	0.622	0.649
		$\text{SpinQuant}_{had}$	13.55	19.53	0.474	0.716	0.669	0.677	0.762	0.616	0.652
		$\text{CodeQuant}_{had}$	10.98	16.80	0.481	0.725	0.719	0.702	0.766	0.650	0.674

Appendix

Table 8: Rotation-based method performance comparison.

\text{CodeQuant}_{had}

indicates that online Hadamard transforms are enabled during the quantization process.

Bit Width	Method	Phi-mini-MoE-Instruct		DeepSeek-V2-Lite		Qwen3-30B-A3B
Bit Width	Method	Mem. (GB) $\downarrow$	Lat. (s) $\downarrow$	Mem. (GB) $\downarrow$	Lat. (s) $\downarrow$	Mem. (GB) $\downarrow$	Lat. (s) $\downarrow$
BF16	Llama.cpp (CPU)	14.3	40.1	29.3	50.0	56.9	66.1
A8W4	Llama.cpp (CPU)	4.1	15.0	8.8	17.1	16.2	20.1
A8W4	CodeQuant (CPU)	4.1	13.3	8.9	14.2	16.5	15.9

Appendix

Table 9: Latency and Memory Evaluation on CPU

Table 13

Mixture-of-Experts Quantization

Mixture-of-Experts quantization compresses expert weights and computations to reduce memory use and speed up inference while preserving model quality.

See all 32 papers

ProQuant: Preserving Expert Quality over Quantity in Ultra-Low-Bit Quantization of Mixture-of-Experts Models
Oct 22, 2026Jiaqi Chen, Xianglong Yan, Chengzhu Bao +2Mixture-of-Experts Quantization Post-Training Quantization
Shanghai Jiao Tong University
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
May 21, 2026Jianing Deng, Song Wang, Dongwei Wang +4Mixed-Precision Quantization LLM Quantization
University of Pittsburgh · University of Central Florida · University of Arizona +1
Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
Apr 7, 2026Mohammed Nowaz Rabbani Chowdhury, Kaoutar El Maghraoui, Hsinyu Tsai +4Mixture-of-Experts Quantization Neural Network Quantization
Rensselaer Polytechnic Institute · IBM Research

Oct 22, 2026J/K move · Enter open · S save

ProQuant: Preserving Expert Quality over Quantity in Ultra-Low-Bit Quantization of Mixture-of-Experts Models

Jiaqi Chen, Xianglong Yan, Chengzhu Bao, Xie Chen+1

Shanghai Jiao Tong University

Mixture-of-Experts Quantization Post-Training Quantization Mixed-Precision Quantization Weight-Only Quantization Expert Routing Mixture-of-Experts Pruning

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) large language models reduce computation through sparse activation but still incur substantial storage costs. Mixed-precision post-training quantization (PTQ) reduces these costs by quantizing different model components at different bit widths, but existing methods often struggle to preserve generation quality under tight bit budgets. We identify two key challenges: (1) nonlinear quantization errors at extremely low precision are underestimated when cross-layer propagation is ignored; (2) direct router fine-tuning may cause routing drift, and calibration gains may not generalize to downstream tasks. To address these issues, we propose ProQuant, a weight-only PTQ framework for MoE models. Prune-or-Prioritize Mixed-Precision Allocation (PoP) jointly optimizes expert pruning and bit-width assignment. It allows higher-precision experts to substitute for severely degraded ones while improving bit-budget utilization. Routing-Consistent RMSNorm Finetune (RoCo) adapts expert input representations through mathematically equivalent transformations. It compensates for quantization errors to reduce routing drift. Extensive experiments demonstrate that ProQuant outperforms existing methods across diverse generative tasks. Code is available at https://github.com/ThrowBug/ProQuant.git.

Abstract

Mixture-of-Experts Quantization

Explore similar work

ProQuant: Preserving Expert Quality over Quantity in Ultra-Low-Bit Quantization of Mixture-of-Experts Models

GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs

Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees