Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts

Kwanhee Lee; Namhoon Lee; Dan Alistarh

cs.ARSep 29, 2026

Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts

Authors: Kwanhee Lee, Namhoon Lee, Dan Alistarh

Organizations: POSTECH & ISTA · POSTECH · ISTA

Abstract

Mixture-of-Experts (MoE) architectures allow frontier language models to scale to trillions of parameters, but their deployment is constrained by massive memory footprints and memory-bandwidth limitations. Although modern accelerators provide Sparse Tensor Cores (SpTCs) that reduce weight storage and increase throughput through low-precision semi-structured sparsity, exploiting them for MoEs remains challenging because of substantial model-quality degradation and the lack of grouped sparse GEMM primitives. We present an end-to-end hardware-software co-design framework that compresses expert weights into hardware-native, low-precision sparse representations and accelerates their execution on SpTCs. Algorithmically, our framework relaxes discrete semi-structured support selection through continuous reparameterization, enabling differentiable joint optimization with quantized weights under a router-weighted reconstruction objective and scalable expert-parallel compression. Systemically, we develop a custom grouped sparse GEMM kernel tailored to low-precision sparse MoE inference on SpTCs. Across MoE models ranging from 30 billion to one trillion parameters, our framework improves state-of-the-art joint sparse-quantization accuracy by up to 4.35 percentage points while preserving 96.09% of the original model's performance. On NVIDIA B200 GPUs, our kernel outperforms the vendor baseline by up to $1.65\times$ , increasing serving throughput by $1.18\times$ and reducing end-to-end latency by up to $4.03\times$ . These results establish hardware-software co-design as a practical path toward scalable and efficient MoE deployment.

Figures & tables

Model	Method	ARC-c	GSM8K	MMLU	WG	HSwag	TQA	Avg	Rec.
Kimi-K2.5	Reference	74.06	94.39	89.53	81.93	91.96	62.69	82.43	100%
	Dense NVFP4	73.21	93.35	89.23	81.45	91.87	62.29	81.90	99.36%
	JSQ	59.22	68.92	71.87	73.88	80.95	50.58	67.57	81.97%
	SGPTQ	63.05	79.30	82.10	79.72	84.39	54.92	73.91	89.67%
	OBR	63.65	82.79	82.49	80.35	85.34	55.78	75.07	91.07%
	Ours	69.97	88.70	85.34	82.40	88.57	60.28	79.21	96.09%

Table 1: OpenLLM Leaderboard v1 results across three MoE scales under the paired

4{:}8

NVFP4 W4A4 expert representation. Recovery (Rec.) is relative to each model’s original average accuracy; bold marks the best compressed result.

	AIME25 $N{=}10$		GPQA Diamond $N{=}5$		MATH500 $N{=}5$
Method	Acc.	Trunc.	Acc.	Trunc.	Acc.	Trunc.	Avg	Rec.
Reference	95.67	3.7	89.49	0.7	96.36	0.1	93.84	100%
JSQ	7.33	88.3	13.43	72.0	27.76	67.0	16.17	17.24%
SGPTQ	68.33	15.3	68.48	2.5	94.24	1.5	77.02	82.07%
OBR	74.67	12.0	71.62	1.0	94.64	0.6	80.31	85.58%
Ours	84.67	6.3	76.46	0.3	96.28	0.4	85.80	91.44%

Table 2: Kimi-K2.5 reasoning results under paired

4{:}8

NVFP4 W4A4 compression. Acc. is mean pass@1 over

N

runs; Trunc. is the fraction exhausting the

65{,}536

-token budget, and Rec. is relative to the reference.

Panel of Figure 1 — Figure 1: Kernel-level evaluation of the grouped sparse GEMM on the Kimi-K2.5 gate/up layer. (a) Median runtime speedup of the grouped sparse GEMM kernel over dense (CuTeDSL) baseline. (b) Energy to solution at iso-frequency, where the label above each bar indicates the energy per call (mJ) and the label inside each bar indicates the mean board power (W) of each kernel; measured on 1xB300. (c) Median per-call latency of the deployment kernels across a range of token counts.

Panel of Figure 2 — Figure 2: End-to-end serving of Kimi-K2.5 on $8\times$ B200 with a shared vLLM stack, varying only the expert representation and corresponding grouped-GEMM backend ( Table 12(b) contains full numbers). (a) Prefill throughput (total tokens/s) across concurrency, with ( $4096\ \text{in}/1\ \text{out}$ ) requests. (b)–(d) Decode ( $384\ \text{in}/128\ \text{out}$ ) throughput (output tokens/s), mean Time Per Output Token (TPOT), and mean end-to-end request latency, with all $500$ requests in flight. Bars in (b)–(d) are the mean of three consecutive repeats, and whiskers show their range.

			OpenLLM v1		Serving
Method	Expert format	bits/expert weight	Avg. acc. $\uparrow$	Recovery $\uparrow$	KV cache (tokens) $\uparrow$	Output tok/s $\uparrow$	TTFT (ms) $\downarrow$	E2E (s) $\downarrow$
Reference	Dense INT4 W4A16	4.50	82.43	100%	1,055k	2,108	7,306	25.0
GSQ	Dense INT2 W2A16	2.13	77.66	94.22%	1,594k	3,811	4,560	14.6
REAP	$50\%$ expert pruning + INT4	2.25	80.77	97.99%	1,632k	3,190	5,366	17.1
Ours	Paired $4{:}8$ NVFP4 W4A4	2.75	79.21	96.09%	1,433k	8,983	2,120	6.2

Table 3: Comparison with alternative expert-compression methods on Kimi-K2.5. Serving metrics are measured with vLLM on

8\times

B200 under the same decode workload and matched concurrency (

384

input/

128

output tokens per request, all

500

requests in flight) and averaged over three repeats. Bit counts include quantization scales and sparse metadata, KV cache is the engine-reported capacity, and bold marks the best compressed result.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	Kimi-K2.5	Qwen3.5-397B-A17B	Qwen3-30B-A3B
MoE blocks compressed	60	60	48
Routed experts per block	384	512	128
Calibration sequences	8,192	8,192	8,192
Batch size (micro-batch)	64 (8)	64 (8)	64 (16)
Epochs per block	10	10	10
Optimizer	Lion ( Chen et al., 2023 )

Appendix

Table 4: Per-model compression hyperparameters. Learning rates are for the support logits

A

and latent weights

V

; the Gumbel temperature

\tau_{t}

of Equation 2 is annealed over the schedule shown.

Objective	Updated	ARC-c	GSM8K	MMLU	WG	HSwag	TQA	Avg	Rec.
$\mathcal{L}^{\mathrm{unif}}$	$(A,V)$	59.30	77.18	69.84	66.93	63.25	51.55	64.68	88.26%
$\mathcal{L}^{\mathrm{rw}}$	$V$ only	57.42	79.15	70.18	66.38	63.41	50.08	64.44	87.95%
$\mathcal{L}^{\mathrm{rw}}$ ( m o esq )	$(A,V)$	59.73	79.45	69.89	67.80	63.60	52.15	65.44	89.30%

Appendix

Table 5: Ablations on Qwen3-30B-A3B under the paired

4{:}8

NVFP4 W4A4 target.

\mathcal{L}^{\mathrm{unif}}

denotes Equation 4 with

w_{l,e}=\mathbf{1}

, and “

V

only” keeps the initial SGPTQ support fixed. Avg. is the unweighted mean across the six OpenLLM tasks, Rec. is relative to the BF16 reference in Table 1 , and bold marks the best result in each column.

Method	Expert format	ARC-c	GSM8K	MMLU	WG	HSwag	TQA	Avg	Rec.
Reference	Dense INT4 W4A16	74.06	94.39	89.53	81.93	91.96	62.69	82.43	100%
GSQ	Dense INT2 W2A16	64.51	91.96	84.35	80.90	86.01	58.24	77.66	94.22%
REAP	$50\%$ expert pruning + INT4	72.35	93.71	87.72	82.00	89.25	59.60	80.77	97.99%
Ours	Paired $4{:}8$ NVFP4 W4A4	69.97	88.70	85.34	82.40	88.57	60.28	79.21	96.09%

Appendix

Table 6: OpenLLM Leaderboard v1 comparison with alternative expert-compression methods on Kimi-K2.5. GSQ uses dense INT2 W2A16 weights, REAP prunes

50\%

of routed experts while retaining the released INT4 weights, and m o esq uses paired

4{:}8

NVFP4 W4A4. Recovery is relative to the reference row, and bold marks the best compressed result.

	Accuracy on terminated generations			Matched subset				Median terminated (chars)
Benchmark	Ours	OBR	JSQ	Coverage	Ours	OBR	JSQ	Ours	JSQ
AIME25	90.39	84.85	62.86	11/30	100.00	100.00	20.00	42,580	83,504
GPQA Diamond	76.70	72.35	48.01	123/198	75.12	71.06	21.63	43,810	105,104
MATH500	96.63	95.21	84.12	379/500	96.89	95.41	36.62	5,426	34,486

Appendix

Table 7: Termination behavior of sparse-quantized Kimi-K2.5 models. Accuracy on terminated generations isolates answer quality after successful termination, while the matched-subset comparison controls for JSQ terminating on substantially fewer problems. Coverage is the number of problems for which JSQ terminates at least once out of the benchmark total.

Model	Layer	$(M,K)$	$N$ sweep	$G$ sweep
Kimi-K2.5	gate/up	$(4096,7168)$	$\{32,64,128,256,512,1024\}$	$\{24,48,96,192,384\}$
	down	$(7168,2048)$
Qwen3-30B-A3B	gate/up	$(1536,2048)$	$\{16,32,64,128,256,512,1024\}$	$\{8,16,32,64,128\}$
	down	$(2048,768)$

Appendix

Table 8: Grouped-GEMM benchmark shapes.

(M,K)

are the output and contraction dimensions;

N

is the maximum per-expert token count and

G

the expert-group count.

Kernel	SM mode (tile- $M$ )	tile- $N$	tile- $K$
Sparse (ours)	1-SM ( $128$ ), 2-SM ( $256$ )	$\{128,256\}$	$256$
Dense (native)	1-SM ( $128$ ), 2-SM ( $256$ )	$\{128,192,256\}$	$256$
Dense (transposed)	1-SM ( $128$ ), 2-SM ( $256$ )	$\{128,192\}$	$256$

Appendix

Table 9: Autotuner search space. Tile-

M

is fixed by the SM mode and tile-

K{=}256

throughout; supported tile-

N

values differ across kernels.

Figure 5: Autotuned speedup of the grouped sparse GEMM over the dense NVFP4 grouped GEMM (native layout) across the four benchmarked MoE layers on B200. Rows are expert-group count $G$ and columns are maximum per-expert token count $N$ .

	Iso-frequency ( $1702$ MHz)			Power-capped ( $\approx 1095$ W)
	Sparse	Dense (transposed)	Dense (native)	Sparse	Dense (transposed)	Dense (native)
Avg. power (W)	$1063$	$865$	$945$	$1095$	$1096$	$1092$
SM clock (MHz)	$1702$	$1702$	$1702$	$1725$	$1897$	$1867$
Latency (ms)	$0.687$	$1.348$	$1.164$	$0.679$	$1.183$	$1.131$
Throughput (TFLOP/s)	$2627$	$1339$	$1550$	$2657$	$1525$	$1595$
Energy/call (mJ)	$727$	$1166$	$1098$	$745$	$1298$	$1237$

Appendix

Table 10: Energy-to-solution for the Kimi-K2.5 gate/up grouped GEMM (

M{=}4096

N{=}128

K{=}7168

G{=}384

) on B300. Iso-frequency is the primary comparison; the power-capped regime is included for completeness. Dense columns are the two operand layouts of the same CUTLASS kernel ( Section D.2 ); speedup and energy reduction are those of the sparse kernel over each dense column.

	bits/p	$N{=}8$	$N{=}16$	$N{=}32$	$N{=}64$	$N{=}128$	$N{=}256$
Ours (paired $4{:}8$ W4A4)	2.750	760	793	843	925	912	1,370
NVFP4-W4A4 CuTeDSL	4.500	979	1,012	1,036	1,100	1,226	1,717
INT4-W4A16 Marlin	4.500	1,584	1,805	2,519	3,717	6,163	11,272
INT2-W2A16 Humming	2.125	1,538	1,545	2,959	3,299	6,446	9,733

Appendix

Table 11: Cross-kernel grouped-GEMM latency (

\mu

s) on the Kimi-K2.5 gate/up layer (

M{=}4096

K{=}7168

G{=}384

) on B200. Timings include activation quantization/dequantization and use the same ragged token pattern as the main sweep.

Table 12: End-to-end serving on Kimi-K2.5 (

8\times

B200, matched concurrency); the results of Figure 2 in full.

Mixture-of-Experts Quantization

Mixture-of-Experts quantization compresses expert weights and computations to reduce memory use and speed up inference while preserving model quality.

See all 32 papers

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
May 11, 2026Chenyang Song, Weilin Zhao, Xu Han +3Sparse Mixture-of-Experts Mixture-of-Experts Inference
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China
Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
Apr 7, 2026Mohammed Nowaz Rabbani Chowdhury, Kaoutar El Maghraoui, Hsinyu Tsai +4Mixture-of-Experts Quantization Neural Network Quantization
Rensselaer Polytechnic Institute · IBM Research
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
Feb 5, 2026Jingze Shi, Zhangyang Peng, Yizhang Zhu +3Mixture-of-Experts Inference LLM Inference Acceleration
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence

May 11, 2026J/K move · Enter open · S save

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices

Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao+2

Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China

Sparse Mixture-of-Experts Mixture-of-Experts Inference Expert Routing On-Device Inference

Open paper AskPDF arXiv

While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, which hinder efficient end-side deployment that simultaneously requires high performance, low computational cost, and small storage overhead. To achieve these properties, we present DECO, a sparse MoE architecture designed to match the performance of dense Transformers under identical total parameter budgets and training tokens. DECO utilizes the differentiable and flexible ReLU-based routing enhanced by learnable expert-wise scaling, which adaptively balances the contributions of routed and shared experts. Furthermore, we introduce NormSiLU, an activation function that normalizes inputs prior to SiLU operators, producing a more stable trend of routed-expert activation ratio and a higher intrinsic sparsity level. We also identify an empirical advantage in using non-gated MLP experts with ReLU-based routing, indicating the possibility of MoE architecture simplification. Experiments demonstrate that DECO, activating only 20% of routed experts, matches dense performance and outperforms established MoE baselines. Our specialized acceleration kernel delivers a 2.93

\times

speedup on Jetson AGX Orin compared with dense inference. Code and checkpoints are available at https://github.com/thunlp/DECO.

Abstract

Mixture-of-Experts Quantization

Explore similar work

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices

Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees

OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale