Accelerating Dense LLMs via L0-regularized Mixture-of-Experts

Zhenyu Zhang; Jiudong Yang; Zhaowen Tao; Meng Chen

cs.AISep 18, 2026

Accelerating Dense LLMs via L0-regularized Mixture-of-Experts

Authors: Zhenyu Zhang, Jiudong Yang, Zhaowen Tao, Meng Chen

Organizations: YZW, Chengdu, China · FuTu AI, Shenzhen, China · Wise AI, Melbourne, Australia

Abstract

Large language models (LLMs) achieve strong performance but suffer from slow and costly inference. Existing acceleration methods often lead to noticeable performance degradation, while Mixture-of-Experts (MoE) models require extensive computational resources. In this paper, we propose L0-MoE, a lightweight MoE approach using L0-regularization to accelerate dense LLMs nearly without performance loss. Our method introduces a cluster confusion matrix for domain-aware dataset curation and applies dynamic batching for efficient training. Experiments show that L0-MoE achieves up to 2.5x speedup over dense models while maintaining competitive performance, outperforming existing LLM acceleration baselines.

Figures & tables

Figure 1: Overview of the L0-MoE Architecture, which includes three main stages: (1) cluster confusion matrix based sampling, (2) expert formation using L0 regularization, and (3) dynamic batching for MoE training. The figure above illustrates the process of building an L0-MoE with four experts over $n$ iterations of dataset sampling.

Model	MMLU	GSM8K	HumanEval	BBH	Average	Speedup
Llama-3-8B	66.6	56.0	33.5	57.7	53.5
Llama-3-8B w/ L0-MoE	66.3	55.9	33.7	57.2	53.3	2.0x
Mistral-7B	64.1	52.2	29.3	56.1	50.4
Mistral-7B w/ L0-MoE	64.8	53.6	31.1	55.9	51.4	2.1x
Qwen2-7B	70.3	79.9	51.2	62.6	66.0
Qwen2-7B w/ L0-MoE	70.4	80.5	52.0	61.5	66.1	2.5x

Table 1: Evaluation of different LLMs on MMLU, GSM8K, HumanEval and BBH benchmarks.

Model	MMLU	GSM8K	Speedup
Qwen2-7B	70.3	79.9	-
L0-MoE	70.4	80.5	2.5x
GPTQ	67.8	73.8	1.8x
LLM Shearing	68.2	75.5	2.6x
RKD + CoT	61.2	60.2	5.1x

Table 2: Comparison with other inference acceleration baselines. We employ Qwen2-7B as the base LLM.

Model	MMLU	GSM8K
L0-MoE	70.4	80.5
CCM w/o K-means	68.2	78.1
w/ random order batching	68.2	75.5
w/ random batch batching	66.6	77.1
Random MoE	48.1	69.6
Magnitude	52.6	69.1

Table 3: Ablation study of CCM and L0-regularization. We conduct experiments on MMLU and GSM8K datasets with Qwen2-7B.

Model	MMLU	#Para.(B)	Speedup
L0-MoE	70.4	23.3	2.5x
CCM Iter. ( $Q=2$ )	68.2	23.3	2.5x
CCM Iter. ( $Q=5$ )	68.8	23.3	2.5x
CCM Iter. ( $Q=10$ )	69.7	23.3	2.5x
Expert size ( $K=8$ )	50.9	4.8	4.6x
Expert size ( $K=32$ )	69.2	12.7	3.2x

Table 4: Hyper-parameter tuning of sampling iterations (

Q

) and cluster size (

K

), keeping 2.8B activated parameters for L0-MoE. Qwen2-7B is the base LLM. #Para.(B) is the number of model parameters.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Parameters(B)	Layer	Hidden	Q/KV	FFN	MoE FFN	Experts
Llama-3-8B	7.5	32	4096	32/8	14336
Llama-3-8B w/ L0-MoE	25.1/3.2	32/24	4096	32/8	14336	1024	64:2
Mistral-7B	7.1	32	4096	32/8	14336
Mistral-7B w/ L0-MoE	24.7/2.9	32/24	4096	32/8	14336	1024	64:2
Qwen2-7B	7	28	3584	28/4	18944
Qwen2-7B w/ L0-MoE	23.3/2.8	28/24	3584	28/4	18944	1280	64:2

Appendix

Table 5: Detailed model architecture parameters. We denote the total and activated parameters of the MoEs, as well as the total layers and MoE layers, using the format “32/24”, etc. All models utilize GQA, and we present the query/key-value heads. “FFN” refers to the dense decoder MLP size, while “MoE FFN” indicates the intermediate size of the expert for the MoE layer. The total and activated experts are represented as “64:2”, etc.

Model	MMLU	GSM8K	SpeedUp
Llama-3-8B	66.6	56.0	-
Llama-3-8B w/ L0-MoE	66.3	55.9	2.0 $\times$
DeQuant/W4A4	57.9	51.6	0.46 $\times$
DeQuant + LWC/W4A4	62.2	51.1	0.46 $\times$

Appendix

Table 6: Performance comparison of L0-MoE and DuQuant on MMLU and GSM8K benchmarks. Llama-3-8B is the base LLM. Due to the lack of DuQuant support in SGlang, we tested inference speed using naive PyTorch transformers (batch size = 64, sequence length = 1200). Without optimized kernels, DuQuant is slow (0.46x speedup), but future SGlang support could make it comparable to GPTQ (1.8x speedup).

CCM Hyper-parameters
$Q$	21
$K$	64
$d_{sv}$	1024
$D_{sl}^{(0)}$	12B tokens; $\|D_{s_{i}}^{(0)}\|\approx 0.15B$
$D_{sl}^{(l)}$ , $l\geq 1$	0.9B tokens; $\|D_{s_{i}}^{(0)}\|\approx 0.007B$
MoE Training Hyper-parameters

Appendix

Table 7: Hyper-parameters for CCM and MoE training.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,101 papers

DOT-MoE: Differentiable Optimal Transport for MoEfication
Jun 1, 2026Udbhav Bamba, Arnav Chavan, Aryamaan Thakur +2LLM Inference Efficiency Mixture-of-Experts Models
Amazon
Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
May 9, 2026Jongseok Park, Sunga Kim, Zhenyu Gu +2Activation Sparsity Mixture-of-Experts Inference
University of California, Berkeley · Work done during an internship at Advanced Micro Devices, Inc. · Advanced Micro Devices, Inc.
BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE
May 14, 2026Juntong Wu, Jialiang Cheng, Qishen Yin +5Sparse Mixture-of-Experts Mixture-of-Experts Inference
Taobao & Tmall Group of Alibaba · Shenzhen Graduate School, Peking University

Jun 1, 2026J/K move · Enter open · S save

DOT-MoE: Differentiable Optimal Transport for MoEfication

Udbhav Bamba, Arnav Chavan, Aryamaan Thakur, Steve Teig+1

Amazon

LLM Inference Efficiency Mixture-of-Experts Models

Open paper AskPDF arXiv

The scaling of Large Language Models (LLMs) has driven significant performance gains but created substantial challenges in inference efficiency. While Mixture of Experts (MoEs) architectures address this by decoupling model size from inference cost, training MoEs from scratch is often unstable and compute intensive. Conversion of pre-trained dense models into sparse MoEs has emerged as an alternative solution; however, existing methods typically rely on heuristic neuron clustering or random splitting to partition the Feed-Forward Network (FFN) into experts. In this work, we propose DOT-MoE, a novel framework that formulates the decomposition of dense layers as a Differentiable Optimal Transport (DOT) problem. Instead of static heuristics, we model neuron assignment as a balanced transport problem, utilizing differentiable Sinkhorn-Knopp iterations to enforce strict expert capacity constraints. Furthermore, we utilize Straight-Through Estimators (STE) to jointly learn the discrete neuron-to-expert assignment and the token-to-expert routing policy end-to-end. Extensive experiments across multiple architectures and benchmarks demonstrate that DOT-MoE significantly outperforms structured pruning, heuristic clustering, and random-split baselines, retaining 90% of the original dense model's performance while reducing active parameters by 50%.

Abstract

LLM Inference Acceleration

Explore similar work

DOT-MoE: Differentiable Optimal Transport for MoEfication

Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution

BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE