Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion

Zachary Broveak; Christian Kubicka; Thierry Tambe

cs.LGOct 13, 2026

Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion

Authors: Zachary Broveak, Christian Kubicka, Thierry Tambe

Organizations: Stanford University

Abstract

Large language models impose substantial memory demands and arithmetic costs during inference. Dense-to-mixture-of-experts (MoE) conversion is a model optimization technique that aims to reduce these arithmetic costs. Dense-to-MoE conversion partitions pretrained feed-forward network (FFN) neurons into experts and selects a subset for each token. Leading methods fall into two categories: analytic methods produce a converted model quickly using heuristics, while learning methods produce a model much more slowly by optimizing via gradients. Both approaches, however, fix the same active expert count, or "utilization," at every layer. We introduce Spectral-MoE, which learns to allocate a utilization budget heterogeneously across layers. Its name draws a metaphor between separating a dense FFN into experts and separating a light beam into its spectrum. Spectral-MoE consists of two distinct approaches: a static approach that uses a single per-layer scalar which is shared across tokens, and a dynamic approach which emits a per-token schedule. We demonstrate that both allocators improve performance over uniform on both the analytic-based and learning-based SOTA dense-to-MoE methods. At 25% FFN utilization, dynamic allocation reduces WikiText-2 perplexity relative to uniform by 1.04, 2.96, and 2.71 points on the learned base for Llama-2-7B, Llama-3-8B, and Qwen2.5-7B, respectively. The corresponding reductions on the analytic base are 2.45, 6.26, and 2.17 points. On the learned base, dynamic allocation improves six-task mean accuracy over uniform by an average of 1.5, 4.2, and 3.6 percentage points across the three models at FFN utilizations of 12.5%, 25%, and 50%, respectively. These results establish per-layer expert utilization as a novel optimization axis in dense-to-MoE conversion.

Figures & tables

Figure 1: Benchmark accuracy versus utilization. Dynamic, heterogeneous allocation (learned) extends the frontier of dense-to-MoE performance.

Figure 2: Spectral-MoE learned-base dynamic allocator architecture. During the forward pass, the allocator uses hidden states to initialize and revise per-token schedules, layers commit budget-constrained expert counts, and the final layer consumes the remainder. During the backward pass, loss gradients pass through selections and budget commitments via STEs to train only the allocator, routers, and partition affinities.

Panel of Table 1 — Table 1: Dense-to-MoE conversion at 25% FFN utilization. Headers give [shots, metric]. Downstream benchmarks use the evaluation harness from Gao et al. (2024) . Analytic and Learned use CMoE and DOT-MoE bases, respectively. Uniform denotes our custom implementation of the bases. Uniform, static, and dynamic runs are seed matched (Appendix ). $\dagger$ : published Pei et al. (2026) . $\ddagger$ : published Bamba et al. (2026) .

FFN Util.	Model	Allocator	PPL ( $\downarrow$ )	BoolQ	SciQ	PIQA	WinoG.	ARC-C	HellaS.	Avg.
FFN Util.	Model	Allocator	PPL ( $\downarrow$ )	[32, acc]	[0, acc]	[0, acc]	[5, acc]	[25, acc-n]	[10, acc-n]	Avg.
	Llama-2-7B	Uniform	15.93	55.4	84.7	60.9	54.3	27.0	41.4	54.0
		Dynamic	14.87	60.5	85.2	60.3	53.0	26.9	44.6	55.1
	Llama-3-8B	Uniform	27.69	59.9	84.6	60.8	52.5	25.9	39.5	53.9
		Dynamic	25.46	62.1	86.3	62.6	54.5	28.2	40.9	55.8
	Qwen2.5-7B	Uniform	23.60	53.5	84.2	63.1	55.6	28.5	41.6	54.4

Table 2: Spectral-MoE learned-base utilization sweep. Dynamic allocation improves perplexity and six-task mean accuracy over uniform across all tested models and utilizations. Metrics follow Table .

Figure 3: Spectral-MoE on the Llama-2-7B learned base at a nominal 25% FFN utilization. (a) WikiText-2 test perplexity. (b) Each layer’s mean utilization under dynamic allocation. The dashed line marks the uniform start (

25.6\%

). (c) Final-step schedules of 16 sampled tokens. The two unique schedules for the token “on” showcase the allocator adapting to context.

Figure 4: Fraction of routed neurons assigned to a different expert between consecutive training steps, measured on a sample learned-base, dynamic dense-to-MoE conversion.

Seed	Router	Overlap	Coverage	BoolQ	SciQ	PIQA	WinoG.	ARC-C	HellaS.	Avg.
A	Original	50.11	87.66	69.5	89.5	65.8	57.4	32.0	56.2	61.74
	Refit	69.42	95.62	66.8	89.7	67.5	60.9	34.9	56.6	62.73
B	Original	50.02	87.66	67.1	88.8	68.4	60.5	35.8	56.5	62.84
	Refit	69.53	95.66	65.4	88.4	67.8	60.8	35.8	56.6	62.48

Table 3: Router refitting on learned dynamic Llama-2-7B at

25\%

FFN utilization. Original and Refit denote checkpoint and ridge-regression routers. Entries are percentages; metrics follow Table .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Definition	Symbol	Definition
$L$ , $l$	FFN layers and zero-based layer index.	$q_{l}$ , $\pi_{l}$	Uncommitted layers $L-l$ , and the current layer’s softmax share.
$n$ , $t$ , $e$	Token count, token index, and routed-expert index.	$\bm{v}^{(l)}$ , $\bm{r}_{l}$ , $\bm{\varDelta}_{l}$	Vector of schedule logits, proposed revisions, and accumulated corrections in $\mathbb{R}^{q_{l}}$ .
$m$ , $d_{\text{ffn}}$ , $d_{\text{model}}$	Neurons per expert, FFN width, and residual-stream width.	$S_{l}$ , $R_{l}$ , $\widehat{S}_{l}$	Committed budget, remainder $B-S_{l}$ , and the STE for $S_{l}$ .
$E_{t}$ , $E_{r}$ , $E_{s}$	Total, routed, and shared experts per layer, with $E_{t}=E_{r}+E_{s}$ .	$a_{l}$ , $b_{l}$	Feasible integer bounds on the current commitment.
$E_{\min}$ , $E_{l}$	Minimum routed count and selected routed count at layer $l$ .	$\bm{h}_{l}$ , $\bm{x}_{l}$ , $\bm{z}_{l}$	FFN input, its $\rms$ -normalized vector, and the allocator’s shared-projection output.
$f_{l}$ , $\bar{f}$	Free count $E_{l}-E_{\min}$ and its layer mean.	$\mathbf{W}_{\mathrm{in}}$ , $\mathbf{W}_{0}$ , $\mathbf{W}_{l}$	Matrices for the shared projection, initial output head, and revision output head.

Appendix

Table 4: Notation used in the main text and appendices. Routed counts measure selected routed experts, whereas FFN utilization includes the shared expert. Bold italic symbols denote vectors and bold upright symbols denote matrices.

Model	$L$	$E_{t}$	$d_{\text{ffn}}$	Nominal $u$	$\bar{E}$	Realized $u$	$B$
Llama-2-7B	32	86	11,008	$12.5\%$	10	$12.79\%$	320
				$25\%$	21	$25.58\%$	672
				$50\%$	42	$50.00\%$	1,344
Llama-3-8B	32	112	14,336	$12.5\%$	13	$12.50\%$	416
				$25\%$	27	$25.00\%$	864
				$50\%$	55	$50.00\%$	1,760

Appendix

Table 5: Expert geometry, the uniform routed count derived from each requested utilization, the utilization it realizes, and the conserved routed budget. All models use

m=128

neurons per expert, one shared expert per layer, and

E_{\min}=1

Task	Shots	Metric	Examples	Split
BoolQ ( Clark et al., 2019 )	32	accuracy	3,270	validation
SciQ ( Welbl et al., 2017 )	0	accuracy	1,000	test
PIQA ( Bisk et al., 2019 )	0	accuracy	1,838	validation
WinoGrande ( Sakaguchi et al., 2019 )	5	accuracy	1,267	validation
ARC-Challenge ( Clark et al., 2018 )	25	length-normalized accuracy	1,172	test
HellaSwag ( Zellers et al., 2019 )	10	length-normalized accuracy	10,042	validation

Appendix

Table 6: Evaluation settings, identical for every model and allocation method.

	Learned base	Analytic base
Trained tensors	$\mathbf{A}$ , $\mathbf{W}_{r}$ , allocator	allocator only
Frozen	pretrained FFN weights	weights, partition, router
Steps / tokens	3,500 / 458.8M	1,024 / 4.19M
Preceding stage	–	LoRA recovery, 4.19M tokens
$\mathbf{A}$ , $\mathbf{W}_{r}$ peak rate	$5\times 10^{-4}$	–
Static $\bm{\epsilon}$ peak rate	$5\times 10^{-3}\bar{E}/10$	$0.875\,\bar{E}/512$

Appendix

Table 7: Training configurations. The analytic static rate is set from the step count,

0.875\bar{E}

spread over half the run, rather than from the learned base’s constant.

Allocation	Initialization	Step-0 PPL	Final PPL	Six-task avg. (%)
Dynamic	seed C	1115.01	10.72	62.2
	seed A	1279.96	10.59	61.7
	seed B (Table )	1365.54	10.57	62.8

Appendix

Table 8: Seed variation on the learned Llama-2-7B base at

25\%

utilization (

B=672

). Metrics follow Table .

Figure 5: Learned per-layer FFN utilization at

25\%

utilization on the three main-body models.

Inclusion temperature	Final PPL
Uniform control	16.30
$T=0.25$	16.57
$T=1$	16.15
$T=1.5$	15.89

Appendix

Table 9: Inclusion temperature for learned static allocation on Llama-2-7B at

\bar{E}=10

(

B=320

Mixture-of-Experts Language Models

Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.

See all 216 papers

Jun 1, 2026

DOT-MoE: Differentiable Optimal Transport for MoEfication

The scaling of Large Language Models (LLMs) has driven significant performance gains but created substantial challenges in inference efficiency. While Mixture of Experts (MoEs) architectures address this by decoupling model size from inference cost, training MoEs from scratch is often unstable and compute intensive. Conversion of pre-trained dense models into sparse MoEs has emerged as an alternative solution; however, existing methods typically rely on heuristic neuron clustering or random splitting to partition the Feed-Forward Network (FFN) into experts. In this work, we propose DOT-MoE, a novel framework that formulates the decomposition of dense layers as a Differentiable Optimal Transport (DOT) problem. Instead of static heuristics, we model neuron assignment as a balanced transport problem, utilizing differentiable Sinkhorn-Knopp iterations to enforce strict expert capacity constraints. Furthermore, we utilize Straight-Through Estimators (STE) to jointly learn the discrete neuron-to-expert assignment and the token-to-expert routing policy end-to-end. Extensive experiments across multiple architectures and benchmarks demonstrate that DOT-MoE significantly outperforms structured pruning, heuristic clustering, and random-split baselines, retaining 90% of the original dense model's performance while reducing active parameters by 50%.

Udbhav Bamba, Arnav Chavan, Aryamaan Thakur +2

Amazon

LLM Inference Efficiency Mixture-of-Experts Models

Ask PDF

Feb 17, 2026

ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns

Mixture-of-Experts (MoE) effectively scales model capacity while preserving computational efficiency through sparse expert activation. However, training high-quality MoEs from scratch is prohibitively expensive. A promising alternative is to convert pretrained dense models into sparse MoEs. Existing dense-to-MoE methods fall into two categories: dynamic structural pruning that converts dense models into MoE architectures with moderate sparsity to balance performance and inference efficiency, and downcycling approaches that use pretrained dense models to initialize highly sparse MoE architectures. However, existing methods break the intrinsic activation patterns within dense models, leading to suboptimal expert construction. In this work, we argue that the Gated Linear Unit (GLU) mechanism provides a natural blueprint for dense-to-MoE conversion. We show that the fine-grained neural-wise activation patterns of GLU reveal a coarse-grained structure, uncovering an inherent MoE architecture composed of consistently activated universal neurons and dynamically activated specialized neurons. Leveraging this discovery, we introduce ExpertWeaver, a training-free framework that partitions neurons according to their activation patterns and constructs shared experts and specialized routed experts with layer-adaptive configurations. Our experiments demonstrate that ExpertWeaver significantly outperforms existing methods, both as a training-free dynamic structural pruning technique and as a downcycling strategy for superior MoE initialization.

Ziyu Zhao, Tong Zhu, Zhi Zhang +6

Zhejiang University · Shanghai Innovation Institute · Shanghai AI Laboratory +3

Mixture-of-Experts Language Models Mixture-of-Experts Pruning LLM Compression

Ask PDF

Apr 9, 2026

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference

Mixture-of-Experts (MoE) has become a dominant architecture for scaling large language models due to their sparse activation mechanism. However, the substantial number of expert activations creates a critical latency bottleneck during inference, especially in resource-constrained deployment scenarios. Existing approaches that reduce expert activations potentially lead to severe model performance degradation. In this work, we introduce the concept of activation budget as a constraint on the number of expert activations and propose Alloc-MoE, a unified framework that optimizes budget allocation coordinately at both the layer and token levels to minimize performance degradation. At the layer level, we introduce Alloc-L, which leverages sensitivity profiling and dynamic programming to determine the optimal allocation of expert activations across layers. At the token level, we propose Alloc-T, which dynamically redistributes activations based on routing scores, optimizing budget allocation without increasing latency. Extensive experiments across multiple MoE models demonstrate that Alloc-MoE maintains model performance under a constrained activation budget. Especially, Alloc-MoE achieves

1.15\times

prefill and

1.34\times

decode speedups on DeepSeek-V2-Lite at half of the original budget.

Baihui Liu, Kaiyuan Tian, Wei Wang +3

National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology

Mixture-of-Experts Inference Mixture of Experts Efficient Language Model Inference

Ask PDF

Abstract

Mixture-of-Experts Language Models

Explore similar work

DOT-MoE: Differentiable Optimal Transport for MoEfication

ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference