DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks

Gökdeniz Gülmez

cs.LGMar 2, 2026

DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks

Authors: Gökdeniz Gülmez

Organizations: Machine Learning Research, Stuttgart, Germany

Abstract

Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling neural networks while maintaining computational efficiency. However, standard MoE implementations rely on two rigid design assumptions: (1) fixed Top-K routing where exactly K experts are activated per token, and (2) uniform expert allocation across all layers. This paper introduces DynaMoE, a novel MoE framework that relaxes both constraints through dynamic token-level expert activation and layer-wise adaptive capacity allocation. DynaMoE introduces a principled routing mechanism where the number of active experts per token varies based on input complexity. Concurrently, the framework implements six distinct scheduling strategies for distributing expert capacity across network depth, including descending, ascending, pyramid, and wave patterns. We theoretically analyze the expressivity gains of dynamic routing and derive bounds on computational efficiency. Through extensive experiments on MNIST, Fashion-MNIST, CIFAR-10 (image classification), and Recycling-the-Web (language modeling) across multiple model scales, we demonstrate that DynaMoE achieves superior parameter efficiency compared to static baselines. Our key finding is that optimal expert schedules are task- and scale-dependent: descending schedules (concentrating capacity in early layers) outperform uniform baselines on image classification. For language modeling, optimal schedules vary by model size, descending for Tiny, ascending for Small, and uniform for Medium. Furthermore, dynamic routing reduces gradient variance during training, leading to improved convergence stability. DynaMoE establishes a new framework for adaptive computation in neural networks, providing principled guidance for MoE architecture design.

Figures & tables

Figure 1: Comparison of expert scheduling strategies showing how expert capacity is distributed across 12 network layers. Descending concentrates capacity in early layers, ascending in later layers, while pyramid and uniform strategies show intermediate patterns. $N_{\max}=16$ , $N_{\min}=1$ .

Figure 2: DynaMoE architecture with descending expert schedule. Expert capacity decreases from 8 experts in Layer 1 to 1 expert in Layer 4, concentrating computational resources in early feature extraction layers.

Size	Layers	Hidden Dim	Parameters
Tiny	2	64	$\sim$ 85K
Small	4	128	$\sim$ 340K
Medium	6	256	$\sim$ 1.4M
Large	8	512	$\sim$ 5.6M

Table 1: Model size configurations

Schedule	Accuracy (%)	Parameters	Epochs to 95%	Time (s)
MLP Baseline	89.42	340K	8	145
Uniform	91.35	380K	7	168
Descending	92.68	380K	6	162
Ascending	90.12	380K	9	175
Pyramid-Up	91.08	380K	7	170
Pyramid-Down	90.85	380K	8	173

Table 2: Expert schedule comparison on MNIST (Small model, 4 layers, 8 max experts)

Figure 3: Expert activation probability heatmaps across 6 layers for three scheduling strategies. Brighter colors indicate higher activation probability. Descending schedule shows strong activation in early layers (L1-L2), uniform maintains consistent patterns, while ascending concentrates activation in deeper layers.

Config	Max	Min	Accuracy (%)	Parameters	Efficiency
E2–1	2	1	90.85	290K	3.13
E4–1	4	1	91.62	325K	2.82
E8–1	8	1	92.68	380K	2.44
E16–1	16	1	92.45	520K	1.78
E8–2	8	2	91.98	390K	2.36
E8–4	8	4	91.24	410K	2.22

Table 3: Expert count sensitivity on MNIST (Small model, descending schedule)

Figure 4: Performance comparison across model sizes (Tiny, Small, Medium) for language modeling. Left: Best validation perplexity (lower is better). Right: Best validation accuracy (higher is better). Results show the descending schedule achieving the best validation perplexity across all model sizes, consistent with image classification findings.

Panel of Figure 5 — Figure 5: Scaling analysis on MNIST across model sizes

Dataset	MLP	Uniform	Descending	Improvement
MNIST	89.42	91.35	92.68	+3.26 %
Fashion-MNIST	84.15	86.82	88.34	+4.19 %
CIFAR-10	62.38	65.12	67.85	+5.47 %

Table 4: Cross-dataset performance (Small model, 20 epochs)

Model Size	Configuration	Best Val PPL	Final Val PPL	Best Val Acc
Tiny	MLP Baseline	1003.08	1762.19	0.1805
	Descending	1011.80	1845.42	0.1797
	Ascending	1063.87	1698.30	0.1762
	Pyramid-Up	1056.17	1739.95	0.1778
	Uniform	1078.31	1394.63	0.1761
Small	MLP Baseline	2311.02	2886.10	0.0505

Table 5: Next token prediction performance (validation perplexity and accuracy)

Panel of Figure 6 — Figure 6: Best validation perplexity across model sizes for language modeling

Figure 7: Training dynamics for next token prediction across model sizes. Top row: Validation perplexity over epochs (log scale). Bottom row: Validation accuracy progression. The plots reveal different convergence patterns for each schedule, with ascending and pyramid schedules showing more stable training for larger models.

Figure 8: Performance heatmap showing best validation perplexity across model sizes and configurations. Colors indicate normalized performance scores (green = better). Numbers show actual perplexity values. The heatmap clearly illustrates task-dependent optimal schedules.

Panel of Figure 9 — Figure 9: Average active experts per token by layer

Figure 10: Training loss convergence analysis across model sizes (log scale). DynaMoE configurations show smooth convergence with different schedules exhibiting distinct convergence rates. Descending schedules achieve fastest initial convergence for Tiny and Small models, while ascending schedules show more stable convergence for Medium models.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 11 — Figure 11: Training convergence comparison

Layer	Experts	Avg Active	Utilization (%)	Entropy
1	8	2.4	30.0	2.12
2	6	2.1	35.0	1.98
3	3	1.5	50.0	1.45
4	1	1.0	100.0	0.00

Appendix

Table 6: Expert distribution by layer (descending schedule,

N_{\max}=8

N_{\min}=1

)

Hyperparameter	Values Tested	Best Value
Temperature $T$	{0.1, 0.5, 1.0, 2.0}	0.5
Noise scale $\sigma$	{0.01, 0.05, 0.1, 0.2}	0.1
Threshold $\tau$	{0.5, 0.6, 0.7, 0.8, 0.9}	0.7
Learning rate	{1e-4, 5e-4, 1e-3, 5e-3}	1e-3

Appendix

Table 7: Hyperparameter sensitivity on MNIST

Explore similar work

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
Apr 9, 2026Baihui Liu, Kaiyuan Tian, Wei Wang +3Mixture-of-Experts Inference Mixture of Experts
National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology
Routing-Free Mixture-of-Experts
Apr 1, 2026Yilun Liu, Jinru Han, Sikuan Yan +2Mixture-of-Experts Models Expert Load Balancing
Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
Feb 5, 2026Jingze Shi, Zhangyang Peng, Yizhang Zhu +3Mixture-of-Experts Inference LLM Inference Acceleration
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence

Apr 9, 2026J/K move · Enter open · S save

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference

Baihui Liu, Kaiyuan Tian, Wei Wang, Zhaoning Zhang+2

National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology

Mixture-of-Experts Inference Mixture of Experts Efficient Language Model Inference

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) has become a dominant architecture for scaling large language models due to their sparse activation mechanism. However, the substantial number of expert activations creates a critical latency bottleneck during inference, especially in resource-constrained deployment scenarios. Existing approaches that reduce expert activations potentially lead to severe model performance degradation. In this work, we introduce the concept of activation budget as a constraint on the number of expert activations and propose Alloc-MoE, a unified framework that optimizes budget allocation coordinately at both the layer and token levels to minimize performance degradation. At the layer level, we introduce Alloc-L, which leverages sensitivity profiling and dynamic programming to determine the optimal allocation of expert activations across layers. At the token level, we propose Alloc-T, which dynamically redistributes activations based on routing scores, optimizing budget allocation without increasing latency. Extensive experiments across multiple MoE models demonstrate that Alloc-MoE maintains model performance under a constrained activation budget. Especially, Alloc-MoE achieves

1.15\times

prefill and

1.34\times

decode speedups on DeepSeek-V2-Lite at half of the original budget.

Abstract

Explore similar work

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference

Routing-Free Mixture-of-Experts

OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale