Excitation: Momentum For Experts

Sagi Shaier

cs.LGFeb 25, 2026

Excitation: Momentum For Experts

Authors: Sagi Shaier

Organizations: Aleph Alpha Research, Heidelberg, Germany.

Abstract

We propose Excitation, a novel optimization framework designed to accelerate learning in sparse architectures such as Mixture-of-Experts (MoEs). Unlike traditional optimizers that treat all parameters uniformly, Excitation dynamically modulates updates using batch-level expert utilization. It introduces a competitive update dynamic that amplifies updates to highly-utilized experts and can selectively suppress low-utilization ones, effectively sharpening routing specialization. Notably, we identify a phenomenon of "structural confusion" in deep MoEs, where standard optimizers fail to establish functional signal paths; Excitation acts as a specialization catalyst, "rescuing" these models and enabling stable training where baselines remain trapped. Excitation is optimizer-, domain-, and model-agnostic, requires minimal integration effort, and introduces neither additional per-parameter optimizer state nor learnable parameters, making it highly viable for memory-constrained settings. Across language and vision tasks, Excitation consistently improves convergence speed and final performance in MoE models, indicating that active update modulation is a key mechanism for effective conditional computation.

Figures & tables

Figure 1: Overview of Excitation . Top: Batch-level expert activations drive competitive update modulation—three strategies are available (here: ZeroSum with 1.5×/0.5× multipliers). Bottom: Optimization trajectory on a 2D toy problem using ZeroSum. Excited SGD (pink) converges faster by amplifying high-consensus expert updates (Expert 1, $w_{0}$ ) while suppressing low-utilization ones (Expert 2, $w_{1}$ ), outperforming standard SGD (gray) and SGD+Momentum (cyan).

Figure 2: CIFAR-10 Foundation Results. Targeted mechanisms ( $\Phi_{ZS},\Phi_{PS}$ ) significantly increase accuracy. The failure of Inverted , Random , and Global-Exp variants demonstrates that success is driven by spatial reinforcement of expert specialization rather than simple learning rate scaling.

Base	Optimizer	Dev Accuracy (%)	$\Delta$
SGD	Vanilla	41.01 $\pm$ 0.33	—
	Excited (zs)	46.58 $\pm$ 0.32	+5.57
	Excited (ps)	46.59 $\pm$ 0.68	+5.58
SGD (mom=0.9)	Vanilla	50.65 $\pm$ 0.20	—
	Excited (zs)	51.75 $\pm$ 0.24	+1.10
	Excited (ps)	51.96 $\pm$ 0.30	+1.31

Table 1: Comparison of Base and Excited optimizers on CIFAR-10 (Top-

K

MLP,

d=128,L=4

, sparsity

0.9

\Delta

denotes absolute percentage improvement over the corresponding vanilla baseline.

Panel of Figure 3 — Figure 3: Rescuing deep sparse models from training collapse. Standard optimizers remain trapped at near-random accuracy due to poor routing in deep layers. Excitation overcomes this bottleneck, enabling both SGD and Adam to “escape” the random-guessing regime and initiate meaningful specialization.

Panel of Figure 4 — Figure 4: Routing Specialization by Depth. Gini coefficients of expert utilization. While standard optimizers exhibit a specialization “dip” in intermediate layers, Excitation maintains high selectivity and sharper routing across all network depths.

Panel of Figure 5 — Figure 5: Evolution of Routing Entropy. Comparison of mean entropy across training. Excitation exhibits faster decay and a lower entropy floor, demonstrating that update modulation enables earlier convergence to highly specialized states.

Panel of Figure 6 — Figure 6: Granularity vs. Capacity (MoE-ViT). (a) Structural Rescue: In “tall and skinny” regimes Excitation mitigates gradient dilution, enabling accuracy leap over vanilla plateaus. (b) Safety Net: In high-capacity regimes Excitation matches tuned baselines, ensuring robustness without over-regularization.

N	k	$d_{ff}$ =256	$d_{ff}$ =384	$d_{ff}$ =512	$d_{ff}$ =768	$d_{ff}$ =1024
4	2	77M $\mid$ +0.64	83M $\mid$ +0.42	89M $\mid$ +0.15	102M $\mid$ +0.75	115M $\mid$ +0.55
8	2	89M $\mid$ +0.72	102M $\mid$ +0.83	115M $\mid$ +0.64	140M $\mid$ +0.92	165M $\mid$ +1.15
	4	89M $\mid$ +0.48	102M $\mid$ +0.31	115M $\mid$ +0.19	140M $\mid$ +0.29	165M $\mid$ +0.47
16	2	115M $\mid$ +1.18	140M $\mid$ +0.49	165M $\mid$ +0.27	216M $\mid$ +0.35	266M $\mid$ +0.67
	4	115M $\mid$ +0.75	140M $\mid$ +0.70	165M $\mid$ +0.51	216M $\mid$ +0.61	266M $\mid$ +0.65
	8	115M $\mid$ +0.45	140M $\mid$ +0.35	165M $\mid$ +0.15	216M $\mid$ +0.39	266M $\mid$ +0.39

Table 2: Relative Perplexity (

\Delta

PPL) Improvement across 60 GPT-MoE Architectures. Values denote absolute PPL reduction (

\text{PPL}_{\text{Adam}}-\text{PPL}_{\text{Excited}}

); positive values indicate improved (lower) perplexity over the baseline.

Figure 7: GPT-MoE Scaling Dynamics. Relative improvement (

\Delta

PPL) over Adam as a function of the number of experts (

N

) for different routing densities (

k

) and hidden widths (

d_{ff}

	Dev Accuracy (%)
LR ( $\eta$ )	Vanilla	Excited (zs)	Excited (ps)
Adam Variants
1.00e-4	$48.47$ 0.23	$49.81$ 0.29	50.55 $\pm$ 0.24
1.50e-4	$49.85$ 0.20	$51.22$ 0.15	51.65 $\pm$ 0.33
2.00e-4	$51.09$ 0.11	$51.66$ 0.29	52.63 $\pm$ 0.29
5.00e-4	$53.69$ 0.63	$52.95$ 0.37	54.07 $\pm$ 0.41

Table 3: Learning Rate Sensitivity (CIFAR-10). Comparison of Vanilla vs. Excited variants ( zerosum , positivesum ) for Adam and SGD. Bold denotes peak accuracy per learning rate.

Sparsity	Vanilla (%)	Excited (%)	$\Delta$
Adam Variants
90%	$48.47$ 0.23	49.81 $\pm$ 0.29	+1.34
70%	$54.01$ 0.15	54.81 $\pm$ 0.28	+0.80
50%	55.49 $\pm$ 0.31	$55.34$ 0.08	-0.15
30%	$55.79$ 0.18	55.86 $\pm$ 0.45	+0.07
10%	56.08 $\pm$ 0.40	$56.00$ 0.15	-0.08

Table 4: Sparsity Analysis (CIFAR-10). Comparison between Vanilla and Excited ( zerosum ) variants.

\Delta

denotes the absolute percentage improvement over the baseline.

Batch Size	Vanilla (%)	Excited (%)	$\Delta$
Adam Variants
16	53.54 $\pm$ 0.26	$53.41$ 0.16	-0.13
32	$52.83$ 0.11	52.87 $\pm$ 0.23	+0.04
64	$51.92$ 0.33	52.08 $\pm$ 0.34	+0.16
128	$50.77$ 0.07	51.69 $\pm$ 0.23	+0.92
256	$49.42$ 0.44	50.55 $\pm$ 0.24	+1.13

Table 5: Batch Size Sensitivity (CIFAR-10). Comparison between Vanilla and Excited ( zerosum ) variants.

\Delta

denotes absolute percentage improvement.

Scheduler	Vanilla (%)	Excited (%)	$\Delta$
Adam Variants
None (Constant)	$50.85$ 0.25	51.28 $\pm$ 0.57	+0.43
Cosine Decay	$48.47$ 0.23	49.81 $\pm$ 0.29	+1.34
SGD Variants
None (Constant)	$44.94$ 0.10	49.10 $\pm$ 0.55	+4.16
Cosine Decay	$41.01$ 0.33	46.58 $\pm$ 0.32	+5.57

Table 6: Scheduler Sensitivity Analysis on CIFAR-10. Comparison between Vanilla and Excited (zerosum) variants with and without Cosine LR scheduling.

\Delta

represents absolute percentage improvement.

Power ( $\gamma$ )	Variant	Dev Accuracy (%)	$\Delta$
Adam Variants (Vanilla: 48.47 $\pm$ 0.23)
1.0	Excited (zs)	$49.81$ 0.29	+1.34
1.0	Excited (ps)	50.55 $\pm$ 0.24	+2.08
2.0	Excited (zs)	$49.47$ 0.64	+1.00
2.0	Excited (ps)	$50.45$ 0.42	+1.98
3.0	Excited (zs)	$49.36$ 0.50	+0.89

Table 7: Power (

p

) Sensitivity Analysis on CIFAR-10. Comparison of Vanilla and Excited variants (zerosum and positivesum) across different power values.

\Delta

represents the absolute percentage improvement over the Vanilla baseline.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Foundational Benchmarks across Datasets. Comparison of Excitation variants against standard SGD and controls. The consistent superiority of targeted excitation confirms that the framework’s ability to resolve structural confusion is domain-agnostic.

Panel of Figure 9 — Figure 9: Evolution of Dev Accuracy Across Base Optimizers. Comparison between vanilla baselines and Excitation variants ( $\Phi_{ZS}$ , $\Phi_{PS}$ ) on CIFAR-10. The framework consistently accelerates the “burn-in” phase of training and achieves higher terminal stability. The most pronounced gains are observed in first-order and simple adaptive methods like SGD and Adagrad, where the structural signal from Excitation compensates for the lack of complex per-parameter state.

Panel of Figure 10 — Figure 10: CIFAR-100 MoE-ViT Scaling. Results mirror the CIFAR-10 trends, with Excitation providing substantial gains in high-granularity regimes while maintaining parity in high-capacity settings.

Panel of Figure 11 — Figure 11: SVHN MoE-ViT Scaling. The consistency of performance across datasets suggests that Excitation ’s ability to facilitate functional differentiation is a domain-agnostic architectural benefit.

Architecture	( $W,L,E$ )	Params	$N_{e}$	Overhead $\chi$ (%)
Neuron-level	2048, 2, –	5.8M	4,096	$33.2$ 28.4
(Top- $k$ MLP)	4096, 2, –	20.0M	8,192	$15.9$ 0.1
	1024, 16, –	16.6M	16,384	$29.7$ 0.4
	16384, 4, –	818.4M	65,536	$18.7$ 0.5
Expert-level	512, 6, 8	56.8M	48	$0.1$ 0.1
(Sparse MoE)	1024, 12, 32	1.64B	384	$-1.2$ 1.9

Appendix

Table 8: Computational overhead analysis (NVIDIA A100-80GB). Model configuration defined by width (

W

), layers (

L

), and experts per layer (

E

N_{e}

denotes total specialized components (

L\times W

for MLP;

L\times E

for MoE). We report relative latency overhead

\chi

(

\pm

std) across 3 sessions (500 trials each, 100 burn-in).

Sparse Mixture-of-Experts

Sparse mixture-of-experts models route each input to a small subset of specialized networks to increase capacity while limiting computation.

See all 121 papers

Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
May 9, 2026Jongseok Park, Sunga Kim, Zhenyu Gu +2Activation Sparsity Mixture-of-Experts Inference
University of California, Berkeley · Work done during an internship at Advanced Micro Devices, Inc. · Advanced Micro Devices, Inc.
DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks
Mar 2, 2026Gökdeniz GülmezMixture-of-Experts Models Expert Routing
Machine Learning Research, Stuttgart, Germany
EMO: Frustratingly Easy Progressive Training of Extendable MoE
May 13, 2026Linghao Jin, Chufan Shi, Huijuan Wang +4Sparse Mixture-of-Experts Efficient Language Model Training
USC-ISI · MBZUAI-IFM

May 9, 2026J/K move · Enter open · S save

Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution

Jongseok Park, Sunga Kim, Zhenyu Gu, Ion Stoica+1

University of California, Berkeley · Work done during an internship at Advanced Micro Devices, Inc. · Advanced Micro Devices, Inc.

Activation Sparsity Mixture-of-Experts Inference LLM Inference Acceleration Mixture-of-Experts Models

Open paper AskPDF arXiv

Mixture of Experts (MoE) architecture has become the standard for state-of-the-art large language models, owing to its computational efficiency through sparse expert activation. However, sparsity through finer expert granularity is becoming increasingly difficult to achieve due to fundamental training challenges such as expert collapse and load imbalance. In this work, we explore and leverage intra-expert activation sparsity as a complementary and underexplored dimension of sparsity in MoE models. Surprisingly, substantial intra-expert sparsity is readily available in existing pre-trained MoE models, without any modification to the activation function or model parameters, providing up to 90% sparsity within each expert without significant accuracy loss. We explore intra-expert activation sparsity across eight off-the-shelf MoE models ranging from 1B to 400B parameters, and extend the MoE execution pipeline of vLLM to leverage intra-expert activation sparsity by skipping the computations of inactive neurons, on top of its existing optimizations, achieving up to 2.5 times speedup in MoE layer execution and 1.2 times end-to-end speedup compared to the original dense vLLM baseline.

Abstract

Sparse Mixture-of-Experts

Explore similar work

Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution

DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks

EMO: Frustratingly Easy Progressive Training of Extendable MoE