EAT: Expert Account Tracker for Efficient MoE Inference

Yuexian Li; Yifei Yang; Zouying Cao; Hai Zhao

cs.AISep 27, 2026

EAT: Expert Account Tracker for Efficient MoE Inference

Authors: Yuexian Li, Yifei Yang, Zouying Cao, Hai Zhao

Organizations: Paris Elite Institute of Technology, Shanghai Jiao Tong University · AGI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University · Ant Group

Abstract

Mixture-of-Experts (MoE) models have emerged as a revolutionary method to scale Transformer models. However, traditional MoE architecture still suffers from inefficiency since a large number of experts are unnecessarily activated. Existing approaches for reducing the number of activated experts often overlook the historical performance of each expert. In this paper, we propose EAT, a novel method called Expert Account Tracker (EAT), which utilizes history-awareness metrics and adaptive thresholding to dynamically select the most important experts, thereby reducing the activated expert number while effectively maintaining the model performance. Experiments show that EAT outperforms the existing baseline Top-P method across multiple models and datasets, achieving over 25% an average reduction compared to the vanilla method in the number of activated experts and performing better token generation speed compared to the baseline. Furthermore, the performance of pruned models can be efficiently recovered via OPD using only 9K data. Additionally, through ablation studies, we find that excessively reducing the number of activated experts can significantly harm model performance, and the importance of experts varies across layers, with higher-level experts being generally more critical.

Figures & tables

Figure 1: Overview of MoE architecture and our proposed EAT method. EAT selects the most useful experts by continuously tracking their historical contributions and incorporating this information into the routing decision.

Hyperparameter	Value	Description
$\alpha$	0.95	Smoothing factor for $\mathcal{C}_{s}^{(t)}$ .
$w_{1},w_{2},w_{3}$	0.3, 0.3, 0.4	Weights to compute $\mathcal{S}_{h,i}$ .
$\beta$	0.6	Balance factor for final importance $\mathcal{I}_{i}$ .
$\alpha_{1},\alpha_{2}$	0.5, 1.0	Scaling coefficients for threshold $\tau$ .
$\kappa_{1},\kappa_{2}$	0.7, 0.3	Weights for threshold adjustment.
$K$ (Mixtral/Phi)	2	Max activated experts per token.

Table 1: Key hyperparameters of the proposed EAT routing strategy.

LLM	Method	Reasoning		Language		Know.	Examination		Und.
LLM	Method	HeSw	PIQA	CHID	WSC	BoolQ	MMLU	CMMLU	XSum
Mixtral -8x7B -v0.1	Vanilla	77.11	81.07	37.51	61.54	69.11	71.67	53.11	9.19
	Top-P	75.72	79.71	32.85	60.58	66.15	65.38	46.74	8.70
	EAT	76.79	80.30	33.97	63.46	68.40	70.37	51.04	9.08
	EAT + OPD	77.05	80.96	35.42	64.83	69.25	71.59	52.66	9.10
Phi -3.5-MoE -instruct	Vanilla	75.17	80.20	66.75	68.27	75.32	76.64	61.03	14.68

Table 2: Main experimental results evaluated on the OpenCompass Platform. Know. denotes Knowledge benchmarks and Und. denotes Understanding benchmark.

LLM	Method	Reasoning		Language		Know.	Examination		Und.
LLM	Method	HeSw	PIQA	CHID	WSC	BoolQ	MMLU	CMMLU	XSum
Mixtral -8x7B -v0.1	Vanilla	2.00	2.00	2.00	2.00	2.00	2.00	2.00	2.00
	Top-P	1.50	1.44	1.75	1.52	1.46	1.51	1.74	1.51
	EAT	1.47	1.46	1.44	1.50	1.45	1.47	1.48	1.47
	EAT + OPD	1.50	1.47	1.46	1.50	1.45	1.69	1.57	1.71
Phi -3.5-MoE -instruct	Vanilla	2.00	2.00	2.00	2.00	2.00	2.00	2.00	2.00

Table 3: Comparison of the average number of activated experts under different expert routing strategies.

Length	Mixtral-8x7B			Phi-3.5-MoE
	Vanilla	Top-P	EAT	Vanilla	Top-P	EAT
2048+256	0.5214	0.5100	0.5196	1.1319	0.9479	1.1600
1024+128	0.3931	0.3739	0.3864	0.9295	0.9073	0.9158
512+32	0.3814	0.1596	0.3671	0.8060	0.8719	1.1421

Table 4: Average token generation speed (token/sec).

Figure 2: Curve of the Relationship Between PPL and Number of Activated Experts.

Figure 3: Curve of the Relationship Between layer number and Number of Activated Experts.

Figure 4: Curve of PPL on OPD with different subset size on Qwen30B on PIQA dataset.

Figure 5: Curve of PPL on OPD with different steps on Qwen30B on BOOLQ dataset.

Explore similar work

It Takes a MAESTRO To Prune Bad Experts
Jul 9, 2026Palaash Goel, Ayush Maheshwari, Tanmoy ChakrabortyExperts Latent Trajectories
Indian Institute of Technology Delhi, India · NVIDIA, India
Post-Trained MoE Can Skip Half Experts via Self-Distillation
May 18, 2026Xingtai Lv, Li Sheng, Kaiyan Zhang +12Mixture-Of-Experts Experts
Tsinghua University · Shanghai AI Lab · WeChat AI +1
OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
Sep 9, 2026Dezhi Li, Lujun Li, Qiyuan Zhu +4Mixture-Of-Experts Large Language Models Large Language Model Compression
The Hong Kong University of Science and Technology

Jul 9, 2026, cs.CLJ/K move · Enter open · S save

It Takes a MAESTRO To Prune Bad Experts

Palaash Goel, Ayush Maheshwari, Tanmoy Chakraborty

Indian Institute of Technology Delhi, India · NVIDIA, India

Experts Latent Trajectories

Open paper AskPDF arXiv

Sparsely-activated Mixture-of-Experts (MoE) language models achieve remarkable inference efficiency by activating only a small fraction of parameters per token, yet their full expert banks reside in memory at all times, creating a prohibitive deployment bottleneck. Existing structured pruning methods, largely designed for dense transformers, assess expert importance using locally derived heuristics that are blind to the interdependent nature of MoE routing. We introduce MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based ROuting), a structured pruning framework designed for MoE architectures that models autoregressive expert activation trajectories as Ergodic Markov chains whose stationary distributions encode cross-layer dependencies, yielding a globally aware importance heuristic. Evaluated across five diverse domains including Safety, Bias, and Ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention under a strict 50% compression regime, while exhibiting substantially lower cross-task variance, indicating that global, routing-congruent pruning produces models that generalize more consistently across heterogeneous tasks.

Abstract

Explore similar work

It Takes a MAESTRO To Prune Bad Experts

Post-Trained MoE Can Skip Half Experts via Self-Distillation

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit