DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge

Yuegui Huang; Zhiyuan Fang; Weiqi Luo; Ruoyu Wu; Wuhui Chen; Zibin Zheng

cs.LGMar 19, 2026

DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge

Authors: Yuegui Huang, Zhiyuan Fang, Weiqi Luo, Ruoyu Wu, Wuhui Chen, Zibin Zheng

Organizations: Sun Yat-sen University, Guangzhou, China · Tencent, Shenzhen, China

Abstract

Despite the computational efficiency of MoE models, the excessive memory footprint and I/O overhead inherent in multi-expert architectures pose formidable challenges for real-time inference on resource-constrained edge platforms. While existing static methods struggle with a rigid latency-accuracy trade-off, we observe that expert importance is highly skewed and depth-dependent. Motivated by these insights, we propose DyMoE, a dynamic mixed-precision quantization framework designed for high-performance edge inference. Leveraging insights into expert importance skewness and depth-dependent sensitivity, DyMoE introduces: (1) importance-aware prioritization to dynamically quantize experts at runtime; (2) depth-adaptive scheduling to preserve semantic integrity in critical layers; and (3) look-ahead prefetching to overlap I/O stalls. Experimental results on commercial edge hardware show that DyMoE reduces Time-to-First-Token (TTFT) by 3.44x-22.7x and up to a 14.58x speedup in Time-Per-Output-Token (TPOT) compared to state-of-the-art offloading baselines, enabling real-time, accuracy-preserving MoE inference on resource-constrained edge devices.

Figures & tables

Figure 1: Pipeline Comparison: DyMoE vs. Two Conventional MoE Baselines.

Panel of Figure 2 — Figure 2: Overview of MoE Structure and its Memory Demands.

Figure 3: Performance evaluation of various expert pruning strategies on the C-Eval benchmark across different retention ratios. Random : experts are retained randomly; Token-based : experts are prioritized based on the volume of assigned critical tokens; Equal : applies a uniform pruning ratio across all layers; Depth-based : adjusts the retention ratio dynamically according to layer depth.

Figure 4: Comparative Visualization of Expert Routing Distributions: heavy-hitter and general Tokens across Different Inputs.

Figure 5: Layer-wise sensitivity of Mixtral-8x7B-Instruct under Int2 quantization, measured on C-Eval and CMMLU benchmarks.

Figure 6: Adjacent Layer Cosine Similarity.

Figure 7: Overview of DyMoE.

Figure 8: Token-Guided Expert Selection (Prefill). Experts processing more heavy-hitter tokens are prioritized.

Figure 9: Gate-Guided Expert Selection (Decode). Experts with higher routing scores are selected as critical.

Figure 10: End-to-End Performance Comparison.

Dataset	Model	Int2	Int4	BF16
MMLU	Mixtral-8 $\times$ 7B	0.4979	0.6795	0.6889
MMLU	Qwen3-30B-A3B	0.4950	0.7705	0.7800
CMMLU	Mixtral-8 $\times$ 7B	0.3048	0.5044	0.5108
CMMLU	Qwen3-30B-A3B	0.4300	0.8044	0.8063
GSM8K	Mixtral-8 $\times$ 7B	0.0781	0.6467	0.6467
GSM8K	Qwen3-30B-A3B	0.5292	0.8908	0.8954

Table 1: Accuracy Comparison under Uniform Quantization.

Panel of Figure 11 — Figure 11: Accuracy vs. Retention Ratio. Higher $r$ yields better accuracy, showing DyMoE’s flexible trade-off capability.

Dataset	Model	High / Low	$r=0.75$	$r=0.9$	$r=1.0$
MMLU	Mixtral-8 $\times$ 7B	4 / 0	0.6714	0.6807	0.6795
	Mixtral-8 $\times$ 7B	4 / 2	0.6782	0.6821	0.6795
	Qwen3-30B-A3B	4 / 0	0.7698	0.7737	0.7705
	Qwen3-30B-A3B	4 / 2	0.7632	0.7704	0.7705
CMMLU	Mixtral-8 $\times$ 7B	4 / 0	0.4822	0.4940	0.5044
	Mixtral-8 $\times$ 7B	4 / 2	0.4947	0.5053	0.5044

Table 2: Evaluation of DyMoE on Mixtral-8

\times

7B and Qwen3-30B-A3B. The retention ratio

r

denotes the average proportion of Critical experts preserved across layers.

Configuration	16 GB		24 GB
Configuration	TTFT(s)	TPOT(s)	TTFT(s)	TPOT(s)
1. Load on Demand	1.0193	0.2795	1.0193	0.2795
2. Cache	0.5922	0.1489	0.4893	0.1273
3. Cache + Prefetch	0.5159	0.1315	0.3954	0.1108
4. Cache+Dyquant(4/2)	0.5431	0.1307	0.2423	0.0796
5. Cache+Dyquant(4/2)	0.4269	0.1150	0.2121	0.0754

Table 3: Ablation study of DyMoE dynamic strategies. Dyquant denotes our proposed Dynamic Quantization scheme.

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 177 papers

APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference
Aug 12, 2026Alish Kanani, Layan Badawi, Umit Y. OgrasEdge Inference Mixture-of-Experts Inference
University of Wisconsin–Madison
MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios
Feb 12, 2026Shuhuai Li, Jianghao Lin, Dongdong Ge +1Mixture-of-Experts Inference Edge Inference
Shanghai University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Standford University, California, United States
OrderMoE: An expert similarity driven distributed edge MoE inference
Jul 19, 2026Xin Yuan, Ning Li, Quan Chen +2Edge Inference Mixture-of-Experts Inference
School of Ocean Engineering, Harbin Institute of Technology, Heilongjiang, China · School of Computer Science and Technology, Harbin Institute of Technology, Heilongjiang, China · School of Computer Science and Technology, Guangdong University of Technology, Guangdong, China +2

Aug 12, 2026J/K move · Enter open · S save

APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference

Alish Kanani, Layan Badawi, Umit Y. Ogras

University of Wisconsin–Madison

Edge Inference Mixture-of-Experts Inference Memory-Efficient Inference

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models are attractive for edge deployment because they provide high model capacity while activating only a small subset of parameters per token, improving compute efficiency. However, MoE inference at the edge is fundamentally limited by memory. Expert parameters are large and often reside in off-chip memory due to capacity, cost, and power constraints, putting expert loading to the critical path. We present APEX: Adaptive Expert Prefetching, a predictive resource management framework that overlaps expert loading with useful computation. APEX introduces a lightweight prefetch router that predicts candidate experts before the attention block to dynamically fetch additional experts using a learned confidence model. This adaptive strategy achieves over 99% overlap accuracy, significantly outperforming fixed top-k prefetching techniques. APEX supports two execution modes: a correctness-preserving mode that guarantees exact routing semantics, and a stall-free mode that eliminates residual stalls by operating on available experts with negligible impact on application accuracy. Across multiple MoE models, the correctness-preserving mode reduces per-token latency by up to 26% and improves energy-delay product (EDP) by up to 41% over state-of-the-art baselines, while the stall-free mode provides additional efficiency gains with negligible impact on application accuracy. These results establish adaptive, confidence-driven expert prefetching as an effective approach for efficient MoE inference on edge systems.

Abstract

Mixture-of-Experts Inference

Explore similar work

APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference

MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios

OrderMoE: An expert similarity driven distributed edge MoE inference