CRISP: Co-designing Pruning and Scheduling for Efficient MoE Inference on Edge Servers

Congwei Zhang; Youming Tao; Yifei Zou; Yu Yang; Ruirui Zhang; Xiaoyu Zhang; Falko Dressler; Xiuzhen Cheng

cs.LGOct 27, 2026

CRISP: Co-designing Pruning and Scheduling for Efficient MoE Inference on Edge Servers

Authors: Congwei Zhang, Youming Tao, Yifei Zou, Yu Yang, Ruirui Zhang, Xiaoyu Zhang, Falko Dressler, Xiuzhen Cheng,

Organizations: School of Electrical Engineering and Computer Science, TU Berlin, Berlin, Germany · School of Computer Science and Technology, Shandong University, Qingdao, China · Department of Data Science, City University of Hong Kong, Hong Kong, China

Abstract

Mixture-of-Experts (MoE) inference on memory-constrained edge platforms is often dominated by expert-weight movement rather than computation. Although each token activates only a small subset of experts, routing is input-dependent, so all experts must remain accessible and memory movement dominates latency. Existing approaches such as uniform pruning and expert dropping overlook two properties of MoE execution: expert combinations are highly skewed in frequency, and experts contribute asymmetrically within each combination. We present CRISP, a combination-aware framework that co-designs pruning and runtime scheduling for efficient MoE serving on memory-constrained edge platforms. CRISP reorders neurons by importance so one stored expert can support multiple execution widths, assigns differentiated execution to hot, warm, and cold combinations based on frequency and expert role, and exploits heterogeneous widths to overlap expert loading with computation. Across Mixtral-8x7B and DeepSeek-MoE-16B, CRISP improves throughput by up to $2.1\times$ at the same memory budget while limiting accuracy loss to 3.6% relative to full-width inference. Our results highlight that efficient edge MoE serving requires jointly optimizing which expert capacity is preserved and how expert execution is scheduled.

Figures & tables

Figure 1. Uniform pruning disproportionately harms the most frequently used expert pairs.

Figure 2. Expert dropping fails to capture the nuanced, combination-dependent roles of experts. This figure shows the accuracy loss when dropping Expert 3. Red bars indicate combinations where Expert 3 is a significant expert, while green bars indicate it is negligible.

Operation	Time (ms)	Percentage
Expert Loading (2 experts)	11.2	78.0%
Router Computation	0.4	2.8%
Expert Computation	2.6	18.1%
Other (memory copy, etc.)	0.2	1.1%
Total	14.4	100%

Table 1. I/O and computation breakdown for a single MoE layer inference on NVIDIA RTX 4090 with PCIe 4.0 (Mixtral-8x7B, 4-bit quantization).

Figure 3. Distribution of expert combination activation frequency (Act. Freq.) and cumulative frequency (Cum. Freq.) in Mixtral-8x7B with top-2 routing, profiled on 2,048 sequences from the C4 validation set. A small fraction of combinations accounts for the majority of token activations, while most combinations are rarely used.

Figure 4. Asymmetric roles of experts within combinations, visualized by perplexity increase (PPL Inc.) when masking each expert in the top 20 most frequent combinations. Most points deviate from the diagonal (Sym. Line), indicating that one dominant expert contributes significantly more than its auxiliary counterpart.

Figure 5. I/O and computation time of a single expert under different pruning ratios. As expert width decreases, both I/O and computation time are reduced, but at different rates, creating opportunities to overlap the I/O of narrow experts with the computation of wider experts.

Figure 6. Overall architecture of CRISP. The offline phase profiles expert combinations and reorders expert weights to produce tier-aware execution metadata and width-flexible experts. During inference, the router activates an expert combination whose tier determines the execution width of each expert. A pruning-aware scheduler prioritizes wider experts so that their longer computation overlaps with loading of narrower experts, hiding I/O latency.

Figure 7. Perplexity comparison on WikiText-103 and C4 datasets (lower is better).

Figure 8. Zero-shot accuracy on downstream tasks (higher is better).

Figure 9. Iso-memory comparison of CRISP with Uniform Pruning and Expert Dropping under matched expert-memory budgets. Lower perplexity and latency are better, while higher average accuracy is better. Dashed horizontal lines show the quality of full-width execution.

Figure 10. Token generation throughput under different GPU memory budgets on Mixtral-8x7B. The x-axis shows the memory available for expert caching as a fraction of total expert size.

Figure 11. Serving throughput and energy per generated token on discrete-GPU and UMA edge platforms under a 10% expert-memory budget. Error bars show 95% confidence intervals.

Figure 12. Perplexity comparison of different neuron ordering strategies on WikiText-103 at 50% pruning ratio.

Figure 13. Comparison of execution policies on Mixtral-8x7B. PPL denotes WikiText-103 perplexity and Throughput is measured in tokens/second.

Figure 14. Throughput comparison of scheduling strategies on Mixtral-8x7B with 20% memory budget.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 15. Tier-conditioned accuracy and Cold-tier auxiliary-expert analysis. The left panel reports accuracy retention within each execution tier and its share of routing events. The right panel compares accuracy retention, I/O volume, and latency when the Cold tier executes only the dominant expert, retains 50% of the auxiliary expert, or retains the full auxiliary expert.

Figure 16. Cross-domain robustness of CRISP when profiling metadata is transferred between MMLU (MM), HellaSwag (HS), and WinoGrande (WG). Target matched uses metadata profiled on the evaluation workload, Transferred directly applies source-workload metadata, and Metadata refresh updates the metadata on the target workload.

Figure 17. Throughput (tokens/second) across different sequence lengths on Mixtral-8x7B with 20% memory budget.

Component	Mixtral-8x7B	DeepSeek-MoE
IC	18 min	12 min
WR	7 min	4 min
CP	15 min	22 min
DEI	8 min	10 min
Total Offline Time	48 min	48 min

Appendix

Table 2. Offline preparation overhead for CRISP. IC: Importance Computation, WR: Weight Reordering, CP: Combination Profiling, DEI: Dominant Expert Identification.

Component	Mixtral-8x7B	DeepSeek-MoE
TL	0.8 $\mu$ s	1.2 $\mu$ s
PQO	2.3 $\mu$ s	4.7 $\mu$ s
SC	45 $\mu$ s	38 $\mu$ s
Total overhead	48.1 $\mu$ s	43.9 $\mu$ s
Token latency	0.05 %	0.06 %

Appendix

Table 3. Runtime scheduling overhead per token. TL: Tier Lookup, PQO: Priority Queue Operations, SC: Stream Coordination.

Warm-Tier Strategy	PPL $\downarrow$	I/O Volume
Both Full-Width	3.84	100%
Both 75%-Width	4.18	75%
Both 50%-Width	4.67	50%
Asymmetric (100% + 50%)	4.02	75%

Appendix

Table 4. Comparison of Warm-tier execution strategies on Mixtral-8x7B.

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 175 papers

OrderMoE: An expert similarity driven distributed edge MoE inference
Jul 19, 2026Xin Yuan, Ning Li, Quan Chen +2Edge Inference Mixture-of-Experts Inference
School of Ocean Engineering, Harbin Institute of Technology, Heilongjiang, China · School of Computer Science and Technology, Harbin Institute of Technology, Heilongjiang, China · School of Computer Science and Technology, Guangdong University of Technology, Guangdong, China +2
APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference
Aug 12, 2026Alish Kanani, Layan Badawi, Umit Y. OgrasEdge Inference Mixture-of-Experts Inference
University of Wisconsin–Madison
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Sep 16, 2026Yu Lin, Yiming Wang, Runyuan Cai +2Expert Offloading Mixture-of-Experts Inference
AutoArk

Jul 19, 2026J/K move · Enter open · S save

OrderMoE: An expert similarity driven distributed edge MoE inference

Xin Yuan, Ning Li, Quan Chen, Wenchao Xu+1

School of Ocean Engineering, Harbin Institute of Technology, Heilongjiang, China · School of Computer Science and Technology, Harbin Institute of Technology, Heilongjiang, China · School of Computer Science and Technology, Guangdong University of Technology, Guangdong, China+2

Edge Inference Mixture-of-Experts Inference

Open paper AskPDF arXiv

Although mixture-of-experts, MoE, models have been increasingly adopted to scale large language models with moderate computation cost, it remains challenging to deploy MoE inference over resource-constrained and bandwidth-limited edge infrastructures. Existing distributed MoE serving methods mainly rely on exact expert placement, caching, replication, or communication scheduling, while overlooking the functional similarity among experts, which provides an opportunity to reduce cross-server token transmission. Therefore, this paper introduces a similarity-aware expert allocation and distributed deployment framework, dubbed OrderMoE, which aims to accelerate edge MoE inference while balancing inference latency, communication overhead, server workload, and inference quality. OrderMoE first constructs an expert similarity model based on router-induced logits representations and partitions experts in each MoE layer into multiple similarity groups. Then, it develops a similarity-aware expert grouping and deployment strategy to improve local similarity coverage across edge servers. Since reducing remote expert invocation and preserving exact inference quality are conflicting objectives, OrderMoE further designs a quality-aware and trajectory-aware runtime server-expert selection algorithm to decide whether a token should invoke its remote target expert or use a feasible local substitute expert. Experimental results on a real distributed edge testbed show that OrderMoE significantly reduces average latency, tail latency, cross-server traffic, and remote expert invocation ratio, while introducing only small and controllable inference quality degradation.

Abstract

Mixture-of-Experts Inference

Explore similar work

OrderMoE: An expert similarity driven distributed edge MoE inference

APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference

The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction