Mixture-of-Experts Inference

Follow this topic Ask about this topic

Momentum

16 papers in the last four weeks, up 167% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 109

All topics

EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding
Aug 5, 2026Sangwoo Ha, Hyunwoo Seo, Yurim Jo +2AI Accelerator Inference On-Device Language Model Inference
KAIST Daejeon, South Korea
Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection
Aug 5, 2026Robin Pan, Raymond Liu, Daniel Fang +2Mixture-of-Experts Inference LLM Inference Acceleration
John A. Paulson School of Engineering and Applied Sciences, Harvard University · Department of Mathematics, Harvard University
AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding
Aug 4, 2026Shuang Liang, Hao Mark Chen, Zhiwen Mo +4Mixture-of-Experts Inference Speculative Decoding
Imperial College London · Tile-AI
Uncertainty Is Not Enough: Value-of-Information Routing for Mixtures of LoRA Experts
Aug 3, 2026Tom Saliencro, Rohan Desai, Priya Nair +2Adaptive Model Routing Selective Prediction
University of California, Irvine · University of Washington
REFLEX: Rethinking MoE Inference as Refinement-Aware Compute Allocation in Diffusion Language Models
Aug 3, 2026Xiang Xia, Cheng Yan, Yiming Zhang +3Expert Load Balancing Mixture-of-Experts Inference
University of Science and Technology of China, Hefei, China
TopoCompress: Topology Aware Token Compression Algorithm for Distributed Edge MoE Inference
Aug 1, 2026Ning Li, Xinyu Wang, Xin Yuan +3Edge Inference Mixture-of-Experts Inference
School of Artificial Intelligence, University of Science and Technology Beijing, China · School of Computer Science and Technology, Harbin Institute of Technology, Heilongjiang, China · School of Ocean Engineering, Harbin Institute of Technology, Heilongjiang, China +2
HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference
Aug 1, 2026Xin Yuan, Ning Li, Wenchao Xu +2Edge Inference Cost-Aware Inference
School of Ocean Engineering, Harbin Institute of Technology, Heilongjiang, China · School of Artificial Intelligence, University of Science and Technology Beijing, China · Division of Integrative Systems and Design, Hong Kong University of Science and Technology, Hong Kong +1
TrimMoE A communication aware and adaptive depth framework for distributed edge inference
Aug 1, 2026Ning Li, Shuting Bai, Xin Yuan +3Adaptive Inference Edge Inference
School of Artificial Intelligence, University of Science and Technology Beijing, China · School of Computer Science and Technology, Harbin Institute of Technology, Heilongjiang, China · School of Ocean Engineering, Harbin Institute of Technology, Heilongjiang, China +2
From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference
Jul 30, 2026Tianyang ZhuSparse Mixture-of-Experts Mixture-of-Experts Inference
Independent Researcher
Incast-Free MoE Rate-Based Scheduling
Jul 28, 2026Evyatar Cohen, Jose Yallouz, Alexander Shpiner +3LLM Inference Scheduling Mixture-of-Experts Inference
Technion · Nvidia · UC Berkeley
OPERA: Offline Policy-guided Expert Routing and Adaptation for Universal Biomedical Image Analysis
Jul 27, 2026Zihan Li, Feiyang Liu, Dandan Shan +2Test-Time Adaptation Medical Image Analysis
University of Washington Seattle, USA · Delft University of Technology Delft, Netherlands · Xiamen University Xiamen, China
DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
Jul 27, 2026Dengke HanExpert Offloading On-Device Language Model Inference
Context-Adaptive Inference: A Unified Statistical and Foundation-Model View
Jul 25, 2026Yue Yao, Caleb N. Ellington, Jingyun Jia +9Adaptive Inference Meta-Learning
Department of Statistics, University of Wisconsin-Madison · Computational Biology Department, Carnegie Mellon University · Department of Computer Science, Yale University +5
Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts
Jul 21, 2026Minyu Cui, Anna Wingkvist, Morgan EricssonGPU Kernel Optimization Communication-Efficient Distributed Training
Linnaeus University · Växjö, Sweden
OrderMoE: An expert similarity driven distributed edge MoE inference
Jul 19, 2026Xin Yuan, Ning Li, Quan Chen +2Edge Inference Mixture-of-Experts Inference
School of Ocean Engineering, Harbin Institute of Technology, Heilongjiang, China · School of Computer Science and Technology, Harbin Institute of Technology, Heilongjiang, China · School of Computer Science and Technology, Guangdong University of Technology, Guangdong, China +2
ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts
Jul 19, 2026Pratyush Dhingra, Pramit Kumar Pal, Janardhan Rao Doppa +1AI Accelerator Inference Mixture-of-Experts Inference
Department of Electrical Engineering and Computer Science, Washington State University, Pullman, WA 99163 USA
Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
Jul 14, 2026Jincheng Xie, Runheng Liu, Heyan Huang +4Mixture-of-Experts Inference Speculative Decoding
Department of Mathematical Sciences, Tsinghua University, Beijing, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JDT AI Infra +2
HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference
Jul 13, 2026Yongqin ZhangAI Accelerator Inference Mixture-of-Experts Inference
Nanjing Vocational College of Information Technology, 99 Wenlan Road, Xianlin College Community, Nanjing, Jiangsu, 210023, P.R.C.
UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods
Jul 7, 2026Yipeng Liu, Chang Liu, Si Shen +16Expert Parallelism High-Performance Computing
State Key Laboratory for Novel Software Technology, Nanjing University · Huawei Technologies Co., Ltd.
Stable Global Weighting of Flow Mixtures using Simplex Exponential Moving Average
Jul 4, 2026Benjamin Wiriyapong, Oktay Karakus, Can Eyupoglu +1Normalizing Flows Mixture-of-Experts Inference
School of Computer Science and Informatics, Cardiff University, Abacws, Cardiff, CF24 4AG, U.K. · Department of Computer Engineering, Turkish Air Force Academy, National Defence University, Istanbul, Türkiye.
Separating Expert Retention from Autonomous Source Inference in Raw-ECG-Replay-Free Continual ECG Deployment
Jul 2, 2026Yufan Lu, Xinhui Liu, Chenyang Xu +2Continual Learning Domain Generalization
Xidian University, China · Department of Computer Science, Tianjin University of Technology, China · DCST, BNRist, RIIT, Institute of Internet Industry, Tsinghua University, China
WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
Jun 20, 2026Jiamu Zhang, Liang Wu, Mayank Darbari +1Expert Offloading Mixture-of-Experts Inference
Nokia, Sunnyvale, CA 94085, USA
Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study
Jun 19, 2026Alfarizy Alfarizy, Hung Truong Thanh Nguyen, René Richard +2LLM Inference Efficiency On-Device Language Model Inference
Analytics Everywhere Lab, University of New Brunswick, Fredericton, NB, Canada · TSAI Lab, University of New Brunswick, Fredericton, NB, Canada
CogniRoute: Learning to Route Social Evidence in Omni-Modal Models
Jun 18, 2026Yifan Shen, Pei Tian, Xinzhuo Li +8RL for Language Model Reasoning Mixture-of-Experts Inference
University of Illinois Urbana-Champaign · 2Columbia University · 3Google
MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference
Jun 17, 2026Craig OpieExpert Offloading On-Device Language Model Inference
Holocron Security, Inc.
A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference
Jun 13, 2026Yingnan Zhao, Razvan Bunescu, Ahmed Louri +2AI Accelerator Inference Mixture-of-Experts Inference
George Washington University Washington, DC, USA · University of North Carolina at Charlotte Charlotte, NC, USA · Ohio University Athens, OH, USA
Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement
Jun 13, 2026Qianli Liu, Kaibin Guo, Zicong Hong +5Expert Parallelism LLM Serving
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong · School of Software Engineering, Sun Yat-Sen University, China · School of Cyber Science and Engineering, Xi’an Jiaotong University, China +1
Sticky Routing: Training MoE Models for Memory-Efficient Inference
Jun 12, 2026Ali KayyamMixture-of-Experts Inference Memory-Efficient Inference
BrainChip Inc., Laguna Hills, CA
Decoupled Mixture-of-Experts for Parametric Knowledge Injection
Jun 12, 2026Baoqing Yue, Weihang Su, Qingyao Ai +5Mixture-of-Experts Language Models Knowledge Augmentation for Language Models
Department of Computer Science and Technology, Tsinghua University
Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design
Jun 9, 2026Wenxin Wang, Yule Hou, Yu Ji +2Expert Parallelism On-Device Language Model Inference
1Tsinghua University · 2Xingyun Integrated Circuits Co., Ltd. · 3Beijing National Research Center for Information Science and Technology

Aug 5, 2026, cs.ARJ/K move · Enter open · S save

EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding

Sangwoo Ha, Hyunwoo Seo, Yurim Jo, Youngjin Moon+1

KAIST Daejeon, South Korea

AI Accelerator Inference On-Device Language Model Inference Mixture-of-Experts Inference Memory-Efficient Inference Speculative Decoding

Open paper AskPDF arXiv

On-device deployment of Large Language Models (LLMs) has become essential for personalized edge applications. A primary bottleneck is external memory access (EMA) in feed-forward network (FFN) layers. Speculative decoding and mixture-of-experts (MoE) are promising solutions. Speculative decoding reduces the number of decoding stages by generating multiple tokens per stage, and MoE minimizes per-stage cost through sparse expert activation. However, there is an incompatibility when combining these two techniques. We propose EdgeXpert, a software-hardware co-designed LLM accelerator that resolves this incompatibility. In the prefill stage, the prompt-wise expert reuse reformulates routing as prompt-level expert reuse rather than independent per-token expert selection. It identifies important tokens using a lightweight encoder, constructs a shared expert set from them, and routes less important tokens with a reduced expert budget to lower expert EMA. In the decode stage, depth-aware expert coalescing exploits the contextual similarity and mutual exclusivity of same-depth candidate tokens. Rather than loading the union of all required channels, EdgeXpert loads only salient channels and applies computational calibration to recover accuracy without additional memory access. Synthesized in Samsung 28nm technology at 800 MHz, EdgeXpert achieves up to 56.3% latency reduction and 44.1% energy reduction compared to prior works, while maintaining near-baseline accuracy.

Momentum

Related topics

Latest papers 109