Mixture-of-Experts Inference

Follow this topic Ask about this topic

Momentum

16 papers in the last four weeks, up 167% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 109

All topics

Democratizing MoE inference on commodity GPUs with CoMoE
Oct 7, 2026Ruwen Fan, Yuezhi Zu, Junru Li +5Mixture-of-Experts Inference
Jimmy · Alibaba Cloud Computing · Tsinghua University
How Sparse Probability Maps Shape Mixture-of-Experts Routing
Oct 5, 2026Tomás Brogueira, Marcos Treviso, Miguel CouceiroMixture-of-Experts Language Models Mixture-of-Experts Inference
Técnico, Universidade de Lisboa · INESC-ID · Instituto de Telecomunicações +2
ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models
Oct 1, 2026Lianjun Liu, Shipeng Li, You Huang +5Mixture-of-Experts Language Models LLM Compression
Hainan University, Haikou, China · Xiamen University, Xiamen, China · Zhejiang Normal University, Jinhua, China
Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs
Sep 30, 2026Jaehwan Lee, Sangmin Lee, Chaewon Kim +2Expert Parallelism Mixture-of-Experts Inference
Dept. of Computer Science and Engineering Seoul National University · Dept. of Computer Science and Engineering Graduate School of Data Science Seoul National University
DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference
Sep 30, 2026Xudong Tan, Peng Ye, Ming Xie +4Mixture-of-Experts Pruning Efficient Multimodal Inference
Fudan University · The Chinese University of Hong Kong
MoRE: Scaling mixture of experts with hardware-aware low-rank routing
Sep 28, 2026Honam Wong, Surbhi Goel, Enric Boix-AdseràParameter-Free Mixture-of-Experts Routing Low-Rank Matrix Decomposition
University of Pennsylvania · The Wharton School, University of Pennsylvania
SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving
Sep 28, 2026Gunho Park, Kyoungho Jeun, Juntaek Oh +3Mixture-of-Experts Pruning Disaggregated LLM Serving
a2sys · KAIST
MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference
Sep 28, 2026Junfeng Wu, Zehao Fan, Hadjer Benmeziane +3Expert Offloading Mixture-of-Experts Language Models
Department of Industrial and Systems Engineering, Rensselaer Polytechnic Institute · Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute · IBM Research
EEG-Fusion: Failure-Informed Source-Free Expert Routing for Robust Motor Imagery EEG Decoding
Sep 27, 2026Abdul Basit, Saim Rehman, Muhammad ShafiqueSource-Free Domain Adaptation MI Classification
eBRAIN Lab, Division of Engineering New York University (NYU) Abu Dhabi, Abu Dhabi, UAE
EAT: Expert Account Tracker for Efficient MoE Inference
Sep 27, 2026Yuexian Li, Yifei Yang, Zouying Cao +1Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Paris Elite Institute of Technology, Shanghai Jiao Tong University · AGI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University +1
OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloading
Sep 27, 2026Jingyuan Xiao, Jiayue Wang, Yitao Hu +7Expert Offloading Mixture of Experts
Tianjin University, Tianjin, China · Tianjin University Tianjin, China
Paging the Experts: A Reproducible Characterization of Flash-Backed MoE Inference on iPhone
Sep 24, 2026Musa ShamsExpert Offloading On-Device Language Model Inference
Independent Researcher
You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
Sep 22, 2026Yuanteng Chen, Qiwei Lai, Chen Tianqi +7Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +1
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Sep 16, 2026Yu Lin, Yiming Wang, Runyuan Cai +2Expert Offloading Mixture-of-Experts Inference
AutoArk
Evidence-Aligned Local Composition of Discrete Experts for Sequence Restoration
Sep 14, 2026Mohammad Panahazari, Usman A. Khan, Shuchin AeronMixture-of-Experts Inference Mixture-of-Experts Models
Department of Electrical and Computer Engineering, Tufts University, Medford, Massachusetts · Boston College, Chestnut Hill, Massachusetts
Quality-Constrained Routing over a Fixed Pool of Quantized Mixture-of-Experts Instances
Sep 14, 2026Zhenghong Huang, Hongfan Wu, Jiheng ZhangMixture-of-Experts Inference Mixture-of-Experts Quantization
The Hong Kong University of Science and Technology Clear Water Bay, Kowloon, Hong Kong
SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
Sep 14, 2026Zihan Wang, Yuqi Wang, Lei Gong +5Expert Offloading LLM Inference Scheduling
University of Science and Technology of China Hefei, China · Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, China
Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts
Sep 10, 2026Dohyeon Kim, Bedionita Soro, Sung Ju HwangParameter-Free Mixture-of-Experts Routing Sparse Mixture-of-Experts
KAIST
PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition
Sep 1, 2026Ziyan Gan, Fangxin Liu, Chenyang Guan +10Mixture-of-Experts Pruning Mixture-of-Experts Inference
1Shanghai Jiao Tong University, China · 2Alibaba Group, China
Candidate-Expanding Routing with Permutation-Stabilized Experts for Mixed-Format Medical VQA
Sep 1, 2026Hai-Dang Nguyen, Huy-Hieu PhamMemory-Augmented VLMs Visual Question Answering
College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam · VinUni-Illinois Smart Health Center, VinUniversity, Hanoi, Vietnam · Center for Innovations in Health Sciences, VinUniversity, Hanoi, Vietnam
DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inference
Aug 31, 2026Xiaoyang Lu, Belthangady Akash Vi Narayana Pai, Xian-He SunEfficient Neural Network Inference LLM Inference Scheduling
Illinois Institute of Technology Chicago, Illinois, USA
TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes
Aug 13, 2026Jie Li, Chenxin Jia, Jinliang Shen +5Expert Parallelism Expert Load Balancing
KlingAI Research
APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference
Aug 12, 2026Alish Kanani, Layan Badawi, Umit Y. OgrasEdge Inference Mixture-of-Experts Inference
University of Wisconsin–Madison
Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
Aug 11, 2026Gongli Zhang, Zhulin Liu, C. L. Philip ChenMixture of Experts Mixture-of-Experts Inference
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China · Pazhou Lab, Guangzhou 510335, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou 510641, China
Tied Trit-Planes: Constraining PTQTP to a Uniform Nine-Level Quantizer, with a Persistent Folded Format for Disk-Streamed Mixture-of-Experts Serving
Aug 9, 2026Matteo GrellaTernary Quantization Mixture-of-Experts Inference
Crisis24
The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
Aug 9, 2026Jiguo LiExpert Parallelism Expert Load Balancing
This report was completed with the assistance of Codex.
UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models
Aug 9, 2026Lei Xin, Bin Gu, Peize Li +8Mixture of Experts Mixture-of-Experts Inference
Kuaishou Technology · Hohai University · Wuhan University +4
EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
Aug 8, 2026Yize Wu, Ke Gao, Ling Li +1Expert Load Balancing Mixture-of-Experts Inference
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
When Does Trace-Driven Evaluation Mislead MoE Expert Caching? Replay Semantics, Workload Contamination, and Operating Regimes
Aug 8, 2026Yu ZhangExpert Offloading Mixture of Experts
China National Chemical Equipment Co. Ltd.
Shape Mutating Expert Compression:LorExperts and BTExperts
Aug 7, 2026Inesh Chakrabarti, Sourjya Roy, Bowen Bao +3Mixture-of-Experts Language Models Low-Rank Matrix Decomposition
Advanced Micro Devices

Oct 7, 2026, cs.DCJ/K move · Enter open · S save

Democratizing MoE inference on commodity GPUs with CoMoE

Ruwen Fan, Yuezhi Zu, Junru Li, Qingda Hu+4

Jimmy · Alibaba Cloud Computing · Tsinghua University

Mixture-of-Experts Inference

Open paper AskPDF arXiv

Deploying Mixture-of-Experts (MoE) models relies heavily on Expert Parallelism, which generates intense inter-GPU communication. Consequently, state-of-the-art inference systems require high-bandwidth, P2P interconnects (e.g., NVLink) in datacenter GPUs to handle massive token routing, making deployment prohibitively expensive. Consumer GPUs offer comparable compute power at significantly lower cost, promising to democratize MoE inference for individuals and enable privacy-preserving local deployments. However, their bandwidth-limited (only weak PCIe bus bandwidth) and host-mediated interconnects (no P2P support) introduce severe communication bottlenecks. We present CoMoE, a communication-efficient MoE inference system that resolves this mismatch through novel host-centric routing. Our key insight is that the unique communication topology provides the opportunity to elevate the host to an active routing hub, which can fundamentally reduce communication volume and eliminate global synchronization-induced stalls. Specifically, for token dispatch, we introduce host-backed token multicast to write shared tokens to the host exactly once, eliminating outbound transmission redundancy. For token combine, we propose a fine-grained, token-level aggregation mechanism using host staging buffers, which replaces rigid global synchronization and mitigates straggler effects. Evaluation on RTX 5090 GPUs shows that CoMoE improves inference throughput by up to 1.46x, approaching the performance of NVLink-capable A800 GPUs at only 23.4% of the hardware cost.

Momentum

Related topics

Latest papers 109