Expert Routing

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 225% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 77

All topics

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
May 11, 2026Chenyang Song, Weilin Zhao, Xu Han +3Sparse Mixture-of-Experts Mixture-of-Experts Inference
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China
Hierarchical Mixture-of-Experts with Two-Stage Optimization
May 8, 2026Gleb Molodtsov, Alexander Miasnikov, Aleksandr BeznosikovExpert Load Balancing Mixture of Experts
BRAIn Lab, Moscow, Russia
StrLoRA: Towards Streaming Continual Visual Instruction Tuning for MLLMs
May 8, 2026Chang Che, Ziqi Wang, Hui Ma +2VLM Adaptation Low-Rank Adaptation
Hefei University of Technology · Tsinghua University
Expert Routing for Communication-Efficient MoE via Finite Expert Banks
May 6, 2026Mohammad Reza Deylam Salehi, Ali KhalesiMixture of Experts Information-Theoretic Generalization Bounds
IEEE Graduate Member · Nice, France · Institut Polytechnique des Sciences Avancées (IPSA) +1
Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts
May 6, 2026Klaus-Rudolf Kladny, Maximilian Mordig, Bernhard Schölkopf +1Mixture-of-Experts Inference Expert Routing
Max Planck Institute for Intelligent Systems, Tübingen · Tübingen AI Center · ETH Zurich +1
Soft-to-Hard Routing in Sparse Mixture-of-Experts Models
May 4, 2026Reza RastegarSparse Mixture-of-Experts Expert Routing
Meta Platforms, Inc Woodinville, WA
Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts
May 1, 2026Man Yung WongSparse Mixture-of-Experts Mixture-of-Experts Inference
Russell · Independent Researcher
SceneSelect: Selective Learning for Trajectory Scene Classification and Expert Scheduling
Apr 27, 2026Xinrun Wang, Deshun Xia, Yuxi Sun +1Multi-Agent Trajectory Prediction Expert Routing
School of Computer Science, China University of Geosciences (Wuhan) · School of Information Engineering, Wuhan University of Technology · School of Mathematics and Statistics, Wuhan University of Technology
SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
Apr 27, 2026Zi-Hao Bo, Yaqian Li, Anzhou Hou +6Vision-Language Models Efficient VLM Inference
Li Auto Inc.
Teacher-Guided Routing for Sparse Vision Mixture-of-Experts
Apr 23, 2026Masahiro Kada, Ryota Yoshihashi, Satoshi Ikehata +2Adaptive Model Routing Sparse Mixture-of-Experts
Institute of Science Tokyo · Tokyo, Japan · DENSO IT Laboratory
Multi-Domain Learning with Global Expert Mapping
Apr 20, 2026Pourya Shamsolmoali, Masoumeh Zareapoor, Huiyu Zhou +3Vision Foundation Model Adaptation Expert Routing
Department of Computer Science, University of York, UK · School of Computing and Mathematical Sciences, University of Leicester, UK · Centre for Vision Speech and Signal Processing, University of Surrey, UK +2
Layer-wise MoE Routing Locality under Shared-Prefix Code Generation: Token-Identity Decomposition and Compile-Equivalent Fork Redundancy
Apr 19, 2026Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino +1Expert Routing Efficient Language Model Inference
Graduate School of Informatics, Nagoya University, Japan · Information Technology Center, Nagoya University, Japan
CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answering
Apr 18, 2026Xiyin Zeng, Yi Lu, Hao WangVisual Question Answering Expert Routing
Hong Kong University of Science and Technology (Guangzhou)
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
Feb 5, 2026Jingze Shi, Zhangyang Peng, Yizhang Zhu +3Mixture-of-Experts Inference LLM Inference Acceleration
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence
Routing-Aware Safety Alignment for Mixture-of-Experts Models
Feb 4, 2026Jiacheng Liang, Yuhui Wang, Tanqiu Jiang +1Mixture-of-Experts Language Models LLM Alignment
Department of Computer Science Stony Brook University
L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts
Jan 29, 2026Minghao Yang, Ren Togo, Guang Li +2Expert Routing Mixture-of-Experts Models
Hokkaido University
Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference
Date pendingZhenhe Wu, Yaping Jin, Qinghua Xing +6Mixture of Experts Memory-Efficient Optimization
Huawei Technologies · Beihang University · Tianjin University +2

May 11, 2026, cs.LGJ/K move · Enter open · S save

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices

Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao+2

Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China

Sparse Mixture-of-Experts Mixture-of-Experts Inference Expert Routing On-Device Inference

Open paper AskPDF arXiv

While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, which hinder efficient end-side deployment that simultaneously requires high performance, low computational cost, and small storage overhead. To achieve these properties, we present DECO, a sparse MoE architecture designed to match the performance of dense Transformers under identical total parameter budgets and training tokens. DECO utilizes the differentiable and flexible ReLU-based routing enhanced by learnable expert-wise scaling, which adaptively balances the contributions of routed and shared experts. Furthermore, we introduce NormSiLU, an activation function that normalizes inputs prior to SiLU operators, producing a more stable trend of routed-expert activation ratio and a higher intrinsic sparsity level. We also identify an empirical advantage in using non-gated MLP experts with ReLU-based routing, indicating the possibility of MoE architecture simplification. Experiments demonstrate that DECO, activating only 20% of routed experts, matches dense performance and outperforms established MoE baselines. Our specialized acceleration kernel delivers a 2.93

\times

speedup on Jetson AGX Orin compared with dense inference. Code and checkpoints are available at https://github.com/thunlp/DECO.

Momentum

Related topics

Latest papers 77