Mixture-Of-Experts Architectures

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

Jul 6Week of Sep 21

Latest papers 46

All topics

SpikeMoE: Brain-Inspired Competitive Routing for Flexible Spiking Mixture-of-Experts
Oct 1, 2026Xiaoli Liu, Yujie Liang, Jialin Li +1Spiking Neural Networks Mixture-Of-Experts Architectures
School of Computer Science and Engineering, University of Electronic Science and Technology of China
ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control
Sep 30, 2026Peng Jin, Zihan Qiu, Zekun Wang +8Load Balancing Mixture-Of-Experts Architectures
Qwen Team, Alibaba Token Hub, Alibaba Group
Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
Sep 29, 2026Yury Nahshan, Nati Daniel, Jacob Goldberger +1Mixture-Of-Experts Large Language Models Mixture-Of-Experts Architectures
Bar-Ilan University, Ramat-Gan, Israel · NVIDIA, Israel
X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depths
Sep 28, 2026Bowen Dong, Yilong Fan, Tengyu Pan +6Mixture-Of-Experts Architectures Scaling Laws
Tsinghua University · Tianjin University
SMAT: Simple and Efficient Merge-Aware Training
Sep 27, 2026Yanggan Gu, Yuanyi Wang, Zhen Li +5Continual Model Merging Soft-Token Representations
The Hong Kong Polytechnic University (PolyU) · The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong +1
ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Experts
Sep 14, 2026Dang Hoai Nam, Nguyen Duy Hieu, Quang Huu Hieu +1Handwritten Text Recognition Handwriting
University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · AJ Technologies, Nagoya, Japan +1
Chimaera: A Mixture-of-Graph-Experts Architecture for Cross-Task and Cross-Dataset Graph Learning
Sep 8, 2026Jonathan Frank, David Richerby, Ansgar ScherpGraph Foundation Model Text-Attributed Graph
University of Ulm · University of Essex
Hyperparameter Scaling Laws Across MoE Sparsity
Sep 8, 2026Changxin Tian, Kunlong Chen, Jia Liu +3Hyperparameter Mixture-Of-Experts Architectures
Ling Team, Ant Group
Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts
Sep 1, 2026Nikolaos Xiros, Dimitrios Damianos, Maria-Eleni Zoumpoulidi +3Mixture-Of-Experts Architectures Mixture-Of-Experts
Institute for Language and Speech Processing, Athena Research Center, Greece
Structure Aware Neural Architecture Search for Mixture of Experts
Aug 30, 2026Petr Babkin, Oleg BakhteevMixture-Of-Experts Architectures Neural Architecture Search
Regime-Gated Residual Mixture-of-Experts for Cross-Sectional Volatility Forecasting
Aug 12, 2026Junyi Ye, Gargi Vijay BordeVolatility Heteroskedasticity
School of Computing, Montclair State University · Montclair, New Jersey, USA
Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts
Aug 11, 2026Soumajyoti Sarkar, Yuxin Tang, Sheng ZhaMixture-Of-Experts Architectures Sparsity
Amazon AGI Foundations
The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
Aug 9, 2026Jiguo LiMixture-Of-Experts Architectures Mixture-Of-Experts
This report was completed with the assistance of Codex.
Policy-Masked Private Experts: Auditable and Reversible Capability Access Control in Sparse MoE Models
Aug 7, 2026Zhuoheng Huang, Mukesh SinghMixture-Of-Experts Architectures Experts
Independent Researcher
MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Jul 24, 2026Qingyu Yang, Haonan He, Minglei Li +4Mixture-Of-Experts Large Language Models Training-Side Stage-Aware Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory · KTH Royal Institute of Technology · University of Science and Technology of China +2
NEST: Tackling Dataset-Level Distribution Shifts via Regime-Oriented Mixture-of-Experts
Jul 7, 2026Lanhao Li, Bingshu Xie, Lijun Sun +3Mixture-Of-Experts Architectures Distribution Shifts
1Beihang University
Gemma 4 Technical Report
Jul 2, 2026Gemma Team, Sherif El Abd, Vaibhav Aggarwal +298Technical Report Mixture-Of-Experts Architectures
Learning Subset-Shared Invariances for Domain Generalization with Mixture-of-Experts
Jun 24, 2026Tien-Hung Nguyen, Tien-Dat Tran, M. -Duong Nguyen +1Multimodal Domain Generalization Mixture-Of-Experts Architectures
VinUniversity, Vietnam
Training-free Task Classification for Multi-Task Model Merging
Jun 21, 2026Jungyong Son, Jinwook Jung, Sungyong BaikContinual Model Merging Multi--Task Learning
Department of Artificial Intelligence Hanyang University · Department of Data Science Hanyang University
SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling
Jun 15, 2026Weiqiao Shan, Ruixiang Mao, Yuang Li +10Mixture-Of-Experts Architectures Mixture-Of-Experts
Northeastern University, China · Huawei TSC, China · CUHK-Shenzhen, China +4
FAME: Forecastability-Aware Mixture of Experts for Heterogeneous Time Series Forecasting
Jun 8, 2026Qianyang Li, Xingjun Zhang, Shaoxun Wang +2Mixture-Of-Experts Architectures Experts
School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China · Department of Research and Development, Shandong New Beiyang Information Technology Co., Ltd., WeiHai 264200, China · Department of Computer Science and Technology, Tsinghua University, Beijing 100084
Variational Proximal Policy Optimization
Jun 6, 2026Ousmane Amadou DiaProximal Policy Optimization Variational Inference
Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling
Jun 5, 2026Rohan ShravanMixture-Of-Experts Architectures Large Language Model Training
The School of AI Bengaluru, India
DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
May 31, 2026Jiarui Feng, Hanqing Zeng, Karish Grover +11Mixture-Of-Experts Architectures Mixture-Of-Experts
Meta MRS · Washington University in St. Louis · Carnegie Mellon University +1
Learning Multi-Modal Trajectory Policies for Data-Efficient Robotic Manipulation
May 31, 2026Zijia Chen, Yuenan Hou, Xinhua Jiang +3Multimodal Action Distributions Robotic Manipulation
College of Electronic Science and Technology, National University of Defense Technology Changsha, 410073, China · Shanghai AI Laboratory Shanghai, 200000, China
Eigenvectors of Experts are Training-free Non-collapsing Routers
May 29, 2026Giang Do, Hung Le, Truyen TranMixture-Of-Experts Architectures Experts
Applied Artificial Intelligence Intiative (A2I2), Deakin University, Victoria, Australia.
VEN-VL: A Visual Ensemble MoE Framework for Effective and Efficient Multi-Modal Understanding
May 25, 2026Yinghao Wu, Zhuoyan Luo, Yiyao Yu +3Multimodal Understanding Mixture-Of-Experts Architectures
Tsinghua University
Task-Routed Mixture-of-Experts with Cognitive Appraisal for Implicit Sentiment Analysis
May 20, 2026Yaping Chai, Haoran Xie, Joe S. QinSentiment Mixture-Of-Experts Architectures
Division of Artificial Intelligence, Lingnan University, Hong Kong
Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts
May 20, 2026Gene Tangtartharakul, Katherine R. StorrsMixture-Of-Experts Architectures Mixture-Of-Experts
School of Psychology · University of Auckland · Auckland, New Zealand
Dynamic Model Merging Made Slim
May 17, 2026Guodong Du, Wanyu LinContinual Model Merging Mixture-Of-Experts Architectures
The Hong Kong Polytechnic University
UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Models
May 15, 2026Van-Tuan Tran, Hong-Hanh Nguyen-Le, Marco Ruffini +1Heterogeneous Federated Learning Mixture-Of-Experts Architectures
School of Computer Science and Statistics, Trinity College Dublin, Ireland · School of Computer Science, University College Dublin, Ireland
When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing
May 15, 2026Libo Sun, Po-wei Harn, Peixiong He +1Mixture-Of-Experts Architectures Mixture-Of-Experts
Department of Computer Science and Software Engineering Auburn University, Auburn, AL 36849, USA · Department of Information Management National Central University, Taoyuan 320317, Taiwan · Department of Computer Science and Software Engineering Auburn University
HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts
May 13, 2026Tao Zhong, Dongzhe Zheng, Christine Allen-BlanchetteMixture-Of-Experts Architectures Data Compression Methods
Princeton University
ReForge: Refining Merged Models with Anchor-Regularized Regression
May 13, 2026Kaiyang Li, Shaobo Han, Qing Su +1Continual Model Merging Multi--Task Learning
School of Computing, University of Connecticut, Storrs, CT 06269 · Optical Networking and Sensing, NEC Labs America, Princeton, NJ 08540
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
May 11, 2026Chenyang Song, Weilin Zhao, Xu Han +3Mixture-Of-Experts Architectures Mixture-Of-Experts
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China
FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning
May 10, 2026Xing Han, Shravan Chaudhari, Tanvi Ranade +2Mixture-Of-Experts Architectures
Department of Computer Science Johns Hopkins University · Department of ECE Johns Hopkins University · Department of Computer Science Johns Hopkins University, Bayesian Health
SDG-MoE: Signed Debate Graph Mixture-of-Experts
May 8, 2026Stepan Kulibaba, Kirill Labzin, Artem Dzhalilov +4Mixture-Of-Experts Architectures Mixture-Of-Experts
Innopolis University, Innopolis · Sirius University, Sirius · Innopolis University, Innopolis; Sirius University, Sirius +1
AxMoE: Characterizing the Impact of Approximate Multipliers on Mixture-of-Experts DNN Architectures
May 6, 2026Omkar B Shende, Marcello Traiola, Gayathri AnanthanarayananMixture-Of-Expert Inference Neural Network Inference
Dept. of Computer Science and Engineering · Univ Rennes, CNRS, Inria, IRISA
GEM: Graph-Enhanced Mixture-of-Experts with ReAct Agents for Dialogue State Tracking
May 6, 2026Ziqi Zhu, Adithya Suresh, Tomal Deb +1Mixture-Of-Experts Architectures Mixture-Of-Experts
Generative AI Innovation Center, Amazon Web Services
The Thinking Pixel: Recursive Sparse Reasoning in Multimodal Diffusion Latents
Apr 28, 2026Yuwei Sun, Yuxuan Yao, Hui Li +1Multimodal Diffusion Models Diffusion Models
Shanghai Academy of AI for Science · Fudan University
Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution
Apr 24, 2026Haiyun Qiu, Xingyu Wu, Kay Chen TanMixture-Of-Experts Architectures Mixture-Of-Experts
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong, China
Teacher-Guided Routing for Sparse Vision Mixture-of-Experts
Apr 23, 2026Masahiro Kada, Ryota Yoshihashi, Satoshi Ikehata +2Mixture-Of-Experts Architectures Mixture-Of-Experts
Institute of Science Tokyo · Tokyo, Japan · DENSO IT Laboratory
Quadruped Parkour Learning: Sparsely Gated Mixture of Experts with Visual Input
Apr 21, 2026Michael Ziegltrum, Jianhao Jiao, Tianhu Peng +2Mixture-Of-Experts Architectures Mixture-Of-Experts
Robot Perception and Learning Lab, Intelligent Robotics, Department of Computer Science, University College London, Gower Street, WC1E 6BT, London, UK.
Multi-Domain Learning with Global Expert Mapping
Apr 20, 2026Pourya Shamsolmoali, Masoumeh Zareapoor, Huiyu Zhou +3Mixture-Of-Experts Architectures Mixture-Of-Experts
Department of Computer Science, University of York, UK · School of Computing and Mathematical Sciences, University of Leicester, UK · Centre for Vision Speech and Signal Processing, University of Surrey, UK +2
Application of a Mixture of Experts-based Foundation Model to the GlueX DIRC Detector
Apr 17, 2026Cristiano Fanelli, James Giroux, Cole Granger +1Wireless Foundation Models Mixture-Of-Experts Architectures
William & Mary, Department of Data Science, Williamsburg, VA 23185, USA · William & Mary, Department of Physics, Williamsburg, VA 23185, USA
StanceMoE: Mixture-of-Experts Architecture for Stance Detection
Apr 1, 2026Abdullah Al Shafi, Md. Milon Islam, Sk. Imran Hossain +1Cross-Topic Arabic Stance Detection Bert-Based Models
Department of Computer Science and Engineering, Khulna University of Engineering & Technology

Oct 1, 2026, cs.AIJ/K move · Enter open · S save

SpikeMoE: Brain-Inspired Competitive Routing for Flexible Spiking Mixture-of-Experts

Xiaoli Liu, Yujie Liang, Jialin Li, Malu Zhang

School of Computer Science and Engineering, University of Electronic Science and Technology of China

Spiking Neural Networks Mixture-Of-Experts Architectures Mixture-Of-Experts Neuromorphic Computing Localized Lora-Moe

Open paper AskPDF arXiv

Spiking Neural Networks (SNNs) enable event-driven computation through biologically inspired dynamics at the neuronal scale, while Mixture-of-Experts (MoE) perform conditional computation through expert selection at the model scale. Integrating their strengths offers potential for flexible neural architectures. A key challenge, however, lies in designing an expert selection mechanism based on spiking activity. To address this, we introduce a spike-based k-WTA Router inspired by competition-inhibition observed in the hippocampal CA1 region. The router incorporates lateral inhibition and refractory period to select Top-K experts according to discrete spike counts. Building on this, we present SpikeMoE, a framework that integrates neuronal-scale spiking dynamics with model-scale expert selection. To address incomplete multisensory inputs in multimodal tasks, we further equip SpikeMoE with a two-stage missing-modality modeling module that combines empirical prototypes from an observed-modality pool with modality-specific learnable embeddings to construct missing-modality representations. Experiments on vision, language, and multimodal benchmarks demonstrate that SpikeMoE achieves state-of-the-art performance among the SNN baselines, matches or exceeds the performance of ANN counterparts, and maintains robustness across diverse missing-modality conditions. These results demonstrate a favorable trade-off between performance and energy efficiency, validating the integration of spiking dynamics with sparse expert computation and highlighting SpikeMoE as a promising approach to energy-efficient brain-inspired computing.

Momentum

Related topics

Latest papers 46