Mixture-of-Experts Language Models

Follow this topic Ask about this topic

Mixture-of-Experts language models divide parts of a transformer into expert networks and use a router to select experts for each token, so a model can have many parameters without activating them all at once. Research explores top-k and other routing rules, load-balancing methods, and training strategies such as pretraining sparse models or converting dense models into experts; evaluations commonly include language-model loss and reasoning, coding, and multilingual benchmarks. A large body of work also aims to make these models easier to fine-tune, compress, interpret, and serve through pruning, quantization, expert sharing, and routing-aware systems. A central tension is how to encourage useful expert specialization while keeping routing reliable and balanced, since communication and memory costs can limit the practical gains from sparse computation.

Latest papers

229 papers37 papers in the last four weeks, up 42% on the four weeks before. 0.2% of all new papers.

SpaceMoE: Realizing Distributed Mixture-of-Experts Inference over Space Networks
May 1, 2026Zhanwei Wang, Huiling Yang, Min Sheng +2Mixture-of-Experts Language Models Mixture-of-Experts Inference
Department of Electrical and Computer Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China · State Key Laboratory of Integrated Service Networks, Institute of Information Science, Xidian University (XDU), Xi’an, Shaanxi, China · Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology (HKUST), Hong Kong SAR, China
Path-Lock Expert: Separating Reasoning Mode in Hybrid Thinking via Architecture-Level Separation
Apr 29, 2026Shouren Wang, Wang Yang, Chuang Ma +7Mixture-of-Experts Language Models LLM Reasoning
1Case Western Reserve University · 2NII LLMC, Japan · 3Michigan State University
FaaSMoE: A Serverless Framework for Multi-Tenant Mixture-of-Experts Serving
Apr 29, 2026Minghe Wang, Trever Schirmer, Mohammadreza Malekabbasi +1Mixture-of-Experts Language Models LLM Serving
TU Berlin · Berlin, Germany
Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
Apr 28, 2026Fan Jiang, Yu Zhao, Chenyang Lyu +5Multilingual Language Models Mixture-of-Experts Language Models
Alibaba International Digital Commerce
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Apr 26, 2026DeepSeek-AI, Anyi Xu, Bangcai Lin +316Mixture-of-Experts Language Models Long-Context Language Model Inference
Mixture of Heterogeneous Grouped Experts for Language Modeling
Apr 25, 2026Zhicheng Ma, Xiang Liu, Zhaoxiang Liu +5Expert Load Balancing Mixture-of-Experts Language Models
1Data Science & Artificial Intelligence Research Institute, China Unicom · 2Unicom Data Intelligence, China Unicom
Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts
Apr 20, 2026Jacob Morrison, Sanjay Adhikesaven, Akshita Bhagia +3Continual Learning for LLMs Mixture-of-Experts Language Models
University of Washington · Allen Institute for AI · University of California, Berkeley
MoVE: Translating Laughter and Tears via Mixture of Vocalization Experts in Speech-to-Speech Translation
Apr 19, 2026Szu-Chi Chen, I-Ning Tsai, Yi-Cheng Lin +2Speech Translation Mixture-of-Experts Language Models
National Taiwan University, Taipei, Taiwan · NVIDIA, Taiwan
Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
Apr 15, 2026Ivan Ternovtsii, Yurii BilakMixture-of-Experts Language Models Expert Routing
Department of Software Systems, Uzhhorod National University Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes
PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models
Apr 14, 2026Han Bao, Penghao Zhang, Yue Huang +9LLM Evaluation Mixture-of-Experts Language Models
University of Notre Dame · Independent Researcher
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Apr 14, 2026NVIDIA, :, Aakshita Chandiramani +544Hybrid Transformer-State Space Model Architectures Mixture-of-Experts Language Models
MoEITS: A Green AI approach for simplifying MoE-LLMs
Apr 12, 2026Luis Balderas, Miguel Lastra, José M. BenítezLLM Compression Mixture-of-Experts Pruning
Department of Computer Science and Artificial Intelligence, DiCITS, iMUDS, DaSCI, University of Granada, Granada, Spain 18071 · Department of Software Engineering, DiCITS, iMUDS, DaSCI, University of Granada, Granada, Spain 18071
TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models
Apr 7, 2026Lin Mu, Haiyang Wang, Li Ni +4Mixture-of-Experts Language Models Low-Rank Adaptation
Anhui University · Hefei University · University of Science and Technology of China
Do Domain-specific Experts exist in MoE-based LLMs?
Apr 7, 2026Giang Do, Hung Le, Truyen TranMixture-of-Experts Language Models Mixture-of-Experts Models
Applied Artificial Intelligence Intiative (A2I2), Deakin University
REAM: Merging Improves Pruning of Experts in LLMs
Apr 6, 2026Saurav Jha, Maryam Hashemzadeh, Ali Saheb Pasand +3LLM Compression Mixture-of-Experts Pruning
Mila – Quebec AI Institute · Polytechnique Montréal · Université de Montréal +3
Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training
Apr 5, 2026Charafeddine MouzouniCongestion Games Expert Load Balancing
OPIT – Open Institute of Technology, and Cohorte AI, Paris, France.
Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
Apr 4, 2026Kening Zheng, Wei-Chieh Huang, Jiahao Huo +9Mixture-of-Experts Language Models Mixture of Experts
University of Illinois Chicago · University of Maryland · HKUST (Guangzhou)
JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency
Apr 3, 2026Aichen Cai, Anmeng Zhang, Anyu Li +66Language Modeling Mixture-of-Experts Language Models
The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level
Apr 2, 2026Jeremy Herbst, Stefan Wermter, Jae Hee LeeMixture-of-Experts Interpretability Mixture-of-Experts Models
Department of Informatics, University of Hamburg, Hamburg, Germany.
FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models
Apr 2, 2026Juyong Jiang, Fan Wang, Hong Qi +2LLM Fine-Tuning Parameter-Efficient Fine-Tuning
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology
Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Models
Apr 2, 2026Shuibai Zhang, Caspian Zhuang, Chihan Cui +8Expert Routing Mixture-of-Experts Language Models
University of Wisconsin-Madison · Scitix · Cornell University +4
Brainstacks: Cross-Domain Cognitive Capabilities via Frozen MoE-LoRA Stacks for Continual LLM Learning
Apr 1, 2026Mohammad R. Abu AyyashContinual Learning for LLMs LLM Fine-Tuning
Brains Build Research, Ramallah, Palestine
Scalable Pretraining of Large Mixture of Experts Language Models on Aurora Super Computer
Apr 1, 2026Dharma Teja Vooturi, Dhiraj Kalamkar, Dipankar Das +1Language Model Pretraining LLM Training
Parallel Computing Lab (India) Intel Corporation
Training-Free Dynamic Upcycling of Expert Language Models
Mar 31, 2026Eros Fanì, Oğuzhan ErsoyMixture-of-Experts Language Models Efficient Language Model Training
Gensyn
Rethinking Language Model Scaling under Transferable Hypersphere Optimization
Mar 30, 2026Liliang Ren, Yang Liu, Yelong Shen +1Language Model Scaling Laws Hyperparameter Transfer
Microsoft
Limited Stereotype Control Through Routing Reweighting in MoE Language Models
Mar 28, 2026Junhyeok Lee, Han Jang, Kyu Sung ChoiSocial Bias in Language Models Demographic Bias in Language Models
Interdisciplinary Program in Cancer Biology, Seoul National University College of Medicine · Department of Radiology, Seoul National University Hospital · Department of Radiology, Seoul National University College of Medicine +1
KALAVAI: Predicting When Independent Specialist Fusion Works -- A Quantitative Model for Post-Hoc Cooperative LLM Training
Mar 24, 2026Ramchand KumaresanLanguage Model Merging Mixture-of-Experts Expert Specialization
Murai Labs
Federated Mixture-of-Experts Alignment on Mobile Edge Networks under Data Heterogeneity
Mar 22, 2026Zihan Fang, Qianru Wang, Haonan An +4Mixture-of-Experts Language Models Federated Learning Aggregation
Hong Kong JC STEM Lab of Smart City and Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong SAR, China · School of Computer Science and Technology, Xidian University, Xi’an, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Pok Fu Lam, Hong Kong, China +1
AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning
Mar 19, 2026Zongfang Liu, Guangyi Chen, Shengkun Tang +3Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Zhejiang University · Westlake University · Carnegie Mellon University +1
Path-Constrained Mixture-of-Experts
Mar 18, 2026Zijin Gu, Tatiana Likhomanenko, Vimal Thilak +2Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Apple · †Google

May 1, 2026J/K move · Enter open · S save

SpaceMoE: Realizing Distributed Mixture-of-Experts Inference over Space Networks

Zhanwei Wang, Huiling Yang, Min Sheng, Khaled B. Letaief+1

Department of Electrical and Computer Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China · State Key Laboratory of Integrated Service Networks, Institute of Information Science, Xidian University (XDU), Xi’an, Shaanxi, China · Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology (HKUST), Hong Kong SAR, China

Mixture-of-Experts Language Models Mixture-of-Experts Inference LEO Satellite Networks Distributed Inference

Open paper AskPDF arXiv

Leveraging continuous solar energy harvesting at high efficiency, space data centers are envisioned as a promising platform for executing energy-intensive large language models (LLMs). Recognizing this advantage, space and AI conglomerates (e.g., SpaceX, Google) are actively investing in this vision. One key challenge, however, is the efficient distributed deployment of a large-scale LLM in a satellite network due to the limited onboard computing and communication resources. This gives rise to a placement problem that involves partitioning and mapping model components to satellites such that the fundamentally different model architecture and network topology can be reconciled to ensure low-latency token generation. To address this problem, we present the Space Network of Mixture-of-Experts (SpaceMoE) framework targeting the distributed execution of a popular mixture-of-experts (MoE) model in space. The proposed placement strategies are two-level: (1) layer placement, which assigns MoE layers to satellite subnets; and (2) intra-layer expert placement, which assigns individual experts to satellites associated with the same layer/subnet. For layer placement, we exploit the ring-like communication pattern of autoregressive inference to partition the satellite constellation along the orbiting direction into subnets arranged on a ring, each hosting one MoE layer. Based on this architecture, we formulate and solve an optimization problem for intra-layer expert placement to map experts with heterogeneous activation probabilities onto satellites. The derived strategy reveals an intuitive principle: a frequently activated expert should be mapped to a satellite on a routing path with low expected latency. Experiments over a thousand-satellite constellation show that SpaceMoE achieves at least a threefold latency reduction compared with conventional random and ablation-based placement strategies.