Mixture-of-Experts Language Models

Follow this topic Ask about this topic

Mixture-of-Experts language models divide parts of a transformer into expert networks and use a router to select experts for each token, so a model can have many parameters without activating them all at once. Research explores top-k and other routing rules, load-balancing methods, and training strategies such as pretraining sparse models or converting dense models into experts; evaluations commonly include language-model loss and reasoning, coding, and multilingual benchmarks. A large body of work also aims to make these models easier to fine-tune, compress, interpret, and serve through pruning, quantization, expert sharing, and routing-aware systems. A central tension is how to encourage useful expert specialization while keeping routing reliable and balanced, since communication and memory costs can limit the practical gains from sparse computation.

Latest papers

229 papers37 papers in the last four weeks, up 42% on the four weeks before. 0.2% of all new papers.

Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency
Mar 17, 2026Lucas Bandarkar, Alan Ansell, Trevor CohnConcept Localization in Language Models LLM Interpretability
Google Research · University of California, Los Angeles · University of Melbourne
EngGPT2: Sovereign, Efficient and Open Intelligence
Mar 17, 2026G. Ciarfaglia, A. Rosanova, S. Cipolla +17Language Modeling Open-Weight Language Models
LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing
Mar 13, 2026Jiawei Hao, Zhiwei Hao, Jianyuan Guo +4Mixture-of-Experts Language Models LLM Compression
School of Information and Electronics, Beijing Institute of Technology, Beijing, China · Department of Computer Science, City University of Hong Kong, Hong Kong, China · Shenzhen Campus of Sun Yat-sen University, Shenzhen, China +2
Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers
Mar 11, 2026Mynampati Sri Ranganadha AvinashMixture-of-Experts Interpretability Mixture-of-Experts Language Models
Independent Researcher | Asthra Labs
Speculating Experts Accelerates Inference for Mixture-of-Experts
Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts Inference LLM Inference Acceleration
University of Maryland · TogetherAI
MoE Lens -- An Expert Is All You Need
Mar 6, 2026Marmik Chaudhari, Idhant Gulati, Nishkal Hundia +2Mixture-of-Experts Expert Specialization Mixture-of-Experts Models
The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA
NeuronMoE: Neuron-Guided Mixture-of-Experts for Efficient Multilingual LLM Extension
Mar 5, 2026Rongzhi Li, Hitomi YanakaMultilingual Language Models Mixture-of-Experts Language Models
The University of Tokyo · Riken · Tohoku University
Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guarantees
Mar 3, 2026Mohammed Nowaz Rabbani Chowdhury, Hsinyu Tsai, Geoffrey W. Burr +3Mixture-of-Experts Inference Heterogeneous Computing
Rensselaer Polytechnic Institute · IBM Research
Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds
Feb 19, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaExpert Routing Mixture-of-Experts Language Models
Department of Computer Science, Iowa State University · Department of Civil, Construction and Environmental Engineering, Iowa State University
Arcee Trinity Large Technical Report
Feb 19, 2026Varun Singh, Lucas Krauss, Sami Jaghouar +23LLM Training Mixture-of-Experts Language Models
Arcee AI · Prime Intellect · DatologyAI
ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
Feb 17, 2026Ziyu Zhao, Tong Zhu, Zhi Zhang +6Mixture-of-Experts Language Models Mixture-of-Experts Pruning
Zhejiang University · Shanghai Innovation Institute · Shanghai AI Laboratory +3
Routing-Aware Safety Alignment for Mixture-of-Experts Models
Feb 4, 2026Jiacheng Liang, Yuhui Wang, Tanqiu Jiang +1Mixture-of-Experts Language Models LLM Alignment
Department of Computer Science Stony Brook University
Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning
Jan 24, 2026Fatema Siddika, Md Anwar Hossen, Tanwi Mallick +1Continual Learning for LLMs Continual Learning
Iowa State University, Ames, USA · Argonne National Laboratory, USA
ButterflyMoE: Compression-Scalable Ternary Experts via Structured Butterfly Orbits
Jan 20, 2026Aryan KarmoreTernary Quantization Mixture-of-Experts Language Models
Indian Institute of Information Technology, Nagpur
DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
Jan 8, 2026Guanzhi Deng, Bo Li, Ronghao Chen +5Mixture-of-Experts Language Models Fine-Tuning
City University of Hong Kong · Tsinghua University · Peking University +2
Mixtures of SubExperts for Large Language Continual Learning
Nov 9, 2025Haeyong Kang, Hee Suk Yoon, Dahua Feng +1Continual Learning for LLMs Mixture-of-Experts Language Models
Korea Advanced Institute of Science and Technology (KAIST)
Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization
Sep 30, 2025Yaoxiang Wang, Qingguo Hu, Yucheng Ding +6Mixture-of-Experts Language Models Mixture-of-Experts Models
Xiamen University · Shanghai Jiao Tong University · University of Science and Technology of China +1
Ensemble Learning for Large Language Models in Text and Code Generation: A Survey
Mar 13, 2025Mari Ashiga, Wei Jie, Fan Wu +5Mixture-of-Experts Language Models Language Model Ensembles
School of Computing and Engineering, University of West London, London W5 5RF, United Kingdom · Turing Intelligence Technology Limited, London EC2M 2PF, United Kingdom · School of Computer Science, University of Leeds, Leeds LS2 9JT, United Kingdom
Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing
Apr 25, 2024Peizhuang Cong, Aomufei Yuan, Shimao Chen +4Mixture-of-Experts Language Models Mixture-of-Experts Models
Peking University

Mar 17, 2026J/K move · Enter open · S save

Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency

Lucas Bandarkar, Alan Ansell, Trevor Cohn

Google Research · University of California, Los Angeles · University of Melbourne

Concept Localization in Language Models LLM Interpretability Mixture-of-Experts Language Models

Open paper AskPDF arXiv

Modern LLMs continue to exhibit significant variance in behavior across languages, such as being able to recall factual information in some languages but not others. While typically studied as a problem to be mitigated, in this work, we propose leveraging this cross-lingual inconsistency as a tool for interpretability in mixture-of-experts (MoE) LLMs. Our knowledge localization framework contrasts routing for sets of languages where the model correctly recalls information from languages where it fails. This allows us to isolate model components that play a functional role in answering about a piece of knowledge. Our method proceeds in two stages: (1) querying the model with difficult factual questions across a diverse set of languages to generate "success" and "failure" activation buckets and then (2) applying a statistical contrastive analysis to the MoE router logits to identify experts important for knowledge. To validate the necessity of this small number of experts for answering a knowledge question, we deactivate them and re-ask the question. We find that despite only deactivating about 20 out of 6000 experts, the model no longer answers correctly in over 40% of cases. Generally, this method provides a realistic and scalable knowledge localization approach to address increasingly complex LLMs.