Mixture-of-Experts Language Models

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 38% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 147

All topics

RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
Oct 8, 2026Ilya Lasy, Nora Yinuo Cai, Kola AyonrindeMixture of Experts Expert Routing
Faculty of Informatics, TU Wien · Independent · UK AI Security Institute
Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts
Oct 8, 2026Yunkai Chai, Tong Zhu, Xiaoye Qu +4Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Shanghai AI Laboratory, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Nanyang Technological University, Singapore
Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
Oct 7, 2026Radha Gulhane, Quentin Anthony, Beren MillidgeMixture-of-Experts Language Models Communication-Efficient Distributed Training
Zyphra
Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression
Oct 7, 2026Tianxiao Cao, Jiahe Shao, Yuning Qiu +3LLM Compression Mixture-of-Experts Language Models
Kyoto University · The University of Tokyo · RIKEN AIP
Distributionally Robust Mixture-of-Experts Training
Oct 5, 2026Xin Teng, Muxiao Li, Hongyi WenMixture-of-Experts Language Models Sparse Mixture-of-Experts
New York University Center for Data Science, NYU Shanghai
How Sparse Probability Maps Shape Mixture-of-Experts Routing
Oct 5, 2026Tomás Brogueira, Marcos Treviso, Miguel CouceiroMixture-of-Experts Language Models Mixture-of-Experts Inference
Técnico, Universidade de Lisboa · INESC-ID · Instituto de Telecomunicações +2
Cross-Lingual Alignment for Decoder-Only Models using MoE Routers
Oct 1, 2026Lucas Bandarkar, Clark Peng, Ahmed Haj Ahmed +2Mixture-of-Experts Language Models Cross-Lingual Representation Alignment
University of California, Los Angeles∗ · Haverford College · MILA - Quebec AI Institute & McGill University
ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models
Oct 1, 2026Lianjun Liu, Shipeng Li, You Huang +5Mixture-of-Experts Language Models LLM Compression
Hainan University, Haikou, China · Xiamen University, Xiamen, China · Zhejiang Normal University, Jinhua, China
Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts
Oct 1, 2026Di He, Pengxiang Li, Da Chang +3Mixture-of-Experts Language Models Recurrent Transformers
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Peng Cheng Laboratory · University of Chinese Academy of Sciences +5
Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Experts
Sep 30, 2026Priya Nair, Lukas Brenner, Maya Lindqvist +5Mixture-of-Experts Language Models Parameter-Free Mixture-of-Experts Routing
University of California, Irvine · University of Washington
Scaling Laws for Looped Mixture of Experts
Sep 30, 2026Yanbei Chen, Anirudh Goyal, Raghuraman KrishnamoorthiMixture-of-Experts Language Models Language Model Scaling Laws
Meta AI
ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control
Sep 30, 2026Peng Jin, Zihan Qiu, Zekun Wang +8Expert Load Balancing Mixture-of-Experts Language Models
Qwen Team, Alibaba Token Hub, Alibaba Group
Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
Sep 29, 2026Yury Nahshan, Nati Daniel, Jacob Goldberger +1Mixture-of-Experts Language Models Parameter-Free Mixture-of-Experts Routing
Bar-Ilan University, Ramat-Gan, Israel · NVIDIA, Israel
How to Loop MoE: Flatten the Experts, Untie the Attention
Sep 28, 2026Shouren Wang, Chuang Ma, Mohsen Hariri +6Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Case Western Reserve University · Kyoto University · NII LLMC
MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference
Sep 28, 2026Junfeng Wu, Zehao Fan, Hadjer Benmeziane +3Expert Offloading Mixture-of-Experts Language Models
Department of Industrial and Systems Engineering, Rensselaer Polytechnic Institute · Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute · IBM Research
EAT: Expert Account Tracker for Efficient MoE Inference
Sep 27, 2026Yuexian Li, Yifei Yang, Zouying Cao +1Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Paris Elite Institute of Technology, Shanghai Jiao Tong University · AGI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University +1
RAZOR: Pruning Replaceable Experts in LLMs
Sep 24, 2026Mingyang Song, Mao ZhengMixture-of-Experts Pruning LLM Pruning
Foundation Model Department, Tencent, China
You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
Sep 22, 2026Yuanteng Chen, Qiwei Lai, Chen Tianqi +7Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +1
From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs
Sep 22, 2026Zhentao Tan, Chang Liu, Yao Liu +2Mixture-of-Experts Language Models Sparse Fine-Tuning
Alibaba Group · Southeast University
Higher-order pruning of experts in mixture-of-experts language models
Sep 16, 2026Alex M. Tseng, Prannay Kaul, Luca Zancato +2Mixture-of-Experts Pruning Mixture-of-Experts Language Models
AI Fundamental Research AWS Agentic AI
MoRE: Mixture of Reused Experts
Sep 16, 2026Eric S. Qiu, Utku Umur Acikalin, Justin Lovelace +4Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Department of Computer Science, Cornell University
OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
Sep 9, 2026Dezhi Li, Lujun Li, Qiyuan Zhu +4Mixture-of-Experts Pruning Mixture-of-Experts Language Models
The Hong Kong University of Science and Technology
How Fragile Is Safety Alignment at Frontier Scale? A Single-Direction Attack on a 320B MoE
Sep 9, 2026Yi Shi, Tanyu Chen, Kai ShenMixture-of-Experts Language Models LLM Alignment
Continuum AI
ProtLingo: Efficient Protein Language Modeling via Conditional Memory and Expert Routing
Sep 7, 2026Mingrui Li, Sixian Shen, Minzhang Li +4Mixture-of-Experts Language Models Protein Fitness Prediction
ShanghaiTech University
Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts
Sep 1, 2026Nikolaos Xiros, Dimitrios Damianos, Maria-Eleni Zoumpoulidi +3Mixture-of-Experts Language Models Expert Routing
Institute for Language and Speech Processing, Athena Research Center, Greece
Instella-MoE Technical Report
Sep 1, 2026Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra +10Mixture-of-Experts Language Models Language Model Post-Training
Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMs
Sep 1, 2026Seungwoo Jung, Dohyeok Kwon, Seungmin Cha +4Mixture-of-Experts Language Models LLM Compression
Department of Computer Science and Engineering, Korea University · Department of Computer Science and Artificial Intelligence, Dongguk University
TuringLLM: Efficiently Scaling Foundation Models Toward Physical AI
Aug 31, 2026Yuheng Zhang, Yizhao Wang, Da Zhu +19Mixture-of-Experts Language Models Long-Context Language Model Inference
Foundation Model Team, Xpeng Inc.
Q-Strata: Hierarchical Bit Allocation for Mixed-Precision Quantization of Mixture-of-Experts LLMs
Aug 31, 2026Deokjae Lee, Sihun Chu, Hyun Oh SongMixed-Precision Quantization LLM Quantization
Department of Computer Science and Engineering, Seoul National University · Neural Processing Research Center · 3Neural Processing Research Center +2
A.X K2 Technical Report
Aug 31, 2026Cheolseung Baek, Dhammiko Arya, Eunki Kim +40Mixture-of-Experts Language Models Long-Context Language Modeling

Oct 8, 2026, cs.AIJ/K move · Enter open · S save

RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing

Ilya Lasy, Nora Yinuo Cai, Kola Ayonrinde

Faculty of Informatics, TU Wien · Independent · UK AI Security Institute

Mixture of Experts Expert Routing Mixture-of-Experts Language Models Neural Network Interpretability

Open paper AskPDF arXiv

Sparse Mixture of Experts (MoE) models scale more efficiently than dense models by routing tokens to modular expert networks that are only active for processing a fraction of tokens. A leading hypothesis for the performance of MoE models is that each expert specialises in a single, coherent domain. However, interpretability efforts that assume this hypothesis have generally been unsuccessful. We propose and present evidence for an alternative account that we call the Superposed Specialisation Hypothesis (SSH): experts specialise in a disjoint union of fine-grained features rather than one broad domain. Leveraging the SSH, we introduce RouterInterp, a method for interpreting expert routing that identifies Sparse Autoencoder features most predictive of routing decisions and produces unified natural language explanations. On gpt-oss-20b, RouterInterp explains expert routing with

{\sim}65\%

higher detection accuracy than prior token statistics based methods. This work provides a scalable method for generating more accurate explanations of expert routing and increases our understanding of a previously uninterpretable component of foundation models.

Momentum

Related topics

Latest papers 147