Mixture-of-Experts Language Models

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 38% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 147

All topics

dMoE: dLLMs with Learnable Block Experts
May 29, 2026Sicheng Feng, Zigeng Chen, Gongfan Fang +2Mixture-of-Experts Language Models Memory-Efficient Inference
National University of Singapore
Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation
May 28, 2026Aditi Khandelwal, Marius Mosbach, Verna Dankers +2Multilingual Language Models Continual Learning for LLMs
Mila – Quebec AI Institute & McGill University
Understanding Safety-Sensitive Expert Behavior in Mixture-of-Experts LLMs
May 28, 2026Zhibo Zhang, Yuxi Li, Zhen Ouyang +2Mixture-of-Experts Language Models LLM Auditing
Huazhong University of Science and Technology, Wuhan, China · Nanyang Technological University, Singapore
ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compression
May 28, 2026Yilun Yao, Jiaming Pan, Elsie Dai +3Mixture-of-Experts Language Models LLM Compression
Peking University
RA-MoE: Routing-Aligned Fine-Tuning for Multilingual Adaptation of Mixture-of-Experts Models
May 27, 2026Guanzhi Deng, Kuan Wu, Haibo Wang +6Mixture-of-Experts Language Models LLM Fine-Tuning
City University of Hong Kong, Hong Kong, China · Carnegie Mellon University, Pittsburgh, USA · The University of Hong Kong, Hong Kong, China
PrunePath: Towards Highly Structured Sparse Language Models
May 27, 2026Zhexuan Gu, Zixun Fu, Yancheng YuanMixture-of-Experts Pruning LLM Pruning
Department of Applied Mathematics, The Hong Kong Polytechnic University
Pruning and Distilling Mixture-of-Experts into Dense Language Models
May 27, 2026Junhyuck Kim, Jihun Yun, Haechan Kim +3Mixture-of-Experts Pruning LLM Pruning
1KRAFTON · 2KAIST
Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts
May 27, 2026Liu O. Martin, Lucas Bandarkar, Nanyun PengMixture-of-Experts Pruning LLM Pruning
University of California, Los Angeles
FPMoE: A Sparse Mixture-of-Experts Approach to Functional Code Generation
May 27, 2026Loc Pham, Lang Hong Nguyet Anh, Thanh Le-CongMixture-of-Experts Language Models Code Generation
GreenNode AI, Singapore · Hanoi University of Science and Technology · Singapore University of Technology and Design
Laguna M.1/XS.2 Technical Report
May 26, 2026Julien Abadji, Marah Abdin, Connor Adams +93Mixture-of-Experts Language Models Software Engineering Agents
MobileMoE: Scaling On-Device Mixture of Experts
May 26, 2026Yanbei Chen, Hanxian Huang, Ernie Chang +5Mixture-of-Experts Language Models On-Device Language Model Inference
Meta AI
L2Rec: Towards Dual-View Understanding of LLMs for Personalized Recommendation
May 26, 2026Pingjun Pan, Tingting Zhou, Peiyao Lu +3Mixture-of-Experts Language Models LLM Personalization
Netease Cloud Music Hangzhou, China
The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
May 26, 2026MiniMax, :, Aili Chen +204Mixture-of-Experts Language Models Agentic RL
RouteScan: A Non-Intrusive Approach to Auditing MoE LLMs Safety via Expert Routing Telemetry
May 24, 2026Bo Lv, Zhiheng Xu, KeDong Xiu +4Mixture-of-Experts Language Models LLM Auditing
Zhejiang University Hangzhou, China · Donghua University Shanghai, China · Louisiana State University Baton Rouge, Louisiana, USA
Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
May 23, 2026Bo Li, Tianyu Dong, Shaolin Zhu +1Multilingual Language Models Mixture-of-Experts Language Models
School of Software, Tsinghua University · College of Intelligence and Computing, Tianjin University
Safety-Oriented Routing Analysis of Mixtral MoE Under Benign and Harmful Prompts
May 22, 2026Md Nurul Absar SiddikyMixture-of-Experts Language Models Expert Routing
Department of Electrical and Computer Engineering University of Hawai’i at M¯anoa Honolulu, HI, USA
BitsMoE: Cost-Aware Bit Allocation in Spectral Space for MoE LLM Quantization
May 22, 2026Jiayu Zhao, Zihan Teng, Minhao Fan +4Mixed-Precision Quantization LLM Quantization
School of Microelectronics, University of Science and Technology of China · College of Computing and Data Science, Nanyang Technological University · School of Electrical and Electronic Engineering, Nanyang Technological University
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
May 21, 2026Jianing Deng, Song Wang, Dongwei Wang +4Mixed-Precision Quantization LLM Quantization
University of Pittsburgh · University of Central Florida · University of Arizona +1
Dynamic Mixture of Latent Memories for Self-Evolving Agents
May 21, 2026Dianzhi Yu, Vireo Zhang, Hongru Wang +7Continual Learning for LLMs Mixture-of-Experts Language Models
The Chinese University of Hong Kong · Nanyang Technological University · University of Edinburgh +2
DBES: A Systematic Benchmark and Metric Suite for Evaluating Expert Specialization in Large-Scale MoEs
May 18, 2026Jing Wang, Hongxuan Lu, Jazze Young +2Mixture-of-Experts Language Models Mixture-of-Experts Models
A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM $Δ$ Integration into Upcycled MoE
May 18, 2026Hao Zhou, Tianhao Li, Zhijun Wang +6Multilingual Language Models Mixture-of-Experts Language Models
National Key Laboratory for Novel Software Technology, Nanjing University · Tongyi Lab, Alibaba Group · Zhejiang University
CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
May 18, 2026Yang Liu, Toan Nguyen, Flora D. SalimContinual Learning for LLMs Vision-Language Models
School of Computer Science and Engineering University of New South Wales Australia
Mixture of Experts for Low-Resource LLMs
May 17, 2026Ori Bar Joseph, Smadar Arvatz, Noam Kayzer +2Mixture-of-Experts Language Models Expert Routing
Scalable Knowledge Editing for Mixture-of-Experts LLMs via Tensor-Structured Updates
May 15, 2026Roman Maksimov, Vladimir Aletov, Dmitry Bylinkin +3Knowledge Editing Mixture-of-Experts Language Models
1Moscow Independent Research Institute of Artificial Intelligence · 2Basic Research of Artificial Intelligence Laboratory · 3Innopolis University
Dense vs Sparse Pretraining at Tiny Scale: Active-Parameter vs Total-Parameter Matching
May 13, 2026Abdalrahman WaelLanguage Model Pretraining Mixture-of-Experts Language Models
Independent Researcher
ROMER: Expert Replacement and Router Calibration for Robust MoE LLMs on Analog Compute-in-Memory Systems
May 12, 2026Wenyong Zhou, Yuannuo Feng, Yizhe Chen +6Compute-in-Memory Mixture-of-Experts Language Models
The Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong · The School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China
Slicing and Dicing: Configuring Optimal Mixtures of Experts
May 12, 2026Margaret Li, Sneha Kudugunta, Danielle Rothermel +1Mixture-of-Experts Language Models Mixture of Experts
Paul G Allen School of Computer Science University of Washington Seattle, USA · Courant School of Data Science New York University New York, USA
HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model
May 11, 2026Noam Kayzer, Dan Revital, Ori Bar Joseph +10Mixture-of-Experts Language Models Open-Weight Language Models
PwC Next
HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
May 11, 2026Jia Wei, Zhonghao Zhang, Ping Chen +5Mixture-of-Experts Language Models Fine-Tuning
Department of Computer Science and Technlogy Tsinghua University Beijing 100084, China · School of Computer Science and Technlogy Xi’an Jiaotong University · The State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
Sparse Layers are Critical to Scaling Looped Language Models
May 9, 2026Ryan Lee, Jacob Biloki, Edward J. Hu +1Mixture-of-Experts Language Models Recurrent Transformers
1USC Information Sciences Institute · 2Netflix · 3Independent Researcher

May 29, 2026, cs.CLJ/K move · Enter open · S save

dMoE: dLLMs with Learnable Block Experts

Sicheng Feng, Zigeng Chen, Gongfan Fang, Xinyin Ma+1

National University of Singapore

Mixture-of-Experts Language Models Memory-Efficient Inference Diffusion Language Model Inference Diffusion Language Models

Open paper AskPDF arXiv

Diffusion Large Language Models (dLLMs) have recently emerged as a promising alternative to autoregressive models, offering competitive performance while naturally supporting parallel decoding. However, as dLLMs are increasingly integrated with Mixture-of-Experts (MoE) architectures to scale model capacity, a fundamental mismatch arises between block parallel decoding and token-level expert selection. Specifically, each dLLM forward pass processes multiple tokens with bidirectional dependencies, whereas conventional MoE layers route each token independently. This mismatch substantially increases the number of uniquely activated experts, making inference increasingly memory-bound. To address this, we propose dMoE, a simple yet effective block-level MoE framework. The central idea of dMoE is to aggregate token-level expert distributions within each block into a unified block-level expert distribution, which is then used to guide expert routing in a more coherent manner. In this way, dMoE substantially reduces the number of uniquely activated experts during inference without sacrificing performance, thereby mitigating the memory-bound bottleneck. Extensive experiments across a variety of benchmarks demonstrate the effectiveness of dMoE. On average, dMoE reduces the number of uniquely activated experts from 69.5 to 14.6 while retaining 99.11% of the original performance. Meanwhile, it reduces memory usage by 76.64% to 79.84% and achieves 1.14

\times

to 1.66

\times

end-to-end latency speedup. Code is available at: https://github.com/fscdc/dMoE

Momentum

Related topics

Latest papers 147