Mixture-of-Experts Language Models

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 38% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 147

All topics

XPERT: Expert Knowledge Transfer for Effective Training of Language Models
May 9, 2026Chang Liu, Boyu Shi, Xu Yang +1Mixture-of-Experts Language Models Knowledge Augmentation for Language Models
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China.
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
May 9, 2026Shengkun Tang, Zekun Wang, Bo Zheng +7Mixture-of-Experts Pruning LLM Pruning
Qwen Team, Alibaba Inc. · MBZUAI · KAUST
ZAYA1-VL-8B Technical Report
May 8, 2026Hassan Shapourian, Kasra Hejazi, Olabode M. Sule +1Vision-Language Models Mixture-of-Experts Language Models
Zyphra Technologies Inc, San Francisco, CA
SDG-MoE: Signed Debate Graph Mixture-of-Experts
May 8, 2026Stepan Kulibaba, Kirill Labzin, Artem Dzhalilov +4Mixture-of-Experts Language Models Mixture of Experts
Innopolis University, Innopolis · Sirius University, Sirius · Innopolis University, Innopolis; Sirius University, Sirius +1
Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs
May 8, 2026Andrea Sassella, Andrea Chizzola, Tommaso Bianchi +2LLM Evaluation Mixture-of-Experts Language Models
2DEIB, Politecnico di Milano · 1AIRIC, Politecnico di Milano
When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
May 8, 2026Youngsik Yoon, Siwei Wang, Wei Chen +1Mixture-of-Experts Language Models LLM Routing
Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea
UniPool: Learning Expert-to-Layer Ownership from Brief Global Access
May 7, 2026Minbin Huang, Han Shi, Chuanyang Zheng +5Mixture-of-Experts Language Models Mixture of Experts
The Chinese University of Hong Kong · Huawei Technologies · The University of Hong Kong
EMO: Pretraining Mixture of Experts for Emergent Modularity
May 7, 2026Ryan Wang, Akshita Bhagia, Sewon MinLanguage Model Pretraining Mixture-of-Experts Language Models
University of California, Berkeley · Allen Institute for AI
A Few Good Clauses: Comparing LLMs vs Domain-Trained Small Language Models on Structured Contract Extraction
May 7, 2026Nicole Lincoln, Nick Whitehouse, Jaron Mar +1Legal NLP Mixture-of-Experts Language Models
Onit AI Labs, Onit Inc., New Zealand
ZAYA1-8B Technical Report
May 6, 2026Robert Washbourne, Rishi Iyer, Tomas Figliolia +15Mixture-of-Experts Language Models RL for Language Model Reasoning
Zyphra San Francisco, CA
RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMs
May 1, 2026Zhiyuan Xu, Joseph Gardiner, Sana Belguith +1Mixture-of-Experts Language Models Adversarial Attacks on LLMs
University of Bristol Bristol, United Kingdom
SpaceMoE: Realizing Distributed Mixture-of-Experts Inference over Space Networks
May 1, 2026Zhanwei Wang, Huiling Yang, Min Sheng +2Mixture-of-Experts Language Models Mixture-of-Experts Inference
Department of Electrical and Computer Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China · State Key Laboratory of Integrated Service Networks, Institute of Information Science, Xidian University (XDU), Xi’an, Shaanxi, China · Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology (HKUST), Hong Kong SAR, China
Path-Lock Expert: Separating Reasoning Mode in Hybrid Thinking via Architecture-Level Separation
Apr 29, 2026Shouren Wang, Wang Yang, Chuang Ma +7Mixture-of-Experts Language Models LLM Reasoning
1Case Western Reserve University · 2NII LLMC, Japan · 3Michigan State University
FaaSMoE: A Serverless Framework for Multi-Tenant Mixture-of-Experts Serving
Apr 29, 2026Minghe Wang, Trever Schirmer, Mohammadreza Malekabbasi +1Mixture-of-Experts Language Models LLM Serving
TU Berlin · Berlin, Germany
Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
Apr 28, 2026Fan Jiang, Yu Zhao, Chenyang Lyu +5Multilingual Language Models Mixture-of-Experts Language Models
Alibaba International Digital Commerce
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Apr 26, 2026DeepSeek-AI, Anyi Xu, Bangcai Lin +316Mixture-of-Experts Language Models Long-Context Language Model Inference
Mixture of Heterogeneous Grouped Experts for Language Modeling
Apr 25, 2026Zhicheng Ma, Xiang Liu, Zhaoxiang Liu +5Expert Load Balancing Mixture-of-Experts Language Models
1Data Science & Artificial Intelligence Research Institute, China Unicom · 2Unicom Data Intelligence, China Unicom
Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts
Apr 20, 2026Jacob Morrison, Sanjay Adhikesaven, Akshita Bhagia +3Continual Learning for LLMs Mixture-of-Experts Language Models
University of Washington · Allen Institute for AI · University of California, Berkeley
MoVE: Translating Laughter and Tears via Mixture of Vocalization Experts in Speech-to-Speech Translation
Apr 19, 2026Szu-Chi Chen, I-Ning Tsai, Yi-Cheng Lin +2Speech Translation Mixture-of-Experts Language Models
National Taiwan University, Taipei, Taiwan · NVIDIA, Taiwan
Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
Apr 4, 2026Kening Zheng, Wei-Chieh Huang, Jiahao Huo +9Mixture-of-Experts Language Models Mixture of Experts
University of Illinois Chicago · University of Maryland · HKUST (Guangzhou)
Limited Stereotype Control Through Routing Reweighting in MoE Language Models
Mar 28, 2026Junhyeok Lee, Han Jang, Kyu Sung ChoiSocial Bias in Language Models Demographic Bias in Language Models
Interdisciplinary Program in Cancer Biology, Seoul National University College of Medicine · Department of Radiology, Seoul National University Hospital · Department of Radiology, Seoul National University College of Medicine +1
Federated Mixture-of-Experts Alignment on Mobile Edge Networks under Data Heterogeneity
Mar 22, 2026Zihan Fang, Qianru Wang, Haonan An +4Mixture-of-Experts Language Models Federated Learning Aggregation
Hong Kong JC STEM Lab of Smart City and Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong SAR, China · School of Computer Science and Technology, Xidian University, Xi’an, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Pok Fu Lam, Hong Kong, China +1
Routing-Aware Safety Alignment for Mixture-of-Experts Models
Feb 4, 2026Jiacheng Liang, Yuhui Wang, Tanqiu Jiang +1Mixture-of-Experts Language Models LLM Alignment
Department of Computer Science Stony Brook University
ButterflyMoE: Compression-Scalable Ternary Experts via Structured Butterfly Orbits
Jan 20, 2026Aryan KarmoreTernary Quantization Mixture-of-Experts Language Models
Indian Institute of Information Technology, Nagpur
DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
Jan 8, 2026Guanzhi Deng, Bo Li, Ronghao Chen +5Mixture-of-Experts Language Models Fine-Tuning
City University of Hong Kong · Tsinghua University · Peking University +2
Mixtures of SubExperts for Large Language Continual Learning
Nov 9, 2025Haeyong Kang, Hee Suk Yoon, Dahua Feng +1Continual Learning for LLMs Mixture-of-Experts Language Models
Korea Advanced Institute of Science and Technology (KAIST)
Ensemble Learning for Large Language Models in Text and Code Generation: A Survey
Mar 13, 2025Mari Ashiga, Wei Jie, Fan Wu +5Mixture-of-Experts Language Models Language Model Ensembles
School of Computing and Engineering, University of West London, London W5 5RF, United Kingdom · Turing Intelligence Technology Limited, London EC2M 2PF, United Kingdom · School of Computer Science, University of Leeds, Leeds LS2 9JT, United Kingdom

May 9, 2026, cs.CLJ/K move · Enter open · S save

XPERT: Expert Knowledge Transfer for Effective Training of Language Models

Chang Liu, Boyu Shi, Xu Yang, Xin Geng

School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China.

Mixture-of-Experts Language Models Knowledge Augmentation for Language Models LLM Training

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) language models organize knowledge into explicitly routed expert modules, making expert-level representations traceable and analyzable. By analyzing expert activation patterns in MoE large language models (LLMs), we find that a subset of experts is consistently activated across diverse knowledge domains. These common experts encode cross-domain, generalizable knowledge that is closely related to model generalization, naturally raising the question of how such identifiable expert knowledge can be practically reused. Motivated by this observation, we propose XPERT, a framework that extracts, consolidates, and reuses expert knowledge from pre-trained MoE LLMs to support more effective training of language models across different model scales. XPERT identifies cross-domain experts via inference-only analysis, refines their representations through tensor decomposition, and adapts the extracted knowledge to reuse in downstream models. Experiments on language understanding and dialogue generation benchmarks show that models benefiting from reused expert knowledge achieve consistently stronger performance and faster convergence compared to strong baselines. These results highlight MoE LLMs as structured and reusable knowledge sources, and demonstrate the value of expert-level knowledge reuse for improving model training.

Momentum

Related topics

Latest papers 147