Mixture-of-Experts Language Models

Overview

Mixture-of-Experts language models divide parts of a transformer into expert networks and use a router to select experts for each token, so a model can have many parameters without activating them all at once. Research explores top-k and other routing rules, load-balancing methods, and training strategies such as pretraining sparse models or converting dense models into experts; evaluations commonly include language-model loss and reasoning, coding, and multilingual benchmarks. A large body of work also aims to make these models easier to fine-tune, compress, interpret, and serve through pruning, quantization, expert sharing, and routing-aware systems. A central tension is how to encourage useful expert specialization while keeping routing reliable and balanced, since communication and memory costs can limit the practical gains from sparse computation.

Latest papers

CardsList

248 papers23 papers in the last four weeks, down 39% on the four weeks before. 0.2% of all new papers.

  1. DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models

    Jan 8, 2026Guanzhi Deng, Bo Li, Ronghao Chen +5Mixture-of-Experts Language ModelsFine-Tuning

  2. Mixtures of SubExperts for Large Language Continual Learning

    Nov 9, 2025Haeyong Kang, Hee Suk Yoon, Dahua Feng +1Continual Learning for LLMsMixture-of-Experts Language Models

  3. Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers

    Oct 13, 2025Wenhan Ma, Hailin Zhang, Liang Zhao +4RL StabilityRL for Language Models

  4. Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization

    Sep 30, 2025Yaoxiang Wang, Qingguo Hu, Yucheng Ding +6Mixture-of-Experts Language ModelsMixture-of-Experts Models

  5. BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts

    Apr 24, 2025Qingyue Wang, Qi Pang, Xixun Lin +2LLM Backdoor AttacksBackdoor Attacks

  6. Ensemble Learning for Large Language Models in Text and Code Generation: A Survey

    Mar 13, 2025Mari Ashiga, Wei Jie, Fan Wu +5Mixture-of-Experts Language ModelsLanguage Model Ensembles