Mixture-of-Experts Language Models

Follow this topic Ask about this topic

Mixture-of-Experts language models divide parts of a transformer into expert networks and use a router to select experts for each token, so a model can have many parameters without activating them all at once. Research explores top-k and other routing rules, load-balancing methods, and training strategies such as pretraining sparse models or converting dense models into experts; evaluations commonly include language-model loss and reasoning, coding, and multilingual benchmarks. A large body of work also aims to make these models easier to fine-tune, compress, interpret, and serve through pruning, quantization, expert sharing, and routing-aware systems. A central tension is how to encourage useful expert specialization while keeping routing reliable and balanced, since communication and memory costs can limit the practical gains from sparse computation.

Latest papers

229 papers37 papers in the last four weeks, up 42% on the four weeks before. 0.2% of all new papers.

L2Rec: Towards Dual-View Understanding of LLMs for Personalized Recommendation
May 26, 2026Pingjun Pan, Tingting Zhou, Peiyao Lu +3Mixture-of-Experts Language Models LLM Personalization
Netease Cloud Music Hangzhou, China
The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
May 26, 2026MiniMax, :, Aili Chen +204Mixture-of-Experts Language Models Agentic RL
RouteScan: A Non-Intrusive Approach to Auditing MoE LLMs Safety via Expert Routing Telemetry
May 24, 2026Bo Lv, Zhiheng Xu, KeDong Xiu +4Mixture-of-Experts Language Models LLM Auditing
Zhejiang University Hangzhou, China · Donghua University Shanghai, China · Louisiana State University Baton Rouge, Louisiana, USA
Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
May 23, 2026Bo Li, Tianyu Dong, Shaolin Zhu +1Multilingual Language Models Mixture-of-Experts Language Models
School of Software, Tsinghua University · College of Intelligence and Computing, Tianjin University
Safety-Oriented Routing Analysis of Mixtral MoE Under Benign and Harmful Prompts
May 22, 2026Md Nurul Absar SiddikyMixture-of-Experts Language Models Expert Routing
Department of Electrical and Computer Engineering University of Hawai’i at M¯anoa Honolulu, HI, USA
BitsMoE: Cost-Aware Bit Allocation in Spectral Space for MoE LLM Quantization
May 22, 2026Jiayu Zhao, Zihan Teng, Minhao Fan +4Mixed-Precision Quantization LLM Quantization
School of Microelectronics, University of Science and Technology of China · College of Computing and Data Science, Nanyang Technological University · School of Electrical and Electronic Engineering, Nanyang Technological University
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
May 21, 2026Jianing Deng, Song Wang, Dongwei Wang +4Mixed-Precision Quantization LLM Quantization
University of Pittsburgh · University of Central Florida · University of Arizona +1
Dynamic Mixture of Latent Memories for Self-Evolving Agents
May 21, 2026Dianzhi Yu, Vireo Zhang, Hongru Wang +7Continual Learning for LLMs Mixture-of-Experts Language Models
The Chinese University of Hong Kong · Nanyang Technological University · University of Edinburgh +2
DBES: A Systematic Benchmark and Metric Suite for Evaluating Expert Specialization in Large-Scale MoEs
May 18, 2026Jing Wang, Hongxuan Lu, Jazze Young +2Mixture-of-Experts Language Models Mixture-of-Experts Models
A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM $Δ$ Integration into Upcycled MoE
May 18, 2026Hao Zhou, Tianhao Li, Zhijun Wang +6Multilingual Language Models Mixture-of-Experts Language Models
National Key Laboratory for Novel Software Technology, Nanjing University · Tongyi Lab, Alibaba Group · Zhejiang University
CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
May 18, 2026Yang Liu, Toan Nguyen, Flora D. SalimContinual Learning for LLMs Vision-Language Models
School of Computer Science and Engineering University of New South Wales Australia
Mixture of Experts for Low-Resource LLMs
May 17, 2026Ori Bar Joseph, Smadar Arvatz, Noam Kayzer +2Mixture-of-Experts Language Models Expert Routing
Scalable Knowledge Editing for Mixture-of-Experts LLMs via Tensor-Structured Updates
May 15, 2026Roman Maksimov, Vladimir Aletov, Dmitry Bylinkin +3Knowledge Editing Mixture-of-Experts Language Models
1Moscow Independent Research Institute of Artificial Intelligence · 2Basic Research of Artificial Intelligence Laboratory · 3Innopolis University
Dense vs Sparse Pretraining at Tiny Scale: Active-Parameter vs Total-Parameter Matching
May 13, 2026Abdalrahman WaelLanguage Model Pretraining Mixture-of-Experts Language Models
Independent Researcher
ROMER: Expert Replacement and Router Calibration for Robust MoE LLMs on Analog Compute-in-Memory Systems
May 12, 2026Wenyong Zhou, Yuannuo Feng, Yizhe Chen +6Compute-in-Memory Mixture-of-Experts Language Models
The Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong · The School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China
Slicing and Dicing: Configuring Optimal Mixtures of Experts
May 12, 2026Margaret Li, Sneha Kudugunta, Danielle Rothermel +1Mixture-of-Experts Language Models Mixture of Experts
Paul G Allen School of Computer Science University of Washington Seattle, USA · Courant School of Data Science New York University New York, USA
HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model
May 11, 2026Noam Kayzer, Dan Revital, Ori Bar Joseph +10Mixture-of-Experts Language Models Open-Weight Language Models
PwC Next
HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
May 11, 2026Jia Wei, Zhonghao Zhang, Ping Chen +5Mixture-of-Experts Language Models Fine-Tuning
Department of Computer Science and Technlogy Tsinghua University Beijing 100084, China · School of Computer Science and Technlogy Xi’an Jiaotong University · The State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
Sparse Layers are Critical to Scaling Looped Language Models
May 9, 2026Ryan Lee, Jacob Biloki, Edward J. Hu +1Mixture-of-Experts Language Models Recurrent Transformers
1USC Information Sciences Institute · 2Netflix · 3Independent Researcher
XPERT: Expert Knowledge Transfer for Effective Training of Language Models
May 9, 2026Chang Liu, Boyu Shi, Xu Yang +1Mixture-of-Experts Language Models Knowledge Augmentation for Language Models
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China.
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
May 9, 2026Shengkun Tang, Zekun Wang, Bo Zheng +7Mixture-of-Experts Pruning LLM Pruning
Qwen Team, Alibaba Inc. · MBZUAI · KAUST
ZAYA1-VL-8B Technical Report
May 8, 2026Hassan Shapourian, Kasra Hejazi, Olabode M. Sule +1Vision-Language Models Mixture-of-Experts Language Models
Zyphra Technologies Inc, San Francisco, CA
SDG-MoE: Signed Debate Graph Mixture-of-Experts
May 8, 2026Stepan Kulibaba, Kirill Labzin, Artem Dzhalilov +4Mixture-of-Experts Language Models Mixture of Experts
Innopolis University, Innopolis · Sirius University, Sirius · Innopolis University, Innopolis; Sirius University, Sirius +1
Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs
May 8, 2026Andrea Sassella, Andrea Chizzola, Tommaso Bianchi +2LLM Evaluation Mixture-of-Experts Language Models
2DEIB, Politecnico di Milano · 1AIRIC, Politecnico di Milano
When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
May 8, 2026Youngsik Yoon, Siwei Wang, Wei Chen +1Mixture-of-Experts Language Models LLM Routing
Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea
UniPool: Learning Expert-to-Layer Ownership from Brief Global Access
May 7, 2026Minbin Huang, Han Shi, Chuanyang Zheng +5Mixture-of-Experts Language Models Mixture of Experts
The Chinese University of Hong Kong · Huawei Technologies · The University of Hong Kong
EMO: Pretraining Mixture of Experts for Emergent Modularity
May 7, 2026Ryan Wang, Akshita Bhagia, Sewon MinLanguage Model Pretraining Mixture-of-Experts Language Models
University of California, Berkeley · Allen Institute for AI
A Few Good Clauses: Comparing LLMs vs Domain-Trained Small Language Models on Structured Contract Extraction
May 7, 2026Nicole Lincoln, Nick Whitehouse, Jaron Mar +1Legal NLP Mixture-of-Experts Language Models
Onit AI Labs, Onit Inc., New Zealand
ZAYA1-8B Technical Report
May 6, 2026Robert Washbourne, Rishi Iyer, Tomas Figliolia +15Mixture-of-Experts Language Models RL for Language Model Reasoning
Zyphra San Francisco, CA
RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMs
May 1, 2026Zhiyuan Xu, Joseph Gardiner, Sana Belguith +1Mixture-of-Experts Language Models Adversarial Attacks on LLMs
University of Bristol Bristol, United Kingdom

May 26, 2026J/K move · Enter open · S save

L2Rec: Towards Dual-View Understanding of LLMs for Personalized Recommendation

Pingjun Pan, Tingting Zhou, Peiyao Lu, Tingting Fei+2

Netease Cloud Music Hangzhou, China

Mixture-of-Experts Language Models LLM Personalization Low-Rank Adaptation Large Language Model-Based Recommendation User Preference Modeling

Open paper AskPDF arXiv

Adapting large language models (LLMs) for personalized recommendation requires aligning their general-purpose capabilities with user-specific preferences while effectively leveraging both behavioral and semantic signals. Existing approaches typically integrate these signals at either the input level (e.g., injecting behavioral embeddings into the token space) or the output level (e.g., contrastive alignment of separate encoders), suffering from distribution gaps or lack of end-to-end task supervision. In this work, we introduce L2Rec, which unifies behavioral and semantic understanding at the parameter level of LLMs. Our key insight is that the same set of Transformer parameters can serve as a shared medium for both views: by applying view-specific, personalized low-rank perturbations via a Dual-view Personalized Mixture-of-Experts (DPMoE) mechanism, L2Rec enables a single LLM backbone to produce complementary behavioral and semantic adaptations for each user with minimal representation-level misalignment. An adaptive cross-view fusion module further integrates the dual-view outputs into a unified user preference. Experiments on four datasets show that L2Rec consistently outperforms state-of-the-art baselines, and online A/B testing on a large-scale industrial platform validates significant improvements in key engagement metrics.