Mixture-of-Experts Language Models

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 38% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 147

All topics

Evolutionary Soups: Evolving Mixture-of-Experts for Multi-Objective LLM Alignment
Aug 30, 2026Lingxiao Kong, Steffen Staab, Cong Yang +2Mixture-of-Experts Language Models Multi-Objective Language Model Alignment
Fraunhofer Institute for Applied Information Technology FIT · University of Cologne · University of Stuttgart +3
RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learning
Aug 12, 2026Yibo Shen, Xudong Han, Xiaowei Zhu +2RL for Language Models Expert Load Balancing
Ant Group
TradingMoE: Routing the Right Experts in Evolving Markets
Aug 12, 2026Chang Zhou, Xingtong Yu, Minbin Huang +4Mixture-of-Experts Language Models Sparse Mixture-of-Experts
University of Science and Technology of China · The Chinese University of Hong Kong · The University of Tokyo +1
Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts
Aug 11, 2026Soumajyoti Sarkar, Yuxin Tang, Sheng ZhaMixture-of-Experts Language Models Language Model Scaling Laws
Amazon AGI Foundations
DistMoE: Private-data Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuning
Aug 10, 2026Mainak Singha, Niccolò Biondi, Elisa Ricci +1Mixture-of-Experts Language Models VLM Adaptation
University of Trento, Italy · Fondazione Bruno Kessler, Italy · University of Bergamo, Italy
MoRSE: Task-Oriented Multi-Agent System with Mixture of Role-Subtask Experts
Aug 10, 2026Peiwen Li, Shiyang Zhang, Yangtian Zhang +3Multi-Agent LLM Systems Mixture-of-Experts Language Models
Yale University, New Haven, CT, USA
Motif 3: Technical Report
Aug 10, 2026Junghwan Lim, Joon Son Chung, Sungmin Lee +24Mixture-of-Experts Language Models Long-Context Language Modeling
Motif Technologies
Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts
Aug 9, 2026Zongfei LiMixture-of-Experts Language Models Sparse Mixture-of-Experts
RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention
Aug 8, 2026Anthony. Lui, Mohamed. Elsaied, N. P. SavaniExpert Offloading Mixture-of-Experts Language Models
Cognizant AI & Analytics, London, UB8 3PH, UK · Southern Methodist University (SMU), Engineering Management, Lyle School of Engineering, Dallas, TX, USA · Cognizant AI Lab, Responsible AI Office, San Francisco, USA +1
Decided Upstream, Written Late: Locating and Pricing the Cross-Lingual Refusal Circuit of a Multilingual MoE
Aug 8, 2026Ramakrishna P. Kompella, Aadit MahajanMultilingual Language Models Mixture-of-Experts Language Models
Gödel Machines · IIT Madras
Shape Mutating Expert Compression:LorExperts and BTExperts
Aug 7, 2026Inesh Chakrabarti, Sourjya Roy, Bowen Bao +3Mixture-of-Experts Language Models Low-Rank Matrix Decomposition
Advanced Micro Devices
Mind the Gaps: Mixture-of-Minds for Human Simulation
Aug 6, 2026Pranav DahiyaMixture-of-Experts Language Models Human Behavior Simulation
Semilattice
K-EXAONE 2.0 Technical Report
Aug 5, 2026Eunbi Choi, Kibong Choi, Sehyun Chun +74Multilingual Language Models Mixture-of-Experts Language Models
AFD-Ledger: Deployment Provisioning for Attention--FFN Disaggregation
Aug 5, 2026Chengyu Qiu, Xiao Fu, Fengcun Li +6Mixture-of-Experts Language Models Disaggregated LLM Serving
Tsinghua University · Meituan · Beijing, China
LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
Aug 4, 2026Fengqi Zhu, Shaoxuan Xu, Jingyang Ou +11Mixture-of-Experts Language Models Language Model Scaling Laws
Gaoling School of AI, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE +2
TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation
Jul 31, 2026Guanzhi Deng, Haibo Wang, Kuan Wu +5Mixture-of-Experts Language Models Mixture of Experts
City University of Hong Kong, Hong Kong, China · Carnegie Mellon University, Pittsburgh, USA · University of Waterloo, Ontario, Canada +1
Kimi K3: Open Frontier Intelligence
Jul 27, 2026Kimi Team, Tongtong Bai, Yifan Bai +399Mixture-of-Experts Language Models Long-Context Language Modeling
MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Jul 24, 2026Qingyu Yang, Haonan He, Minglei Li +4Mixture-of-Experts Language Models Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory · KTH Royal Institute of Technology · University of Science and Technology of China +2
Solar Open 2 Technical Report
Jul 22, 2026Sungrae Park, Sanghoon Kim, Gyoungjin Gim +50Mixture-of-Experts Language Models Long-Context Language Modeling
Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training
Jul 21, 2026Nuemaan MalikDeep Learning Optimization Mixture-of-Experts Language Models
Independent Researcher
Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Coding
Jul 18, 2026Anik JhaMixture-of-Experts Pruning Mixture-of-Experts Language Models
1Independent Researcher
Multi-level context Modeling for consistent expert selection in Mixture-of-Experts
Jul 17, 2026Shuhan Huang, Naifan Zhang, Yuanbo Tang +2Mixture-of-Experts Language Models Expert Routing
Tsinghua Shenzhen International Graduate School, Tsinghua University · School of AI, The Chinese University of Hong Kong (Shenzhen)
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
Jul 17, 2026Yuchen Yang, Yifan Zhao, Anisha Dasgupta +1LLM Quantization Mixture-of-Experts Language Models
University of Illinois Urbana-Champaign
Loop the Loopies!
Jul 17, 2026Zitian Gao, Yilong Chen, Yihao Xiao +4Mixture-of-Experts Language Models Recurrent Transformers
IQuest Research
QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides
Jul 17, 2026Zhengyang Zhuge, Hao Yu, Xin Wang +4LLM Quantization RL for Language Models
Qwen Team, Alibaba Inc.
UMoE:Unlocking Every Expert in Domain-Specific Training
Jul 13, 2026Xuefeng Li, Pengfei LiuMixture-of-Experts Pruning Mixture-of-Experts Language Models
1SII · 2SJTU · 3GAIR
A Sovereign, Open-Source Foundation Model for German and English
Jul 10, 2026The Soofi-Team, :, Benedikt Droste +29Language Model Pretraining Multilingual Language Models
ellamind · Fraunhofer IAIS · Lamarr +8
It Takes a MAESTRO To Prune Bad Experts
Jul 9, 2026Palaash Goel, Ayush Maheshwari, Tanmoy ChakrabortyMixture-of-Experts Pruning LLM Pruning
Indian Institute of Technology Delhi, India · NVIDIA, India
Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs
Jul 5, 2026Akhiad Bercovich, Talor Abramovich, Daniel Afrimi +67Mixture-of-Experts Pruning LLM Pruning
EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning
Jul 2, 2026Ahin Lee, Sehyun Yun, Taesik GongMixture-of-Experts Pruning Mixture-of-Experts Language Models
Ulsan National Institute of Science & Technology (UNIST)

Aug 30, 2026, cs.CLJ/K move · Enter open · S save

Evolutionary Soups: Evolving Mixture-of-Experts for Multi-Objective LLM Alignment

Lingxiao Kong, Steffen Staab, Cong Yang, Oya Beyan+1

Fraunhofer Institute for Applied Information Technology FIT · University of Cologne · University of Stuttgart+3

Mixture-of-Experts Language Models Multi-Objective Language Model Alignment Model Merging Inference-Time Language Model Alignment

Open paper AskPDF arXiv

Large language models are increasingly required to generate responses that satisfy multiple competing objectives. Since optimal trade-offs depend on both user preferences and input prompts, controllable multi-objective generation must dynamically adapt models at inference time without retraining. To address this, we propose Evolutionary Soups, a mixture-of-experts framework for fine-grained generation control, with gating networks trained via an evolutionary algorithm. The per-layer gating networks dynamically produce expert-merging coefficients from hidden-state representations, while the evolutionary algorithm incorporates greedy hypervolume contribution for effective evolution of these gating networks, achieving consistent improvements on large and noisy training datasets and broader coverage of the non-convex Pareto front. Experiments across three tasks demonstrate the effectiveness of Evolutionary Soups over baselines: it achieves the best hypervolume, linear utility, and Tchebyshev utility (~20% improvement) among controllable methods on all tasks.

Momentum

Related topics

Latest papers 147