Mixture-of-Experts Fine-Tuning

Follow this topic Ask about this topic

Latest papers

16 papers2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

An Empirical Study of Parameter-Efficient Fine-Tuning for Mixture-of-Experts LLMs
Oct 19, 2026Minglei Li, Peng Ye, Haonan He +3Mixture-of-Experts Fine-Tuning Parameter-Efficient Fine-Tuning
Fudan University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +3
MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference
Sep 28, 2026Junfeng Wu, Zehao Fan, Hadjer Benmeziane +3Expert Offloading Mixture-of-Experts Language Models
Department of Industrial and Systems Engineering, Rensselaer Polytechnic Institute · Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute · IBM Research
ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs
Sep 5, 2026Ahin Lee, Sehyun Yun, Joonha Park +1Mixture-of-Experts Fine-Tuning Parameter-Efficient Fine-Tuning
Ulsan National Institute of Science and Technology (UNIST) Ulsan, Republic of Korea
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
Aug 8, 2026Ali Janati, Kaoutar El Maghraoui, Chengke Zou +2Mixture-of-Experts Pruning Mixture-of-Experts Models
Data Science Institute, Columbia University, New York, NY, USA · Department of Computer Science, Columbia University, New York, NY, USA
MoEGen: Mixture-of-Experts for Instance-Adaptive LoRA Generation
Aug 4, 2026Yiming Zeng, Lei Lu, Zexin Li +10LLM Fine-Tuning Low-Rank Adaptation
University of Connecticut · Northeastern University · Nanyang Technological University +5
TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation
Jul 31, 2026Guanzhi Deng, Haibo Wang, Kuan Wu +5Mixture-of-Experts Language Models Mixture of Experts
City University of Hong Kong, Hong Kong, China · Carnegie Mellon University, Pittsburgh, USA · University of Waterloo, Ontario, Canada +1
MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Jul 24, 2026Qingyu Yang, Haonan He, Minglei Li +4Mixture-of-Experts Language Models Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory · KTH Royal Institute of Technology · University of Science and Technology of China +2
UMoE:Unlocking Every Expert in Domain-Specific Training
Jul 13, 2026Xuefeng Li, Pengfei LiuMixture-of-Experts Pruning Mixture-of-Experts Language Models
1SII · 2SJTU · 3GAIR
FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts
Jun 30, 2026Tom Saliencro, Maya Lindqvist, Rohan Desai +2Low-Rank Adaptation Mixture-of-Experts Models
University of California, Irvine · University of Washington
HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
May 11, 2026Jia Wei, Zhonghao Zhang, Ping Chen +5Mixture-of-Experts Language Models Fine-Tuning
Department of Computer Science and Technlogy Tsinghua University Beijing 100084, China · School of Computer Science and Technlogy Xi’an Jiaotong University · The State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
VLA-GSE: Boosting Parameter-Efficient Fine-Tuning in VLA with Generalized and Specialized Experts
May 7, 2026Yuhua Jiang, Junjie Lu, Xinyao Qin +4Fine-Tuning Vision-Language-Action Models
Microsoft Research Asia · Tsinghua University
Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
Apr 24, 2026Haoze He, Xingyuan Ding, Xuan Jiang +5Supervised Fine-Tuning Fine-Tuning
Carnegie Mellon University · MIT · UCSD +1
Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts
Apr 20, 2026Jacob Morrison, Sanjay Adhikesaven, Akshita Bhagia +3Continual Learning for LLMs Mixture-of-Experts Language Models
University of Washington · Allen Institute for AI · University of California, Berkeley
MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning
Mar 25, 2026Andrea ManzoniMixture-of-Experts Fine-Tuning Fine-Tuning
KALAVAI: Predicting When Independent Specialist Fusion Works -- A Quantitative Model for Post-Hoc Cooperative LLM Training
Mar 24, 2026Ramchand KumaresanLanguage Model Merging Mixture-of-Experts Expert Specialization
Murai Labs
DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
Jan 8, 2026Guanzhi Deng, Bo Li, Ronghao Chen +5Mixture-of-Experts Language Models Fine-Tuning
City University of Hong Kong · Tsinghua University · Peking University +2

Oct 19, 2026J/K move · Enter open · S save

An Empirical Study of Parameter-Efficient Fine-Tuning for Mixture-of-Experts LLMs

Minglei Li, Peng Ye, Haonan He, Jingqi Ye+2

Fudan University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong+3

Mixture-of-Experts Fine-Tuning Parameter-Efficient Fine-Tuning Mixture-of-Experts Language Models Low-Rank Adaptation

Open paper AskPDF arXiv

Parameter-efficient fine-tuning (PEFT) methods are critical for adapting large models under resource-constrained settings. However, as mixture-of-experts (MoE) architectures, which offer efficient inference at massive parameter scales, gain increasing prevalence in the community, a critical gap arising from the lack of systematic recipes for applying PEFT to MoE models. To narrow this gap, we conduct a large-scale systematic study that investigates a set of essential design recipes, including which modules should be adapted, which granularity should be used to adapt MoE modules, which PEFT method to use, and to what extent should the load-balancing objective be used, across three MoE backbones on downstream tasks and general knowledge tasks. Our experiments yield several key insights. First, adapting MoE experts alone achieves the highest downstream performance with competitive general capability retention under a comparable parameter budget. Second, native expert-weight updates offer a better downstream--retention trade-off than adapters that bypass experts or entire MoE modules. Third, LoRA and DoRA together define the strongest operating region for high downstream accuracy among the evaluated PEFT methods. Fourth, no nonzero load-balancing coefficient Pareto-dominates the zero baseline. We finally evaluate a weight-level redesign of MoE

^2

-LoRA on Qwen3. Moving its module-level bypass pool into native expert weights improves all four upstream endpoints by 2.4 to 17.9 points with 17% fewer trainable parameters and a 3.2-point average downstream cost.