An Empirical Study of Parameter-Efficient Fine-Tuning for Mixture-of-Experts LLMs

Minglei Li; Peng Ye; Haonan He; Jingqi Ye; Qingyu Yang; Tao Chen

cs.LGOct 19, 2026

An Empirical Study of Parameter-Efficient Fine-Tuning for Mixture-of-Experts LLMs

Authors: Minglei Li, Peng Ye, Haonan He, Jingqi Ye, Qingyu Yang, Tao Chen

Organizations: Fudan University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong · University of Science and Technology of China ·

Abstract

Parameter-efficient fine-tuning (PEFT) methods are critical for adapting large models under resource-constrained settings. However, as mixture-of-experts (MoE) architectures, which offer efficient inference at massive parameter scales, gain increasing prevalence in the community, a critical gap arising from the lack of systematic recipes for applying PEFT to MoE models. To narrow this gap, we conduct a large-scale systematic study that investigates a set of essential design recipes, including which modules should be adapted, which granularity should be used to adapt MoE modules, which PEFT method to use, and to what extent should the load-balancing objective be used, across three MoE backbones on downstream tasks and general knowledge tasks. Our experiments yield several key insights. First, adapting MoE experts alone achieves the highest downstream performance with competitive general capability retention under a comparable parameter budget. Second, native expert-weight updates offer a better downstream--retention trade-off than adapters that bypass experts or entire MoE modules. Third, LoRA and DoRA together define the strongest operating region for high downstream accuracy among the evaluated PEFT methods. Fourth, no nonzero load-balancing coefficient Pareto-dominates the zero baseline. We finally evaluate a weight-level redesign of MoE $^2$ -LoRA on Qwen3. Moving its module-level bypass pool into native expert weights improves all four upstream endpoints by 2.4 to 17.9 points with 17% fewer trainable parameters and a 3.2-point average downstream cost.

Figures & tables

Figure 1: Illustration of Position, PEFT Methods, and Auxiliary Loss.

Figure 2: Three evaluated adaptation granularities of the MoE module.

Figure 3: Visualized results of the study of adapted modules. Attn denotes attention, and gray labels report trainable parameters.

Figure 4: Visualized results of the study of MoE adaptation granularity.

Figure 5: Visualized results of the study of PEFT methods.

Figure 6: Visualized results of the study of the strength of auxiliary loss.

Checkpoint	Benchmark	Original	Weight Level
Trainable params		47.71M	39.58M
Math	MATH-500	60.0	55.8
	MMLU-Pro	48.19	50.63
	IFEval	35.12	40.30
Intent	Intent	84.0	81.8
	MMLU-Pro	45.26	51.20

Table 1: Original MoE 2 -LoRA versus the insight-guided expert-weight redesign on Qwen3.

Figure 7: Functional response diversity across all 48 Qwen3 MoE layers on synthetic probes. Points denote layers, boxes show medians and interquartile ranges, and diamonds mark means. Weight-level rank-one updates produce broad layer responses on both adapted checkpoints.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Math	Intent
Training data	MetaMathQA-R1	ESFT_Intent
Maximum samples	80,000	80,000
Epochs	1	3
Sequence cutoff	2,048	4,096
Packing	default	disabled
Evaluation set size	500	500

Appendix

Table 2: Task-specific training and downstream-evaluation settings.

Base arm	Attention	Expert adaptation	Router variant
Attention only	$r64/a128$	–	off or $r8/a16$
Experts only	–	$r256/a512$	off or $r8/a16$
Attention+experts	$r32/a64$	$r128/a256$	off or $r8/a16$
Higher budget	$r64/a128$	$r256/a512$	off or $r8/a16$

Appendix

Table 3: Adaptation-position arms. Expert adaptation uses one independently routed top-1 branch per layer (

N_{L}=1

), shared by the complete expert block.

Granularity	Routing and placement	Expert setting
Module Level	Independent, $N_{L}=1$ , top-1	$r256/a512$
Expert Level	Follow native router; one branch per expert	$r2/a4$
Weight Level	Follow native router; gate/up/down weights	$r1/a2$
Weight Level	Follow native router; gate/up/down weights	$r2/a4$

Appendix

Table 4: The four final granularity arms. Native routing uses 128 experts and top-8 for Qwen3/Gemma-4, and 32 experts and top-4 for GPT-OSS.

Method	Expert update	Router update	Backbone-specific values
Weight-level LoRA	Expert gate/up/down weights, $r1/a2$	LoRA $r8/a16$	Same on all three backbones
Weight-level DoRA	Expert gate/up/down weights, $r1/a2$ , plus magnitude	DoRA $r8/a16$	Same ranks/scales on all three backbones
Weight-level OFT	Blockwise Cayley transform	Blockwise OFT	Expert/router blocks: Qwen3 $4/128$ ; Gemma-4 $8/128$ ; GPT-OSS $6/32$
FourierFT	Fixed-support spectral delta	FourierFT	Expert/router frequencies: Qwen3 $4096/17408$ ; Gemma-4 $5632/23552$ ; GPT-OSS $5760/23296$ ; scale 150, seed 2024
IA3	Per-expert down-input scale	Router-logit output scale	One-initialized vectors; no rank

Appendix

Table 5: Weight-level methods in the final PEFT method comparison. Attention is disabled, so attention-specific capacity values are intentionally omitted.

Method	Expert update	Router update	Backbone-specific values
Expert-level LoRA	One adaptation branch per routed expert, follows the native router’s top- $k$ , $r2/a4$	LoRA $r8/a16$	$E/k=128/8$ for Qwen3 and Gemma-4; $32/4$ for GPT-OSS
MoE-LoRA	Independent adapter router, $N_{L}=8$ , adapter top-2	LoRA $r8/a16$	FFN $r32/a64$ on Qwen3/Gemma-4; $r8/a16$ on GPT-OSS
Module-level LoRA	One independent top-1 adaptation branch per layer	LoRA $r8/a16$	FFN $r256/a512$ on Qwen3/Gemma-4; $r64/a128$ on GPT-OSS
DAS-LoRA	Highest-DAS expert subset per layer	Enabled common LoRA router anchor	Select 8 experts with $r32/a64$ on Qwen3/Gemma-4; 4 experts with $r16/a32$ on GPT-OSS
Partial FT	Full expert weights in last decoder layer	Full router in last layer	No adapter and no parameter matching

Appendix

Table 6: MoE-structured and partial-fine-tuning methods in the final PEFT method comparison. MoE 2 -LoRA is excluded from this controlled ten-method grid and analyzed separately in Section A.1 .

Configuration	Trainable params	Intent	MMLU-Pro	IFEval
Experts only	47.71M	84.00	45.26	18.11
Attention+experts	48.07M	81.60	47.93	14.23
Balance $10^{-4}$	47.71M	85.00	47.46	17.56
Balance $10^{-3}$	47.71M	82.20	45.18	5.91
Balance $10^{-2}$	47.71M	80.80	22.37	7.76
Balance $10^{-1}$	47.71M	67.80	16.89	13.31

Appendix

Table 7: Additional Qwen3 MoE 2 -LoRA validation on Intent. All metrics are percentages. Intent uses 500 held-out examples, MMLU-Pro uses 12,032 examples, and IFEval is prompt-level accuracy on 541 prompts.

Configuration	Trainable params	MATH-500	MMLU-Pro	IFEval
Experts only	47.71M	60.00	48.19	35.12
Attention+experts	48.07M	56.60	47.56	36.23
Balance $10^{-4}$	47.71M	56.40	47.27	40.67
Balance $10^{-3}$	47.71M	55.80	46.69	43.81
Balance $10^{-2}$	47.71M	55.60	44.83	45.47
Balance $10^{-1}$	47.71M	53.40	42.50	46.58

Appendix

Table 8: Additional Qwen3 MoE 2 -LoRA validation on Math. All metrics are percentages.

Backbone	Layers	Hidden	Q heads	KV heads	Head dim	Attention pattern
Qwen3-30B-A3B-Base	48	2048	32	4	128	Full attention; no sliding window
Gemma-4-26B-A4B	30	2816	16	8 local / 2 global	256 local / 512 global	Five sliding layers followed by one global layer; window 1024; global K=V and p-RoPE
GPT-OSS-20B	24	2880	64	8	64	Alternating sliding and full layers; window 128

Appendix

Table 9: Text-backbone and attention differences. “Q heads” and “KV heads” distinguish query heads from grouped key/value heads. Gemma-4 changes both KV-head count and head dimension in its global-attention layers and ties their key and value projections.

Backbone	Routed experts	Top- $k$	Shared/always-on expert path	Routed width	Expert adaptation target
Qwen3-30B-A3B-Base	128	8	No explicit shared expert	768	Routed experts host
Gemma-4-26B-A4B	128	8	One always-active shared expert; dense width 2112 in config	704	The 128 routed experts ; shared path frozen
GPT-OSS-20B	32	4	No explicit shared expert	2880	Routed experts host

Appendix

Table 10: MoE structure and the actual expert-adaptation boundary. “No explicit shared expert” means that no such branch is defined in the released checkpoint configuration or reference implementation; it is not a claim about all models in that family. The Gemma-4 shared path is always active, while the repository adapts only its separate routed-expert container.

Granularity	Local $r$	Math	Intent	Mean	Down. avg.	General-capability observation
Module Level	256	25.25	20.54	22.90	68.4	MMLU-Pro $-0.5$ ; IFEval $-3.7$
Expert Level	2	16.31	15.91	16.11	64.7	MMLU-Pro $-5.0$ ; IFEval $-4.6$ ; unstable across backbones
Weight Level	1	63.90	69.08	66.49	68.5	MMLU-Pro $+0.8$ ; IFEval $-1.4$ ; best retention
Weight Level	2	86.37	85.22	85.80	69.1	MMLU-Pro $-0.9$ ; IFEval $-3.4$ ; nearly twice the trainable parameters

Appendix

Table 11: Functional effective rank and the corresponding granularity results. Math, Intent, and Mean report Qwen3 layer-level effective ranks; Mean averages the 96 task–layer observations. Downstream and general-capability results are the main-paper aggregates over three backbones and two task endpoints. The local

r

values constrain different objects and are included only to identify each configuration.

Checkpoint	Configuration	$S_{16}$	$S_{64}$	$S_{256}$	$C_{\mathrm{spec}}$
Reference	Pretrained down spectrum	0.09368	0.25758	0.64587	1.000
Reference	Uniform directions	0.02083	0.08333	0.33333	0.000
Math	Weight Level, $r=1$	0.08921	0.23353	0.58366	0.812
Math	Weight Level, $r=2$	0.08950	0.23470	0.58642	0.822
Math	Expert Level, $r=2$	0.03762	0.11561	0.37202	0.141
Intent	Weight Level, $r=1$	0.08308	0.21957	0.55615	0.733

Appendix

Table 12: Mean cumulative spectral profiles and pretrained spectral-consistency scores. Each adapted row averages 6,144 layer–expert positions. The reference rows average the corresponding pretrained expert spectra and use

m=768

for the uniform profile.

Configuration	$C_{\mathrm{spec}}$	Down.	MMLU	IFEval
	(M / I)	avg.	$\Delta$	$\Delta$
Weight Level $r=1$	0.812 / 0.733	68.5	$+0.8$	$-1.4$
Weight Level $r=2$	0.822 / 0.741	69.1	$-0.9$	$-3.4$
Expert Level $r=2$	0.141 / 0.087	64.7	$-5.0$	$-4.6$

Appendix

Table 13: Pretrained spectral consistency alongside the main granularity results. M/I denotes Math/Intent, and MMLU denotes MMLU-Pro. Spectral scores are measured on Qwen3; downstream and general-capability values are the main-paper aggregates over three backbones and two task endpoints.

Figure 9: Perturbation energy in the pretrained expert output subspace on Qwen3 synthetic probes. The left and middle panels show layer means over 128 experts; the right panel shows distributions over 6,144 layer–expert positions per configuration. Boxes show medians and interquartile ranges, whiskers extend to the most extreme observations within 1.5 interquartile ranges, and diamonds mark means. The dashed line is the isotropic reference of 0.375. Weight-level updates concentrate more energy in pretrained output subspaces across both checkpoints.

	Intent			Math
Configuration	LR	Samples/s	Steps/s	LR	Samples/s	Steps/s
Adaptation position
$A_{64}$	$10^{-3}$	$72.034\pm 3.596$	$0.504\pm 0.025$	$10^{-5}$	$30.903\pm 0.298$	$0.215\pm 0.002$
$E_{256}$	$10^{-4}$	$44.719\pm 1.206$	$0.313\pm 0.008$	$10^{-5}$	$29.615\pm 0.124$	$0.206\pm 0.001$
$A_{32}+E_{128}$	$10^{-4}$	$38.396\pm 1.004$	$0.269\pm 0.007$	$10^{-5}$	$27.960\pm 0.317$	$0.194\pm 0.002$
$A_{64}+R$	$10^{-4}$	$70.154\pm 1.695$	$0.491\pm 0.012$	$10^{-5}$	$30.835\pm 0.061$	$0.214\pm 0.001$

Appendix

Table 14: Selected learning rates and training throughput on Qwen3-30B-A3B-Base. Throughput is reported as mean

\pm

sample standard deviation across the runs used for each reported configuration. In the position block,

A_{r}

and

E_{r}

denote attention and expert-block adaptation at rank

r

;

R

adds router adaptation.

Mixture-of-Experts Fine-Tuning

See all 16 papers

EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning
Jul 2, 2026Ahin Lee, Sehyun Yun, Taesik GongMixture-of-Experts Pruning Mixture-of-Experts Language Models
Ulsan National Institute of Science & Technology (UNIST)
MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Jul 24, 2026Qingyu Yang, Haonan He, Minglei Li +4Mixture-of-Experts Language Models Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory · KTH Royal Institute of Technology · University of Science and Technology of China +2
ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs
Sep 5, 2026Ahin Lee, Sehyun Yun, Joonha Park +1Mixture-of-Experts Fine-Tuning Parameter-Efficient Fine-Tuning
Ulsan National Institute of Science and Technology (UNIST) Ulsan, Republic of Korea

Jul 2, 2026J/K move · Enter open · S save

EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning

Ahin Lee, Sehyun Yun, Taesik Gong

Ulsan National Institute of Science & Technology (UNIST)

Mixture-of-Experts Pruning Mixture-of-Experts Language Models Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models scale efficiently but remain costly to adapt due to redundant experts and uniform parameter allocation. Existing parameter-efficient fine-tuning (PEFT) methods such as LoRA ignore MoE routing dynamics, leading to suboptimal resource use. We propose EPnG, an adaptive prune-and-grow framework that reallocates LoRA capacity based on expert importance derived from router gate probabilities. EPnG prunes under-utilized experts and expands high-importance experts via rank growth with orthogonal initialization, while maintaining a fixed parameter budget. Across OLMoE and Qwen1.5-MoE, EPnG consistently outperforms LoRA under the same budget and achieves performance comparable to full fine-tuning while updating only 0.55%-0.72% of parameters (up to 140x-180x fewer). These results demonstrate that aligning PEFT with MoE routing yields a more effective and scalable fine-tuning strategy.

Abstract

Mixture-of-Experts Fine-Tuning

Explore similar work

EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning

MoE2^22-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs

MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation