MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning

Andrea Manzoni

cs.LGMar 25, 2026

MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning

Authors: Andrea Manzoni

Abstract

Standard LoRA fine-tuning of Mixture-of-Experts (MoE) models applies adapters to every expert, yet our profiling shows that per-layer expert routing is highly skewed: a small subset of experts handles most tokens in each layer, while many others are rarely activated ("cold"). We propose MoE-Sieve, a simple routing-guided framework for LoRA fine-tuning, and pair it with a systematic profiling study of expert routing across architectures and tasks. The method is simple: profile routing counts on a small calibration set, select the top-k most-routed experts per layer, and apply LoRA only to those experts. Across two architecturally distinct MoE models and three diverse tasks, tuning only the top 25% routed experts per layer remains competitive with full LoRA, with mean differences within +/-1 percentage point across all conditions. This reduces LoRA trainable parameters by 70-73%, adapter checkpoint size by 71-73%, and wall-clock training time by up to 50%. We also observe a non-monotonic relationship between expert count and seed-to-seed variance, consistent with the hypothesis that adapting cold experts can introduce gradient noise without improving accuracy. Further ablations show that random expert selection at matched budget is about 2.5 percentage points worse, indicating that the routing signal matters, while greedy per-layer budget optimization does not improve over uniform top-k.

Figures & tables

Model	Layers	Routed	Shared	Top- $k$	Expert / FFN
OLMoE-1B-7B	16	64	0	8	1.0 $\times$
Qwen1.5-MoE-A2.7B	24	60	4	4	0.25 $\times$
DeepSeek-MoE-16B	27	64	2	6	0.13 $\times$

Table 1: Architectural summary of the profiled MoE models. All three activate eight experts per token in total, but differ in how much capacity is routed versus shared and in expert granularity.

Model	Global CV	Layer CV	Ratio	Cold%	Cov@25%
OLMoE-1B-7B	0.216	0.869	4.0 $\times$	28.5%	53.0%
Qwen1.5-MoE-A2.7B	0.077	0.374	4.9 $\times$	5.2%	37.4%
DeepSeek-MoE-16B	0.106	0.514	4.9 $\times$	11.0%	42.6%

Table 2: Routing imbalance averaged over 10 datasets. Global CV : coefficient of variation (standard deviation / mean) of per-expert totals aggregated across all layers. Layer CV : mean within-layer CV. Cold% : fraction of experts per layer receiving

<

50% of the uniform share. Cov@25% : fraction of token activations captured by the top-25% most-routed experts per layer.

Figure 1: Global vs. per-layer routing skew across all three models. Each dot is one model–dataset pair. Per-layer CV is consistently 4.0–4.9

\times

higher than global CV, demonstrating the “globally balanced, locally imbalanced” phenomenon.

Figure 2: Expert coverage as a function of routed expert budget for all three models. Left: fraction of per-layer token activations captured by the top-

k

routed experts. Right: per-token expert coverage, counting both shared experts (always active) and the selected routed experts. Qwen’s four shared experts provide a 50% floor before any routed expert is selected; OLMoE has no shared experts and starts from 0%. Dots mark the 25% operating point.

Figure 3: Cross-dataset Jaccard similarity of top-16 expert sets for OLMoE. Code/programming tasks (MBPP, Spider, CodeAlpaca) share experts with each other but not with broad-domain tasks, confirming domain-driven task-dependent specialisation.

Model	Task	Full LoRA	Hot (25%)	$\Delta$ (pp)	95% CI (pp)	Eqv@2pp
OLMoE	Spider	$.396\pm.026$	$.399\pm.015$	$+0.30$	[ $-2.04,\ +2.64$ ]	$\times$
OLMoE	GSM8K	$.304\pm.011$	$.304\pm.006$	$-0.08$	[ $-1.45,\ +1.30$ ]	$\checkmark$
OLMoE	HellaSwag	$.805\pm.005$	$.807\pm.008$	$+0.17$	[ $-0.71,\ +1.05$ ]	$\checkmark$
Qwen	Spider	$.520\pm.014$	$.511\pm.005$	$-0.93$	[ $-1.88,\ +0.03$ ]	$\checkmark$
Qwen	GSM8K	$.590\pm.011$	$.592\pm.007$	$+0.20$	[ $-0.77,\ +1.17$ ]	$\checkmark$
Qwen	HellaSwag	$.885\pm.002$	$.893\pm.001$	$+0.73$	[ $+0.53,\ +0.93$ ]	$\checkmark$

Table 3: Main results: mean accuracy

\pm

std (8 seeds), paired delta

\Delta

= hot

-

full (pp), paired 95% CI, and equivalence at a pre-declared

\pm

2 pp margin (

\checkmark

= established,

\times

= inconclusive). Attention, router, and shared-expert adapters are always active.

Model	Params (Full $\rightarrow$ Hot)	Red.	Ckpt (Full $\rightarrow$ Hot)	Red.	Time (Full $\rightarrow$ Hot)
OLMoE	311.5M $\rightarrow$ 85.0M	72.7%	1.25 GB $\rightarrow$ 340 MB	73.4%	1h 48m $\rightarrow$ 54m (50%)
Qwen	509.7M $\rightarrow$ 151.3M	70.3%	2.04 GB $\rightarrow$ 606 MB	71.0%	3h 23m $\rightarrow$ 1h 44m (49%)

Table 4: Efficiency of hot-25% relative to full LoRA. Trainable-parameter counts are from the final PEFT model. Wall-clock times are one example measured on GSM8K (3 epochs, 351 steps) on a single GPU; actual speedup varies by task and configuration.

Model	Task	Uniform	Greedy	$k$ range	Cov. (%)
OLMoE	GSM8K	$.304\pm.006$	$.304\pm.008$	12–22	56.8
OLMoE	Spider	$.399\pm.015$	$.405\pm.019$	12–20	68.4
OLMoE	HellaSwag	$.807\pm.008$	$.806\pm.008$	0 9–21	43.0
Qwen	GSM8K	$.592\pm.007$	$.591\pm.010$	12–19	34.2
Qwen	Spider	$.511\pm.005$	$.509\pm.012$	12–19	48.2
Qwen	HellaSwag	$.893\pm.001$	$.895\pm.001$	11–20	36.4

Table 5: Greedy marginal-gain allocation vs. uniform hot-25%. Both strategies use the same total expert–layer budget; greedy varies the per-layer count (

k

range) while uniform fixes it. Mean routing coverage (%) is virtually identical, and accuracy differences are negligible across all conditions (8 seeds each).

Model	Task	Full	Hot (25%)	Hot / Full
OLMoE	Spider	.026	.015	0.57 $\times$
OLMoE	GSM8K	.011	.006	0.58 $\times$
OLMoE	HellaSwag	.005	.008	1.68 $\times$
Qwen	Spider	.014	.005	0.36 $\times$
Qwen	GSM8K	.011	.007	0.59 $\times$
Qwen	HellaSwag	.002	.001	0.67 $\times$

Table 6: Seed-to-seed standard deviation (8 seeds). Ratio = std(hot-25%) / std(full LoRA); values

<

1 indicate variance reduction.

Figure 4: OLMoE

\times

GSM8K

k

-sweep (8 seeds per

k

). Dots show individual seeds; red markers show mean

\pm

95% CI; the green band marks full LoRA.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Layer	CV	Cold%	Top-16 Cov%	Norm. Entropy
0	0.81	25%	53%	0.935
1	0.85	23%	52%	0.938
2	1.08	41%	59%	0.902
3	1.09	47%	64%	0.886
4	1.38	55%	71%	0.845
5	1.46	47%	70%	0.837

Appendix

Table 7: Per-layer profiling for OLMoE on Spider. CV and cold-expert fraction rise sharply from layer 4 onward. Cold% = fraction of experts receiving

<

50% of the uniform routing share.

Figure 5: Per-layer expert activation heatmaps (row-normalised) for OLMoE across all 10 profiling datasets. Each row is a layer and each column an expert. All datasets exhibit a hot-head plus cold-tail structure, but its sharpness varies with task domain: code and programming tasks (MBPP, Spider, CodeAlpaca) show more concentrated patterns than broad-domain tasks (Wikitext, BoolQ, MMLU).

Figure 6: Subsample stability of top-

k

expert sets under 10% profiling data, for all three models across all datasets. Bars show mean Jaccard across 50 bootstrap trials; error bars show the gap to the minimum per-layer overlap. MBPP (374 examples) is the only dataset with mean

J<0.90

for fine-grained architectures; all other datasets achieve mean

J\geq 0.90

across all models.

Model	Task	$\Delta$ (pp)	$\varepsilon$ =1pp	$\varepsilon$ =2pp	$\varepsilon$ =3pp	$t$ -test $p$
OLMoE	GSM8K	$-$ 0.08	.003 ✓	$<$ .001 ✓	$<$ .001 ✓	.753
OLMoE	HellaSwag	$+$ 0.17	$<$ .001 ✓	$<$ .001 ✓	$<$ .001 ✓	.243
OLMoE	Spider	$+$ 0.30	.251	.065	.015 ✓	.771
Qwen	GSM8K	$+$ 0.20	.006 ✓	$<$ .001 ✓	$<$ .001 ✓	.425
Qwen	HellaSwag	$+$ 0.73	$<$ .001 ✓	$<$ .001 ✓	$<$ .001 ✓	$<$ .001
Qwen	Spider	$-$ 0.93	.429	.016 ✓	$<$ .001 ✓	.055

Appendix

Table 8: TOST equivalence

p

-values and paired

t

-test results for hot-25% vs. full LoRA. ✓= equivalence established at

\alpha

= 0.05.

Layer	$k$	Cov%	$k$	Cov%
	Uniform		Greedy
0	16	48.5	15	46.8
1	16	48.0	14	44.5
2	16	52.8	16	52.8
3	16	52.4	19	57.8
4	16	57.5	16	57.5

Appendix

Table 9: Per-layer allocation: uniform (

k

= 16) vs. greedy for OLMoE

\times

GSM8K. Despite greedy’s variable allocation (12–22 experts/layer), mean coverage is virtually identical (0.567 vs. 0.568), explaining why downstream accuracy does not differ.

Explore similar work

MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Jul 24, 2026Qingyu Yang, Haonan He, Minglei Li +4Mixture-of-Experts Language Models Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory · KTH Royal Institute of Technology · University of Science and Technology of China +2
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
Aug 8, 2026Ali Janati, Kaoutar El Maghraoui, Chengke Zou +2Mixture-of-Experts Pruning Mixture-of-Experts Models
Data Science Institute, Columbia University, New York, NY, USA · Department of Computer Science, Columbia University, New York, NY, USA
ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs
Sep 5, 2026Ahin Lee, Sehyun Yun, Joonha Park +1Parameter-Efficient Fine-Tuning Fine-Tuning
Ulsan National Institute of Science and Technology (UNIST) Ulsan, Republic of Korea

Jul 24, 2026J/K move · Enter open · S save

MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

Qingyu Yang, Haonan He, Minglei Li, Jingqi Ye+3

Shanghai Artificial Intelligence Laboratory · KTH Royal Institute of Technology · University of Science and Technology of China+2

Mixture-of-Experts Language Models Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) architectures have been widely adopted in large language models, yet parameter-efficient fine-tuning (PEFT) for MoE models remains underexplored. Existing PEFT methods for MoE either ignore router priors with uniform adapters, reducing efficiency and risking forgetting, or rely on static expert selection, limiting per-token capacity and cross-expert feature learning. In this paper, we make the first attempt to fine-tune MoE models with MoE-style low-rank adaptation: our method, entitled MoE

^2

-LoRA, deeply couples the pretrained expert specialization with task-specific adaptivity via a dual-channel Routing-Conditioned Projection (RCP) module, which reuses base router activations to inform LoRA routing. We further introduce a single global LoRA expert pool shared across all layers, enabling model-wide adaptation with emergent layer-wise affinities and balanced expert utilization. MoE

^2

-LoRA simultaneously benefits from the advantages of prior reuse, dynamic adapter routing, and model-wide knowledge sharing. Evaluated on multiple MoE backbones with varying scales and expert granularities, MoE

^2

-LoRA consistently achieves state-of-the-art downstream accuracy while retaining stronger general capabilities.

Abstract

Explore similar work

MoE2^22-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models

ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs

MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation