MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning

Andrea Manzoni

cs.LGMar 25, 2026

MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning

Authors: Andrea Manzoni

Abstract

Standard LoRA fine-tuning of Mixture-of-Experts (MoE) models applies adapters to every expert, yet our profiling shows that per-layer expert routing is highly skewed: a small subset of experts handles most tokens in each layer, while many others are rarely activated ("cold"). We propose MoE-Sieve, a simple routing-guided framework for LoRA fine-tuning, and pair it with a systematic profiling study of expert routing across architectures and tasks. The method is simple: profile routing counts on a small calibration set, select the top-k most-routed experts per layer, and apply LoRA only to those experts. Across two architecturally distinct MoE models and three diverse tasks, tuning only the top 25% routed experts per layer remains competitive with full LoRA, with mean differences within +/-1 percentage point across all conditions. This reduces LoRA trainable parameters by 70-73%, adapter checkpoint size by 71-73%, and wall-clock training time by up to 50%. We also observe a non-monotonic relationship between expert count and seed-to-seed variance, consistent with the hypothesis that adapting cold experts can introduce gradient noise without improving accuracy. Further ablations show that random expert selection at matched budget is about 2.5 percentage points worse, indicating that the routing signal matters, while greedy per-layer budget optimization does not improve over uniform top-k.

Figures & tables

Model	Layers	Routed	Shared	Top- $k$	Expert / FFN
OLMoE-1B-7B	16	64	0	8	1.0 $\times$
Qwen1.5-MoE-A2.7B	24	60	4	4	0.25 $\times$
DeepSeek-MoE-16B	27	64	2	6	0.13 $\times$

Table 1: Architectural summary of the profiled MoE models. All three activate eight experts per token in total, but differ in how much capacity is routed versus shared and in expert granularity.

Model	Global CV	Layer CV	Ratio	Cold%	Cov@25%
OLMoE-1B-7B	0.216	0.869	4.0 $\times$	28.5%	53.0%
Qwen1.5-MoE-A2.7B	0.077	0.374	4.9 $\times$	5.2%	37.4%
DeepSeek-MoE-16B	0.106	0.514	4.9 $\times$	11.0%	42.6%

Table 2: Routing imbalance averaged over 10 datasets. Global CV : coefficient of variation (standard deviation / mean) of per-expert totals aggregated across all layers. Layer CV : mean within-layer CV. Cold% : fraction of experts per layer receiving

<

50% of the uniform share. Cov@25% : fraction of token activations captured by the top-25% most-routed experts per layer.

Figure 1: Global vs. per-layer routing skew across all three models. Each dot is one model–dataset pair. Per-layer CV is consistently 4.0–4.9

\times

higher than global CV, demonstrating the “globally balanced, locally imbalanced” phenomenon.

Figure 2: Expert coverage as a function of routed expert budget for all three models. Left: fraction of per-layer token activations captured by the top-

k

routed experts. Right: per-token expert coverage, counting both shared experts (always active) and the selected routed experts. Qwen’s four shared experts provide a 50% floor before any routed expert is selected; OLMoE has no shared experts and starts from 0%. Dots mark the 25% operating point.

Figure 3: Cross-dataset Jaccard similarity of top-16 expert sets for OLMoE. Code/programming tasks (MBPP, Spider, CodeAlpaca) share experts with each other but not with broad-domain tasks, confirming domain-driven task-dependent specialisation.

Model	Task	Full LoRA	Hot (25%)	$\Delta$ (pp)	95% CI (pp)	Eqv@2pp
OLMoE	Spider	$.396\pm.026$	$.399\pm.015$	$+0.30$	[ $-2.04,\ +2.64$ ]	$\times$
OLMoE	GSM8K	$.304\pm.011$	$.304\pm.006$	$-0.08$	[ $-1.45,\ +1.30$ ]	$\checkmark$
OLMoE	HellaSwag	$.805\pm.005$	$.807\pm.008$	$+0.17$	[ $-0.71,\ +1.05$ ]	$\checkmark$
Qwen	Spider	$.520\pm.014$	$.511\pm.005$	$-0.93$	[ $-1.88,\ +0.03$ ]	$\checkmark$
Qwen	GSM8K	$.590\pm.011$	$.592\pm.007$	$+0.20$	[ $-0.77,\ +1.17$ ]	$\checkmark$
Qwen	HellaSwag	$.885\pm.002$	$.893\pm.001$	$+0.73$	[ $+0.53,\ +0.93$ ]	$\checkmark$

Table 3: Main results: mean accuracy

\pm

std (8 seeds), paired delta

\Delta

= hot

-

full (pp), paired 95% CI, and equivalence at a pre-declared

\pm

2 pp margin (

\checkmark

= established,

\times

= inconclusive). Attention, router, and shared-expert adapters are always active.

Model	Params (Full $\rightarrow$ Hot)	Red.	Ckpt (Full $\rightarrow$ Hot)	Red.	Time (Full $\rightarrow$ Hot)
OLMoE	311.5M $\rightarrow$ 85.0M	72.7%	1.25 GB $\rightarrow$ 340 MB	73.4%	1h 48m $\rightarrow$ 54m (50%)
Qwen	509.7M $\rightarrow$ 151.3M	70.3%	2.04 GB $\rightarrow$ 606 MB	71.0%	3h 23m $\rightarrow$ 1h 44m (49%)

Table 4: Efficiency of hot-25% relative to full LoRA. Trainable-parameter counts are from the final PEFT model. Wall-clock times are one example measured on GSM8K (3 epochs, 351 steps) on a single GPU; actual speedup varies by task and configuration.

Model	Task	Uniform	Greedy	$k$ range	Cov. (%)
OLMoE	GSM8K	$.304\pm.006$	$.304\pm.008$	12–22	56.8
OLMoE	Spider	$.399\pm.015$	$.405\pm.019$	12–20	68.4
OLMoE	HellaSwag	$.807\pm.008$	$.806\pm.008$	0 9–21	43.0
Qwen	GSM8K	$.592\pm.007$	$.591\pm.010$	12–19	34.2
Qwen	Spider	$.511\pm.005$	$.509\pm.012$	12–19	48.2
Qwen	HellaSwag	$.893\pm.001$	$.895\pm.001$	11–20	36.4

Table 5: Greedy marginal-gain allocation vs. uniform hot-25%. Both strategies use the same total expert–layer budget; greedy varies the per-layer count (

k

range) while uniform fixes it. Mean routing coverage (%) is virtually identical, and accuracy differences are negligible across all conditions (8 seeds each).

Model	Task	Full	Hot (25%)	Hot / Full
OLMoE	Spider	.026	.015	0.57 $\times$
OLMoE	GSM8K	.011	.006	0.58 $\times$
OLMoE	HellaSwag	.005	.008	1.68 $\times$
Qwen	Spider	.014	.005	0.36 $\times$
Qwen	GSM8K	.011	.007	0.59 $\times$
Qwen	HellaSwag	.002	.001	0.67 $\times$

Table 6: Seed-to-seed standard deviation (8 seeds). Ratio = std(hot-25%) / std(full LoRA); values

<

1 indicate variance reduction.

Figure 4: OLMoE

\times

GSM8K

k

-sweep (8 seeds per

k

). Dots show individual seeds; red markers show mean

\pm

95% CI; the green band marks full LoRA.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Layer	CV	Cold%	Top-16 Cov%	Norm. Entropy
0	0.81	25%	53%	0.935
1	0.85	23%	52%	0.938
2	1.08	41%	59%	0.902
3	1.09	47%	64%	0.886
4	1.38	55%	71%	0.845
5	1.46	47%	70%	0.837

Appendix

Table 7: Per-layer profiling for OLMoE on Spider. CV and cold-expert fraction rise sharply from layer 4 onward. Cold% = fraction of experts receiving

<

50% of the uniform routing share.

Figure 5: Per-layer expert activation heatmaps (row-normalised) for OLMoE across all 10 profiling datasets. Each row is a layer and each column an expert. All datasets exhibit a hot-head plus cold-tail structure, but its sharpness varies with task domain: code and programming tasks (MBPP, Spider, CodeAlpaca) show more concentrated patterns than broad-domain tasks (Wikitext, BoolQ, MMLU).

Figure 6: Subsample stability of top-

k

expert sets under 10% profiling data, for all three models across all datasets. Bars show mean Jaccard across 50 bootstrap trials; error bars show the gap to the minimum per-layer overlap. MBPP (374 examples) is the only dataset with mean

J<0.90

for fine-grained architectures; all other datasets achieve mean

J\geq 0.90

across all models.

Model	Task	$\Delta$ (pp)	$\varepsilon$ =1pp	$\varepsilon$ =2pp	$\varepsilon$ =3pp	$t$ -test $p$
OLMoE	GSM8K	$-$ 0.08	.003 ✓	$<$ .001 ✓	$<$ .001 ✓	.753
OLMoE	HellaSwag	$+$ 0.17	$<$ .001 ✓	$<$ .001 ✓	$<$ .001 ✓	.243
OLMoE	Spider	$+$ 0.30	.251	.065	.015 ✓	.771
Qwen	GSM8K	$+$ 0.20	.006 ✓	$<$ .001 ✓	$<$ .001 ✓	.425
Qwen	HellaSwag	$+$ 0.73	$<$ .001 ✓	$<$ .001 ✓	$<$ .001 ✓	$<$ .001
Qwen	Spider	$-$ 0.93	.429	.016 ✓	$<$ .001 ✓	.055

Appendix

Table 8: TOST equivalence

p

-values and paired

t

-test results for hot-25% vs. full LoRA. ✓= equivalence established at

\alpha

= 0.05.

Layer	$k$	Cov%	$k$	Cov%
	Uniform		Greedy
0	16	48.5	15	46.8
1	16	48.0	14	44.5
2	16	52.8	16	52.8
3	16	52.4	19	57.8
4	16	57.5	16	57.5

Appendix

Table 9: Per-layer allocation: uniform (

k

= 16) vs. greedy for OLMoE

\times

GSM8K. Despite greedy’s variable allocation (12–22 experts/layer), mean coverage is virtually identical (0.567 vs. 0.568), explaining why downstream accuracy does not differ.

Explore similar work

Jul 24, 2026

MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

Mixture-of-Experts (MoE) architectures have been widely adopted in large language models, yet parameter-efficient fine-tuning (PEFT) for MoE models remains underexplored. Existing PEFT methods for MoE either ignore router priors with uniform adapters, reducing efficiency and risking forgetting, or rely on static expert selection, limiting per-token capacity and cross-expert feature learning. In this paper, we make the first attempt to fine-tune MoE models with MoE-style low-rank adaptation: our method, entitled MoE

^2

-LoRA, deeply couples the pretrained expert specialization with task-specific adaptivity via a dual-channel Routing-Conditioned Projection (RCP) module, which reuses base router activations to inform LoRA routing. We further introduce a single global LoRA expert pool shared across all layers, enabling model-wide adaptation with emergent layer-wise affinities and balanced expert utilization. MoE

^2

-LoRA simultaneously benefits from the advantages of prior reuse, dynamic adapter routing, and model-wide knowledge sharing. Evaluated on multiple MoE backbones with varying scales and expert granularities, MoE

^2

-LoRA consistently achieves state-of-the-art downstream accuracy while retaining stronger general capabilities.

Qingyu Yang, Haonan He, Minglei Li +4

Shanghai Artificial Intelligence Laboratory · KTH Royal Institute of Technology · University of Science and Technology of China +2

Mixture-of-Experts Language Models Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Aug 8, 2026

Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models

Mixture-of-Experts (MoE) models decouple parameter count from per-token compute, but deployment still requires hosting every expert in memory. Recent theory shows that experts whose router weights change least during fine-tuning can be pruned with provable accuracy preservation, yet the guarantee assumes full fine-tuning. We show that the signal can be elicited through a brief parameter-efficient adaptation. We fine-tune with a lightweight adapter, rank experts by the induced router change, and prune the least-changed experts in one shot. On Mixtral-8

\times

7B-Instruct, router-only LoRA trains 0.002% of parameters and retains 27.54% MMLU-Pro accuracy with half the experts removed, against roughly 16% for magnitude and random pruning. Signal quality improves monotonically with adapter size, reaching 28.76%, and declines as adaptation spreads beyond the router. Under their shared budget, IA3 reaches 28.04% while Houlsby reaches 25.39%. The criterion transfers to Qwen1.5-MoE fine-tuned for mathematical reasoning, retaining 49.7% mean accuracy over eleven benchmarks with half the experts removed. Structural pruning reduces memory by 49% and per-token latency by 37%. Lightweight router sensitivity therefore makes provably motivated, task-conditioned expert pruning practical at scale.

Ali Janati, Kaoutar El Maghraoui, Chengke Zou +2

Data Science Institute, Columbia University, New York, NY, USA · Department of Computer Science, Columbia University, New York, NY, USA

Mixture-of-Experts Pruning Mixture-of-Experts Models

Ask PDF

Sep 5, 2026

ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs

Parameter-efficient fine-tuning (PEFT) of mixture-of-experts (MoE) models commonly attaches a separate low-rank adapter to each expert. This expert-wise design fragments adaptation in three ways: capacity is split across narrow low-rank updates, gradient supervision becomes sparse and imbalanced under sparse routing, and execution is decomposed into many small GEMMs. We find that such expert-wise separation is often unnecessary, as subsets of LoRA adapters become functionally similar during fine-tuning, revealing redundancy among expert-specific adapters. Based on this redundancy, we propose ACE (Adapter Consolidation across Experts), which groups redundant experts and replaces their expert-specific adapters with group-shared higher-rank LoRA modules under the same PEFT budget. ACE further introduces grouped adapter execution, which consolidates fragmented expert-wise adapter computations into fewer, larger group-level GEMMs. Across evaluations covering 12 datasets and four MoE backbones, ACE achieves the highest observed mean accuracy among the parameter-matched PEFT methods on the three backbones with complete baseline coverage, while providing

1.31\times

1.48\times

wall-clock training speedup over expert-wise LoRA without increasing peak memory. Our code is available at https://github.com/UbiquitousAILab/ACE.

Ahin Lee, Sehyun Yun, Joonha Park +1

Ulsan National Institute of Science and Technology (UNIST) Ulsan, Republic of Korea

Parameter-Efficient Fine-Tuning Fine-Tuning Mixture-of-Experts Language Models Low-Rank Adaptation

Ask PDF

Abstract

Explore similar work

MoE2^22-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models

ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs

MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation