ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs

Ahin Lee; Sehyun Yun; Joonha Park; Taesik Gong

cs.LGSep 5, 2026

ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs

Authors: Ahin Lee, Sehyun Yun, Joonha Park, Taesik Gong

Organizations: Ulsan National Institute of Science and Technology (UNIST) Ulsan, Republic of Korea

Abstract

Parameter-efficient fine-tuning (PEFT) of mixture-of-experts (MoE) models commonly attaches a separate low-rank adapter to each expert. This expert-wise design fragments adaptation in three ways: capacity is split across narrow low-rank updates, gradient supervision becomes sparse and imbalanced under sparse routing, and execution is decomposed into many small GEMMs. We find that such expert-wise separation is often unnecessary, as subsets of LoRA adapters become functionally similar during fine-tuning, revealing redundancy among expert-specific adapters. Based on this redundancy, we propose ACE (Adapter Consolidation across Experts), which groups redundant experts and replaces their expert-specific adapters with group-shared higher-rank LoRA modules under the same PEFT budget. ACE further introduces grouped adapter execution, which consolidates fragmented expert-wise adapter computations into fewer, larger group-level GEMMs. Across evaluations covering 12 datasets and four MoE backbones, ACE achieves the highest observed mean accuracy among the parameter-matched PEFT methods on the three backbones with complete baseline coverage, while providing $1.31\times$ to $1.48\times$ wall-clock training speedup over expert-wise LoRA without increasing peak memory. Our code is available at https://github.com/UbiquitousAILab/ACE.

Figures & tables

Figure 1: Overview of ACE . Left : Standard expert-wise LoRA in MoE uses a separate rank- $r$ adapter for each expert, fragmenting adaptation capacity, yielding sparse updates, and causing many small computations. Right : ACE groups experts with similar adapter-induced representations and replaces their separate adapters with group-shared higher-rank adapters under the same trainable parameter budget. Each shared adapter is updated by all experts in its group and computed once per group, improving capacity, supervision density, and execution efficiency.

Figure 2: Functional similarity among expert LoRA adapters. Each heatmap shows pairwise CKA similarity within one MoE layer, with x- and y-axes denoting expert indices. Values closer to 1 indicate more similar adapter-induced representations. High-similarity blocks reveal groups of experts with similar LoRA corrections. Panels correspond to layers 0, 4, and 14 of the gate projection in OLMoE on HellaSwag. Full details are in Appendix F .

Model	Method	# Params (%) $\downarrow$	HS $\uparrow$	WG $\uparrow$	ARC-c $\uparrow$	ARC-e $\uparrow$	BoolQ $\uparrow$	OBQA $\uparrow$	PIQA $\uparrow$	SIQA $\uparrow$	Avg. $\uparrow$
OLMoE (1B/7B)	LoRA	0.27	90.82 $\mkern 1.0mu{\scriptstyle\pm 0.23}$	70.08 $\mkern 1.0mu{\scriptstyle\pm 0.34}$	69.42 $\mkern 1.0mu{\scriptstyle\pm 1.00}$	85.61 $\mkern 1.0mu{\scriptstyle\pm 0.09}$	66.71 $\mkern 1.0mu{\scriptstyle\pm 1.12}$	74.60 $\mkern 1.0mu{\scriptstyle\pm 0.92}$	83.10 $\mkern 1.0mu{\scriptstyle\pm 0.22}$	52.51 $\mkern 1.0mu{\scriptstyle\pm 0.90}$	74.11 $\mkern 1.0mu{\scriptstyle\pm 0.39}$
	DoRA	0.33	90.98 $\mkern 1.0mu{\scriptstyle\pm 0.27}$	70.25 $\mkern 1.0mu{\scriptstyle\pm 0.34}$	69.46 $\mkern 1.0mu{\scriptstyle\pm 0.31}$	85.98 $\mkern 1.0mu{\scriptstyle\pm 0.14}$	66.96 $\mkern 1.0mu{\scriptstyle\pm 0.71}$	74.93 $\mkern 1.0mu{\scriptstyle\pm 1.21}$	83.31 $\mkern 1.0mu{\scriptstyle\pm 0.30}$	52.35 $\mkern 1.0mu{\scriptstyle\pm 0.89}$	74.28 $\mkern 1.0mu{\scriptstyle\pm 0.23}$
	AdaLoRA	0.54 $\rightarrow$ 0.27	90.68 $\mkern 1.0mu{\scriptstyle\pm 0.14}$	68.14 $\mkern 1.0mu{\scriptstyle\pm 0.25}$	68.51 $\mkern 1.0mu{\scriptstyle\pm 0.23}$	84.69 $\mkern 1.0mu{\scriptstyle\pm 0.84}$	65.52 $\mkern 1.0mu{\scriptstyle\pm 0.39}$	73.67 $\mkern 1.0mu{\scriptstyle\pm 0.12}$	83.08 $\mkern 1.0mu{\scriptstyle\pm 0.47}$	51.31 $\mkern 1.0mu{\scriptstyle\pm 0.61}$	73.20 $\mkern 1.0mu{\scriptstyle\pm 0.11}$
	PiSSA	0.27	90.98 $\mkern 1.0mu{\scriptstyle\pm 0.19}$	70.46 $\mkern 1.0mu{\scriptstyle\pm 0.25}$	69.96 $\mkern 1.0mu{\scriptstyle\pm 0.44}$	86.38 $\mkern 1.0mu{\scriptstyle\pm 0.27}$	68.71 $\mkern 1.0mu{\scriptstyle\pm 0.34}$	75.27 $\mkern 1.0mu{\scriptstyle\pm 0.50}$	83.44 $\mkern 1.0mu{\scriptstyle\pm 0.55}$	52.83 $\mkern 1.0mu{\scriptstyle\pm 0.53}$	74.75 $\mkern 1.0mu{\scriptstyle\pm 0.16}$
	PERFT-R	0.27	87.36 $\mkern 1.0mu{\scriptstyle\pm 0.89}$	71.46 $\mkern 1.0mu{\scriptstyle\pm 2.43}$	66.38 $\mkern 1.0mu{\scriptstyle\pm 1.13}$	82.76 $\mkern 1.0mu{\scriptstyle\pm 0.28}$	64.04 $\mkern 1.0mu{\scriptstyle\pm 4.73}$	73.87 $\mkern 1.0mu{\scriptstyle\pm 0.23}$	80.05 $\mkern 1.0mu{\scriptstyle\pm 0.85}$	51.62 $\mkern 1.0mu{\scriptstyle\pm 0.10}$	72.19 $\mkern 1.0mu{\scriptstyle\pm 0.49}$
	EPnG	0.27	90.82 $\mkern 1.0mu{\scriptstyle\pm 0.34}$	68.95 $\mkern 1.0mu{\scriptstyle\pm 0.55}$	68.83 $\mkern 1.0mu{\scriptstyle\pm 0.60}$	85.75 $\mkern 1.0mu{\scriptstyle\pm 0.60}$	66.96 $\mkern 1.0mu{\scriptstyle\pm 0.52}$	74.47 $\mkern 1.0mu{\scriptstyle\pm 1.40}$	83.15 $\mkern 1.0mu{\scriptstyle\pm 0.39}$	52.27 $\mkern 1.0mu{\scriptstyle\pm 0.21}$	73.90 $\mkern 1.0mu{\scriptstyle\pm 0.33}$

Table 1: Commonsense reasoning results across MoE backbones. Model entries show active/total backbone parameters. The # Params column denotes the percentage of trainable adapter parameters. We report accuracy for each benchmark; the Avg. column reports the mean benchmark accuracy averaged over three runs, with standard deviation across runs. The best result for each model and metric is shown in bold.

Model	Peak Mem. (GB)		Train Time (hr)		Speed up
Model	LoRA	ACE	LoRA	ACE	Speed up
OLMoE	98.21	98.18	2.47	1.89	1.31 $\times$
Qwen	89.90	89.84	6.18	4.19	1.48 $\times$
Moonlight	48.80	48.80	23.75	16.77	1.42 $\times$
Kimi	118.22	118.22	87.39	60.41	1.45 $\times$

Table 2: Training efficiency over three epochs. We compare ACE with expert-wise LoRA under the same batch size and training setup. Speedup is computed as LoRA time divided by ACE time.

	OLMoE						Qwen
Method	# Params (%) $\downarrow$	GSM8K $\uparrow$	MATH $\uparrow$	HEval $\uparrow$	CHold $\uparrow$	Avg. $\uparrow$	# Params (%) $\downarrow$	GSM8K $\uparrow$	MATH $\uparrow$	HEval $\uparrow$	CHold $\uparrow$	Avg. $\uparrow$
LoRA	1.08	65.73	25.95	30.49	78.53	50.18	0.86	66.34	36.31	77.44	72.11	63.05
AdaLoRA	2.16 $\rightarrow$ 1.08	65.20	24.68	26.22	75.65	47.94	1.72 $\rightarrow$ 0.86	66.64	38.33	79.27	69.06	63.33
DoRA	1.14	66.11	25.78	28.05	78.64	49.65	0.91	67.10	37.41	77.44	72.26	63.55
PiSSA	1.08	65.88	24.68	32.32	79.83	50.68	0.86	66.19	39.34	75.61	74.93	64.02
ACE (ours)	1.08	66.49	26.96	31.10	82.57	51.78	0.86	67.48	37.49	79.27	75.24	64.87

Table 3: Results on mathematical reasoning, code generation, and legal understanding across MoE backbones. We report accuracy for GSM8K, MATH, and CaseHOLD (CHold), and pass@10 for HumanEval (HEval). The Avg. reports the mean across the four benchmarks. The best result for each model and metric is shown in bold.

Figure 3: Distribution of effective ranks for LoRA-

r{=}2

, LoRA-

r{=}64

, and ACE . The parameter counts denote the total number of trainable parameters for each method.

Figure 4: Gradient-supervision fragmentation for up-projection LoRA adapters across three MoE backbones. ACE reduces both gradient-norm imbalance and the zero-gradient ratio relative to expert-wise LoRA.

Metric	LoRA	ACE
Matrix multiplication calls	106.0K	62.2K ( $-41.3\%$ )
Matrix multiplication CUDA time	838ms	558ms ( $-33.4\%$ )
CUDA kernel events	482.9K	389.0K ( $-19.4\%$ )
CUDA kernel time	10.36s	8.14s ( $-21.4\%$ )
Wall-clock time	20.24s	17.68s ( $-12.6\%$ )

Table 4: CUDA profiling prefill using torch.profiler . Matrix multiplication rows correspond to PyTorch aten::mm operations in the profiled region. Kernel-level rows include all CUDA operations in the same region. Values in parentheses indicate the relative reduction compared with LoRA.

Figure 5: Adapter swapping analysis. Swapping high-CKA adapters causes much smaller NLL degradation than swapping low-CKA or random adapters, indicating that CKA similarity reflects functional interchangeability.

Method	Sim. Group	Rank Cons.	Grp. Exec.	Avg. $\uparrow$	Time $\downarrow$	Mem. $\downarrow$
LoRA	$\times$	$\times$	$\times$	74.11	2.47	98.21
w/o Sim. Group
Random	$\times$	✓	✓	74.72 $(-0.89)$	1.89	98.18
Full Merge	$\times$	✓	✓	73.57 $(-2.04)$	2.03 $(+0.14)$	97.61
w/o Rank Cons.	✓	$\times$	✓	72.63 $(-2.98)$	1.84	96.54
w/o Grp. Exec.	✓	✓	$\times$	75.61	2.83 $(+0.94)$	98.18

Table 5: Component ablation of ACE . Sim. Group denotes CKA-based functional grouping; Rank cons. pools per-expert ranks into a higher-rank group adapter; Grp. exec. evaluates each shared adapter once per active group. Under w/o Sim. Group , Random groups experts randomly, and Full Merge consolidates all experts into one group. Time in hours, memory in GB.

	OLMoE		Qwen
Adaptation Scope	# Params	Avg. $\uparrow$	# Params	Avg. $\uparrow$
Attention only (Q, K, V)	0.36	$71.46\mkern 1.0mu{\scriptstyle\pm 0.20}$	0.26	$79.72\mkern 1.0mu{\scriptstyle\pm 0.20}$
MoE only (LoRA)	0.27	$74.11\mkern 1.0mu{\scriptstyle\pm 0.39}$	0.21	$79.75\mkern 1.0mu{\scriptstyle\pm 0.03}$
MoE only ( ACE )	0.27	$\mathbf{75.61}\mkern 1.0mu{\scriptstyle\pm 0.28}$	0.21	$\mathbf{80.27}\mkern 1.0mu{\scriptstyle\pm 0.18}$
Attn + MoE (LoRA)	0.63	$75.12\mkern 1.0mu{\scriptstyle\pm 0.38}$	0.47	$80.46\mkern 1.0mu{\scriptstyle\pm 0.13}$
Attn + MoE ( ACE )	0.63	$\mathbf{75.99}\mkern 1.0mu{\scriptstyle\pm 0.16}$	0.47	$\mathbf{80.59}\mkern 1.0mu{\scriptstyle\pm 0.12}$

Table 6: Effect of adaptation scope on OLMoE and Qwen, averaged over eight commonsense reasoning benchmarks and three runs. The # Params columns report the percentage of trainable parameters.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	OLMoE-1B-7B	Qwen1.5-MoE-A2.7B	Moonlight-16B-A3B	Kimi-Linear-48B-A3B
Total Parameters	6.9B	14.3B	16B	48B
Active Parameters	1.3B	2.7B	3B	3B
Layers ( $L$ )	16	24	27	27
Hidden Dimension ( $d_{\text{model}}$ )	2048	2048	2048	2304
Number of Experts ( $N$ )	64	60 + 1 (shared)	64 + 2 (shared)	256 + 1 (shared)
Top-K Routing ( $k$ )	8	4	6	8

Appendix

Table 7: A summary of the MoE model configurations used in our experiments.

Backbone	Method	Seed 0	Seed 1	Seed 2
OLMoE	PiSSA	74.58	74.90	74.78
	ACE	75.70	75.84	75.30
Qwen	PiSSA	80.11	79.96	79.89
	ACE	80.32	80.42	80.07
Moonlight	EPnG	79.89	80.13	80.36
	ACE	80.98	81.35	81.19

Appendix

Table 8: Seed-wise average accuracy against the strongest fully evaluated non- ACE baseline for each backbone.

Method	$T_{\text{init}}$	HS	WG	ARC-c	ARC-e	BoolQ	OBQA	PIQA	SIQA	Avg.
LoRA	–	90.82	70.08	69.42	85.61	66.71	74.60	83.10	52.51	74.11
ACE	50	92.14	72.45	71.33	86.62	69.36	77.40	82.37	53.38	75.63
ACE	150	92.27	73.32	70.82	86.45	69.14	76.60	84.06	52.97	75.70
ACE	500	92.52	70.88	69.62	86.41	68.78	76.40	83.73	53.53	75.23

Appendix

Table 9: Effect of warm-up duration on ACE . We vary

T_{\text{init}}

while keeping all other settings fixed, and include LoRA as a baseline. ACE shows limited sensitivity to the exact warm-up duration and outperforms LoRA at every

T_{\text{init}}

, with

T_{\text{init}}=150

giving the best average performance in this sweep. Results are from one random seed.

Figure 6: Local stability of expert grouping over training in the all-layer view. Every 15 steps, we recompute the grouping and compare it with the grouping from the previous measurement step using Jaccard similarity. The left and right panels correspond to training on the HellaSwag and WinoGrande datasets, respectively. In both settings, the local similarity increases quickly and then plateaus, indicating that the grouping structure has stabilized.

Step	Load Entropy $\uparrow$	Aux. Loss $\downarrow$
	LoRA / ACE	LoRA / ACE
500	0.9613 / 0.9605	1.1168 / 1.1214
1000	0.9614 / 0.9616	1.1161 / 1.1162
Final	0.9619 / 0.9620	1.1142 / 1.1133

Appendix

Table 10: Routing statistics for LoRA and ACE during training.

Method	Group Count	Avg. $\uparrow$	Eval Time (s) $\downarrow$
LoRA	–	74.11	20.82
ACE ( $\gamma=3.2$ )	242	74.59	19.76
ACE ( $\gamma=1.6$ )	215	75.04	17.67
ACE ( $\gamma=1.2$ )	141	75.21	17.24
ACE ( $\gamma=0.8$ , default)	97	75.61	17.22
ACE ( $\gamma=0.4$ )	67	75.75	16.97

Appendix

Table 11: Effect of the Louvain resolution parameter

\gamma

on OLMoE. Group Count is the total number of consolidated adapter groups across all MoE layers. Avg. is the eight-benchmark commonsense reasoning average. Eval time is the wall-clock duration of one full evaluation pass on a single H200.

Metric	Centered CKA	Uncentered CKA
Raw MSE Gap $\uparrow$	-2.417e-6	-3.911e-6
Louvain Q $\uparrow$	0.395	0.374
Downstream Average Accuracy $\uparrow$	75.70	75.44

Appendix

Table 12: Comparison between centered and uncentered CKA on OLMoE commonsense reasoning. Both variants use the same ACE pipeline and differ only in whether the representation matrices are mean-centered before computing CKA similarity. Results are from one random seed.

Probe Tokens	Jaccard	Pair-F1	ARI
512	0.923	0.953	0.922
1,000	0.946	0.969	0.948
2,000	0.958	0.976	0.960
5,000	0.989	0.994	0.991
10,000	1.000	1.000	1.000

Appendix

Table 13: Similarity of group assignments obtained with different probe-set sizes to the default 10k-token reference on OLMoE.

Panel of Figure 7 — Figure 7: Rank sensitivity analysis of ACE and LoRA. ACE maintains stronger average performance across different adapter ranks and shows lower final training loss, indicating robustness to the choice of adaptation capacity.

Method	HellaSwag	WinoGrande	OBQA	Average
Avg. $\Delta W$	92.52	67.17	76.60	78.76
Max $\Delta W$	92.46	68.27	76.40	79.04
CKA-center	92.27	68.98	76.60	79.28

Appendix

Table 14: Analysis of group-anchor strategies for shared-adapter initialization.

Model	Method	# Params (%)	HellaSwag	WinoGrande	ARC-c	ARC-e	BoolQ	OBQA	PIQA	Avg.
OLMoE	LoRA	0.27	87.58	64.09	53.92	67.85	73.55	64.40	70.13	68.79
	AdaLoRA	0.54 $\rightarrow$ 0.27	86.06	62.27	55.63	73.99	71.83	62.60	64.64	68.15
	DoRA	0.33	87.41	63.06	52.13	69.99	75.90	60.80	74.97	69.18
	ACE (Ours)	0.27	89.08	71.11	55.46	75.29	78.90	64.60	78.51	73.28
Qwen	LoRA	0.21	91.82	77.03	73.98	87.84	83.70	78.60	84.77	82.53
	AdaLoRA	0.42 $\rightarrow$ 0.21	90.19	65.90	75.26	88.30	73.24	60.60	82.48	76.57

Appendix

Table 15: Accuracy (%) on commonsense reasoning benchmarks. All methods use the same adapter rank on MoE experts, resulting in nearly identical trainable parameter budgets.

Method	ROUGE-1 $\uparrow$	ROUGE-2 $\uparrow$	ROUGE-L $\uparrow$
LoRA	32.98	13.77	26.27
DoRA	33.16	13.90	26.45
AdaLoRA	30.01	11.82	24.17
PiSSA	30.48	12.26	24.72
PERFT-R	33.57	14.31	26.90
ACE	33.82	14.37	27.01

Appendix

Table 16: XSum summarization results on OLMoE under matched trainable-parameter budgets.

Figure 8: Effect of adapter swapping based on expert similarity applied on up projection. Replacing adapters with high similarity experts causes only minor increases in NLL, while dissimilar or random experts lead to substantially larger degradation.

Model	Method	# Params (%) $\downarrow$	HS $\uparrow$	WG $\uparrow$	ARC-c $\uparrow$	ARC-e $\uparrow$	BoolQ $\uparrow$	OBQA $\uparrow$	PIQA $\uparrow$	SIQA $\uparrow$	Avg. $\uparrow$
OLMoE (1B/7B)	LoRA	0.27	90.82	70.08	69.42	85.61	66.71	74.60	83.10	52.51	74.11
OLMoE (1B/7B)	ACE	0.27	92.24	72.85	70.88	86.21	68.04	77.33	83.93	53.41	75.61
Qwen (3B/14B)	LoRA	0.21	93.23	77.66	79.44	90.63	70.84	82.67	86.80	56.72	79.75
Qwen (3B/14B)	ACE	0.21	93.81	79.19	79.50	90.66	71.49	83.87	86.67	56.99	80.27
Moonlight (3B/16B)	LoRA	0.22	92.23	74.59	84.67	94.28	67.17	82.00	86.92	52.47	79.29
Moonlight (3B/16B)	ACE	0.22	93.49	79.95	84.30	94.75	69.31	84.93	87.71	54.93	81.17

Appendix

Table 17: Commonsense reasoning comparison between LoRA and ACE across four MoE backbones. Results for OLMoE, Qwen1.5-MoE, and Moonlight are from Table 1 ; Kimi-Linear-48B-A3B is added here. ACE achieves higher average accuracy than LoRA on every backbone, including the largest one. The best result per backbone-row pair is in bold.

Group.	HS	WG	ARC-c	ARC-e	BoolQ	OBQA	PIQA	SIQA	Avg.
Louvain	92.24	72.85	70.88	86.21	68.04	77.33	83.93	53.41	75.61
Thresh.	92.17	71.77	71.05	86.66	67.94	77.07	84.21	53.15	75.50

Appendix

Table 18: Comparison between Louvain-based grouping and fixed-threshold grouping on OLMoE commonsense reasoning. “Thresh.” denotes fixed-threshold grouping with

t=0.9

. Both variants use the same ACE pipeline and differ only in how CKA-based expert groups are formed.

Component	LoRA (ms)	ACE (ms)	Reduction
Total step	$3010.38$ $\mkern 1.0mu{\scriptstyle\pm 33.86}$	$1688.13$ $\mkern 1.0mu{\scriptstyle\pm 9.31}$	43.92%
Forward	$1351.95$ $\mkern 1.0mu{\scriptstyle\pm 17.20}$	$845.43$ $\mkern 1.0mu{\scriptstyle\pm 6.49}$	37.47%
Adapter forward	$517.48$ $\mkern 1.0mu{\scriptstyle\pm 3.14}$	$56.09$ $\mkern 1.0mu{\scriptstyle\pm 0.28}$	89.16%
Backward	$1619.53$ $\mkern 1.0mu{\scriptstyle\pm 21.18}$	$809.91$ $\mkern 1.0mu{\scriptstyle\pm 3.30}$	49.99%
Optimizer	$24.35$ $\mkern 1.0mu{\scriptstyle\pm 0.66}$	$21.88$ $\mkern 1.0mu{\scriptstyle\pm 0.25}$	–
Router linear	$0.330$ $\mkern 1.0mu{\scriptstyle\pm 0.001}$	$0.330$ $\mkern 1.0mu{\scriptstyle\pm 0.001}$	–

Appendix

Table 19: Fine-grained runtime profiling of expert-wise LoRA and ACE with grouped adapter execution on OLMoE. Adapter forward is a subrange of the forward pass.

Figure 9: Gradient-supervision fragmentation for gate-projection LoRA adapters across three MoE backbones. ACE reduces both gradient-norm imbalance and the zero-gradient ratio relative to expert-wise LoRA.

Model	Method	Train Time (hr)	Peak Memory (GB)
Moonlight	LoRA	23.75 $\pm$ 0.60	48.80 $\pm$ 0.00
	AdaLoRA	43.84 $\pm$ 2.05	50.07 $\pm$ 0.62
	DoRA	41.10 $\pm$ 0.95	80.88 $\pm$ 0.02
	PiSSA	23.64 $\pm$ 0.33	48.80 $\pm$ 0.00
	ACE (ours)	16.77 $\pm$ 0.23	48.80 $\pm$ 0.00
Qwen	LoRA	6.18 $\pm$ 0.05	89.90 $\pm$ 0.01

Appendix

Table 20: Additional training efficiency results across PEFT methods and MoE backbones. Training time is reported as wall-clock hours over 3 epochs, and peak memory is reported in GB. Means and standard deviations are computed over three seeds (0, 1, and 2). All methods use the same effective batch size within each backbone; methods with higher memory requirements may use smaller per-device batches with additional gradient accumulation. For Kimi-Linear, we report the available LoRA comparison.

Model	$E$	Layers	Total Experts	Warm-up Overhead (s)	CKA (s)	Louvain (s)	Consolidation (s)	Total Overhead (% of ACE)	ACE Speedup over LoRA
OLMoE-1B-7B	64	16	1,024	0.00	123.90	0.12	0.37	0.035 hr (1.83%)	1.31 $\times$
Qwen1.5-MoE-A2.7B	60	24	1,440	0.00	275.17	0.15	0.94	0.077 hr (1.83%)	1.48 $\times$
Moonlight-16B-A3B	64	27	1,728	0.00	328.89	0.19	0.56	0.092 hr (0.55%)	1.42 $\times$
Kimi-Linear-48B-A3B	256	27	6,912	0.00	2393.11	0.79	3.19	0.666 hr (1.10%)	1.45 $\times$

Appendix

Table 21: Per-stage breakdown of the ACE consolidation pipeline. The three stages (pairwise CKA computation, Louvain community detection, and adapter consolidation) are reported in seconds. Total overhead is the sum of the three stages relative to total ACE training time. The warm-up phase itself adds no extra cost, as it is identical to standard LoRA training. All models are trained for 3 epochs, following the configuration of Table 1 . Shared experts, which are not included in the ACE grouping procedure, are excluded from the expert counts.

Figure 10: Training loss curves on OLMoE, Qwen, Moonlight with Commonsense.

Figure 11: Training loss curves on OLMoE, Qwen with MetaMathQA.

Figure 12: Training loss curves on OLMoE, Qwen with CodeFeedback.

Figure 13: Final training loss of LoRA and ACE across datasets and MoE backbones. Lower values indicate better convergence.

Explore similar work

Jul 24, 2026

MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

Mixture-of-Experts (MoE) architectures have been widely adopted in large language models, yet parameter-efficient fine-tuning (PEFT) for MoE models remains underexplored. Existing PEFT methods for MoE either ignore router priors with uniform adapters, reducing efficiency and risking forgetting, or rely on static expert selection, limiting per-token capacity and cross-expert feature learning. In this paper, we make the first attempt to fine-tune MoE models with MoE-style low-rank adaptation: our method, entitled MoE

^2

-LoRA, deeply couples the pretrained expert specialization with task-specific adaptivity via a dual-channel Routing-Conditioned Projection (RCP) module, which reuses base router activations to inform LoRA routing. We further introduce a single global LoRA expert pool shared across all layers, enabling model-wide adaptation with emergent layer-wise affinities and balanced expert utilization. MoE

^2

-LoRA simultaneously benefits from the advantages of prior reuse, dynamic adapter routing, and model-wide knowledge sharing. Evaluated on multiple MoE backbones with varying scales and expert granularities, MoE

^2

-LoRA consistently achieves state-of-the-art downstream accuracy while retaining stronger general capabilities.

Qingyu Yang, Haonan He, Minglei Li +4

Shanghai Artificial Intelligence Laboratory · KTH Royal Institute of Technology · University of Science and Technology of China +2

Mixture-of-Experts Language Models Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Mar 25, 2026

MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning

Standard LoRA fine-tuning of Mixture-of-Experts (MoE) models applies adapters to every expert, yet our profiling shows that per-layer expert routing is highly skewed: a small subset of experts handles most tokens in each layer, while many others are rarely activated ("cold"). We propose MoE-Sieve, a simple routing-guided framework for LoRA fine-tuning, and pair it with a systematic profiling study of expert routing across architectures and tasks. The method is simple: profile routing counts on a small calibration set, select the top-k most-routed experts per layer, and apply LoRA only to those experts. Across two architecturally distinct MoE models and three diverse tasks, tuning only the top 25% routed experts per layer remains competitive with full LoRA, with mean differences within +/-1 percentage point across all conditions. This reduces LoRA trainable parameters by 70-73%, adapter checkpoint size by 71-73%, and wall-clock training time by up to 50%. We also observe a non-monotonic relationship between expert count and seed-to-seed variance, consistent with the hypothesis that adapting cold experts can introduce gradient noise without improving accuracy. Further ablations show that random expert selection at matched budget is about 2.5 percentage points worse, indicating that the routing signal matters, while greedy per-layer budget optimization does not improve over uniform top-k.

Andrea Manzoni

Fine-Tuning Selective Fine-Tuning Sparse Fine-Tuning Low-Rank Adaptation Expert Routing

Ask PDF

Jul 2, 2026

EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning

Mixture-of-Experts (MoE) models scale efficiently but remain costly to adapt due to redundant experts and uniform parameter allocation. Existing parameter-efficient fine-tuning (PEFT) methods such as LoRA ignore MoE routing dynamics, leading to suboptimal resource use. We propose EPnG, an adaptive prune-and-grow framework that reallocates LoRA capacity based on expert importance derived from router gate probabilities. EPnG prunes under-utilized experts and expands high-importance experts via rank growth with orthogonal initialization, while maintaining a fixed parameter budget. Across OLMoE and Qwen1.5-MoE, EPnG consistently outperforms LoRA under the same budget and achieves performance comparable to full fine-tuning while updating only 0.55%-0.72% of parameters (up to 140x-180x fewer). These results demonstrate that aligning PEFT with MoE routing yields a more effective and scalable fine-tuning strategy.

Ahin Lee, Sehyun Yun, Taesik Gong

Ulsan National Institute of Science & Technology (UNIST)

Mixture-of-Experts Pruning Mixture-of-Experts Language Models Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Explore similar work

MoE2^22-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning

EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning

MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation