CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

Jie Cao; Zhenxuan Fan; Zhuonan Wang; Tianwei Lin; Ziyuan Zhao; Rolan Yan; Wenqiao Zhang; Feifei Shao

cs.CLFeb 28, 2026

CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

Authors: Jie Cao, Zhenxuan Fan, Zhuonan Wang, Tianwei Lin, Ziyuan Zhao, Rolan Yan, Wenqiao Zhang, Feifei Shao, +3 more

Organizations: Zhejiang University · Wechat, Tencent

Abstract

Large language models (LLMs) achieve remarkable performance on diverse downstream and domain-specific tasks via parameter-efficient fine-tuning (PEFT). However, existing PEFT methods, particularly MoE-LoRA architectures, suffer from limited parameter efficiency and coarse-grained adaptation due to the proliferation of LoRA experts and instance-level routing. To address these issues, we propose Core Space Mixture of LoRA (CoMoL), a novel MoE-LoRA framework that incorporates expert diversity, parameter efficiency, and fine-grained adaptation. Specifically, CoMoL introduces two key components: core space experts and core space routing. Core space experts store each expert in a compact core matrix, preserving diversity while controlling parameter growth. Core space routing dynamically selects and activates the appropriate core experts for each token, enabling fine-grained, input-adaptive routing. Activated core experts are then merged via a soft-merging strategy into a single core expert, which is combined with a shared LoRA to form a specialized LoRA module. Besides, the routing network is projected into the same low-rank space as the LoRA matrices, further reducing parameter overhead without compromising expressiveness. Extensive experiments demonstrate that CoMoL retains the adaptability of MoE-LoRA architectures while achieving parameter efficiency comparable to standard LoRA, consistently outperforming existing methods across multiple tasks.

Figures & tables

Figure 1: Comparison of LoRA-based methods. (a) LoRA injects trainable low-rank matrices $\boldsymbol{A}$ and $\boldsymbol{B}$ into transformer layers to approximate weight updates. (b) MoE-LoRA employs multiple LoRA experts with a routing network to select and activate experts based on input tokens. (c) CoMoL fuses expert parameters in the low-rank core space, preserving token-level routing while maintaining expert-specific expressiveness with minimal parameters.

Methods	Params ( $\times$ )	FLOPs ( $\times$ )	Routing Level	Routing Latency
LoRA	$1.0\times$	$1.0\times$	-	-
Soft-weighted MoE-LoRA	$N\times$	$N\times$	Token	Low
Sparse MoE-LoRA	$N\times$	$k\times$	Token	High
Soft-merging MoE-LoRA	$N\times$	$1.0\times$	Instance	Low
CoMoL	$1.0\times$	$1.0\times$	Token	Low

Table 1: Comparison of LoRA and MoE-LoRA variants. Metrics for "Params" and "FLOPs" reflect only the LoRA experts; routing overhead is omitted for simplicity.

N\times

indicates the ratio relative to the LoRA baseline.

Models	AddSub	AQuA	GSM8k	MultiArith	SingleEq	SVAMP	Average	Param
LoRA	$93.67_{\pm 0.39}$	$37.40_{\pm 2.58}$	$81.80_{\pm 0.67}$	$98.17_{\pm 0.86}$	$96.85_{\pm 0.23}$	$88.80_{\pm 0.53}$	$82.78_{\pm 0.47}$	24.77M
MoLoRA	$93.76_{\pm 1.30}$	$42.26_{\pm 2.41}$	$82.51_{\pm 2.35}$	$98.11_{\pm 0.38}$	$96.06_{\pm 1.04}$	$90.40_{\pm 0.30}$	$83.85_{\pm 1.17}$	107.35M
HydraLoRA	$93.16_{\pm 1.27}$	$39.11_{\pm 2.79}$	$81.78_{\pm 1.48}$	$98.11_{\pm 0.25}$	$95.21_{\pm 0.75}$	$88.80_{\pm 0.26}$	$82.70_{\pm 0.88}$	49.55M
MoLA	$93.33_{\pm 0.89}$	$42.78_{\pm 0.45}$	$81.50_{\pm 1.14}$	$98.33_{\pm 0.44}$	$96.13_{\pm 0.50}$	$89.83_{\pm 1.16}$	$83.65_{\pm 0.26}$	107.35M
AdaMoLE	$92.24_{\pm 1.05}$	$41.73_{\pm 4.26}$	$81.65_{\pm 1.00}$	$98.22_{\pm 0.25}$	$96.19_{\pm 0.80}$	$89.10_{\pm 0.75}$	$83.19_{\pm 0.85}$	108.38M
SparseMoA	$93.33_{\pm 0.15}$	$40.68_{\pm 2.95}$	$83.85_{\pm 0.35}$	$98.17_{\pm 0.44}$	$97.31_{\pm 0.11}$	$88.80_{\pm 1.66}$	$83.69_{\pm 0.76}$	24.49M

Table 2: Experimental results on Qwen3-8B for mathematical reasoning benchmarks. Accuracy is used as the evaluation metric. All methods are run with three random seeds; the mean and standard deviation are reported. “Param” indicates trainable parameters. "CoMoL w/o CR" denotes that Core Space Routing is not applied in CoMoL.

Models	AddSub	AQuA	GSM8k	MultiArith	SingleEq	SVAMP	Average	Param
LoRA	$96.20_{\pm 0.25}$	$41.34_{\pm 3.39}$	$87.49_{\pm 0.86}$	$97.83_{\pm 0.29}$	$97.64_{\pm 0.41}$	$90.60_{\pm 0.56}$	$85.18_{\pm 0.86}$	35.39M
MoLoRA	$96.37_{\pm 0.29}$	$44.88_{\pm 2.73}$	$87.59_{\pm 1.12}$	$98.33_{\pm 0.29}$	$97.90_{\pm 0.45}$	$91.50_{\pm 0.69}$	$86.10_{\pm 0.72}$	76.84M
HydraLoRA	$95.78_{\pm 1.39}$	$43.70_{\pm 2.39}$	$88.05_{\pm 1.20}$	$98.17_{\pm 1.04}$	$97.77_{\pm 0.41}$	$91.37_{\pm 0.81}$	$85.81_{\pm 0.34}$	40.47M
MoLA	$96.96_{\pm 0.67}$	$42.26_{\pm 4.55}$	$88.98_{\pm 0.68}$	$98.67_{\pm 0.17}$	$97.83_{\pm 0.52}$	$91.63_{\pm 0.57}$	$86.06_{\pm 0.66}$	153.68M
AdaMoLE	$96.20_{\pm 0.25}$	$40.55_{\pm 4.54}$	$88.02_{\pm 1.26}$	$98.50_{\pm 0.17}$	$98.10_{\pm 0.23}$	$91.13_{\pm 0.70}$	$85.42_{\pm 0.54}$	78.36M
SparseMoA	$96.29_{\pm 0.89}$	$44.62_{\pm 3.44}$	$88.20_{\pm 1.02}$	$98.11_{\pm 0.35}$	$98.10_{\pm 0.11}$	$91.63_{\pm 0.99}$	$\underline{86.16}_{\pm 0.77}$	34.50M

Table 3: Experimental results on Qwen3-14B for mathematical reasoning benchmarks.

	Llama3.1-8B				Qwen3-8B
Methods	Pass@1	Pass@5	Pass@10	# Parameters	Pass@1	Pass@5	Pass@10	# Parameters
LoRA	26.22	54.41	67.07	23.06M	39.69	72.29	81.71	24.77M
MoLoRA	34.15	63.15	72.56	100.14M	43.78	73.49	82.93	107.35M
HydraLora	33.35	60.96	69.51	45.09M	46.89	75.61	84.15	49.55M
MoLA	28.72	56.53	67.68	100.14M	41.10	71.60	81.10	107.35M
AdaMoLE	21.52	44.58	56.10	101.12M	40.91	66.65	76.83	108.38M

Table 4: Experimental results for the code generation task on Llama3.1-8B and Qwen3-8B. Pass@k is used as the evaluation metric. "# Parameters" indicates trainable parameters. Bold numbers indicate the highest performance scores, and underlined numbers indicate the second-highest performance scores.

Figure 2: Performance and trainable parameter comparison between CoMoL and LoRA across different ranks on the code generation benchmark.

	CoMoL w/o CR		CoMoL
# Experts	Math	# Param	Math	# Param
Qwen3-8B
8	84.47	33.39M	84.48	25.16M
16	82.50	42.02M	84.00	25.56M
32	82.49	59.28M	82.24	26.34M
64	84.50	93.78M	82.72	27.91M

Table 5: Comparison between CoMoL and CoMoL w/o CR with varying numbers of experts on mathematical reasoning. "# Param" indicates trainable parameters.

Method	Training Time	Inference Time
LoRA	00:56:47	18.40
MoLoRA	04:44:50	19.67
HydraLoRA	04:15:15	19.16
MoLA	07:59:02	30.03
AdaMoLE	08:18:54	40.78
CoMoL w/o CR	02:33:27	18.62

Table 6: Wall-clock time comparison of training and inference across methods. Training time (HH:MM:SS) is measured for one epoch on CodeAlpaca-20k (batch size 1). Inference time (minutes) is the cumulative duration for 10 generations per sample on HumanEval (batch size 16). All MoE-LoRA variants use

k{=}8

experts.

Explore similar work

TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models
Apr 7, 2026Lin Mu, Haiyang Wang, Li Ni +4Mixture-of-Experts Language Models Low-Rank Adaptation
Anhui University · Hefei University · University of Science and Technology of China
MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Jul 24, 2026Qingyu Yang, Haonan He, Minglei Li +4Mixture-of-Experts Language Models Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory · KTH Royal Institute of Technology · University of Science and Technology of China +2
Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
Apr 29, 2026Weihang Li, Jianchun Liu, Hongli XuMixture-of-Experts Pruning LLM Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, China

Apr 7, 2026J/K move · Enter open · S save

TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models

Lin Mu, Haiyang Wang, Li Ni, Lei Sang+3

Anhui University · Hefei University · University of Science and Technology of China

Mixture-of-Experts Language Models Low-Rank Adaptation Parameter-Efficient Fine-Tuning LLM Routing

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of Large Language Models (LLMs), and recent Mixture-of-Experts (MoE) extensions further enhance flexibility by dynamically combining multiple LoRA experts. However, existing MoE-augmented LoRA methods assume that experts operate independently, often leading to unstable routing, expert dominance. In this paper, we propose TalkLoRA, a communication-aware MoELoRA framework that relaxes this independence assumption by introducing expert-level communication prior to routing. TalkLoRA equips low-rank experts with a lightweight Talking Module that enables controlled information exchange across expert subspaces, producing a more robust global signal for routing. Theoretically, we show that expert communication smooths routing dynamics by mitigating perturbation amplification while strictly generalizing existing MoELoRA architectures. Empirically, TalkLoRA consistently outperforms vanilla LoRA and MoELoRA across diverse language understanding and generation tasks, achieving higher parameter efficiency and more balanced expert routing under comparable parameter budgets. These results highlight structured expert communication as a principled and effective enhancement for MoE-based parameter-efficient adaptation. Code is available at https://github.com/why0129/TalkLoRA.

Abstract

Explore similar work

TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models

MoE2^22-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning

MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation