TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models

Lin Mu; Haiyang Wang; Li Ni; Lei Sang; Zhize Wu; Peiquan Jin; Yiwen Zhang

cs.LGApr 7, 2026

TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models

Authors: Lin Mu, Haiyang Wang, Li Ni, Lei Sang, Zhize Wu, Peiquan Jin, Yiwen Zhang

Organizations: Anhui University · Hefei University · University of Science and Technology of China

Abstract

Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of Large Language Models (LLMs), and recent Mixture-of-Experts (MoE) extensions further enhance flexibility by dynamically combining multiple LoRA experts. However, existing MoE-augmented LoRA methods assume that experts operate independently, often leading to unstable routing, expert dominance. In this paper, we propose TalkLoRA, a communication-aware MoELoRA framework that relaxes this independence assumption by introducing expert-level communication prior to routing. TalkLoRA equips low-rank experts with a lightweight Talking Module that enables controlled information exchange across expert subspaces, producing a more robust global signal for routing. Theoretically, we show that expert communication smooths routing dynamics by mitigating perturbation amplification while strictly generalizing existing MoELoRA architectures. Empirically, TalkLoRA consistently outperforms vanilla LoRA and MoELoRA across diverse language understanding and generation tasks, achieving higher parameter efficiency and more balanced expert routing under comparable parameter budgets. These results highlight structured expert communication as a principled and effective enhancement for MoE-based parameter-efficient adaptation. Code is available at https://github.com/why0129/TalkLoRA.

Figures & tables

Figure 1: Average expert weights of different experts during inference in the MoELoRA architecture on the OBQA Mihaylov et al. (2018) dataset.

Panel of Figure 2 — Figure 2: Framework comparison of LoRA(left) and TalkLoRA(right).

LLM	Method	#Param(%)	BoolQ	PIQA	SIQA	ARC-c	ARC-e	OBQA	HellaS.	WinoG.	Avg.
Qwen2.5-7B	LoRA † ( $r=16$ )	0.4	60.0	73.6	70.0	71.7	85.9	74.4	78.6	75.8	73.8
	HiRA † ( $r=16$ )	0.4	69.0	88.3	80.8	88.7	95.4	88.0	92.3	81.0	85.4
	TeamLoRA † ( $r=16$ )	0.4	74.6	90.0	82.3	88.5	95.9	92.2	95.4	89.0	88.5
	TalkLoRA ( $r=16$ )	0.2	73.6	90.9	83.0	89.6	96.5	92.8	95.5	89.9	89.0
LLaMA2-7B	LoRA ( $r=32$ )	0.8	69.8	79.9	79.5	64.7	79.8	81.0	83.6	82.6	77.6
	DoRA ( $r=32$ )	0.8	71.8	83.7	76.0	68.2	83.7	82.4	89.1	82.6	79.7

Table 1: Accuracy(%) comparison of various methods fine-tuning Qwen2.5-7B, LLaMA2-7B and LLaMA3-8B on the commonsense reasoning tasks. Results for LoRA, DoRA, and HiRA are sourced from Huang et al. (2025) , MixLoRA and MoELoRA are sourced from Li et al. (2024) and Yang et al. (2025) , respectively. † indicates results reproduced using the same configuration as TalkLoRA. ∗ denotes the use of 8 experts.

Method	#Param	SST-2	MRPC	CoLA	QNLI	RTE	STS-B	Avg
BitFit	0.1M	94.0 ±0.87	88.1 ±1.57	54.0 ±3.07	91.0 ±0.05	69.8 ±1.51	89.5 ±0.35	81.1
IA3	0.06M	93.4 ±0.00	86.4 ±0.00	57.8 ±0.00	91.1 ±0.00	73.5 ±0.00	88.5 ±0.00	81.8
LoReFT	0.02M	93.4 ±0.64	89.2 ±2.62	60.4 ±2.60	91.2 ±0.25	79.0 ±1.76	90.0 ±0.29	83.9
RED	0.02M	93.9 ±0.31	89.2 ±0.98	61.0 ±2.96	90.7 ±0.35	78.0 ±2.06	90.4 ±0.32	83.9
LoRA	0.3M	93.9 ±0.49	88.7 ±0.76	59.7 ±4.36	92.6 ±0.10	75.3 ±2.79	90.3 ±0.54	83.4
Adapter	0.4M	93.3 ±0.40	88.4 ±1.54	60.9 ±3.09	92.5 ±0.02	76.5 ±2.26	90.5 ±0.35	83.7

Table 2: Performance comparison of RoBERTa-base model fine-tuned by TalkLoRA and other PEFT baseline methods on the GLUE benchmark (without MNLI and QQP). We report Matthew’s correlation for CoLA,Pearson correlation for STS-B, and accuracy for the remaining tasks. All baseline results are sourced from Bini et al. (2025) .

#Param(%)	LoRA	TalkLoRA	Avg.
0.74	QKVUD	-	80.8
0.17	-	QKV	87.0
0.24	-	UD	87.3
0.41	-	QKVUD	87.6

Table 3: Accuracy(%) comparison of TalkLoRA with several different tuning granularity fine-tuning LLaMA3-8B. Each module is represented by its first letter as follows: (Q)uery, (K)ey, (V)alue, (U)p, (D)own.

Figure 3: Robustness analysis of TalkLoRA. "2e, 3e, 4e" denotes different numbers of experts.

Method	# Param(%)	Avg.
TalkLoRA	0.4	87.6
w/o Sharing	0.7	87.7
w/o Talking	0.4	86.5

Table 4: Accuracy (%) comparison of several variants of TalkLoRA.

Figure 4: Visualization of the learned communication matrix $C$ . All entries in each matrix are normalized to [-1, 1]. Due to space limitations, we select only even-numbered layers for visualization.

Panel of Figure 5 — Figure 5: Router load visualization of without Talking Module (left) and TalkLoRA (right).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Splits Sizes	SST-2	MRPC	CoLA	QNLI	RTE	STS-B
Training Set	67K	3.7K	8.5K	105K	2.5K	5.7K
New Validation Set	436	204	522	1K	139	750
New Test Set	436	204	521	4.5K	138	750

Appendix

Table 5: GLUE dataset sizes.

HyperParmaters	Qwen2.5-7B	LLaMA2-7B		LLaMA3-8B
Rank r	16	16	32	16	32
$\alpha$	16	16	32	16	32
Dropout	0.05
Optimizer	AdamW
LR	1e-4	3e-4
LR Scheduler	Linear

Appendix

Table 6: The hyperparameters for TalkLoRA on the commonsense reasoning tasks.

HyperParmaters	SST-2	MRPC	CoLA	QNLI	RTE	STS-B
Rank r	16
$\alpha$	32
Seed	42,43,44,45,46
Optimizer	AdamW
LR Schedule	Linear
Warmup Ratio	6e-2

Appendix

Table 7: The hyperparameters for TalkLoRA on the Natural Language Understanding.

Figure 6: Distribution of spectral norms for the communication matrix across different layers and modules. The red dashed line ( $y=1.0$ ) denotes the non-expansive threshold. The results demonstrate that the learned matrices predominantly satisfy the non-expansive property, ensuring signal stability throughout the network.

#Param(%)	TalkLoRA	BoolQ	PIQA	SIQA	ARC-c	ARC-e	OBQA	HellaS.	WinoG.	Avg.
0.17	QKV	74.9	89.4	81.2	83.3	93.4	89.4	96.2	88.4	87.0
0.24	UD	74.0	90.2	82.7	83.2	93.1	89.4	96.4	89.2	87.3

Appendix

Table 8: Accuracy(%) comparison of several different tuning granularity of TalkLoRA fine-tuning LLaMA3-8B. Each module is represented by its first letter as follows: (Q)uery, (K)ey, (V)alue, (U)p, (D)own.

#Param(%)	$r_{e}$	$n$	BoolQ	PIQA	SIQA	ARC-c	ARC-e	OBQA	HellaS.	WinoG.	Avg.
0.2	8	2	76.1	88.4	82.6	81.5	93.7	88.0	96.0	88.5	86.8
0.3	8	3	75.7	89.0	82.4	82.6	93.4	89.6	95.9	89.0	87.2
0.4	8	4	76.1	89.6	82.4	84.5	93.9	89.4	96.0	89.2	87.6
0.4	16	2	75.6	89.9	82.5	83.4	93.7	88.8	96.1	88.2	87.3
0.6	16	3	74.7	88.5	81.5	82.7	94.0	89.0	95.8	89.0	86.9
0.8	16	4	76.3	89.2	82.5	83.5	93.6	88.8	95.8	89.7	87.4

Appendix

Table 9: Accuracy(%) comparison of robustness for TalkLoRA fine-tuning LLaMA3-8B.

#Param(%)	Method	BoolQ	PIQA	SIQA	ARC-c	ARC-e	OBQA	HellaS.	WinoG.	Avg.
0.7	w/o Sharing	75.4	89.8	83.0	84.5	93.7	89.4	96.0	89.8	87.7
0.4	w/o Talking	72.6	88.8	82.2	83.4	93.0	88.4	95.7	88.2	86.5

Appendix

Table 10: Accuracy(%) comparison of variant of TalkLoRA fine-tuning LLaMA3-8B.

Explore similar work

MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Jul 24, 2026Qingyu Yang, Haonan He, Minglei Li +4Mixture-of-Experts Language Models Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory · KTH Royal Institute of Technology · University of Science and Technology of China +2
HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
May 11, 2026Jia Wei, Zhonghao Zhang, Ping Chen +5Mixture-of-Experts Language Models Fine-Tuning
Department of Computer Science and Technlogy Tsinghua University Beijing 100084, China · School of Computer Science and Technlogy Xi’an Jiaotong University · The State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging
Feb 28, 2026Jie Cao, Zhenxuan Fan, Zhuonan Wang +8Mixture-of-Experts Models Parameter-Efficient Fine-Tuning
Zhejiang University · Wechat, Tencent

Jul 24, 2026J/K move · Enter open · S save

MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

Qingyu Yang, Haonan He, Minglei Li, Jingqi Ye+3

Shanghai Artificial Intelligence Laboratory · KTH Royal Institute of Technology · University of Science and Technology of China+2

Mixture-of-Experts Language Models Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) architectures have been widely adopted in large language models, yet parameter-efficient fine-tuning (PEFT) for MoE models remains underexplored. Existing PEFT methods for MoE either ignore router priors with uniform adapters, reducing efficiency and risking forgetting, or rely on static expert selection, limiting per-token capacity and cross-expert feature learning. In this paper, we make the first attempt to fine-tune MoE models with MoE-style low-rank adaptation: our method, entitled MoE

^2

-LoRA, deeply couples the pretrained expert specialization with task-specific adaptivity via a dual-channel Routing-Conditioned Projection (RCP) module, which reuses base router activations to inform LoRA routing. We further introduce a single global LoRA expert pool shared across all layers, enabling model-wide adaptation with emergent layer-wise affinities and balanced expert utilization. MoE

^2

-LoRA simultaneously benefits from the advantages of prior reuse, dynamic adapter routing, and model-wide knowledge sharing. Evaluated on multiple MoE backbones with varying scales and expert granularities, MoE

^2

-LoRA consistently achieves state-of-the-art downstream accuracy while retaining stronger general capabilities.

Abstract

Explore similar work

MoE2^22-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models

CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

MoE $^2$ -LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation