NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptation

Yuxin Yang; Haoran Zhang; Mingxuan Li; Jiachen Xu; Ruoxi Shen; Zhenyu Wang; Tianhao Liu; Siqi Chen

cs.LGMar 12, 2026

NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptation

Authors: Yuxin Yang, Haoran Zhang, Mingxuan Li, Jiachen Xu, Ruoxi Shen, Zhenyu Wang, Tianhao Liu, Siqi Chen,

Organizations: Shanghai University · Fudan University

Abstract

Parameter-Efficient Fine-Tuning (PEFT) techniques, particularly Low-Rank Adaptation (LoRA), have become essential for adapting Large Language Models (LLMs) to downstream tasks. While the recent FlyLoRA framework successfully leverages bio-inspired sparse random projections to mitigate parameter interference, it relies on a static, magnitude-based routing mechanism that is agnostic to input context. In this paper, we propose NeuroLoRA, a novel Mixture-of-Experts (MoE) based LoRA framework inspired by biological neuromodulation -- the dynamic regulation of neuronal excitability based on context. NeuroLoRA retains the computational efficiency of frozen random projections while introducing a lightweight, learnable neuromodulation gate that contextually rescales the projection space prior to expert selection. We further propose a Contrastive Orthogonality Loss to explicitly enforce separation between expert subspaces, enhancing both task decoupling and continual learning capacity. Extensive experiments on MMLU, GSM8K, and ScienceQA demonstrate that NeuroLoRA consistently outperforms FlyLoRA and other strong baselines across single-task adaptation, multi-task model merging, and sequential continual learning scenarios, while maintaining comparable parameter efficiency.

Figures & tables

Method	Params (%)	MMLU	SciQA	GSM8K	Avg.
Full FT	100	66.8	96.2	62.4	75.1
LoRA	0.83	64.2	94.0	56.2	71.5
AdaLoRA	0.81	64.5	93.6	56.8	71.6
DoRA	0.84	64.8	94.3	57.4	72.2
MoLE	0.35	63.9	93.1	55.8	70.9
FlyLoRA	0.13	65.1	94.1	58.7	72.6

Table 1: Single-task performance comparison on Llama-3-8B. Params (%) denotes the percentage of trainable parameters relative to the full model. NeuroLoRA achieves the best performance across all three benchmarks while maintaining comparable parameter efficiency to FlyLoRA.

Method	Avg. (Indiv.)	Task Arith.	TIES
LoRA	71.5	58.2 (-18.6%)	60.5 (-15.4%)
DoRA	72.2	59.8 (-17.2%)	61.7 (-14.5%)
FlyLoRA	72.6	68.9 (-5.1%)	69.6 (-4.1%)
NeuroLoRA	74.3	71.8 (-3.4%)	72.3 (-2.7%)

Table 2: Multi-task model merging results (average accuracy across three tasks). Percentages indicate relative degradation from individual task performance. NeuroLoRA exhibits the smallest performance drop under both merging strategies.

Method	MMLU	SciQA	GSM8K	Avg.	BWT
LoRA-Seq	51.8	84.2	55.4	63.8	$-$ 11.3
EWC+LoRA	56.1	87.5	54.8	66.1	$-$ 7.5
O-LoRA	58.2	89.4	55.6	67.7	$-$ 5.8
FlyLoRA-Seq	59.7	90.1	57.6	69.1	$-$ 4.3
NeuroLoRA-Seq	62.1	92.0	60.4	71.5	$\mathbf{-2.6}$

Table 3: Continual learning results (MMLU

\rightarrow

ScienceQA

\rightarrow

GSM8K). BWT measures backward transfer; values closer to zero indicate less forgetting. NeuroLoRA achieves the best trade-off between plasticity (learning new tasks) and stability (retaining old tasks).

Variant	GSM8K	$\Delta$
NeuroLoRA (full)	61.2	—
w/o $\mathcal{L}_{orth}$ ( $\lambda=0$ )	60.1	$-$ 1.1
w/o Modulation Gate	58.7	$-$ 2.5
w/ Static Gate ( $m_{x}=\text{const}$ )	59.4	$-$ 1.8
w/ Trainable $A$	57.8	$-$ 3.4
$d_{h}=32$	60.6	$-$ 0.6

Table 4: Ablation study on GSM8K. Removing the modulation gate (equivalent to FlyLoRA) incurs the largest degradation. Making

A

trainable degrades performance, confirming the importance of the frozen projection for generalization.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Base model	Llama-3-8B
LoRA target modules	$W_{q},W_{k},W_{v},W_{o}$
Total rank $r$	32
Active rank $k$	8
Sparsity $\rho$	0.25
Gate bottleneck $d_{h}$	64

Appendix

Table 5: Complete hyperparameter configuration.

Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning adapts pretrained models by updating small parameter subsets or lightweight modules instead of all weights.

See all 609 papers

CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging
Feb 28, 2026Jie Cao, Zhenxuan Fan, Zhuonan Wang +8Mixture-of-Experts Models Parameter-Efficient Fine-Tuning
Zhejiang University · Wechat, Tencent
Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
May 8, 2026Haozhan Tang, Xiuqi Zhu, Xinyin Zhang +3LLM Fine-Tuning Low-Rank Adaptation
Carnegie Mellon University · Tsinghua University · Infinigence AI
HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
May 11, 2026Jia Wei, Zhonghao Zhang, Ping Chen +5Mixture-of-Experts Language Models Fine-Tuning
Department of Computer Science and Technlogy Tsinghua University Beijing 100084, China · School of Computer Science and Technlogy Xi’an Jiaotong University · The State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security

Feb 28, 2026J/K move · Enter open · S save

CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

Jie Cao, Zhenxuan Fan, Zhuonan Wang, Tianwei Lin+7

Zhejiang University · Wechat, Tencent

Mixture-of-Experts Models Parameter-Efficient Fine-Tuning Low-Rank Adaptation

Open paper AskPDF arXiv

Large language models (LLMs) achieve remarkable performance on diverse downstream and domain-specific tasks via parameter-efficient fine-tuning (PEFT). However, existing PEFT methods, particularly MoE-LoRA architectures, suffer from limited parameter efficiency and coarse-grained adaptation due to the proliferation of LoRA experts and instance-level routing. To address these issues, we propose Core Space Mixture of LoRA (CoMoL), a novel MoE-LoRA framework that incorporates expert diversity, parameter efficiency, and fine-grained adaptation. Specifically, CoMoL introduces two key components: core space experts and core space routing. Core space experts store each expert in a compact core matrix, preserving diversity while controlling parameter growth. Core space routing dynamically selects and activates the appropriate core experts for each token, enabling fine-grained, input-adaptive routing. Activated core experts are then merged via a soft-merging strategy into a single core expert, which is combined with a shared LoRA to form a specialized LoRA module. Besides, the routing network is projected into the same low-rank space as the LoRA matrices, further reducing parameter overhead without compromising expressiveness. Extensive experiments demonstrate that CoMoL retains the adaptability of MoE-LoRA architectures while achieving parameter efficiency comparable to standard LoRA, consistently outperforming existing methods across multiple tasks.

Abstract

Parameter-Efficient Fine-Tuning

Explore similar work

CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models