CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning

Qian Zha; Jinda Liu; Yuan Wu; Yi Chang

cs.LGAug 22, 2026

CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning

Authors: Qian Zha, Jinda Liu, Yuan Wu, Yi Chang

Organizations: School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

Abstract

While Multi-Task Learning (MTL) is essential for adapting Large Language Models (LLMs) to diverse domains, prevailing LoRA-based methods rely on complex routing mechanisms that partition task-specific knowledge. In this work, we reveal that such routing-based designs are prone to a training-inference discrepancy, where stochastic routing decisions under distribution shifts compromise inference stability. Driven by a second-order Taylor analysis that exposes the instability induced by routing variance, we challenge the training-inference discrepancy and propose Consistency-Driven Low-Rank Adaptation (CD-LoRA). By eliminating routers entirely, CD-LoRA employs a consistency-driven alignment mechanism to enforce representation congruence across tasks in a shared low-rank space. This paradigm fosters robust, task-agnostic features without explicit partitioning overhead. Extensive experiments show that CD-LoRA consistently outperforms state-of-the-art multi-adapter baselines, offering a simpler, router-free, and more stable solution for multi-task PEFT. The code is available at the anonymous link https://github.com/zhaqian21/CD-LoRA.

Figures & tables

Figure 1: Architectural comparison of fine-tuning paradigms. (a) Full Parameter Fine-Tuning (FPT); (b) Vanilla LoRA; (c) Multi-Adapter architectures with task-specific low-rank modules; (d) Multi-Head or asymmetric LoRA variants with routing mechanisms.

Architecture	Multi-Adapter	Multi-Head	Dynamic Routing
Weight Update	$\displaystyle\Delta W=\sum_{i=1}^{N}B_{i}A_{i}$	$\displaystyle\Delta W=\sum_{i=1}^{N}B_{i}A$	$\displaystyle\Delta W=\sum_{i=1}^{N}\omega_{i}(x)\,B_{i}A$

Table 1: Architectural Variants of LoRA for Multi-Task Learning

Figure 2: t-SNE visualization of LoRA-A activations in the down_proj module. Different colors indicate different tasks. Representations from multiple tasks exhibit substantial overlap in the shared low-rank space.

Figure 3: Overview of the CD-LoRA framework.

Figure 4: t-SNE visualization of LoRA-A activations learned by CD-LoRA. Different colors correspond to different tasks. CD-LoRA exhibits compact and well-separated clusters, indicating improved task-level representation consistency.

Method	QNLI	CoLA	MNLI	QQP	SST-2	PIQA	Wino	SIQA	BoolQ	Hella	ARC	GSM8K	Avg.	Mem $\ast$
Qwen2.5-7B
LoRA	82.40	84.30	87.20	86.40	95.44	88.30	78.70	60.30	88.10	94.20	91.34	48.50	82.10	60062
HydraLoRA	83.70	85.20	88.10	87.30	96.33	89.70	79.20	60.70	88.60	95.30	91.60	51.60	83.11	87860
R-LoRA	83.70	85.40	88.50	86.90	96.44	90.10	79.20	61.00	88.90	95.20	91.83	50.30	83.12	66070
M-LoRA	84.00	85.60	88.60	87.20	96.56	90.20	80.90	60.40	89.20	95.50	92.29	50.70	83.43	49868
CD-LoRA	84.60	85.70	89.60	87.80	96.79	90.20	83.50	61.40	89.60	95.30	92.17	50.40	83.92	56882

Table 2: Performance comparison of multi-task fine-tuning methods on Qwen2.5-7B and Qwen2.5-14B.

Method	Router	Consistency	QNLI	MNLI	Wino	BoolQ	ARC	Avg.	Mem $\ast$
R-LoRA $\dagger$	✓	✗	84.30	90.30	83.00	88.40	91.94	87.59	66071
CR-LoRA $\ddagger$	✓	✓	84.20	90.20	83.90	89.10	92.17	87.91	80943
DR-LoRA $\ast$	✗	✗	85.10	90.60	84.60	89.00	92.17	88.29	49869
CD-LoRA $\lx@sectionsign$	✗	✓	85.40	90.90	85.00	89.20	92.29	88.56	56883

Table 3: Module-level ablation on five NLU tasks using 7B models.

Method	up_proj	down_proj	gate_proj	Avg.
R-LoRA $\dagger$	77.98	80.15	74.80	77.64
CR-LoRA $\ddagger$	79.30	82.00	75.23	78.84
DR-LoRA $\ast$	82.20	82.50	81.52	82.07
CD-LoRA $\lx@sectionsign$	85.03	85.26	84.49	84.93

Table 4: Quantification of

\mathbf{B}

matrix similarity (%) across projection layers for Qwen2.5-14B.

Figure 5: Layer-wise $\mathbf{B}$ matrix similarity heatmap for CD-LoRA. The compact and uniform patterns across all projections validate the efficacy of consistency-driven alignment.

Figure 6: Sensitivity analysis of the regularization weight

\lambda

on Qwen2.5-3B. The model maintains superior performance over the baseline within a wide range of

\lambda

, with optimal multi-task accuracy achieved at

\lambda=0.03

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Router	Consistency	BBH (Avg)	BBH (Ov.)	Mem (MB)
R-LoRA $\dagger$	✓	✗	51.70	50.77	66070
CR-LoRA $\ddagger$	✓	✓	52.50	51.69	80942
DR-LoRA $\ast$	✗	✗	52.07	51.27	45510
CD-LoRA $\lx@sectionsign$	✗	✓	53.31	52.58	60288

Appendix

Table 5: Performance comparison on the BBH benchmark with 12-task training.

Figure 7: Layer-wise $B_{i}$ similarity matrix for R-LoRA on Qwen2.5-14B, showing fragmented task-specific subspaces induced by nonlinear routing.

Figure 8: Layer-wise $B_{i}$ similarity matrix for CR-LoRA. While consistency is enforced, routing still leads to uneven and unstable subspace alignment.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 754 papers

From Isolation to Alignment: Unified LoRA for Efficient Multi-Task Learning
Aug 7, 2025Jinda Liu, Yi Chang, Yuan WuMulti-Task Learning Low-Rank Adaptation
School of Artificial Intelligence, Jilin University · 2Southern University of Science and Technology · 3International Center of Future Science, Jilin University +1
Scalable Multi-Task Low-Rank Model Adaptation
Mar 2, 2026Zichen Tian, Antoine Ledent, Qianru SunLow-Rank Adaptation Multi-Task Learning
Singapore Management University
Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation
Mar 15, 2026Xi Xiao, Chenrui Ma, Yunbei Zhang +7Low-Rank Adaptation Parameter-Efficient Fine-Tuning
University of Alabama at Birmingham · University of Virginia · Tulane University +5

Aug 7, 2025J/K move · Enter open · S save

From Isolation to Alignment: Unified LoRA for Efficient Multi-Task Learning

Jinda Liu, Yi Chang, Yuan Wu

School of Artificial Intelligence, Jilin University · 2Southern University of Science and Technology · 3International Center of Future Science, Jilin University+1

Multi-Task Learning Low-Rank Adaptation Representation Alignment Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Parameter-Efficient Fine-Tuning (PEFT) is essential for adapting Large Language Models (LLMs) to multi-task scenarios. A prevailing trend in this field involves complex LoRA variants with multiple adapters or heads, which rely on the premise that architectural isolation of task-specific knowledge is necessary. However, this design often introduces dynamic routing, preventing weight merging and causing significant inference latency. In this work, we present a direct challenge to this paradigm. We first reveal a paradox where a simplified, router-free multi-head model with high inter-head redundancy outperforms complex, diversity-driven baselines. Furthermore, we demonstrate that a unified, single-adapter LoRA with increased rank achieves highly competitive performance, questioning the necessity of multi-component structures. Based on these findings, we propose Align-LoRA, a unified and efficient framework that shifts the focus from architectural isolation to representation alignment. Align-LoRA incorporates an explicit alignment loss to encourage the learning of task-shared representations within a shared latent space. Crucially, our method maintains the standard LoRA architecture, ensuring zero inference latency via weight merging. Theoretical analysis and extensive experiments confirm that Align-LoRA significantly surpasses prevailing approaches, establishing a simpler, more effective, and production-friendly paradigm for multi-task PEFT. The code is available at https://github.com/jinda-liu/Align-LoRA.

Abstract

LLM Fine-Tuning

Explore similar work

From Isolation to Alignment: Unified LoRA for Efficient Multi-Task Learning

Scalable Multi-Task Low-Rank Model Adaptation

Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation