Scalable Multi-Task Low-Rank Model Adaptation

Zichen Tian; Antoine Ledent; Qianru Sun

cs.LGMar 2, 2026

Scalable Multi-Task Low-Rank Model Adaptation

Authors: Zichen Tian, Antoine Ledent, Qianru Sun

Organizations: Singapore Management University

Abstract

Scaling multi-task low-rank adaptation (LoRA) to a large number of tasks induces catastrophic performance degradation, such as an accuracy drop from 88.2% to 2.0% on DOTA when scaling from 5 to 15 tasks. This failure is due to parameter and representation misalignment. We find that existing solutions, like regularization and dynamic routing, fail at scale because they are constrained by a fundamental trade-off: strengthening regularization to reduce inter-task conflict inadvertently suppresses the essential feature discrimination required for effective routing. In this work, we identify two root causes for this trade-off. First, uniform regularization disrupts inter-task knowledge sharing: shared underlying knowledge concentrates in high-SV components (89% alignment on Flanv2->BBH). Uniform regularization forces high-SV components to update in orthogonal directions, directly disrupting the shared knowledge. Second, Conflict Amplification: Applying LoRA at the component-level (e.g., W_q, W_v) amplifies gradient conflicts; we show block-level adaptation reduces this conflict by 76% with only 50% parameters. Based on these insights, we propose mtLoRA, a scalable solution with three novel designs: 1) Spectral-Aware Regularization to selectively orthogonalize low-SV components while preserving high-SV shared knowledge, 2) Block-Level Adaptation to mitigate conflict amplification and largely improve parameter efficiency, and 3) Fine-Grained Routing using dimension-specific weights for superior expressive power. On four large-scale (15-25 tasks) vision (DOTA and iNat2018) and NLP (Dolly-15k and BBH) benchmarks, mtLoRA achieves 91.7%, 81.5%, 44.5% and 38.5% accuracy on DOTA, iNat2018, Dolly-15k and BBH respectively, outperforming the state-of-the-art by 2.3% on average while using 47% fewer parameters and 24% less training time.

Figures & tables

Figure 1: Motivating observations for our three novel designs. (A) Orthogonal regularization introduces a trade-off between conflict reduction and routing uncertainty. Specifically, through orthogonal regularization, the model accuracy ( blue curve) peaks at $\lambda=0.25$ (+1.7%) but degrades at $\lambda=1.0$ (-1.8%), due to increased routing uncertainty (i.e., Routing Entropy in orange curve). (B) Shared knowledge concentrates in high-SV components. Specifically, high-SV (top-20%, highlighted) shows 89% inter-task alignment and encodes 54% of total singular values, while low-SV (50-100%) shows only 3% alignment with 22% of singular values (detailed in Sec. 4.1 ). This motivates spectral-aware regularization: preserve high-SV shared knowledge, only orthogonalize low-SV components. (C) Block-level LoRA adaptation reduces gradient conflicts. Specifically, block-level adaptation achieves higher gradient alignment between tasks (measured by cosine similarity, $-0.013$ $\pm 0.169$ ) as compared to component-level adaptation ( $-0.054$ $\pm 0.201$ ), accompanied by a +2.1% accuracy improvement (91.2% vs. 89.0% in Table 4 ). See Sec. 4.3 for detailed experimental setups.

Figure 2: The architectural innovations of mt LoRA. (A) Block-Level Adaptation. The LoRA update is computed in a parallel path that bypasses the block’s internal non-linearities, mitigating gradient conflict amplification. This path takes the same LayerNorm output as the main block. (B) Fine-Grained Routing. Within the parallel path, a router (lightweight MLP) generates dimension-specific weight vectors to compose task experts, allowing different feature subspaces to use different LoRA combinations.

Method	DOTA	iNat2018	Avg.
Uniform Routing †
HydraLoRA	$18.0$	$8.5$	$13.3$
+ Sparsity Reg.	$16.5$	$7.2$	$11.9$
+ Orthogonal Reg.	$20.5$	$10.1$	$15.3$
Dynamic Routing
HydraLoRA	$89.0$	$78.3$	$83.8$

Table 1: Orthogonal regularization requires dynamic routing. Without routing (uniform

1/N

weighting), orthogonal regularization achieves only 20.5% on DOTA; with dynamic routing, it improves to 89.8%. Sparsity regularization harms performance in both settings. All results in accuracy (%)

\uparrow

Method	Block-Level Adaptation	Spectral-Aware Regularization	Fine-Grained Routing	Params (%)	Time	DOTA	iNat2018	Dolly-15k	BBH	Avg.
HydraLoRA				75.5M (1.11%)	1.00x	$89.0$ $\pm 0.4$	$78.3$ $\pm 1.7$	$41.6$ $\pm 1.0$	$35.5$ $\pm 1.7$	61.1
mt LoRA (Ours)	$\checkmark$			37.7M (0.56%)	0.67x	$91.2$ $\pm 0.2$	$79.9$ $\pm 1.0$	$43.7$ $\pm 0.4$	$37.9$ $\pm 0.4$	63.2
	$\checkmark$	$\checkmark$		37.7M (0.56%)	0.70x	91.7 $\pm 0.4$	$81.3$ $\pm 1.1$	$43.6$ $\pm 0.4$	$38.4$ $\pm 0.3$	63.8
	$\checkmark$		$\checkmark$	39.8M (0.59%)	0.69x	$89.9$ $\pm 0.5$	$80.2$ $\pm 0.7$	$44.1$ $\pm 0.3$	$38.2$ $\pm 0.2$	63.1
	$\checkmark$	$\checkmark$	$\checkmark$	39.8M (0.59%)	0.76x	$91.0$ $\pm 0.8$	81.5 $\pm 0.6$	44.5 $\pm 0.3$	38.5 $\pm 0.3$	63.9

Table 2: Contribution of each key design. mt LoRA improves +2.8% over baseline with 47% fewer parameters (highlighted in blue ). Block-level adaptation contributes the most (+2.1%). Improvements are consistent across vision (+2.6% by average) and NLP (+2.9% by average) benchmarks. All results in average accuracy (%)

\uparrow

, reported with std across 3 random seeds. Params shown as trainable parameters and % of LLaMA-2-7B. Wall-clock time breakdown in Appendix C.6 .

Strategy	$g$	Dolly-15k	BBH	Avg.	Router $\Delta$
Module-Wise
Scalar	1	$41.6$	$35.5$	38.5	—
Fine-Grained
Grouped	2	$41.6$	$37.0$	39.3	+0.06%
	8	$41.3$	$36.6$	39.0	+0.44%
	16	$41.7$	$37.0$	39.3	+0.93%

Table 3: Ablation of routing granularity. Fine-grained routing balances performance and parameter efficiency. The

g

=2 already achieves +1.5% with only +0.06% extra router parameters (highlighted in blue ), and

g

=32 achieves best performance (+2.2% on BBH). Results in accuracy (%)

\uparrow

. Router

\Delta

shown as extra param % of LLaMA-2-7B.

Adaptation Level		Params (%)	Time	DOTA	iNat2018	Dolly-15k	BBH	Avg.
Component-Level
$W_{q}$ , $W_{v}$		75.5M (1.11%)	1.00x	$89.0$	$78.3$	$41.6$	$35.5$	61.1
Block-Level
Attn	FFN
$\checkmark$		37.7M (0.56%)	0.67x	$89.4$	$79.3$	$43.3$	$37.2$	62.3
	$\checkmark$	37.7M (0.56%)	0.67x	$91.0$	$79.4$	$43.7$	37.9	63.0

Table 4: Ablation of block-level adaptation. Block-level achieves better performance with 50% fewer parameters and 33% less wall-clock time . Specifically, Attn+FFN achieves 63.1% average accuracy (+2.0% over component-level) with same parameter count (75.5M, 1.1% of foundation model). Notably, FFN alone uses only 50% parameters (37.7M vs 75.5M) yet achieves 63.0% average (highlighted in blue ). All results in accuracy (%)

\uparrow

. Params shown as trainable parameters and % of LLaMA-2-7B. Wall-clock time breakdown in Appendix C.6 .

Method	DOTA	iNat2018	Dolly-15k	BBH	Avg.
LoRAHub ( Huang et al., 2023 )	$88.9$ $\pm 1.7$	$80.2$ $\pm 1.6$	$42.0$ $\pm 0.3$	$34.9$ $\pm 0.4$	61.5
MMoELoRA ( Zadouri et al., 2023 )	$89.4$ $\pm 0.2$	$78.0$ $\pm 0.3$	$42.1$ $\pm 0.8$	$35.4$ $\pm 0.9$	61.2
HydraLoRA ( Tian et al., 2024a ) ‡	$89.1$ $\pm 0.4$	$78.5$ $\pm 1.7$	$42.4$ $\pm 0.7$	$36.9$ $\pm 1.0$	61.7
mt LoRA (Ours)	91.7 $\pm 0.4$	81.5 $\pm 0.6$	44.5 $\pm 0.2$	38.5 $\pm 0.3$	64.0

Table 5: Comparison with SOTA. mt LoRA achieves 64.0% average accuracy across four benchmarks (91.7% on DOTA, 81.5% on iNat2018, 44.5% on Dolly-15k, 38.5% on BBH), outperforming previous SOTA HydraLoRA by 2.3% on average. We compare with multiple SOTA methods including LoRAHub, MMoELoRA, and HydraLoRA with identical experimental setup (rank

r=16

, with identical experts numbers). Results obtained with rank

r=16

. All results in average accuracy (%)

\uparrow

Difficulty	LoRA	MMoELoRA	HydraLoRA	mt LoRA
Easy (7 tasks)	64.32	63.74	67.96	69.52
Medium (8 tasks)	37.82	40.82	37.55	41.01
Hard (12 tasks)	14.63	15.20	18.39	18.78
Overall (27 tasks)	34.38	35.37	36.92	38.52

Table 6: Performance breakdown by task difficulty on BBH. Tasks’ difficulty are categorized by average accuracy across methods: Easy (

>

50%), Medium (30-50%), Hard (

<

30%) (category details in Appendix C.5 ). mt LoRA achieves best across all difficulty levels, especially on Medium tasks (+3.5% over HydraLoRA, highlighted ). All results in accuracy (%)

\uparrow

. Per-task results in Table S6 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Block	Spec.	FGR	Params	%	Time (min)	Rel. Time	FLOPs	Rel. FLOPs
HydraLoRA				75.5M	1.11%	94.6	1.00×	4.73e17	1.00×
mt LoRA	$\checkmark$			37.7M	0.56%	63.0	0.67×	4.70e17	0.99×
	$\checkmark$	$\checkmark$		37.7M	0.56%	66.1	0.70×	4.01e17	0.85×
	$\checkmark$		$\checkmark$	39.8M	0.59%	65.5	0.69×	4.70e17	0.99×
	$\checkmark$	$\checkmark$	$\checkmark$	39.8M	0.59%	72.1	0.76×	4.01e17	0.85×

Appendix

Table S1: Computational efficiency breakdown for main ablation (Table 2 ). Block-level adaptation reduces training time by 33% while using 50% fewer parameters. Full mt LoRA is 24% faster than HydraLoRA with 47% fewer parameters.

Configuration	Params	%	Time (min)	Rel. Time	FLOPs	Rel. FLOPs
Component ( $W_{q}$ , $W_{v}$ )	75.5M	1.11%	94.6	1.00×	4.73e17	1.00×
Block Attn only	37.7M	0.56%	63.4	0.67×	4.70e17	0.99×
Block FFN only	37.7M	0.56%	63.0	0.67×	4.70e17	0.99×
Block Attn+FFN	75.5M	1.11%	80.3	0.85×	4.73e17	1.00×

Appendix

Table S2: Computational efficiency breakdown for block-level ablation (Table 4 ). FFN-only achieves same efficiency as Attn-only but better performance (63.0% vs 62.3%). Attn+FFN is 15% faster than component-level despite same parameter count.

	DOTA			iNat2018				Dolly-15k
	5	10	15	15	25	80	100	4	8	16
Single LoRA	94.5	94.5	94.5	87.0	87.0	87.0	87.0	45.5	45.5	45.5
Naive Averaging	88.2	12.0	2.0	3.5	1.0	0.5	0.3	46.1	40.5	16.0
Conflict Score	6.7	87.3	97.9	96.0	98.9	99.4	99.7	$-1.5$	10.9	64.7

Appendix

Table S3: Multi-task collapse increases with task numbers. Naive averaging degrades from 88.2% (5 tasks) to 2.0% (15 tasks) on DOTA, with conflict score reaching 97.9%. Single LoRA achieves 94.5% on DOTA, 87.0% on iNat2018. All results in accuracy (%)

\uparrow

Figure S2: Spectral-aware regularization selectively suppresses low-SV components. (A) Higher weight $\sigma$ on low-SV. The singular value magnitude $\sigma$ ( blue ) decays rapidly across indices. Our weighting function $w(\sigma)=\exp(-\sigma/\bar{\sigma})$ ( orange ) assigns higher regularization weights for low-SV components. (B) Per-index suppression. The relative change in SV magnitude after applying spectral regularization ( $\lambda=1.0$ ). Low-SV components show stronger suppression. (C) Per-band suppression. Aggregated comparison across three bands: top-20% ( $-2.0\%$ ), 20–50% ( $-4.5\%$ ), and 50–100% ( $-6.0\%$ ).

Figure S3: Per-layer gradient similarity analysis. (A) Per-layer mean pairwise gradient cosine similarity for component-level (red) and block-level (teal) adaptation. Error bands show standard deviation. (B) Per-layer difference (Block $-$ Component). Negative values (teal) indicate block-level has lower gradient similarity (less conflict). Block-level shows up to 36% conflict reduction in later layers (24, 29, 31).

Method	DOTA	iNat2018	Dolly-15k	BBH	Avg.
Block-Level
HydraLoRA	$89.0$	$78.3$	$41.6$	$35.5$	61.1
+ Block-Level Adaptation	91.2	$79.9$	$43.7$	$37.9$	63.2
Routing
+ Fine-Grained	$89.9$	80.2	44.1	38.2	63.1

Appendix

Table S4: Domain difference in NLP vs vision. Block-level adaptation improves both vision (DOTA: +2.2%, iNat2018: +1.6%) and NLP (Dolly-15k: +2.1%, BBH: +2.4%), while fine-grained routing shows mixed effects: it degrades DOTA (

-

1.3%) but improves iNat2018 (+0.3%) and NLP (Dolly-15k: +0.4%, BBH: +0.3%), indicating heterogeneous feature dimensions benefit from fine-grained routing. All results in accuracy (%)

\uparrow

Easy ( $>$ 50%)	Medium (30-50%)	Hard ( $<$ 30%)
formal fallacies	causal judgement	logical deduction 5 objects
boolean expressions	snarks	logical deduction 7 objects
movie recommendation	object counting	penguins in a table
sports understanding	logical deduction 3 objects	reasoning colored objects
hyperbaton	date understanding	salient translation error
navigate	tracking shuffled 3	tracking shuffled 5

Appendix

Table S5: BBH task categorization by difficulty level.

Task	LoRA	MMoELoRA	HydraLoRA	mt LoRA
formal_fallacies	100.00	95.55	100.00	100.00
boolean_expressions	72.47	70.04	94.74	71.66
movie_recommendation	61.94	69.64	36.84	84.21
sports_understanding	60.73	54.66	71.66	63.16
hyperbaton	48.18	48.18	64.37	59.51
navigate	56.28	57.89	57.49	57.49

Appendix

Table S6: Per-task performance on BBH. mt LoRA achieves best results on 12 out of 27 tasks. All results in accuracy (%)

\uparrow

. Best per task in bold .

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 542 papers

CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning
Aug 22, 2026Qian Zha, Jinda Liu, Yuan Wu +1LLM Fine-Tuning Low-Rank Adaptation
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
TLoRA: Task-aware Low Rank Adaptation of Large Language Models
Apr 20, 2026Weicheng Lin, Yi Zhang, Jiawei Dang +1LLM Fine-Tuning Low-Rank Adaptation
College of Computer Science and Software Engineering, Shenzhen University, China
DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation
Oct 13, 2026Yangming Zhang, Ling Yi, Datong Li +4Low-Rank Adaptation Parameter-Efficient Fine-Tuning
Chongqing University of Posts and Telecommunications, China · Anhui University of Science and Technology · Northeastern University, China

Aug 22, 2026J/K move · Enter open · S save

CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning

Qian Zha, Jinda Liu, Yuan Wu, Yi Chang

School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning Multi-Task Learning Consistency Regularization

Open paper AskPDF arXiv

While Multi-Task Learning (MTL) is essential for adapting Large Language Models (LLMs) to diverse domains, prevailing LoRA-based methods rely on complex routing mechanisms that partition task-specific knowledge. In this work, we reveal that such routing-based designs are prone to a training-inference discrepancy, where stochastic routing decisions under distribution shifts compromise inference stability. Driven by a second-order Taylor analysis that exposes the instability induced by routing variance, we challenge the training-inference discrepancy and propose Consistency-Driven Low-Rank Adaptation (CD-LoRA). By eliminating routers entirely, CD-LoRA employs a consistency-driven alignment mechanism to enforce representation congruence across tasks in a shared low-rank space. This paradigm fosters robust, task-agnostic features without explicit partitioning overhead. Extensive experiments show that CD-LoRA consistently outperforms state-of-the-art multi-adapter baselines, offering a simpler, router-free, and more stable solution for multi-task PEFT. The code is available at the anonymous link https://github.com/zhaqian21/CD-LoRA.

Abstract

Low-Rank Adaptation

Explore similar work

CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation