Scalable Multi-Task Low-Rank Model Adaptation

Zichen Tian; Antoine Ledent; Qianru Sun

cs.LGMar 2, 2026

Scalable Multi-Task Low-Rank Model Adaptation

Authors: Zichen Tian, Antoine Ledent, Qianru Sun

Organizations: Singapore Management University

Abstract

Scaling multi-task low-rank adaptation (LoRA) to a large number of tasks induces catastrophic performance degradation, such as an accuracy drop from 88.2% to 2.0% on DOTA when scaling from 5 to 15 tasks. This failure is due to parameter and representation misalignment. We find that existing solutions, like regularization and dynamic routing, fail at scale because they are constrained by a fundamental trade-off: strengthening regularization to reduce inter-task conflict inadvertently suppresses the essential feature discrimination required for effective routing. In this work, we identify two root causes for this trade-off. First, uniform regularization disrupts inter-task knowledge sharing: shared underlying knowledge concentrates in high-SV components (89% alignment on Flanv2->BBH). Uniform regularization forces high-SV components to update in orthogonal directions, directly disrupting the shared knowledge. Second, Conflict Amplification: Applying LoRA at the component-level (e.g., W_q, W_v) amplifies gradient conflicts; we show block-level adaptation reduces this conflict by 76% with only 50% parameters. Based on these insights, we propose mtLoRA, a scalable solution with three novel designs: 1) Spectral-Aware Regularization to selectively orthogonalize low-SV components while preserving high-SV shared knowledge, 2) Block-Level Adaptation to mitigate conflict amplification and largely improve parameter efficiency, and 3) Fine-Grained Routing using dimension-specific weights for superior expressive power. On four large-scale (15-25 tasks) vision (DOTA and iNat2018) and NLP (Dolly-15k and BBH) benchmarks, mtLoRA achieves 91.7%, 81.5%, 44.5% and 38.5% accuracy on DOTA, iNat2018, Dolly-15k and BBH respectively, outperforming the state-of-the-art by 2.3% on average while using 47% fewer parameters and 24% less training time.

Figures & tables

Figure 1: Motivating observations for our three novel designs. (A) Orthogonal regularization introduces a trade-off between conflict reduction and routing uncertainty. Specifically, through orthogonal regularization, the model accuracy ( blue curve) peaks at $\lambda=0.25$ (+1.7%) but degrades at $\lambda=1.0$ (-1.8%), due to increased routing uncertainty (i.e., Routing Entropy in orange curve). (B) Shared knowledge concentrates in high-SV components. Specifically, high-SV (top-20%, highlighted) shows 89% inter-task alignment and encodes 54% of total singular values, while low-SV (50-100%) shows only 3% alignment with 22% of singular values (detailed in Sec. 4.1 ). This motivates spectral-aware regularization: preserve high-SV shared knowledge, only orthogonalize low-SV components. (C) Block-level LoRA adaptation reduces gradient conflicts. Specifically, block-level adaptation achieves higher gradient alignment between tasks (measured by cosine similarity, $-0.013$ $\pm 0.169$ ) as compared to component-level adaptation ( $-0.054$ $\pm 0.201$ ), accompanied by a +2.1% accuracy improvement (91.2% vs. 89.0% in Table 4 ). See Sec. 4.3 for detailed experimental setups.

Figure 2: The architectural innovations of mt LoRA. (A) Block-Level Adaptation. The LoRA update is computed in a parallel path that bypasses the block’s internal non-linearities, mitigating gradient conflict amplification. This path takes the same LayerNorm output as the main block. (B) Fine-Grained Routing. Within the parallel path, a router (lightweight MLP) generates dimension-specific weight vectors to compose task experts, allowing different feature subspaces to use different LoRA combinations.

Method	DOTA	iNat2018	Avg.
Uniform Routing †
HydraLoRA	$18.0$	$8.5$	$13.3$
+ Sparsity Reg.	$16.5$	$7.2$	$11.9$
+ Orthogonal Reg.	$20.5$	$10.1$	$15.3$
Dynamic Routing
HydraLoRA	$89.0$	$78.3$	$83.8$

Table 1: Orthogonal regularization requires dynamic routing. Without routing (uniform

1/N

weighting), orthogonal regularization achieves only 20.5% on DOTA; with dynamic routing, it improves to 89.8%. Sparsity regularization harms performance in both settings. All results in accuracy (%)

\uparrow

Method	Block-Level Adaptation	Spectral-Aware Regularization	Fine-Grained Routing	Params (%)	Time	DOTA	iNat2018	Dolly-15k	BBH	Avg.
HydraLoRA				75.5M (1.11%)	1.00x	$89.0$ $\pm 0.4$	$78.3$ $\pm 1.7$	$41.6$ $\pm 1.0$	$35.5$ $\pm 1.7$	61.1
mt LoRA (Ours)	$\checkmark$			37.7M (0.56%)	0.67x	$91.2$ $\pm 0.2$	$79.9$ $\pm 1.0$	$43.7$ $\pm 0.4$	$37.9$ $\pm 0.4$	63.2
	$\checkmark$	$\checkmark$		37.7M (0.56%)	0.70x	91.7 $\pm 0.4$	$81.3$ $\pm 1.1$	$43.6$ $\pm 0.4$	$38.4$ $\pm 0.3$	63.8
	$\checkmark$		$\checkmark$	39.8M (0.59%)	0.69x	$89.9$ $\pm 0.5$	$80.2$ $\pm 0.7$	$44.1$ $\pm 0.3$	$38.2$ $\pm 0.2$	63.1
	$\checkmark$	$\checkmark$	$\checkmark$	39.8M (0.59%)	0.76x	$91.0$ $\pm 0.8$	81.5 $\pm 0.6$	44.5 $\pm 0.3$	38.5 $\pm 0.3$	63.9

Table 2: Contribution of each key design. mt LoRA improves +2.8% over baseline with 47% fewer parameters (highlighted in blue ). Block-level adaptation contributes the most (+2.1%). Improvements are consistent across vision (+2.6% by average) and NLP (+2.9% by average) benchmarks. All results in average accuracy (%)

\uparrow

, reported with std across 3 random seeds. Params shown as trainable parameters and % of LLaMA-2-7B. Wall-clock time breakdown in Appendix C.6 .

Strategy	$g$	Dolly-15k	BBH	Avg.	Router $\Delta$
Module-Wise
Scalar	1	$41.6$	$35.5$	38.5	—
Fine-Grained
Grouped	2	$41.6$	$37.0$	39.3	+0.06%
	8	$41.3$	$36.6$	39.0	+0.44%
	16	$41.7$	$37.0$	39.3	+0.93%

Table 3: Ablation of routing granularity. Fine-grained routing balances performance and parameter efficiency. The

g

=2 already achieves +1.5% with only +0.06% extra router parameters (highlighted in blue ), and

g

=32 achieves best performance (+2.2% on BBH). Results in accuracy (%)

\uparrow

. Router

\Delta

shown as extra param % of LLaMA-2-7B.

Adaptation Level		Params (%)	Time	DOTA	iNat2018	Dolly-15k	BBH	Avg.
Component-Level
$W_{q}$ , $W_{v}$		75.5M (1.11%)	1.00x	$89.0$	$78.3$	$41.6$	$35.5$	61.1
Block-Level
Attn	FFN
$\checkmark$		37.7M (0.56%)	0.67x	$89.4$	$79.3$	$43.3$	$37.2$	62.3
	$\checkmark$	37.7M (0.56%)	0.67x	$91.0$	$79.4$	$43.7$	37.9	63.0

Table 4: Ablation of block-level adaptation. Block-level achieves better performance with 50% fewer parameters and 33% less wall-clock time . Specifically, Attn+FFN achieves 63.1% average accuracy (+2.0% over component-level) with same parameter count (75.5M, 1.1% of foundation model). Notably, FFN alone uses only 50% parameters (37.7M vs 75.5M) yet achieves 63.0% average (highlighted in blue ). All results in accuracy (%)

\uparrow

. Params shown as trainable parameters and % of LLaMA-2-7B. Wall-clock time breakdown in Appendix C.6 .

Method	DOTA	iNat2018	Dolly-15k	BBH	Avg.
LoRAHub ( Huang et al., 2023 )	$88.9$ $\pm 1.7$	$80.2$ $\pm 1.6$	$42.0$ $\pm 0.3$	$34.9$ $\pm 0.4$	61.5
MMoELoRA ( Zadouri et al., 2023 )	$89.4$ $\pm 0.2$	$78.0$ $\pm 0.3$	$42.1$ $\pm 0.8$	$35.4$ $\pm 0.9$	61.2
HydraLoRA ( Tian et al., 2024a ) ‡	$89.1$ $\pm 0.4$	$78.5$ $\pm 1.7$	$42.4$ $\pm 0.7$	$36.9$ $\pm 1.0$	61.7
mt LoRA (Ours)	91.7 $\pm 0.4$	81.5 $\pm 0.6$	44.5 $\pm 0.2$	38.5 $\pm 0.3$	64.0

Table 5: Comparison with SOTA. mt LoRA achieves 64.0% average accuracy across four benchmarks (91.7% on DOTA, 81.5% on iNat2018, 44.5% on Dolly-15k, 38.5% on BBH), outperforming previous SOTA HydraLoRA by 2.3% on average. We compare with multiple SOTA methods including LoRAHub, MMoELoRA, and HydraLoRA with identical experimental setup (rank

r=16

, with identical experts numbers). Results obtained with rank

r=16

. All results in average accuracy (%)

\uparrow

Difficulty	LoRA	MMoELoRA	HydraLoRA	mt LoRA
Easy (7 tasks)	64.32	63.74	67.96	69.52
Medium (8 tasks)	37.82	40.82	37.55	41.01
Hard (12 tasks)	14.63	15.20	18.39	18.78
Overall (27 tasks)	34.38	35.37	36.92	38.52

Table 6: Performance breakdown by task difficulty on BBH. Tasks’ difficulty are categorized by average accuracy across methods: Easy (

>

50%), Medium (30-50%), Hard (

<

30%) (category details in Appendix C.5 ). mt LoRA achieves best across all difficulty levels, especially on Medium tasks (+3.5% over HydraLoRA, highlighted ). All results in accuracy (%)

\uparrow

. Per-task results in Table S6 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Block	Spec.	FGR	Params	%	Time (min)	Rel. Time	FLOPs	Rel. FLOPs
HydraLoRA				75.5M	1.11%	94.6	1.00×	4.73e17	1.00×
mt LoRA	$\checkmark$			37.7M	0.56%	63.0	0.67×	4.70e17	0.99×
	$\checkmark$	$\checkmark$		37.7M	0.56%	66.1	0.70×	4.01e17	0.85×
	$\checkmark$		$\checkmark$	39.8M	0.59%	65.5	0.69×	4.70e17	0.99×
	$\checkmark$	$\checkmark$	$\checkmark$	39.8M	0.59%	72.1	0.76×	4.01e17	0.85×

Appendix

Table S1: Computational efficiency breakdown for main ablation (Table 2 ). Block-level adaptation reduces training time by 33% while using 50% fewer parameters. Full mt LoRA is 24% faster than HydraLoRA with 47% fewer parameters.

Configuration	Params	%	Time (min)	Rel. Time	FLOPs	Rel. FLOPs
Component ( $W_{q}$ , $W_{v}$ )	75.5M	1.11%	94.6	1.00×	4.73e17	1.00×
Block Attn only	37.7M	0.56%	63.4	0.67×	4.70e17	0.99×
Block FFN only	37.7M	0.56%	63.0	0.67×	4.70e17	0.99×
Block Attn+FFN	75.5M	1.11%	80.3	0.85×	4.73e17	1.00×

Appendix

Table S2: Computational efficiency breakdown for block-level ablation (Table 4 ). FFN-only achieves same efficiency as Attn-only but better performance (63.0% vs 62.3%). Attn+FFN is 15% faster than component-level despite same parameter count.

	DOTA			iNat2018				Dolly-15k
	5	10	15	15	25	80	100	4	8	16
Single LoRA	94.5	94.5	94.5	87.0	87.0	87.0	87.0	45.5	45.5	45.5
Naive Averaging	88.2	12.0	2.0	3.5	1.0	0.5	0.3	46.1	40.5	16.0
Conflict Score	6.7	87.3	97.9	96.0	98.9	99.4	99.7	$-1.5$	10.9	64.7

Appendix

Table S3: Multi-task collapse increases with task numbers. Naive averaging degrades from 88.2% (5 tasks) to 2.0% (15 tasks) on DOTA, with conflict score reaching 97.9%. Single LoRA achieves 94.5% on DOTA, 87.0% on iNat2018. All results in accuracy (%)

\uparrow

Figure S2: Spectral-aware regularization selectively suppresses low-SV components. (A) Higher weight $\sigma$ on low-SV. The singular value magnitude $\sigma$ ( blue ) decays rapidly across indices. Our weighting function $w(\sigma)=\exp(-\sigma/\bar{\sigma})$ ( orange ) assigns higher regularization weights for low-SV components. (B) Per-index suppression. The relative change in SV magnitude after applying spectral regularization ( $\lambda=1.0$ ). Low-SV components show stronger suppression. (C) Per-band suppression. Aggregated comparison across three bands: top-20% ( $-2.0\%$ ), 20–50% ( $-4.5\%$ ), and 50–100% ( $-6.0\%$ ).

Figure S3: Per-layer gradient similarity analysis. (A) Per-layer mean pairwise gradient cosine similarity for component-level (red) and block-level (teal) adaptation. Error bands show standard deviation. (B) Per-layer difference (Block $-$ Component). Negative values (teal) indicate block-level has lower gradient similarity (less conflict). Block-level shows up to 36% conflict reduction in later layers (24, 29, 31).

Method	DOTA	iNat2018	Dolly-15k	BBH	Avg.
Block-Level
HydraLoRA	$89.0$	$78.3$	$41.6$	$35.5$	61.1
+ Block-Level Adaptation	91.2	$79.9$	$43.7$	$37.9$	63.2
Routing
+ Fine-Grained	$89.9$	80.2	44.1	38.2	63.1

Appendix

Table S4: Domain difference in NLP vs vision. Block-level adaptation improves both vision (DOTA: +2.2%, iNat2018: +1.6%) and NLP (Dolly-15k: +2.1%, BBH: +2.4%), while fine-grained routing shows mixed effects: it degrades DOTA (

-

1.3%) but improves iNat2018 (+0.3%) and NLP (Dolly-15k: +0.4%, BBH: +0.3%), indicating heterogeneous feature dimensions benefit from fine-grained routing. All results in accuracy (%)

\uparrow

Easy ( $>$ 50%)	Medium (30-50%)	Hard ( $<$ 30%)
formal fallacies	causal judgement	logical deduction 5 objects
boolean expressions	snarks	logical deduction 7 objects
movie recommendation	object counting	penguins in a table
sports understanding	logical deduction 3 objects	reasoning colored objects
hyperbaton	date understanding	salient translation error
navigate	tracking shuffled 3	tracking shuffled 5

Appendix

Table S5: BBH task categorization by difficulty level.

Task	LoRA	MMoELoRA	HydraLoRA	mt LoRA
formal_fallacies	100.00	95.55	100.00	100.00
boolean_expressions	72.47	70.04	94.74	71.66
movie_recommendation	61.94	69.64	36.84	84.21
sports_understanding	60.73	54.66	71.66	63.16
hyperbaton	48.18	48.18	64.37	59.51
navigate	56.28	57.89	57.49	57.49

Appendix

Table S6: Per-task performance on BBH. mt LoRA achieves best results on 12 out of 27 tasks. All results in accuracy (%)

\uparrow

. Best per task in bold .

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 542 papers

Aug 22, 2026

CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning

While Multi-Task Learning (MTL) is essential for adapting Large Language Models (LLMs) to diverse domains, prevailing LoRA-based methods rely on complex routing mechanisms that partition task-specific knowledge. In this work, we reveal that such routing-based designs are prone to a training-inference discrepancy, where stochastic routing decisions under distribution shifts compromise inference stability. Driven by a second-order Taylor analysis that exposes the instability induced by routing variance, we challenge the training-inference discrepancy and propose Consistency-Driven Low-Rank Adaptation (CD-LoRA). By eliminating routers entirely, CD-LoRA employs a consistency-driven alignment mechanism to enforce representation congruence across tasks in a shared low-rank space. This paradigm fosters robust, task-agnostic features without explicit partitioning overhead. Extensive experiments show that CD-LoRA consistently outperforms state-of-the-art multi-adapter baselines, offering a simpler, router-free, and more stable solution for multi-task PEFT. The code is available at the anonymous link https://github.com/zhaqian21/CD-LoRA.

Qian Zha, Jinda Liu, Yuan Wu +1

School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning Multi-Task Learning Consistency Regularization

Ask PDF

Apr 20, 2026

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well as initialization. Existing LoRA variants typically address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency. In this work, we present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimizes initialization and resource allocation at the outset of training. TLoRA introduces a data-driven initialization strategy that aligns the LoRA

A

matrix with task-relevant subspaces by performing singular value decomposition on the product of pre-trained weights and input activation covariance. After this, the

A

matrix is frozen, and only the

B

matrix is trained. Furthermore, TLoRA employs a sensitivity-based importance metric to adaptively allocate ranks and scaling factors across layers under a fixed parameter budget. We conduct extensive experiments that demonstrate TLoRA consistently performs excellently across various tasks, including natural language understanding, commonsense reasoning, math reasoning, code generation, and chat generation, while significantly reducing the number of trainable parameters.

Weicheng Lin, Yi Zhang, Jiawei Dang +1

College of Computer Science and Software Engineering, Shenzhen University, China

LLM Fine-Tuning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning

Ask PDF

Oct 13, 2026

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation

Low-rank adaptation (LoRA) often underperforms full fine-tuning. We investigate this performance gap through a tangent-space lens: factorized updates are confined to the tangent space of the current low-rank factors, leaving gradient components outside this space unexploited. An analysis of ten representative LoRA variants reveals that these residual components account for most of the gradient energy and exhibit strong spectral concentration. Motivated by this finding, we propose Dynamic Tangent LoRA (DT-LoRA), which partitions a fixed rank budget into a stable branch and a dynamic branch to capture the principal residual gradient directions. We show that these directions define the optimal update of a local quadratic model under orthogonal-complement and rank constraints. Matrix-free randomized SVD recovers them without materializing the residual gradient, while controlled write-back preserves accumulated updates and stabilizes training within the allocated rank. DT-LoRA adds no trainable parameters or inference cost over LoRA. Experiments across computer vision and natural language processing tasks demonstrate consistent improvements over LoRA and its variants. Moreover, despite being a general-purpose adaptation method, DT-LoRA outperforms task-specific approaches to open-vocabulary domain generalization for semantic segmentation, achieving state-of-the-art results. Code is available at https://github.com/Snipine/DT-LoRA.

Yangming Zhang, Ling Yi, Datong Li +4

Chongqing University of Posts and Telecommunications, China · Anhui University of Science and Technology · Northeastern University, China

Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Low-Rank Adaptation

Explore similar work

CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation