CRLoMTL: Conflict-Resolution through Low-rank matrices for Multi-Task Learning

Ahmed Agiza; Tasneem Shaffee; Rahil Badkul; Sherief Reda

cs.CVOct 26, 2026

CRLoMTL: Conflict-Resolution through Low-rank matrices for Multi-Task Learning

Authors: Ahmed Agiza, Tasneem Shaffee, Rahil Badkul, Sherief Reda

Organizations: School of Engineering, Brown University, Providence, RI, USA

Abstract

Multi-task learning (MTL) improves model efficiency by sharing representations across related vision tasks, but it often suffers from negative transfer when task updates interfere with one another. Existing conflict-mitigation methods typically operate in gradient space, requiring explicit task-gradient manipulation and additional optimization steps. We propose Conflict-Resolution through Low-Rank Matrices for Multi-Task Learning (CRLoMTL), a parameter-efficient framework that mitigates task interference in the low-rank parameter-update space. CRLoMTL learns task-specific LoRA updates through task-specific passes, where each task updates its own low-rank matrices, and integrates these updates through global shared-subspace fusion. Specifically, we introduce a shared-subspace resolver based on singular value decomposition (SVD), referred to as the SVD resolver, which extracts a common low-dimensional orthonormal basis across task adapters and projects task updates into this shared space before merging them into a single deployment-time update. The resulting update can be folded into the backbone, preserving the inference structure of the original model without additional runtime branches. Experiments with Swin Transformer backbones on PASCAL-Context, NYUD-v2, and CelebA show that CRLoMTL achieves a favorable accuracy-parameter trade-off, improving over single-task baselines by up to 2.77% on PASCAL-Context, 5.03% on NYUD-v2, and 3.50% on CelebA, while training only a small fraction of the parameters used by single-task models.

Figures & tables

Figure 1: Overview of the CRLoMTL architecture and training modes. The model alternates between task-specific passes, where each task updates its own low-rank matrices, and global passes, where the SVD resolver fuses task-specific LoRA updates into a single fused adapter $B_{s}A_{s}$ . The compact notation $A_{s}$ and $B_{s}$ denote the shared matrices $A_{\ell,\mathrm{shared}}$ and $B_{\ell,\mathrm{shared}}$ produced by the resolver, with the layer index $\ell$ omitted in the diagram for visual clarity.

Method	SemSeg	Human Parts	Saliency	Normals	Trainable	$\Delta_{m}$ (%) $\uparrow$
	(mIoU $\uparrow$ )	(mIoU $\uparrow$ )	(mIoU $\uparrow$ )	(RMSE $\downarrow$ )	Parameters (M)
Single Task	67.21	61.93	62.35	17.97	112.62	0
Vanilla MTL	67.56	60.24	65.21	16.64	29.47	+2.23
MTI-Net	66.93	59.66	64.76	16.23	31.47	+2.37
MTLoRA	67.90	59.84	65.40	16.60	8.34	+2.55
KnOTS	67.90	60.26	64.68	16.81	30.29	+2.12

Table 1: Evaluation of CRLoMTL versus state-of-the-art (SOTA) methods on PASCAL-Context dataset using Swin-Tiny backbone.

\Delta_{m}

represents the relative improvement of the MTL model compared to the single-task models.

Method	SemSeg	Human Parts	Saliency	Normals	Trainable	$\Delta_{m}$ (%) $\uparrow$
	(mIoU $\uparrow$ )	(mIoU $\uparrow$ )	(mIoU $\uparrow$ )	(RMSE $\downarrow$ )	Parameters (M)
Vanilla MTL	70.20	63.30	66.09	16.17	50.78	+5.66
CRLoMTL	70.78	63.60	66.28	16.17	6.75	+6.08

Table 2: Evaluation on PASCAL-Context with a Swin-Small backbone.

Figure 2: Effect of the global pass on task-specific LoRA update alignment.

Figure 3: SVD resolver analysis in parameter-update space. (a) The retained component

K_{t}

has higher cross-task alignment than the original update

\Delta_{t}

and residual

R_{t}

. (b) The retained component has lower negative-pair rate (NPR), while the residual contains most conflict-prone directions. (c) The final fused LoRA update is more aligned with

K_{t}

than with

R_{t}

, indicating that the merge follows the shared SVD-retained subspace. For clarity, the layer index

\ell

is omitted and

\Delta_{t}

abbreviates

\Delta W_{\ell,t}

Figure 4: Memory usage and average training time per epoch.

Model	Parameters (M)	Eval. Time (s)	Time / Image (ms)
CRLoMTL unmerged	31.91	$235.81\pm 1.57$	$46.19\pm 0.31$
CRLoMTL merged	29.47	$229.23\pm 2.64$	$44.90\pm 0.52$
Vanilla MTL	29.47	$230.63\pm 2.40$	$45.18\pm 0.47$

Table 3: Deployment analysis on PASCAL-Context over three evaluation runs.

Table 8

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Retained and residual energy over training.

Panel of Figure 6 — Figure 6: t-SNE visualization of task-specific features learned by the vanilla MTL baseline and CRLoMTL after 175 training epochs on PASCAL-Context.

Method	SemSeg	Human Parts	Saliency	Normals	$\Delta_{m}$
	mIoU $\uparrow$	mIoU $\uparrow$	mIoU $\uparrow$	RMSE $\downarrow$	(%) $\uparrow$
CRLoMTL	$67.86\pm 0.06$	$60.43\pm 0.02$	$64.93\pm 0.02$	$16.48\pm 0.01$	$2.74\pm 0.03$

Appendix

Table 7: Seed stability of CRLoMTL on PASCAL-Context over three random seeds.

Multi-Task Learning

Multi-task learning trains models to solve related tasks together by sharing representations, parameters, or supervision.

See all 392 papers

Scalable Multi-Task Low-Rank Model Adaptation
Mar 2, 2026Zichen Tian, Antoine Ledent, Qianru SunLow-Rank Adaptation Multi-Task Learning
Singapore Management University
CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning
Aug 22, 2026Qian Zha, Jinda Liu, Yuan Wu +1LLM Fine-Tuning Low-Rank Adaptation
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
Frequency Switching Mechanism for Parameter-E!cient Multi-Task Learning
Mar 22, 2026Shih-Wen Liu, Yen-Chang Chen, Wei-Ta Chu +2Multi-Task Learning Parameter-Efficient Fine-Tuning
National Cheng Kung University, Tainan, Taiwan · NVIDIA Research, Taiwan

Mar 2, 2026J/K move · Enter open · S save

Scalable Multi-Task Low-Rank Model Adaptation

Zichen Tian, Antoine Ledent, Qianru Sun

Singapore Management University

Low-Rank Adaptation Multi-Task Learning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Scaling multi-task low-rank adaptation (LoRA) to a large number of tasks induces catastrophic performance degradation, such as an accuracy drop from 88.2% to 2.0% on DOTA when scaling from 5 to 15 tasks. This failure is due to parameter and representation misalignment. We find that existing solutions, like regularization and dynamic routing, fail at scale because they are constrained by a fundamental trade-off: strengthening regularization to reduce inter-task conflict inadvertently suppresses the essential feature discrimination required for effective routing. In this work, we identify two root causes for this trade-off. First, uniform regularization disrupts inter-task knowledge sharing: shared underlying knowledge concentrates in high-SV components (89% alignment on Flanv2->BBH). Uniform regularization forces high-SV components to update in orthogonal directions, directly disrupting the shared knowledge. Second, Conflict Amplification: Applying LoRA at the component-level (e.g., W_q, W_v) amplifies gradient conflicts; we show block-level adaptation reduces this conflict by 76% with only 50% parameters. Based on these insights, we propose mtLoRA, a scalable solution with three novel designs: 1) Spectral-Aware Regularization to selectively orthogonalize low-SV components while preserving high-SV shared knowledge, 2) Block-Level Adaptation to mitigate conflict amplification and largely improve parameter efficiency, and 3) Fine-Grained Routing using dimension-specific weights for superior expressive power. On four large-scale (15-25 tasks) vision (DOTA and iNat2018) and NLP (Dolly-15k and BBH) benchmarks, mtLoRA achieves 91.7%, 81.5%, 44.5% and 38.5% accuracy on DOTA, iNat2018, Dolly-15k and BBH respectively, outperforming the state-of-the-art by 2.3% on average while using 47% fewer parameters and 24% less training time.

Abstract

Multi-Task Learning

Explore similar work

Scalable Multi-Task Low-Rank Model Adaptation

CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning

Frequency Switching Mechanism for Parameter-E!cient Multi-Task Learning