DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation

Yangming Zhang; Ling Yi; Datong Li; Jinliang Ding; Xiaojie Wang; Tiecheng Song; Zhaolong Ning

cs.CVOct 13, 2026

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation

Authors: Yangming Zhang, Ling Yi, Datong Li, Jinliang Ding, Xiaojie Wang, Tiecheng Song, Zhaolong Ning

Organizations: Chongqing University of Posts and Telecommunications, China · Anhui University of Science and Technology · Northeastern University, China

Abstract

Low-rank adaptation (LoRA) often underperforms full fine-tuning. We investigate this performance gap through a tangent-space lens: factorized updates are confined to the tangent space of the current low-rank factors, leaving gradient components outside this space unexploited. An analysis of ten representative LoRA variants reveals that these residual components account for most of the gradient energy and exhibit strong spectral concentration. Motivated by this finding, we propose Dynamic Tangent LoRA (DT-LoRA), which partitions a fixed rank budget into a stable branch and a dynamic branch to capture the principal residual gradient directions. We show that these directions define the optimal update of a local quadratic model under orthogonal-complement and rank constraints. Matrix-free randomized SVD recovers them without materializing the residual gradient, while controlled write-back preserves accumulated updates and stabilizes training within the allocated rank. DT-LoRA adds no trainable parameters or inference cost over LoRA. Experiments across computer vision and natural language processing tasks demonstrate consistent improvements over LoRA and its variants. Moreover, despite being a general-purpose adaptation method, DT-LoRA outperforms task-specific approaches to open-vocabulary domain generalization for semantic segmentation, achieving state-of-the-art results. Code is available at https://github.com/Snipine/DT-LoRA.

Figures & tables

Panel of Figure 1 — Figure 1: Analysis of the tangent-space residual gradient. (a) Tangent-space residual gradient energy ratio $\rho$ of ten LoRA variants. (b) Cumulative spectral energy $E_{k}$ of the tangent-space residual gradient. (c) Gradient coverage $1-\rho$ of four LoRA variants and DT-LoRA during training. For various LoRA variants, the tangent-space residual gradient accounts for a large proportion of the gradient energy and exhibits spectral concentration. Our DT-LoRA explicitly accounts for this property, and its gradient coverage keeps increasing during training, indicating that the principal directions of the tangent-space residual gradient are effectively exploited.

Figure 2: Overview of DT-LoRA. A fixed rank budget is split into a Stable branch and a Dynamic branch. Every $\tau$ steps, Matrix-Free Randomized SVD extracts the principal directions of the tangent-space residual gradient from the cached activations $X$ and output gradients $R$ , yielding a candidate update $H$ . Controlled Write-back then merges $H$ into the Dynamic branch, keeping the branch at rank $q$ . After training, both branches are merged into the pre-trained weights, so inference costs the same as LoRA.

Method	Param. (M)	Natural							Specialized				Structured								Avg.
Method	Param. (M)	Cifar100	Caltech101	DTD	Flower102	Pets	SVHN	Sun397	Camelyon	EuroSAT	Resisc45	Retinopathy	Clevr-Count	Clevr-Dist	DMLab	KITTI-Dist	dSpr-Loc	dSpr-Ori	sNORB-Azim	sNORB-Ele	Avg.
Full FT	85.8	68.9	87.7	64.3	97.2	86.9	84.7	38.8	79.7	95.7	84.2	73.9	56.3	58.6	41.7	65.5	57.5	46.7	25.7	29.1	68.9
Linear	0	64.4	85.0	63.2	97.0	86.3	36.6	51.0	78.5	87.5	68.5	74.0	34.3	30.6	33.2	55.4	12.5	20.0	9.6	19.2	57.6
Adapter [ 11 ]	0.16	69.2	90.1	68.0	98.8	89.9	82.8	54.3	84.0	94.9	81.9	75.5	80.9	65.3	48.6	78.3	74.8	48.5	29.9	41.6	73.9
AdaptFormer [ 3 ]	0.16	70.8	91.2	70.5	99.1	90.9	86.6	54.8	83.0	95.8	84.4	76.3	81.9	64.3	49.3	80.3	76.3	45.7	31.7	41.1	74.7
AdaLoRA [ 44 ]	0.37	70.5	90.3	71.9	99.1	91.4	87.0	56.0	83.9	95.3	85.6	75.2	81.6	68.1	51.8	80.2	75.3	49.1	28.9	41.1	75.1

Table 1: Accuracy (%) on VTAB-1K across natural, specialized, and structured tasks. In all result tables, our method is shaded in gray and the best results are in bold.

Method	# Params (M)	Detection						Instance Segmentation						All Avg.
		mAP	AP 50	AP 75	AP S	AP M	AP L	mAP	AP 50	AP 75	AP S	AP M	AP L
Full FT	104.97	49.0	69.1	54.6	31.7	53.5	61.7	43.4	66.2	47.6	23.4	47.1	60.9	50.7
LoRA r=16 [ 12 ]	17.27	35.5	55.9	38.9	22.7	38.4	45.1	33.6	53.1	36.3	17.4	35.9	48.2	38.4
LieRA [ 29 ]	17.27	39.1	59.9	43.1	25.9	42.6	49.2	37.0	57.6	40.2	20.1	40.0	53.2	42.3
DoRA [ 25 ]	17.27	39.7	60.6	43.6	25.3	43.2	51.4	37.2	57.9	40.1	19.4	40.2	53.9	42.7
DT-LoRA	17.27	44.1	65.2	48.2	28.7	47.9	56.5	40.3	62.6	43.6	21.9	43.5	57.1	46.6

Table 2: Object detection and instance segmentation results of fine-tuned ConvNeXt-V2-B on MS COCO.

Method	Backbone	Training Data	Dv-19				Dv-58
Method	Backbone	Training Data	ACDC-19	BDD-19	Mapi-19	Avg.	ACDC-41	BDD-41	Mapi-30	RW-10	Avg.
OVSeg (CVPR’23) [ 22 ]		CS-7	36.6	40.9	45.8	41.1	41.2	48.3	31.7	32.6	38.5
SAN (CVPR’23) [ 39 ]		CS-7	37.6	39.4	44.0	40.3	44.3	46.8	35.4	30.7	39.3
CAT-Seg (CVPR’24) [ 4 ]		CS-7	38.9	44.0	47.6	43.5	47.6	51.6	38.2	36.5	43.5
MAFT+ (ECCV’24) [ 15 ]		CS-7	40.4	43.5	50.2	44.7	46.4	48.8	38.1	35.4	42.2
ESC-Net (CVPR’25) [ 16 ]	EVA02 [ 30 ] ViT-B/16	CS-7	40.4	43.6	45.9	43.3	47.3	49.2	40.1	34.5	42.7

Table 3: Comparison of different methods across different backbones under the Real-to-Real OVDG-SS setting.

Method	Backbone	Training Data	Dv-19					Dv-58
Method	Backbone	Training Data	CS-19	ACDC-19	BDD-19	Mapi-19	Avg.	ACDC-41	BDD-41	Mapi-30	RW-10	Avg.
OVSeg (CVPR’23) [ 22 ]		GTA-7	41.2	34.9	42.5	45.1	40.9	45.1	52.6	35.5	32.9	41.5
SAN (CVPR’23) [ 39 ]		GTA-7	43.3	36.7	42.0	48.1	42.5	47.7	49.0	37.8	30.3	41.2
CAT-Seg (CVPR’24) [ 4 ]		GTA-7	43.7	37.6	45.4	48.8	43.9	51.2	56.2	39.3	35.7	45.6
MAFT+ (ECCV’24) [ 15 ]		GTA-7	45.7	38.5	43.7	50.2	44.5	49.1	48.3	39.9	31.8	42.3
ESC-Net (CVPR’25) [ 16 ]	EVA02 ViT-B/16	GTA-7	42.2	35.9	44.6	48.1	42.7	49.4	52.4	37.9	33.3	43.3

Table 4: Comparison of different methods across different backbones under the Synthetic-to-Real OVDG-SS setting.

Model	Method	# Params (%)	BoolQ	PIQA	SIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Avg.
ChatGPT	-	–	73.1	85.4	68.5	78.5	66.1	89.8	79.9	74.8	77.0
LLaMA2-7B	LoRA r=32 [ 12 ]	0.83	69.8	79.9	79.5	83.6	79.8	79.3	64.7	81.0	77.6
	LoRA r=16 [ 12 ]	0.43	72.4	82.5	79.1	88.5	80.6	82.3	67.1	78.9	79.3
	DoRA r=32 [ 25 ]	0.84	71.8	83.7	76.0	89.1	82.6	83.7	68.2	82.4	79.7
	DT-LoRA r=16	0.43	72.2	82.9	79.0	90.0	82.8	82.5	69.2	81.2	80.0
LLaMA3-8B	LoRA r=32 [ 12 ]	0.70	70.8	85.2	79.9	91.7	84.3	84.2	71.2	79.0	80.8

Table 5: Accuracy (%) on eight Commonsense170K evaluation benchmarks. HS: HellaSwag; WG: WinoGrande; ARC-e: ARC-Easy; ARC-c: ARC-Challenge; OBQA: OpenBookQA.

Method	Params	CoLA (Mcc.)	MNLI (Acc.)	MRPC (Acc.)	RTE (Acc.)	QNLI (Acc.)	SST-2 (Acc.)	STS-B (Corr.)	QQP (Acc.)	Avg.
Full FT	184.3M	69.2	89.9	90.2	83.8	94.0	95.6	91.6	92.4	88.3
BitFit [ 1 ]	0.1M	67.0	89.4	87.8	78.7	92.2	94.8	91.4	88.4	86.2
H-Adapter [ 11 ]	1.2M	62.6	86.5	89.9	80.4	92.8	93.7	90.2	90.8	85.9
P-Adapter [ 27 ]	1.2M	63.9	86.8	89.5	80.5	92.6	93.8	90.7	90.5	86.0
DoRA [ 25 ]	1.3M	65.4	87.8	90.1	81.7	93.0	91.3	91.3	91.3	86.5
DyLoRA r=8 [ 32 ]	0.9M	59.5	86.8	91.4	77.6	93.0	94.4	91.1	89.9	85.5

Table 6: Results of fine-tuned DeBERTaV3-base on the GLUE development set.

Variant	OVDG-SS		MS COCO
Variant	Dv-19	Dv-58	Det.mAP	Seg.mAP
DT-LoRA w/o Dynamic Branch	52.7	48.5	36.2	34.5
DT-LoRA w/o Controlled Write-back	27.3	16.8	33.7	32.9
DT-LoRA w/ Full-Gradient Refresh	54.2	51.9	43.0	39.3
DT-LoRA	59.2	56.1	44.1	40.3

Table 7: Ablation study of the Dynamic branch in DT-LoRA on OVDG-SS and MS COCO. All variants use the same total rank

r=16

Method	#Params (M)	Extra time	Extra memory	Time (s/iter)	Relative speed
LoRA [ 12 ]	4.33	-	-	0.82	$5.55\times$
MF+RSVD (ours)	4.33	$\mathcal{O}(Nqd/\tau)$	$\mathcal{O}((N{+}d)q)$	0.89	$5.11\times$
DG+RSVD	4.33	$\mathcal{O}(Nd_{\min}d/\tau)$	$\mathcal{O}(d_{\min}d)$	0.93	$4.89\times$
MF+FSVD	4.33	$\mathcal{O}((N{+}d_{\min})d_{\min}d/\tau)$	$\mathcal{O}((N{+}d)d_{\min})$	4.30	$1.06\times$
DG+FSVD	4.33	$\mathcal{O}((N{+}d_{\min})d_{\min}d/\tau)$	$\mathcal{O}(d_{\min}d)$	4.55	$1.00\times$

Table 8: Training efficiency and extra cost of direction extraction. MF: Matrix-Free; DG: Dense gradient; RSVD/FSVD: Randomized/Full SVD. Speed is relative to DG+FSVD. Time is the steady-state per-iteration training time with

\tau=5

, measured on one A100 for OVDG-SS with CAT-Seg and EVA02 ViT-L/14 under the main experimental settings, extra memory excludes the cached

X

and

R

, and

q\ll d_{\min}

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Images	Classes	Usage
train2017	118,287	80	Training
val2017	5,000	80	Evaluation

Appendix

Table 10: Statistics of the MS COCO splits.

Split	Dataset	Source	Images	Resolution	Classes	Group
Train	CS-7	Cityscapes	2,975	$2048\times 1024$	7	–
Train	GTA-7	GTA5	24,999	$2048\times 1024$	7	–
Test	CS-19 (GTA-7 only)	Cityscapes	500	$2048\times 1024$	19	Dv-19
Test	ACDC-19	ACDC	1,600	$1920\times 1080$	19	Dv-19
Test	BDD-19	BDD100K	1,000	$1280\times 720$	19	Dv-19
Test	Mapi-19	Mapillary	2,000	2K–4K	19	Dv-19

Appendix

Table 11: Source and target datasets for OVDG-SS.

Task	Train	Test	Answer
BoolQ	9.4K	3,270	Yes/No
PIQA	16.1K	1,838	Option
SIQA	33.4K	1,954	Option
HellaSwag	39.9K	10,042	Option
WinoGrande	63.2K	1,267	Option
ARC-e	1.1K	2,376	Option

Appendix

Table 12: Statistics of the commonsense reasoning tasks.

Task	Type	Train	Dev	Metric
CoLA	Acceptability	8.5k	1.0k	Matthews Corr.
MNLI	NLI	393k	20k	Accuracy
MRPC	Paraphrase	3.7k	408	Accuracy
RTE	NLI	2.5k	277	Accuracy
QNLI	QA/NLI	105k	5.4k	Accuracy
SST-2	Sentiment	67k	872	Accuracy

Appendix

Table 13: GLUE tasks and statistics.

Hyperparameter	ViT-B/16
Rank $r$ ( $r_{s}$ / $r_{d}$ )	8(6 / 2)
Refresh Interval $\tau$	5 steps
Oversampling $p$	2
Power Iterations $m$	2
$\rho_{\mathrm{refresh}}$	0.25
$\rho_{\mathrm{jump}}$	0.05

Appendix

Table 14: Hyperparameters of DT-LoRA on VTAB-1K.

Hyperparameter	ConvNeXt-V2-B
Adapted Modules	All Conv2d and Linear layers
Rank $r$ ( $r_{s}$ / $r_{d}$ )	16 (12 / 4) / 32 (24 / 8)
$\alpha$	16 / 32
Refresh Interval $\tau$	5 steps
Oversampling $p$	4 / 8
Power Iterations $m$	2

Appendix

Table 15: Hyperparameters of DT-LoRA on MS COCO.

Hyperparameter	ViT-B/16	ViT-L/14
Adapted Modules	$Q$ , $K$ , $V$ , $O$
Adapted Layers (visual / text)	12 / 12	24 / 12
Rank $r$ ( $r_{s}$ / $r_{d}$ )	16 (12 / 4)
$\alpha$	16
Refresh Interval $\tau$	5 steps
Oversampling $p$	4

Appendix

Table 16: Hyperparameters of DT-LoRA on OVDG-SS.

Hyperparameter	LLaMA2-7B / LLaMA3-8B
Adapted Modules	$Q$ , $K$ , $V$ , Up, Down
Rank $r$ ( $r_{s}$ / $r_{d}$ )	16 (12 / 4)
$\alpha$	16
Refresh Interval $\tau$	5 steps
Oversampling $p$	4 / 8
Power Iterations $m$	2

Appendix

Table 17: Hyperparameters of DT-LoRA on commonsense reasoning.

Shared Settings
Adapted Modules		$Q$ , $K$ , $V$ , $O$ , FFN-Int, FFN-Out
Rank $r$ ( $r_{s}$ / $r_{d}$ )		8 (6 / 2)
$\alpha$		16
LoRA Dropout		0.05
Refresh Interval $\tau$		5 steps
Oversampling $p$		2

Appendix

Table 18: Hyperparameters of DT-LoRA on GLUE.

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 542 papers

ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning
Oct 14, 2026Fanyao Li, Yuxi Liu, Xiaohao Liu +3Low-Rank Adaptation Parameter-Efficient Fine-Tuning
Communication University of China · National University of Singapore · Xidian University
Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
Sep 29, 2026Yihao Ouyang, Shiwei Li, Haozhao Wang +5Low-Rank Adaptation Parameter-Efficient Fine-Tuning
Huazhong University of Science and Technology, Wuhan, China · Hebei University of Technology, Tianjin, China
Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
May 29, 2026Valérie Castin, Kimia Nadjahi, Pierre Ablin +1LLM Fine-Tuning Low-Rank Adaptation
1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.

Oct 14, 2026J/K move · Enter open · S save

ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning

Fanyao Li, Yuxi Liu, Xiaohao Liu, He Chang+2

Communication University of China · National University of Singapore · Xidian University

Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning by learning low-rank weight updates. Increasing rank expands adaptation capacity, but does not guarantee that all learned directions improve downstream performance. In a diagnostic experiment with rank-16 LoRA, we show that retaining only the top singular components achieves the best performance, whereas adding further components degrades it. Motivated by this, we propose Residual-Gradient Adaptation (ReGRA), a two-stage method for making more effective use of a fixed rank budget. After the first stage, ReGRA estimates task gradients on a calibration set and removes their projections onto the two-sided subspace induced by the first-stage update. It then aggregates the residual-gradient directions to initialize a second-stage adapter without changing the first-stage update. An orthogonality regularizer encourages the two updates to capture complementary directions. Both updates can be merged into the pretrained weights, incurring no additional inference overhead. Across diverse benchmark sets, ReGRA achieves stronger performance than alternatives with the same total rank and even matches LoRA adapters with twice the rank. Ablation studies and further analysis support the contribution of its core design choices. Code is available at https://github.com/Fanyao-Li/ReGRA.

Abstract

Low-Rank Adaptation

Explore similar work

ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence