D2-LoRA: A Synergistic Approach to Differential and Directional Low-Rank Adaptation

Nozomu Fujisawa; Masaaki Kondo

cs.LGFeb 16, 2026

D2-LoRA: A Synergistic Approach to Differential and Directional Low-Rank Adaptation

Authors: Nozomu Fujisawa, Masaaki Kondo

Organizations: Keio University

Abstract

We systematically investigate the parameter-efficient fine-tuning design space under practical data and compute constraints, and propose D2-LoRA. D2-LoRA achieves 76.4 percent average accuracy across eight question answering and reading comprehension benchmarks using only 5k training samples per task and two epochs, while preserving algebraic mergeability at inference with near-exact numerical equivalence. The method combines signed low-rank residual updates with additive and subtractive components, together with a train-time column-wise projection that keeps each column close to its original norm. After training, the adapter is merged into a single weight matrix, adding zero inference latency. Compared with LoRA, D2-LoRA improves average accuracy by 2.2 percentage points; at matched parameter counts (LoRA rank 2r versus D2-LoRA rank r), the improvement is 1.6 points, indicating gains from architectural design rather than increased parameterization. Compared with DoRA, it matches or exceeds performance on most tasks. Beyond QA and reading comprehension, D2-LoRA improves generative tasks (plus 1.2 ROUGE-L and plus 1.1 percent win rate) and shows 36 percent lower training volatility. The merge preserves numerical fidelity (mean gap about 0.03 percentage points) and recovers about 1.91x evaluation throughput. Training overhead is 19 percent, comparable to DoRA, and decreases with longer input sequences. We provide a geometric analysis explaining how the projection stabilizes training, together with ablation studies isolating the contribution of each design component.

Figures & tables

Method	Mergeable	Norm control	Small-data stability	Rank expressivity
LoRA	✓		$\triangle$	$r$
DoRA	✓	✓	$\triangle$	$r$
D 2 -LoRA	✓	✓	✓	$2r$

Table 1: PEFT design choices and their implications.

Figure 1: Overview of D 2 -LoRA. Left: columnwise magnitude $\,\mathbf{m}=\|W_{0}\|_{2,\mathrm{col}}\,$ from the pretrained weight $W_{0}$ . Middle: differential signed low-rank residual with scale $\tau$ and a train-time directional projection that preserves the column norms of $W_{0}$ . Right: inference uses the merged weight $\widehat{W}=W^{\star}+\Delta W$ , so latency equals a single linear layer.

	Llama-3.2-3B-Instruct (2 epochs)			Qwen2.5-7B-Instruct (1 epoch)
Dataset	LoRA	DoRA	D 2 -LoRA	LoRA	DoRA	D 2 -LoRA
ARC-Challenge	72.6	72.2	73.9	89.6	88.3	89.6
ARC-Easy	87.2	87.6	87.4	96.2	96.2	96.2
BoolQ	62.6	61.8	65.6	82.0	84.2	85.2
CommonsenseQA	73.4	74.8	75.4	82.8	81.8	82.2
HellaSwag	77.6	81.8	82.4	90.2	90.0	91.0

Table 2: Main results: Accuracy (%) on 8 QA/RC tasks with CLL scoring. Best per backbone in bold .

Dataset	LoRA	DoRA	D 2 -LoRA
ARC-Challenge	+2.0	+0.3	+0.7
ARC-Easy	+0.6	+0.4	+0.2
BoolQ	+0.6	$-1.4$	+1.2
CommonsenseQA	+0.0	+0.0	+1.2
HellaSwag	+2.6	+5.4	+5.6
OpenBookQA	$-0.4$	+0.0	+0.6

Table 3: CLL vs generation (Llama-3.2-3B-Instruct,

r{=}32

). Entries show CLL

-

Gen in pp.

Backbone	Dataset	Unmerged	Merged	Speedup
Llama-3.2-3B-Instruct	ARC-Challenge	73.9	73.2	$1.89\times$
	ARC-Easy	87.4	87.4	$1.87\times$
	BoolQ	65.6	65.6	$1.98\times$
	CommonsenseQA	75.4	74.8	$1.82\times$
	HellaSwag	82.4	82.6	$2.03\times$
	OpenBookQA	82.6	83.0	$1.85\times$

Table 4: D 2 -LoRA merge equivalence across backbones. Accuracy (%) and post-merge evaluation speedup. Speedup is computed as (unmerged D 2 -LoRA evaluation time) / (merged D 2 -LoRA evaluation time), i.e., comparing merged vs. unmerged configurations of the same D 2 -LoRA model. This measures the inference benefit of merging, not comparison against other methods.

Table 5: Ablation studies (Llama-3.2-3B-Instruct, CLL scoring): (a) effect of rank

r

; (b) effect of target modules (

r{=}32

Dataset	LoRA ( $r{=}64$ )	DoRA ( $r{=}64$ )	D 2 -LoRA ( $r{=}32$ )
ARC-Challenge	72.9	73.1	73.9
ARC-Easy	87.4	87.5	87.4
BoolQ	63.2	62.8	65.6
CommonsenseQA	74.1	74.6	75.4
HellaSwag	78.8	80.2	82.4
OpenBookQA	82.2	82.8	82.6

Table 8: Parameter-matched comparison (Llama-3.2-3B-Instruct,

\sim

1.58M params). D 2 -LoRA outperforms despite no parameter advantage.

	CNN/DM (ROUGE-L)		AlpacaEval 2.0 (%)
Method	Llama-3.2-3B	Qwen2.5-7B	Llama-3.2-3B	Qwen2.5-7B
LoRA	28.4	31.2	12.3	18.7
DoRA	28.9	31.5	12.8	19.1
D 2 -LoRA	29.6	32.1	13.4	19.8

Table 9: Generative task results (parameter-matched: LoRA/DoRA at

r{=}64

, D 2 -LoRA at

r{=}32

\sim

1.58M params). Left: CNN/DailyMail summarization (ROUGE-L F1). Right: AlpacaEval 2.0 instruction following (Length-Controlled Win Rate %).

Figure 2: Training dynamics on BoolQ (Llama-3.2-3B-Instruct, $r{=}32$ ). (a) Loss curves: D 2 -LoRA converges to a lower final loss (1.894) than LoRA (1.931) or DoRA (1.907). (b) Rolling standard deviation indicates 36% lower volatility for D 2 -LoRA, reflecting more stable optimization. (c) Loss distribution during the final phase (steps 200–312) shows D 2 -LoRA exhibits a tighter concentration around convergence.

Method	Total Time	Overhead vs LoRA
LoRA	24.6 min	—
DoRA	28.8 min	+17.0%
D 2 -LoRA	29.3 min	+19.3%

Table 10: Wall-clock training time (Llama-3.2-3B-Instruct,

r{=}32

, 2 epochs, A100).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Fixed $\tau$ stability. Macro averages for varying $\tau$ (from Table 7 ).

Method	Total Time	Overhead	Note
LoRA	24.6 min	—	fastest
DoRA	28.8 min	+17.0%	magnitude-direction split
D 2 -LoRA	29.3 min	+19.3%	+ signed branch

Appendix

Table 11: Training time summary (Llama-3.2-3B-Instruct,

r{=}32

, 2 epochs).

Dataset	LoRA	DoRA	D 2 -LoRA	Overhead
BoolQ	356	399	405	+13.8%
RACE	452	499	504	+11.5%
HellaSwag	349	393	399	+14.3%
CommonsenseQA	86	123	128	+48.8%
OpenBookQA	94	130	135	+43.6%
ARC-Easy	61	78	80	+31.1%

Appendix

Table 12: Per-dataset training time breakdown (Llama-3.2-3B-Instruct,

r{=}32

, seconds).

BoolQ	CommonsenseQA	RACE	HellaSwag	WinoGrande	ARC-Easy	ARC-Challenge	OpenBookQA	Avg
82.8	82.0	90.2	90.4	62.4	96.2	89.0	88.0	85.1

Appendix

Table 13: Qwen2.5-7B-Instruct, D 2 -LoRA with

r{=}8

(1 epoch).

Dataset	LoRA	DoRA	D 2 -LoRA
BoolQ	1.931	1.907	1.894
HellaSwag	2.201	2.174	2.164
RACE	2.096	2.077	2.065
CommonsenseQA	2.016	1.967	1.868
Average	1.906	1.848	1.843

Appendix

Table 14: Final training loss (Llama-3.2-3B-Instruct,

r{=}32

Method	Parameters	Optimizer State	Total
LoRA	0.79M	2.37M	9.8
DoRA	0.79M	2.37M	10.2
D 2 -LoRA	1.58M	4.74M	11.3

Appendix

Table 15: Peak memory usage during training (GB).

$A_{-}$ init scale	Stable runs	Avg. accuracy	Note
$1.0\sigma$ (standard)	5/8	71.2%	3 runs diverge
$0.1\sigma$ (ours)	8/8	76.4%	all stable
$0.01\sigma$	8/8	75.9%	slower $(-)$ activation

Appendix

Table 16: Effect of

A_{-}

initialization scale (Llama-3.2-3B-Instruct,

r{=}32

Target Modules	Params	Avg Acc	$\Delta$
$q,k,v,o$ (default)	1.58M	76.4	—
$+$ gate, up	2.37M	76.8	+0.4
$+$ gate, up, down	3.16M	77.1	+0.7
All attn + MLP	4.74M	77.2	+0.8

Appendix

Table 17: Effect of adapting MLP layers (Llama-3.2-3B-Instruct).

Initialization	Avg Accuracy	Stability
Same std ( $A_{-}=A_{+}$ )	73.8	Unstable (NaN 3/8)
$A_{-}=0.5\times\text{std}(A_{+})$	75.2	Stable
$A_{-}=0.1\times\text{std}(A_{+})$ (ours)	76.4	Most stable
$A_{-}=0.01\times\text{std}(A_{+})$	75.8	Slow convergence

Appendix

Table 18: Effect of

A_{-}

initialization scale (Llama-3.2-3B-Instruct).

$r_{+}$	$r_{-}$	Avg Accuracy	$\Delta$
16	16	76.4	—
24	8	76.6	+0.2
28	4	75.9	$-$ 0.5
8	24	74.8	$-$ 1.6

Appendix

Table 19: Asymmetric ranks for

\pm

branches (1.58M params total).

Dataset	Avg. context	# choices	Type	Gain
WinoGrande	Short	2	Coref	+4.8 pp
HellaSwag	Medium	4	Completion	+4.8 pp
BoolQ	Long	2	Yes/No	+3.0 pp
CommonsenseQA	Short	5	Commonsense	+2.0 pp

Appendix

Table 20: Task properties and D 2 -LoRA gain over LoRA (Llama-3.2-3B-Instruct).

Hyperparameter	Llama-3.2-3B-Instruct	Qwen2.5-7B-Instruct
Learning rate	$5\times 10^{-5}$	$5\times 10^{-5}$
Warmup steps	100	100
Weight decay	0.01	0.01
Grad clip norm	1.0	1.0
Adam $(\beta_{1},\beta_{2})$	(0.9, 0.999)	(0.9, 0.999)
Adam $\epsilon$	$10^{-8}$	$10^{-8}$

Appendix

Table 21: Exact hyperparameters for reproducibility. Unless stated, both backbones share settings.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 754 papers

Oct 13, 2026

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation

Low-rank adaptation (LoRA) often underperforms full fine-tuning. We investigate this performance gap through a tangent-space lens: factorized updates are confined to the tangent space of the current low-rank factors, leaving gradient components outside this space unexploited. An analysis of ten representative LoRA variants reveals that these residual components account for most of the gradient energy and exhibit strong spectral concentration. Motivated by this finding, we propose Dynamic Tangent LoRA (DT-LoRA), which partitions a fixed rank budget into a stable branch and a dynamic branch to capture the principal residual gradient directions. We show that these directions define the optimal update of a local quadratic model under orthogonal-complement and rank constraints. Matrix-free randomized SVD recovers them without materializing the residual gradient, while controlled write-back preserves accumulated updates and stabilizes training within the allocated rank. DT-LoRA adds no trainable parameters or inference cost over LoRA. Experiments across computer vision and natural language processing tasks demonstrate consistent improvements over LoRA and its variants. Moreover, despite being a general-purpose adaptation method, DT-LoRA outperforms task-specific approaches to open-vocabulary domain generalization for semantic segmentation, achieving state-of-the-art results. Code is available at https://github.com/Snipine/DT-LoRA.

Yangming Zhang, Ling Yi, Datong Li +4

Chongqing University of Posts and Telecommunications, China · Anhui University of Science and Technology · Northeastern University, China

Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Oct 14, 2026

ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning

Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning by learning low-rank weight updates. Increasing rank expands adaptation capacity, but does not guarantee that all learned directions improve downstream performance. In a diagnostic experiment with rank-16 LoRA, we show that retaining only the top singular components achieves the best performance, whereas adding further components degrades it. Motivated by this, we propose Residual-Gradient Adaptation (ReGRA), a two-stage method for making more effective use of a fixed rank budget. After the first stage, ReGRA estimates task gradients on a calibration set and removes their projections onto the two-sided subspace induced by the first-stage update. It then aggregates the residual-gradient directions to initialize a second-stage adapter without changing the first-stage update. An orthogonality regularizer encourages the two updates to capture complementary directions. Both updates can be merged into the pretrained weights, incurring no additional inference overhead. Across diverse benchmark sets, ReGRA achieves stronger performance than alternatives with the same total rank and even matches LoRA adapters with twice the rank. Ablation studies and further analysis support the contribution of its core design choices. Code is available at https://github.com/Fanyao-Li/ReGRA.

Fanyao Li, Yuxi Liu, Xiaohao Liu +3

Communication University of China · National University of Singapore · Xidian University

Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

May 29, 2026

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show--both theoretically and empirically--that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.

Valérie Castin, Kimia Nadjahi, Pierre Ablin +1

1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.

LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

LLM Fine-Tuning

Explore similar work

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation

ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence