ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning

Ashfak Yeafi; Mehedi Hasan; Md Khairul Islam

cs.CLSep 8, 2026

ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning

Authors: Ashfak Yeafi, Mehedi Hasan, Md Khairul Islam

Organizations: Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology, Khulna, Bangladesh · Department of Computer Science and Engineering, Brac University, Dhaka, Bangladesh · Department of Mathematics and Computer Science, Hobart and William Smith Colleges, Geneva, NY, USA

Abstract

Parameter-efficient fine-tuning (PEFT) adapts large language models (LLMs) to downstream tasks while updating only a small fraction of their pretrained parameters. Low-Rank Adaptation (LoRA) uses two trainable low-rank matrices, while Weight-Decomposed Low-Rank Adaptation (DoRA) further separates weight magnitude and direction but retains the dense LoRA-style factorization in its directional branch. We propose ChainDoRA, a weight-decomposed adaptation framework that constructs the directional low-rank factors from a connected Tensor-Train (TT) chain, where the adapter rank forms the boundary rank between input- and output-side TT contractions and an independent TT rank controls representation capacity and parameter cost. Under a controlled 15,119-example response-only adaptation setting with LLaMA-7B, ChainDoRA is evaluated against matched LoRA and DoRA baselines on seven commonsense reasoning benchmarks. ChainDoRA with TT rank 16 achieves a seven-task average accuracy of 72.30%, compared with 69.88% for LoRA and 69.39% for DoRA, while requiring only 5.35M trainable parameters versus 56.10M for LoRA and 56.98M for DoRA, corresponding to a 90.62% reduction relative to DoRA. Ablations over TT rank and adapter placement show controllable parameter-accuracy trade-offs, indicating that connected TT parameterization can substantially reduce the parameter cost of magnitude-direction adaptation while preserving, and in this setting improving, downstream reasoning performance.

Figures & tables

Figure 1: Overview of the proposed ChainDoRA framework. A connected Tensor-Train chain constructs the directional factors $A_{\mathrm{TT}}$ and $B_{\mathrm{TT}}$ , which are coupled through the adapter-rank boundary $r$ . Their product defines the low-rank directional update, while the row-wise magnitude vector $m$ is optimized independently. The pretrained weight $W_{0}$ remains frozen.

Method	Low-Rank Update	Tensor Representation	Magnitude–Direction	Joint $A/B$ Structure	Primary Structural Mechanism
LoRA [ 3 ]	✓	–	–	–	Dense trainable low-rank factors
AdaLoRA [ 9 ]	✓	–	–	–	Adaptive rank-budget allocation
DoRA [ 4 ]	✓	–	✓	–	Magnitude–direction decomposition
LoRETTA [ 7 ]	✓	TT	–	–	Tensorized adapters / reparameterization
TT-LoRA [ 8 ]	✓	TT	–	–	TT-parameterized adaptation
DoTA [ 12 ]	–	MPO	Weight-decomposed	–	Pretrained-weight MPO initialization

Table 1: Conceptual comparison of representative low-rank and tensorized parameter-efficient adaptation methods. “Joint

A/B

structure” indicates whether the two low-rank factors are generated or parameterized through a shared structured representation.

Figure 2: Connected Tensor-Train construction of ChainDoRA. The input-side and output-side TT contractions generate

A_{\mathrm{TT}}

and

B_{\mathrm{TT}}

, respectively. The adapter rank

r

forms the boundary bond between the two contractions, while the remaining internal TT bonds are controlled by rank

\rho

Module	$d_{\mathrm{in}}$	$d_{\mathrm{out}}$	Input Modes	Output Modes
$q_{\mathrm{proj}}$	4096	4096	$(16,16,16)$	$(16,16,16)$
$k_{\mathrm{proj}}$	4096	4096	$(16,16,16)$	$(16,16,16)$
$v_{\mathrm{proj}}$	4096	4096	$(16,16,16)$	$(16,16,16)$
$up_{\mathrm{proj}}$	4096	11008	$(16,16,16)$	$(16,16,43)$
$down_{\mathrm{proj}}$	11008	4096	$(16,16,43)$	$(16,16,16)$

Table 2: Tensorization used for ChainDoRA on LLaMA-7B.

Configuration	Value
Backbone	LLaMA-7B
Adaptation pool	15,119 examples
Optimization examples	14,999
Validation examples	120
Training epochs	3
Optimizer	AdamW

Table 3: Training configuration used for the primary comparison.

Benchmark	Test Examples	Primary Reasoning Type
BoolQ	3,270	Yes/no reasoning
PIQA	1,838	Physical commonsense
SocialIQA	1,954	Social commonsense
WinoGrande	1,267	Coreference / commonsense
ARC-Easy	2,376	Science reasoning
ARC-Challenge	1,172	Challenging science reasoning

Table 4: Commonsense reasoning benchmarks used for evaluation.

Configuration	$\rho$	Target Set	Trainable Params.
TT16-Full	16	QKV+MLP	5,346,816
TT8-Full	8	QKV+MLP	2,784,000
TT4-Full	4	QKV+MLP	1,748,352
TT16-QV	16	QV	1,867,776
TT16-QKV	16	QKV	2,801,664
TT16-MLP	16	MLP	2,545,152

Table 5: ChainDoRA configurations used in the ablation studies.

Method	Params.	BoolQ	PIQA	SocialIQA	WinoGrande	ARC-E	ARC-C	OBQA	Avg.
LoRA	56.10M	64.83	72.09	72.47	65.82	78.03	62.88	73.00	69.88
DoRA	56.98M	65.60	74.54	70.21	65.51	76.60	60.07	73.20	69.39
ChainDoRA	5.35M	66.24	77.64	71.80	66.30	82.32	67.58	74.20	72.30

Table 6: Commonsense reasoning performance of LoRA, DoRA, and ChainDoRA under the controlled 15,119-example adaptation setting. Accuracy is reported in percent. The best accuracy in each benchmark column is shown in bold.

Method	Trainable Params.	Compression	Reduction
LoRA	56,098,816	$10.49\times$	90.47%
DoRA	56,975,360	$10.66\times$	90.62%
ChainDoRA	5,346,816	$1.00\times$	–

Table 7: Trainable-parameter efficiency of the primary methods. The compression factor indicates how many times larger the corresponding baseline adapter is than ChainDoRA.

Figure 3: Parameter–accuracy trade-off among LoRA, DoRA, and representative ChainDoRA configurations.

$\boldsymbol{\rho}$	Params.	BoolQ	PIQA	SocialIQA	WinoGrande	ARC-E	ARC-C	OBQA	Avg.
4	1.748M	57.16	75.63	66.33	59.91	77.90	58.62	66.80	66.05
8	2.784M	64.62	76.61	67.96	63.85	80.98	62.29	71.00	69.61
16	5.347M	66.24	77.64	71.80	66.30	82.32	67.58	74.20	72.30

Table 8: Effect of the internal TT rank on ChainDoRA under full QKV+MLP adaptation with a fixed adapter rank of

r=32

. Accuracy is reported in percent. The best accuracy in each benchmark column is shown in bold.

Figure 4: Effect of internal TT rank

\rho

on the seven-task average accuracy and ChainDoRA parameter count. Increasing

\rho

improves predictive performance while increasing the size of the TT representation.

Target	Params. (M)	Avg. (%)	$\Delta$ (pp)
QV	1.868	70.43	$-1.87$
QKV	2.802	68.60	$-3.70$
MLP	2.545	70.93	$-1.37$
QV+MLP	4.413	70.93	$-1.37$
QKV+MLP	5.347	72.30	–

Table 9: Effect of ChainDoRA adapter placement.

Figure 5: Per-benchmark performance of ChainDoRA under different adapter-placement strategies.

Configuration	Params. (M)	Avg. (%)
TT16-Full	5.347	72.30
TT16-MLP	2.545	70.93
TT8-Full	2.784	69.61
TT8-MLP	1.383	68.97
TT4-Full	1.748	66.05

Table 10: Representative ChainDoRA parameter–accuracy trade-offs.

Model	Params. (M)		Red.	Comp.
	DoRA	CD-TT16	(%)	( $\times$ )
Qwen2.5-3B	40.67	4.97	87.77	8.18
Mistral-7B	57.41	5.46	90.48	10.51
Phi-3-Medium	100.25	6.09	93.92	16.45
OPT-30B	178.91	12.42	93.06	14.40
Falcon-40B	193.72	12.12	93.74	15.98

Table 11: Cross-architecture trainable-parameter scaling.

Figure 6: Trainable-parameter scaling of full-target DoRA and ChainDoRA-TT16 across representative language-model architectures.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 755 papers

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
Sep 29, 2026Yihao Ouyang, Shiwei Li, Haozhao Wang +5Low-Rank Adaptation Parameter-Efficient Fine-Tuning
Huazhong University of Science and Technology, Wuhan, China · Hebei University of Technology, Tianjin, China
Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
May 15, 2026Jing Gao, Zhong-Yi Lu, Pan Zhang +1LLM Fine-Tuning Low-Rank Adaptation
School of Fundamental Physics and Mathematical Sciences, Hangzhou Institute for Advanced Study, UCAS, Hangzhou 310024, China · School of Physical Sciences, University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China · CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China +1
TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models
Apr 15, 2026Yarui Cao, Kai LiuLLM Fine-Tuning Parameter-Efficient Fine-Tuning
Department of Computer Science, Clemson University, Clemson, SC 29634, USA

Sep 29, 2026J/K move · Enter open · S save

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

Yihao Ouyang, Shiwei Li, Haozhao Wang, Xiandi Luo+4

Huazhong University of Science and Technology, Wuhan, China · Hebei University of Technology, Tianjin, China

Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-tuning (FFT). Many LoRA variants improve the initialization or optimization of low-rank factors. At each training step, however, their first-order weight-space directions are constrained by the current parameterization. We characterize the corresponding LoRA-accessible gradient space and show that it coincides with the tangent space induced by the current LoRA parameterization. This characterization yields an orthogonal decomposition of the full weight gradient at the current model parameters. We term the component orthogonal to this space the normal gradient. Based on this decomposition, we propose GDLoRA (Gradient-Decomposed Low-Rank Adaptation). GDLoRA reconstructs the full weight gradient from forward activations and backward signals, extracts its normal component, and directly updates the base weights with this component, while retaining standard AdamW optimization for the LoRA factors. GDLoRA incorporates complementary normal gradients without increasing standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations. Experiments on natural language understanding, mathematical reasoning, commonsense reasoning, and image classification show that GDLoRA consistently improves over LoRA and narrows the performance gap to FFT. The code is available at https://anonymous.4open.science/r/GDLoRA.

Abstract

LLM Fine-Tuning

Explore similar work

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

Strategic Over-Parameterization for Generalizable Low-Rank Adaptation

TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models