TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models

Yarui Cao; Kai Liu

cs.CLApr 15, 2026

TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models

Authors: Yarui Cao, Kai Liu

Organizations: Department of Computer Science, Clemson University, Clemson, SC 29634, USA

Abstract

Fine-tuning large language models (LLMs) aims to adapt pre-trained models to specific tasks using relatively small and domain-specific datasets. Among Parameter-Efficient Fine-Tuning (PEFT) methods, Low-Rank Adaptation (LoRA) stands out by matching the performance of full fine-tuning while avoiding additional inference latency. In this paper, we propose a novel PEFT method that incorporates the TLoRA+ optimizer into the weight matrices of pre-trained models. The proposed approach not only preserves the efficiency of low-rank adaptation but also further enhances performance without significantly increasing computational cost. We conduct experiments on the GLUE benchmark across diverse model architectures. Numerical experiments consistently demonstrate the effectiveness and robustness of our proposed method.

Figures & tables

Figure 1: Structural comparison of adaptation strategies (left to right): full fine-tuning, LoRA and TLoRA.

Figure 2: Three configurations of the tri-matrices adapter. Red indicates trainable matrices, while blue denotes frozen (non-trainable) matrices.

ID	Time(s)	Dataset	Warmup	WD	Tr Acc	Tr Loss	Val Acc	Val Loss	Val MCC
1	542	RTE	0.15	0.05	0.9590	0.1167	0.8375	0.6052	0.6739
2	540	RTE	0.10	0.10	0.9502	0.1291	0.8520	0.5770	0.7132
3	542	RTE	0.10	0.05	0.9555	0.1302	0.8556	0.5545	0.7102
4	540	RTE	0.15	0.10	0.9549	0.1231	0.8592	0.5451	0.7174
5	395	MRPC	0.10	0.05	0.9447	0.1488	0.8946	0.3642	0.7533
6	394	MRPC	0.15	0.10	0.9496	0.1358	0.8971	0.3710	0.7581

Table 1: Hyperparameter search results for RoBERTa-large-MNLI (learning Rate:

1\times 10^{-4}

Figure 3: Average training time per epoch (in seconds) across five GLUE datasets (MRPC, RTE, CoLA, SST-2 and QNLI) for four base models (RoBERTa-large-MNLI, RoBERTa-base, OPT-125M and DeBERTa-base). The grid organizes models by columns and rank configurations (8, 16, 32 and 64) by rows. Within each subplot, grouped bars compare the computational time of our proposed method, LoRA and TLoRA.

Panel of Figure 4 — Figure 4: Parameter efficiency across four transformer architectures. Each subplot shows the percentage of trainable parameters relative to the total model parameters for ranks 8, 16, 32 and 64, along with the distribution of trainable components for our proposed method, LoRA and TLoRA.

Panel of Figure 5 — Figure 5: Comparison of trends for CoLA and QNLI datasets under the OPT-125M model.

Panel of Figure 6 — Figure 6: Validation accuracy and MCC on two datasets at rank 8 across different models.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Validation loss under different learning rates and backbone models. A learning rate of $2\times 10^{-4}$ leads to overfitting, while $5\times 10^{-5}$ results in slow and incomplete convergence. In contrast, $1\times 10^{-4}$ achieves stable and near-converged performance across both datasets (RTE and MRPC) and backbone models under all other hyperparameter combinations.

ID	Time(s)	Dataset	Warmup	WD	Tr Acc	Tr Loss	Val Acc	Val Loss	Val MCC
1	182	RTE	0.10	0.10	0.7786	0.5109	0.6706	0.6441	0.3378
2	183	RTE	0.15	0.05	0.7729	0.4797	0.6679	0.6504	0.3393
3	182	RTE	0.10	0.05	0.7700	0.4759	0.6751	0.6073	0.3498
4	183	RTE	0.15	0.10	0.7810	0.4561	0.6931	0.6180	0.3849
5	137	MRPC	0.15	0.05	0.8451	0.3494	0.7892	0.5155	0.4811
6	137	MRPC	0.10	0.05	0.8576	0.3320	0.7966	0.4883	0.5006

Appendix

Table 2: Hyperparameter search results for OPT-125M (learning Rate:

1\times 10^{-4}

ID	Method	Model	Tr Acc	Tr Loss	Val Acc	Val Loss	Val MCC
1	LoRA	DeBERTa-base	0.9918	0.0404	0.9044	0.3805	0.7747
2	Ours	DeBERTa-base	0.7045	0.4873	0.6887	0.4780	0.1032
3	TLoRA	DeBERTa-base	0.6601	0.6762	0.6838	0.6776	0.0000
4	LoRA	OPT-125M	0.9957	0.0125	0.8064	1.0821	0.5347
5	Ours	OPT-125M	0.8062	0.4326	0.8039	0.4529	0.5211
6	TLoRA	OPT-125M	0.6785	0.6225	0.6765	0.6100	0.0973

Appendix

Table 3: MRPC rank 8 results.

ID	Method	Model	Tr Acc	Tr Loss	Val Acc	Val Loss	Val MCC
1	LoRA	DeBERTa-base	0.9818	0.0619	0.9020	0.3852	0.7693
2	Ours	DeBERTa-base	0.6832	0.4282	0.6838	0.4160	0.1002
3	TLoRA	DeBERTa-base	0.6712	0.6379	0.6814	0.6323	-0.0337
4	LoRA	OPT-125M	0.9976	0.0093	0.8137	1.0633	0.5489
5	Ours	OPT-125M	0.8684	0.3115	0.8094	0.4804	0.5372
6	TLoRA	OPT-125M	0.6768	0.6188	0.6961	0.6144	0.1656

Appendix

Table 4: MRPC rank 16 results.

ID	Method	Model	Tr Acc	Tr Loss	Val Acc	Val Loss	Val MCC
1	LoRA	DeBERTa-base	0.9628	0.1237	0.8922	0.3426	0.7465
2	Ours	DeBERTa-base	0.8851	0.2779	0.8750	0.3072	0.7156
3	TLoRA	DeBERTa-base	0.6783	0.6099	0.6863	0.5850	0.0729
4	LoRA	OPT-125M	0.9978	0.0083	0.8064	0.9143	0.5300
5	Ours	OPT-125M	0.8959	0.2559	0.8113	0.5037	0.5405
6	TLoRA	OPT-125M	0.7205	0.5492	0.7083	0.5371	0.2281

Appendix

Table 5: MRPC rank 32 results.

ID	Method	Model	Tr Acc	Tr Loss	Val Acc	Val Loss	Val MCC
1	LoRA	DeBERTa-base	0.9337	0.1948	0.8995	0.3200	0.7642
2	Ours	DeBERTa-base	0.9188	0.2099	0.8824	0.3077	0.7314
3	TLoRA	DeBERTa-base	0.6767	0.5758	0.6838	0.5611	0.0000
4	LoRA	OPT-125M	0.9905	0.0267	0.8015	0.9194	0.5253
5	Ours	OPT-125M	0.9670	0.1004	0.8098	0.7045	0.5373
6	TLoRA	OPT-125M	0.7615	0.4990	0.7574	0.5119	0.3890

Appendix

Table 6: MRPC rank 64 results.

Panel of Figure 8 — Figure 8: Validation accuracy on QNLI at rank 8 across different backbone models. Each subplot compares three methods.

Panel of Figure 9 — Figure 9: Validation accuracy on QNLI at rank 16 across different backbone models. Each subplot compares three methods.

Panel of Figure 10 — Figure 10: Validation accuracy on QNLI at rank 32 across different backbone models. Each subplot compares three methods.

Panel of Figure 11 — Figure 11: Validation accuracy on QNLI at rank 64 across different backbone models. Each subplot compares three methods.

Panel of Figure 12 — Figure 12: Validation accuracy and MCC on CoLA at rank 8 across different models.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 746 papers

TLoRA: Task-aware Low Rank Adaptation of Large Language Models
Apr 20, 2026Weicheng Lin, Yi Zhang, Jiawei Dang +1LLM Fine-Tuning Low-Rank Adaptation
College of Computer Science and Software Engineering, Shenzhen University, China
Parameter-Efficient Fine-Tuning with Learnable Rank
Jun 3, 2026Arpit Garg, Simon Lucey, Hemanth SaratchandranNonlinear LoRA Low-Rank Adaptation
Australian Institute for Machine Learning Adelaide University
Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
May 15, 2026Jing Gao, Zhong-Yi Lu, Pan Zhang +1LLM Fine-Tuning Low-Rank Adaptation
School of Fundamental Physics and Mathematical Sciences, Hangzhou Institute for Advanced Study, UCAS, Hangzhou 310024, China · School of Physical Sciences, University of Chinese Academy of Sciences, No. 19A Yuquan Road, Beijing 100049, China · CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China +1

Apr 20, 2026J/K move · Enter open · S save

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

Weicheng Lin, Yi Zhang, Jiawei Dang, Liang-Jie Zhang

College of Computer Science and Software Engineering, Shenzhen University, China

LLM Fine-Tuning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well as initialization. Existing LoRA variants typically address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency. In this work, we present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimizes initialization and resource allocation at the outset of training. TLoRA introduces a data-driven initialization strategy that aligns the LoRA

A

matrix with task-relevant subspaces by performing singular value decomposition on the product of pre-trained weights and input activation covariance. After this, the

A

matrix is frozen, and only the

B

matrix is trained. Furthermore, TLoRA employs a sensitivity-based importance metric to adaptively allocate ranks and scaling factors across layers under a fixed parameter budget. We conduct extensive experiments that demonstrate TLoRA consistently performs excellently across various tasks, including natural language understanding, commonsense reasoning, math reasoning, code generation, and chat generation, while significantly reducing the number of trainable parameters.

Abstract

LLM Fine-Tuning

Explore similar work

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

Parameter-Efficient Fine-Tuning with Learnable Rank

Strategic Over-Parameterization for Generalizable Low-Rank Adaptation