ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition

Xindian Ma; Rundong Kong; Peng Zhang; Ruoxiang Huang; Yongyu Jiang

cs.CLFeb 24, 2026

ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition

Authors: Xindian Ma, Rundong Kong, Peng Zhang, Ruoxiang Huang, Yongyu Jiang

Organizations: Tianjin University, Tianjin, China

Abstract

LoRA has become a universal Parameter-Efficient Fine-Tuning (PEFT) technique that equips Large Language Models (LLMs) to adapt quickly to new tasks. However, when these models are scaled up, even the latest LoRA variants still introduce considerable overhead in trainable parameters. Conversely, aggressively lowering the rank to curb this overhead markedly degrades performance in complex multi-task settings. We propose ID-LoRA, a novel PEFT framework that breaks the trade-off. Its core innovation lies in extracting and reusing clustered parameter groups from the pretrained weight matrix. These groups are then used to form multiple low-rank components, all of which share only a single initialized trainable low-rank matrix. This approach cuts the number of trainable parameters while keeping the model's capacity intact. We evaluate ID-LoRA on five diverse benchmarks: Mathematical Reasoning, Code Generation, MMLU, CommonsenseQA, and Safety Alignment. ID-LoRA outperforms both full fine-tuning and existing PEFT baselines (e.g., LoRA, DoRA, HydraLoRA) while using up to 46% fewer trainable parameters than the standard LoRA. In multi-task scenarios, it surpasses LoRA and its recent variants (e.g., DoRA and HydraLoRA) on both Code and MMLU tasks, yet requires only 54% of the trainable parameters demanded by the conventional LoRA.

Figures & tables

Figure 1: Architectural Comparison and Parameter Efficiency: LoRA versus ID-LoRA. (a) LoRA requires training two low-rank matrices: randomly initialized $A\in\mathbb{R}^{r\times d}$ and zero-initialized $B\in\mathbb{R}^{d\times r}$ . (b) ID-LoRA employs the parameter clustering and rank boosting to generate multiple low-rank components while sharing a single B, thereby reducing trainable parameters. (c) Trainable parameters: ID-LoRA achieves $\sim 5\times$ compression versus LoRA at rank $32$ (right) and maintains superior scalability across model sizes (left).

Figure 2: A diagram of the ID-LoRA architecture.

Method	# Params (%)	GSM8K	HumanEval			HEx-PHI
Method	# Params (%)	GSM8K	Pass@1	Pass@5	Pass@10	HEx-PHI
LLaMA-3-8B
FFT	100%	58.8	30.5	39.3	41.7	94.8
LoRA	1.03%	62.8	34.7	46.4	50.8	91.6
DoRA	1.05%	63.5	33.1	44.0	48.6	93.6
HydraLoRA	1.10%	62.8	33.4	44.7	49.3	94.2

Table 1: Performance comparison of different adaptation methods on single tasks, including GSM8K (Math), HumanEval (Code), and HEx-PHI (Safety) benchmarks using LLaMA-3 and Mistral. Bold indicates the best-performing method.

\dagger

indicates statistically significant improvement, i.e., Wilcoxon signed-rank test (

p<0.05

) against LoRA.

Method	# Params (%)	MATH	MMLU	CQA	CODE
Method	# Params (%)	MATH	MMLU	CQA	Pass@1	Pass@5	Pass@10
LLaMA-3-8B
FFT	100%	52.2	37.5	38.7	31.2	40.1	43.9
LoRA	1.03%	60.8	48.1	46.9	39.0	48.8	51.9
DoRA	1.05%	59.4	41.5	35.0	38.3	50.2	55.7
HydraLoRA	1.10%	60.1	47.5	44.1	33.2	43.9	47.8

Table 2: Performance comparison of different adaptation methods on multi tasks, including math (GSM8K), code (HumanEval), world knowledge (MMLU), and common sense (CommonsenseQA) benchmarks using LLaMA-3 and Mistral. Bold indicates the best-performing method. “CQA” is the abbreviation for for CommonsenseQA.

\dagger

indicates statistically significant improvement, i.e., Wilcoxon signed-rank test (

p<0.05

) against LoRA.

Panel of Figure 3 — Figure 3: The inference time and extra memory overhead of different adaptation methods under the same hyperparameter settings as the multi-task experiments on LLaMA-3-8B, tested on an A800 GPU.

Method	# Params (%)	MATH	MMLU	CQA	CODE
Method	# Params (%)	MATH	MMLU	CQA	Pass@1	Pass@5	Pass@10
LoRA	1.49%	38.6	43.8	58.2	28.8	34.7	37.1
ID-LoRA (PMRC+RB)	0.77%	38.1	48.5	65.6	29.4	35.9	38.2
ID-LoRA (w/o RB)	0.83%	38.8	45.2	64.4	28.7	35.3	38.6
ID-LoRA (SS+RB)	0.77%	35.1	48.2	65.3	29.3	35.3	37.6
ID-LoRA (RS+RB)	0.77%	36.6	48.5	65.1	28.9	34.5	37.4

Table 3: Ablation on LLaMA-3.2-3B across a multi-task dataset comparing different strategies. ’w/o RB’ idenotes removing the RB algorithm from ID-LoRA. ‘SS’ abbreviates Serial Selection, where pivots are assigned to groups in sequential order. ’RS‘ abbreviates Random Selection, where pivots are assigned to groups at random.

Figure 4: Performance comparison between ID-LoRA and vanilla LoRA on four representative benchmarks (GSM8K, HumanEval, MMLU, and CommonsenseQA) using LLaMA3-8B as the backbone. Both methods vary the rank while keeping the number of trainable parameters approximately equal, and results are reported under few-shot or zero-shot settings. Detailed results are provided in Appendix B.2

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	GSM8K		CodeAlpaca		Saferpaca
	Llama-3	Mistral	Llama-3	Mistral	Llama-3	Mistral
Rank/ $\alpha$	128 / 256	128 / 256	128 / 256	128 / 256	128 / 256	128 / 256
Learning Rate	5e-5	3e-5	5e-5	5e-5	1e-4	1e-4
Dropout	0.05	0.05	0.05	0.05	0.05	0.05
Optimizer	AdamW	AdamW	AdamW	AdamW	AdamW	AdamW
Batch Size	32	32	32	32	32	32

Appendix

Table 4: Hyperparameter settings for ID-LoRA on GSM8K, CodeAlpaca, and Saferpaca datasets. The ID-LoRA parameters are placed at the q, k, v, o, gate, up, and down positions.

Method	ID-LoRA	ID-LoRA
Base Model	Llama-3	Mistral
Rank $r$	128	128
$\alpha$	256	256
Learning Rate	5e-5	5e-5
Dropout	0.05
Optimizer	AdamW

Appendix

Table 5: Hyperparameter settings for ID-LoRA on the multi-task dataset

		LoRA	DoRA	HydraLoRA	MoELoRA	ID-LoRA
	Rank	# parameters	# parameters	# parameters	# parameters	# parameters
llama3-8B	8	21.0M	22.3M	46.7M	88.9M	7.7M
	16	41.9M	43.3M	89.7M	172.8M	10.4M
	32	83.9M	85.3M	175.7M	340.5M	16.0M
	64	167.8M	169.1M	347.7M	676.1M	27.0M
	128	335.5M	336.9M	691.6M	1.3B	49.0M

Appendix

Table 6: Number of trainable parameters for different methods on LLaMA-3-8B

Method	# Params (%)	MATH	MMLU	CQA	CODE
Method	# Params (%)	MATH	MMLU	CQA	Pass@1	Pass@5	Pass@10
LoRA	0.13 (r=4)	57.3	47.1	39.4	15.9	24.6	29.1
	0.26 (r=8)	57.9	47.2	47.3	20.7	30.3	34.5
	0.52 (r=16)	59.4	47.5	37.8	27.1	37.3	42.2
	1.03 (r=32)	60.8	48.1	46.9	39.0	48.8	51.9
ID-LoRA	0.19 (r=32)	57.7	45.2	44.6	24.5	32.7	36.2

Appendix

Table 7: Performance comparison of LoRA and ID-LoRA on the LLaMA-3-8B backbone under multi-task instruction tuning. All methods are trained on the same mixture (Alpaca + GSM8k + 50% CodeAlpaca) and evaluated on MATH, MMLU, CommonsenseQA (CQA), and HumanEval (Pass@1/5/10). These numbers correspond to the experiment shown in Figure 4 of the paper.

$k$	MATH	MMLU	COM	CODE
$k$	MATH	MMLU	COM	Pass@1	Pass@5	Pass@10
2	38.3	48.7	64.4	29.0	34.7	37.1
4	38.1	48.5	65.6	29.4	35.9	38.2
6	39.6	46.9	65.1	30.6	36.5	38.4
8	38.1	45.5	64.7	30.2	36.0	38.2

Appendix

Table 8: Ablation on the number of pre-trained parameter clusters k under the multi-task setting. Results are obtained with the LLaMA-3.2-3B backbone. The table reports overall multi-task accuracy for each choice of

k

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 754 papers

TLoRA: Task-aware Low Rank Adaptation of Large Language Models
Apr 20, 2026Weicheng Lin, Yi Zhang, Jiawei Dang +1LLM Fine-Tuning Low-Rank Adaptation
College of Computer Science and Software Engineering, Shenzhen University, China
TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models
Apr 15, 2026Yarui Cao, Kai LiuLLM Fine-Tuning Parameter-Efficient Fine-Tuning
Department of Computer Science, Clemson University, Clemson, SC 29634, USA
Parameter-Efficient Fine-Tuning with Learnable Rank
Jun 3, 2026Arpit Garg, Simon Lucey, Hemanth SaratchandranNonlinear LoRA Low-Rank Adaptation
Australian Institute for Machine Learning Adelaide University

Apr 20, 2026J/K move · Enter open · S save

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

Weicheng Lin, Yi Zhang, Jiawei Dang, Liang-Jie Zhang

College of Computer Science and Software Engineering, Shenzhen University, China

LLM Fine-Tuning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well as initialization. Existing LoRA variants typically address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency. In this work, we present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimizes initialization and resource allocation at the outset of training. TLoRA introduces a data-driven initialization strategy that aligns the LoRA

A

matrix with task-relevant subspaces by performing singular value decomposition on the product of pre-trained weights and input activation covariance. After this, the

A

matrix is frozen, and only the

B

matrix is trained. Furthermore, TLoRA employs a sensitivity-based importance metric to adaptively allocate ranks and scaling factors across layers under a fixed parameter budget. We conduct extensive experiments that demonstrate TLoRA consistently performs excellently across various tasks, including natural language understanding, commonsense reasoning, math reasoning, code generation, and chat generation, while significantly reducing the number of trainable parameters.

Abstract

LLM Fine-Tuning

Explore similar work

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models

Parameter-Efficient Fine-Tuning with Learnable Rank