cs.CLFeb 24, 2026

ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition

Authors: Xindian Ma, Rundong Kong, Peng Zhang, Ruoxiang Huang, Yongyu Jiang

Organizations: Tianjin University, Tianjin, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LoRA has become a universal Parameter-Efficient Fine-Tuning (PEFT) technique that equips Large Language Models (LLMs) to adapt quickly to new tasks. However, when these models are scaled up, even the latest LoRA variants still introduce considerable overhead in trainable parameters. Conversely, aggressively lowering the rank to curb this overhead markedly degrades performance in complex multi-task settings. We propose ID-LoRA, a novel PEFT framework that breaks the trade-off. Its core innovation lies in extracting and reusing clustered parameter groups from the pretrained weight matrix. These groups are then used to form multiple low-rank components, all of which share only a single initialized trainable low-rank matrix. This approach cuts the number of trainable parameters while keeping the model's capacity intact. We evaluate ID-LoRA on five diverse benchmarks: Mathematical Reasoning, Code Generation, MMLU, CommonsenseQA, and Safety Alignment. ID-LoRA outperforms both full fine-tuning and existing PEFT baselines (e.g., LoRA, DoRA, HydraLoRA) while using up to 46% fewer trainable parameters than the standard LoRA. In multi-task scenarios, it surpasses LoRA and its recent variants (e.g., DoRA and HydraLoRA) on both Code and MMLU tasks, yet requires only 54% of the trainable parameters demanded by the conventional LoRA.

Figures & tables

Figure 1: Architectural Comparison and Parameter Efficiency: LoRA versus ID-LoRA. (a) LoRA requires training two low-rank matrices: randomly initialized A∈Rr×dA\in\mathbb{R}^{r\times d} and zero-initialized B∈Rd×rB\in\mathbb{R}^{d\times r} . (b) ID-LoRA employs the parameter clustering and rank boosting to generate multiple low-rank components while sharing a single B, thereby reducing trainable parameters. (c) Trainable parameters: ID-LoRA achieves ∼5×\sim 5\times compression versus LoRA at rank 3232 (right) and maintains superior scalability across model sizes (left).
Figure 2: A diagram of the ID-LoRA architecture.
Method# Params (%)GSM8KHumanEvalHEx-PHI
Pass@1Pass@5Pass@10
LLaMA-3-8B
FFT100%58.830.539.341.794.8
LoRA1.03%62.834.746.450.891.6
DoRA1.05%63.533.144.048.693.6
HydraLoRA1.10%62.833.444.749.394.2
Table 1: Performance comparison of different adaptation methods on single tasks, including GSM8K (Math), HumanEval (Code), and HEx-PHI (Safety) benchmarks using LLaMA-3 and Mistral. Bold indicates the best-performing method. †\dagger indicates statistically significant improvement, i.e., Wilcoxon signed-rank test ( p<0.05p<0.05 ) against LoRA.
Method# Params (%)MATHMMLUCQACODE
Pass@1Pass@5Pass@10
LLaMA-3-8B
FFT100%52.237.538.731.240.143.9
LoRA1.03%60.848.146.939.048.851.9
DoRA1.05%59.441.535.038.350.255.7
HydraLoRA1.10%60.147.544.133.243.947.8
Table 2: Performance comparison of different adaptation methods on multi tasks, including math (GSM8K), code (HumanEval), world knowledge (MMLU), and common sense (CommonsenseQA) benchmarks using LLaMA-3 and Mistral. Bold indicates the best-performing method. “CQA” is the abbreviation for for CommonsenseQA. †\dagger indicates statistically significant improvement, i.e., Wilcoxon signed-rank test ( p<0.05p<0.05 ) against LoRA.
Figure 3: The inference time and extra memory overhead of different adaptation methods under the same hyperparameter settings as the multi-task experiments on LLaMA-3-8B, tested on an A800 GPU.
Method# Params (%)MATHMMLUCQACODE
Pass@1Pass@5Pass@10
LoRA1.49%38.643.858.228.834.737.1
ID-LoRA (PMRC+RB)0.77%38.148.565.629.435.938.2
ID-LoRA (w/o RB)0.83%38.845.264.428.735.338.6
ID-LoRA (SS+RB)0.77%35.148.265.329.335.337.6
ID-LoRA (RS+RB)0.77%36.648.565.128.934.537.4
Table 3: Ablation on LLaMA-3.2-3B across a multi-task dataset comparing different strategies. ’w/o RB’ idenotes removing the RB algorithm from ID-LoRA. ‘SS’ abbreviates Serial Selection, where pivots are assigned to groups in sequential order. ’RS‘ abbreviates Random Selection, where pivots are assigned to groups at random.
Figure 4: Performance comparison between ID-LoRA and vanilla LoRA on four representative benchmarks (GSM8K, HumanEval, MMLU, and CommonsenseQA) using LLaMA3-8B as the backbone. Both methods vary the rank while keeping the number of trainable parameters approximately equal, and results are reported under few-shot or zero-shot settings. Detailed results are provided in Appendix B.2
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

HyperparameterGSM8KCodeAlpacaSaferpaca
 Llama-3MistralLlama-3MistralLlama-3Mistral
Rank/ α\alpha128 / 256128 / 256128 / 256128 / 256128 / 256128 / 256
Learning Rate5e-53e-55e-55e-51e-41e-4
Dropout0.050.050.050.050.050.05
OptimizerAdamWAdamWAdamWAdamWAdamWAdamW
Batch Size323232323232
Appendix
Table 4: Hyperparameter settings for ID-LoRA on GSM8K, CodeAlpaca, and Saferpaca datasets. The ID-LoRA parameters are placed at the q, k, v, o, gate, up, and down positions.
MethodID-LoRAID-LoRA
Base ModelLlama-3Mistral
Rank rr128128
α\alpha256256
Learning Rate5e-55e-5
Dropout0.05
OptimizerAdamW
Appendix
Table 5: Hyperparameter settings for ID-LoRA on the multi-task dataset
  LoRADoRAHydraLoRAMoELoRAID-LoRA
 Rank# parameters# parameters# parameters# parameters# parameters
llama3-8B821.0M22.3M46.7M88.9M7.7M
1641.9M43.3M89.7M172.8M10.4M
3283.9M85.3M175.7M340.5M16.0M
 64167.8M169.1M347.7M676.1M27.0M
 128335.5M336.9M691.6M1.3B49.0M
Appendix
Table 6: Number of trainable parameters for different methods on LLaMA-3-8B
Method# Params (%)MATHMMLUCQACODE
Pass@1Pass@5Pass@10
LoRA0.13 (r=4)57.347.139.415.924.629.1
 0.26 (r=8)57.947.247.320.730.334.5
 0.52 (r=16)59.447.537.827.137.342.2
 1.03 (r=32)60.848.146.939.048.851.9
ID-LoRA0.19 (r=32)57.745.244.624.532.736.2
Appendix
Table 7: Performance comparison of LoRA and ID-LoRA on the LLaMA-3-8B backbone under multi-task instruction tuning. All methods are trained on the same mixture (Alpaca + GSM8k + 50% CodeAlpaca) and evaluated on MATH, MMLU, CommonsenseQA (CQA), and HumanEval (Pass@1/5/10). These numbers correspond to the experiment shown in Figure 4 of the paper.
kkMATHMMLUCOMCODE
Pass@1Pass@5Pass@10
238.348.764.429.034.737.1
438.148.565.629.435.938.2
639.646.965.130.636.538.4
838.145.564.730.236.038.2
Appendix
Table 8: Ablation on the number of pre-trained parameter clusters k under the multi-task setting. Results are obtained with the LLaMA-3.2-3B backbone. The table reports overall multi-task accuracy for each choice of kk .
More in this topic

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 753 papers

Explore similar work

CardsList
Apr 20, 2026

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well as initialization. Existing LoRA variants typically address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency. In this work, we present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimizes initialization and resource allocation at the outset of training. TLoRA introduces a data-driven initialization strategy that aligns the LoRA AA matrix with task-relevant subspaces by performing singular value decomposition on the product of pre-trained weights and input activation covariance. After this, the AA matrix is frozen, and only the BB matrix is trained. Furthermore, TLoRA employs a sensitivity-based importance metric to adaptively allocate ranks and scaling factors across layers under a fixed parameter budget. We conduct extensive experiments that demonstrate TLoRA consistently performs excellently across various tasks, including natural language understanding, commonsense reasoning, math reasoning, code generation, and chat generation, while significantly reducing the number of trainable parameters.

Weicheng Lin, Yi Zhang, Jiawei Dang +1

College of Computer Science and Software Engineering, Shenzhen University, China

LLM Fine-TuningLow-Rank AdaptationAdaptive Rank AllocationParameter-Efficient Fine-Tuning
AskPDF
Apr 15, 2026

TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models

Fine-tuning large language models (LLMs) aims to adapt pre-trained models to specific tasks using relatively small and domain-specific datasets. Among Parameter-Efficient Fine-Tuning (PEFT) methods, Low-Rank Adaptation (LoRA) stands out by matching the performance of full fine-tuning while avoiding additional inference latency. In this paper, we propose a novel PEFT method that incorporates the TLoRA+ optimizer into the weight matrices of pre-trained models. The proposed approach not only preserves the efficiency of low-rank adaptation but also further enhances performance without significantly increasing computational cost. We conduct experiments on the GLUE benchmark across diverse model architectures. Numerical experiments consistently demonstrate the effectiveness and robustness of our proposed method.

Yarui Cao, Kai Liu

Department of Computer Science, Clemson University, Clemson, SC 29634, USA

LLM Fine-TuningParameter-Efficient Fine-TuningLow-Rank Adaptation
AskPDF
Jun 3, 2026

Parameter-Efficient Fine-Tuning with Learnable Rank

Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method that restricts weight updates to low-rank adapters, introducing a fixed low-rank inductive bias by optimizing in a low-dimensional subspace. In this work, we question whether a fixed-rank constraint is the most effective inductive bias for parameter-efficient fine-tuning. We introduce Learnable Rank LoRA (LR-LoRA), a PEFT method in which the adapter rank is learned during the training process. Instead of prescribing a uniform rank for all adapter layers, LR-LoRA allows the optimizer to determine the appropriate rank for each layer. Using this approach, we find substantial layer-wise variation in the learned ranks, with the attention and MLP layers in the transformer models exhibiting systematically different rank preferences. Across a range of language understanding and commonsense reasoning benchmarks, LR-LoRA achieves state-of-the-art performance in most settings and consistently outperforms strong PEFT baselines, demonstrating that a learnable rank provides a more flexible and effective inductive bias than fixed-rank adaptations.

Arpit Garg, Simon Lucey, Hemanth Saratchandran

Australian Institute for Machine Learning Adelaide University

Nonlinear LoRALow-Rank AdaptationParameter-Efficient Fine-Tuning
AskPDF