Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization

Zebang Xie; Chuanyang Zheng; Yik-Chung Wu; Yihang Gao

cs.LGSep 24, 2026

Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization

Authors: Zebang Xie, Chuanyang Zheng, Yik-Chung Wu, Yihang Gao

Organizations: National University of Singapore, Singapore · Huawei, Hong Kong · The University of Hong Kong, Hong Kong · Hong Kong Baptist University, Hong Kong

Abstract

Low-rank adaptation (LoRA) has become a popular parameter-efficient fine-tuning method for large language models. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency. Existing adaptive-rank methods typically allocate ranks according to manually designed importance scores, which are not directly derived from an optimization objective. In this work, we propose $\ell_p$ -LoRA, a principled rank-allocation method based on $\ell_p$ regularization with $0<p<1$ , which is a classical sparsity-inducing technique in signal processing and statistics. Specifically, we regularize the energy of each rank-one LoRA component, encouraging redundant components to vanish while preserving important ones. We derive the corresponding proximal subproblem and reduce the matrix optimization to a two-dimensional problem, leading to an implicit thresholding criterion for identifying redundant components. Experiments on natural language understanding and question-answering tasks demonstrate that the proposed method achieves competitive performance with existing LoRA baselines.

Figures & tables

Method	CoLA	MRPC	RTE	STS-B
LoRA	$69.10_{\pm 0.28}$	$90.10_{\pm 0.51}$	$87.40_{\pm 0.85}$	$91.82_{\pm 0.12}$
AdaLoRA	$\textbf{70.57}_{\pm 0.56}$	$90.24_{\pm 0.53}$	$87.81_{\pm 0.93}$	$\textbf{91.94}_{\pm 0.16}$
IGU-LoRA	$69.98_{\pm 0.99}$	$89.37_{\pm 0.41}$	$88.22_{\pm 0.65}$	$91.78_{\pm 0.14}$
$\ell_{p}$ -LoRA	$69.61_{\pm 0.95}$	$\textbf{91.32}_{\pm 0.66}$	$\textbf{88.45}_{\pm 0.92}$	$91.36_{\pm 0.10}$

Table 1: Results on NLU tasks from the GLUE benchmark using DeBERTaV3-base. Higher values indicate better performance. Results are reported as the mean over five random seeds, with the empirical standard deviation shown in the subscript. The best result is shown in bold.

Method	BoolQ	ARC-Easy	OpenBookQA	CSQA
LoRA	$89.25_{\pm 0.12}$	$92.39_{\pm 0.83}$	${91.72}_{\pm 0.83}$	${86.37}_{\pm 0.53}$
AdaLoRA	${89.42}_{\pm 0.22}$	$\mathbf{92.95}_{\pm 0.23}$	$\mathbf{92.20}_{\pm 0.47}$	$85.90_{\pm 0.13}$
IGU-LoRA	$89.31_{\pm 0.19}$	${92.74}_{\pm 0.16}$	$90.52_{\pm 0.54}$	$85.49_{\pm 0.51}$
$\ell_{p}$ -LoRA	$\mathbf{90.20}_{\pm 0.38}$	$92.17_{\pm 0.49}$	$91.12_{\pm 1.04}$	$\mathbf{86.39}_{\pm 0.50}$

Table 2: Results on question answering tasks using Qwen2.5-7B. We report accuracy as the mean over five random seeds, with the empirical standard deviation shown in the subscript. The best result is shown in bold.

Explore similar work

TLoRA: Task-aware Low Rank Adaptation of Large Language Models
Apr 20, 2026Weicheng Lin, Yi Zhang, Jiawei Dang +1Subspace Data-Driven
College of Computer Science and Software Engineering, Shenzhen University, China
Post-Optimization Adaptive Rank Allocation for LoRA
Apr 30, 2026Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. SrijithLow-Rank Adaptation Framework Large Language Model Compression
1Indian Institute of Technology Hyderabad · 2Deakin University
RSLoRA: Training-free Rank Allocation for LoRA via Representational Sensitivity Probing
Jul 5, 2026Jiaqi Liu, Haidong Kang, Qihui Zhao +1Training-Side Stage-Aware Low-Rank Adaptation Parameter-Efficient Adaptation
School of Computer Science and Technology Dalian University of Technology · Hebei Key Laboratory of Marine Perception Network and Data Processing Northeastern University · School of Computer and Communication Engineering Northeastern University +1

Apr 20, 2026, cs.CLJ/K move · Enter open · S save

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

Weicheng Lin, Yi Zhang, Jiawei Dang, Liang-Jie Zhang

College of Computer Science and Software Engineering, Shenzhen University, China

Subspace Data-Driven

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well as initialization. Existing LoRA variants typically address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency. In this work, we present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimizes initialization and resource allocation at the outset of training. TLoRA introduces a data-driven initialization strategy that aligns the LoRA

A

matrix with task-relevant subspaces by performing singular value decomposition on the product of pre-trained weights and input activation covariance. After this, the

A

matrix is frozen, and only the

B

matrix is trained. Furthermore, TLoRA employs a sensitivity-based importance metric to adaptively allocate ranks and scaling factors across layers under a fixed parameter budget. We conduct extensive experiments that demonstrate TLoRA consistently performs excellently across various tasks, including natural language understanding, commonsense reasoning, math reasoning, code generation, and chat generation, while significantly reducing the number of trainable parameters.

Abstract

Explore similar work

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

Post-Optimization Adaptive Rank Allocation for LoRA

RSLoRA: Training-free Rank Allocation for LoRA via Representational Sensitivity Probing