Low-rank adaptation (LoRA) is a widely used strategy for efficient fine-tuning of large language models (LLMs), but its strictly linear structure fundamentally limits expressive capacity. The bilinear formulation of weight updates captures only first-order dependencies between low-rank factors, restricting the modeling of nonlinear and higher-order parameter interactions. In this paper, we propose Polynomial Expansion Rank Adaptation (PERA), a novel method that introduces structured polynomial expansion directly into the low-rank factor space. By expanding each low-rank factor to synthesize high-order interaction terms before composition, PERA transforms the adaptation space into a polynomial manifold capable of modeling richer nonlinear coupling without increasing rank or inference cost. We provide theoretical analysis demonstrating that PERA offers enhanced expressive capacity and more effective feature utilization compare to existing linear adaptation approaches. Empirically, PERA consistently outperforms state-of-the-art methods across diverse benchmarks. Notably, our experiments show that incorporating high-order nonlinear components particularly square terms is crucial for enhancing expressive capacity and maintaining strong and robust performance under various rank settings. Our code is available at https://github.com/zhangwenhao6/PERA
Figures & tables
Figure 1: Comparison of Training Losses between First-Order and High-Order Terms.
Figure 2: The architecture comparison between LoRA and PERA. By applying the polynomial expansion technique, PERA synthesizes new high-order feature terms from the original r feature terms in the low-rank matrices, including r square feature terms and C(r,2) crossed feature terms.
Model
Method
#Params (%)
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Avg.
ChatGPT
-
-
73.10
85.40
68.50
79.90
89.80
74.80
78.50
66.10
77.01
Llama-2-7B
Prompt Tuning
0.0012
55.93
12.35
30.50
6.06
8.63
9.40
6.91
40.57
21.29
P-Tuning
0.7428
58.75
36.02
0.20
0.17
1.98
0.80
0.01
0.00
12.24
LoRA ( r=32 )
0.8256
69.80
79.90
79.50
64.70
79.80
81.00
83.60
82.60
77.61
DoRA ( r=32 )
0.8256
71.80
83.70
76.00
68.20
83.70
82.40
89.10
82.60
79.69
MoRA ( r=32 )
0.8241
72.17
80.79
79.53
71.42
85.31
81.20
29.09
80.19
72.46
Table 1: Accuracy(%) comparison of LLaMA model with various fine-tuned methods on commonsense reasoning datasets. Results of all baseline methods are taken from ( Wu et al., 2024b ; Huang et al., 2025 ) . The best performance within each LLM is indicated in bold, while the second best performance is highlighted in underline.
Model
Method
#Param
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
Avg.
RoBERTa-base
FFT
125M
94.40
87.90
62.40
92.50
78.30
90.60
84.40
RED
0.02M
93.90
89.20
61.00
90.70
78.00
90.40
83.90
LoRA
0.3M
93.90
88.70
59.70
92.60
75.30
90.30
83.40
HiRA
0.3M
94.22
89.53
60.30
92.39
74.68
89.58
83.45
DeLoRA
0.3M
94.10
89.00
63.60
92.80
77.10
90.90
84.60
PERA
0.3M
94.13
89.71
64.80
92.94
77.84
91.13
85.10
Table 2: Comparisons of different methods finetuning RoBERTa-base and RoBERTa-large on GLUE benchmark. Results of all baselines are taken from ( Bini et al., 2025 ; Wu et al., 2024c ) except HiRA. The best performance within each LLM is indicated in bold, while the second best performance is highlighted in underline.
Figure 3: Accuracy (%) comparison with the rank r increases on LLaMA3-8B model. The detailed results are provided in Appendix C.1
Placement
#Params(%)
Avg.
QKV, UD
0.3513
87.38
QKV
0.1176
87.02
UD
0.2345
86.91
QK
0.0849
85.25
QV
0.0849
86.70
Q
0.0522
84.83
Table 3: Average accuracy (%) comparison of different modules on various tasks with LLaMA3-8B model.
Figure 4: Average accuracy under different pruning ratios of Commonsense170K with LLaMA3-8B model.
Method
ΔW
Avg.
LoRA
Eq. 8
82.80
LoRA + square-only
Eq. 10
87.48
LoRA + cross-only
Eq. 11
86.83
PERA
Eq. 9
87.38
Table 4: Ablation of PERA innovations on the commonsense reasoning tasks. We show how different components affect performance from LoRA ( Hu et al., 2022 ) with the LLaMA3-8B model.
Figure 5: The interaction strength matrix of LoRA and PERA. We quantify the combined influence of each feature dimension on the output, not just the individual influence.
Method
Training
Inference
Memory
Speed
Memory
Speed
LoRA
18.70GB
12h 27m
19.50GB
9m30s
DoRA
19.58GB
22h 07m
19.94GB
23m40s
HiRA
18.72GB
14h 02m
19.58GB
20m45s
PERA
19.12GB
13h 30m
19.70GB
14m53s
Table 5: Comparison of training and inference efficiency across methods.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
HyperParameters
LLaMA2-7B
LLaMA3-8B
Rank r
16
32
16
32
α
16
32
16
32
Dropout
0.05
Optimizer
AdamW
LR
1e-4
LR Scheduler
Linear
Appendix
Table 6: The hyperparameters for PERA on the commonsense reasoning tasks.
Splits Sizes
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
Training Set
67K
3.7K
8.5K
105K
2.5K
5.7K
New Validation Set
436
204
522
1K
139
750
New Test Set
436
204
521
4.5K
138
750
Appendix
Table 7: GLUE dataset sizes, with new validation and test splits following ( Wu et al., 2024c ; Bini et al., 2025 ) setup
Model
Hyperparameters
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
RoBERTa-base
Learning Rate
1e-4
1e-3
1e-3
3e-4
5e-4
1e-3
Batch Size
32
32
32
32
32
32
Num. Epochs
50
50
40
30
75
80
Dropout
0
0.25
0
0.25
0
0.25
Warmup Ratio
6e-2
Maximum Sequence Length
512
Appendix
Table 8: GLUE benchmark hyperparameters.
Figure 6: Loss Comparision between Different PEFT Methods
Rank
#Param (%)
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Avg.
2
0.0440
72.39
89.39
83.32
83.45
93.43
88.00
96.14
89.19
86.91
4
0.0880
74.25
89.93
81.88
83.28
93.56
87.60
96.48
89.11
87.01
8
0.1760
75.72
88.68
81.78
83.53
93.64
88.00
96.15
88.95
87.05
12
0.2638
75.11
87.49
82.60
83.79
93.56
88.40
95.87
89.74
87.07
16
0.3515
76.15
88.57
83.16
83.62
93.73
88.60
95.94
89.27
87.38
20
0.4391
74.62
89.83
81.93
83.28
93.86
90.20
95.77
88.56
87.25
Appendix
Table 9: Accuracy (%) comparison with the rank r increases on LLaMA3-8B model.
Placement
#Params (%)
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Avg.
QKV, UD
0.3513
76.15
88.57
83.16
83.62
93.73
88.60
95.94
89.27
87.38
QKV
0.1176
74.71
89.39
82.50
83.62
93.81
87.60
95.92
88.63
87.02
UD
0.2345
72.66
89.61
82.45
83.53
93.39
89.00
96.25
88.40
86.91
QK
0.0849
71.96
88.30
80.96
80.12
93.01
86.80
95.01
85.87
85.25
QV
0.0849
74.80
89.12
81.73
83.19
93.01
87.00
96.03
88.71
86.70
Q
0.0522
71.80
88.63
80.71
79.78
92.26
84.60
94.99
85.87
84.83
Appendix
Table 10: Accuracy (%) comparison of different placement on commonsense reasoning tasks with LLaMA3-8B model.
Pruning Ratio
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Average
0.1
68.78
86.67
77.38
80.46
92.26
82.60
92.03
84.37
83.07
0.2
68.84
84.93
77.79
79.44
91.75
81.60
91.04
82.40
82.22
0.5
73.52
88.41
80.91
82.94
93.22
85.60
95.30
89.19
86.14
0.8
75.60
89.06
82.04
84.56
93.77
87.60
96.19
88.32
87.14
1.0
76.15
88.57
83.16
83.62
93.73
88.60
95.94
89.27
87.38
Appendix
Table 11: Accuracy (%) comparision of different training samples pruning ratio fine-tuning LLaMA3-8B model on the commonsense reasoning tasks.
Component
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Average
LoRA
72.30
86.70
79.30
75.70
87.70
82.80
93.50
84.80
82.80
LoRA + cross features
75.41
88.52
81.78
82.76
93.05
88.80
95.75
88.55
86.83
LoRA + square features
76.17
89.28
82.34
84.72
92.80
89.20
95.83
89.50
87.48
PERA
76.15
88.57
83.16
83.62
93.73
88.60
95.94
89.27
87.38
Appendix
Table 12: Accuracy (%) comparision of different high-order components from LoRA with LLaMA3-8B model on the commonsense reasoning tasks.
Low-rank adaptation (LoRA) has become a popular parameter-efficient fine-tuning method for large language models. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency. Existing adaptive-rank methods typically allocate ranks according to manually designed importance scores, which are not directly derived from an optimization objective. In this work, we propose ℓp-LoRA, a principled rank-allocation method based on ℓp regularization with 0<p<1, which is a classical sparsity-inducing technique in signal processing and statistics. Specifically, we regularize the energy of each rank-one LoRA component, encouraging redundant components to vanish while preserving important ones. We derive the corresponding proximal subproblem and reduce the matrix optimization to a two-dimensional problem, leading to an implicit thresholding criterion for identifying redundant components. Experiments on natural language understanding and question-answering tasks demonstrate that the proposed method achieves competitive performance with existing LoRA baselines.
Zebang Xie, Chuanyang Zheng, Yik-Chung Wu +1
National University of Singapore, Singapore · Huawei, Hong Kong · The University of Hong Kong, Hong Kong +1
Exponential growth in the scale of modern foundation models has led to the widespread adoption of Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning technique. However, standard LoRA implementations disregard the varying intrinsic dimensionality of model layers and enforce a uniform rank, leading to parameter redundancy. We propose Post-Optimization Adaptive Rank Allocation (PARA), a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines. PARA leverages Singular Value Decomposition to prune LoRA ranks using a global threshold over singular values across all layers. This results in non-uniform rank allocation based on layer-wise spectral importance. As a post-hoc method, PARA circumvents the training modifications and resulting instabilities that dynamic architectures typically incur. We empirically demonstrate that PARA reduces parameter count by 75-90% while preserving the predictive performance of the original, uncompressed LoRA across multiple vision and language benchmarks. Code will be published upon acceptance.
Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. Srijith
1Indian Institute of Technology Hyderabad · 2Deakin University
Low-rank adaptation (LoRA) approximates the update of a pretrained weight matrix using the product of two low-rank matrices. However, standard LoRA follows an explicit-rank paradigm, where increasing model capacity requires adding more rows or columns (i.e., basis vectors) to the low-rank matrices, leading to substantial parameter growth. In this paper, we find that these basis vectors exhibit significant parameter redundancy and can be compactly represented by lightweight nonlinear functions. Therefore, we propose Generative Low-Rank Adapter (GenLoRA), which replaces explicit basis vector storage with nonlinear basis vector generation. Specifically, GenLoRA maintains a latent vector for each low-rank matrix and employs a set of lightweight radial basis functions (RBFs) to synthesize the basis vectors. Each RBF requires far fewer parameters than an explicit basis vector, enabling higher parameter efficiency in GenLoRA. Extensive experiments across multiple datasets and architectures show that GenLoRA attains higher effective LoRA ranks under smaller parameter budgets, resulting in superior fine-tuning performance. The code is available at https://anonymous.4open.science/r/GenLoRA.
Yihao Ouyang, Shiwei Li, Haozhao Wang +5
Huazhong University of Science and Technology, Wuhan, China