Polynomial Expansion Rank Adaptation: Enhancing Low-Rank Fine-Tuning with High-Order Interactions

Wenhao Zhang; Lin Mu; Li Ni; Peiquan Jin; Yiwen Zhang

cs.LGApr 12, 2026

Polynomial Expansion Rank Adaptation: Enhancing Low-Rank Fine-Tuning with High-Order Interactions

Authors: Wenhao Zhang, Lin Mu, Li Ni, Peiquan Jin, Yiwen Zhang

Organizations: Anhui University · University of Science and Technology of China

Abstract

Low-rank adaptation (LoRA) is a widely used strategy for efficient fine-tuning of large language models (LLMs), but its strictly linear structure fundamentally limits expressive capacity. The bilinear formulation of weight updates captures only first-order dependencies between low-rank factors, restricting the modeling of nonlinear and higher-order parameter interactions. In this paper, we propose Polynomial Expansion Rank Adaptation (PERA), a novel method that introduces structured polynomial expansion directly into the low-rank factor space. By expanding each low-rank factor to synthesize high-order interaction terms before composition, PERA transforms the adaptation space into a polynomial manifold capable of modeling richer nonlinear coupling without increasing rank or inference cost. We provide theoretical analysis demonstrating that PERA offers enhanced expressive capacity and more effective feature utilization compare to existing linear adaptation approaches. Empirically, PERA consistently outperforms state-of-the-art methods across diverse benchmarks. Notably, our experiments show that incorporating high-order nonlinear components particularly square terms is crucial for enhancing expressive capacity and maintaining strong and robust performance under various rank settings. Our code is available at https://github.com/zhangwenhao6/PERA

Figures & tables

Figure 1: Comparison of Training Losses between First-Order and High-Order Terms.

Figure 2: The architecture comparison between LoRA and PERA. By applying the polynomial expansion technique, PERA synthesizes new high-order feature terms from the original $r$ feature terms in the low-rank matrices, including $r$ square feature terms and $C(r,2)$ crossed feature terms.

Model	Method	#Params (%)	BoolQ	PIQA	SIQA	ARC-c	ARC-e	OBQA	HellaS	WinoG	Avg.
ChatGPT	-	-	73.10	85.40	68.50	79.90	89.80	74.80	78.50	66.10	77.01
Llama-2-7B	Prompt Tuning	0.0012	55.93	12.35	30.50	6.06	8.63	9.40	6.91	40.57	21.29
	P-Tuning	0.7428	58.75	36.02	0.20	0.17	1.98	0.80	0.01	0.00	12.24
	LoRA ( $r=32$ )	0.8256	69.80	79.90	79.50	64.70	79.80	81.00	83.60	82.60	77.61
	DoRA ( $r=32$ )	0.8256	71.80	83.70	76.00	68.20	83.70	82.40	89.10	82.60	79.69
	MoRA ( $r=32$ )	0.8241	72.17	80.79	79.53	71.42	85.31	81.20	29.09	80.19	72.46

Table 1: Accuracy(%) comparison of LLaMA model with various fine-tuned methods on commonsense reasoning datasets. Results of all baseline methods are taken from ( Wu et al., 2024b ; Huang et al., 2025 ) . The best performance within each LLM is indicated in bold, while the second best performance is highlighted in underline.

Model	Method	#Param	SST-2	MRPC	CoLA	QNLI	RTE	STS-B	Avg.
RoBERTa-base	FFT	125M	94.40	87.90	62.40	92.50	78.30	90.60	84.40
	RED	0.02M	93.90	89.20	61.00	90.70	78.00	90.40	83.90
	LoRA	0.3M	93.90	88.70	59.70	92.60	75.30	90.30	83.40
	HiRA	0.3M	94.22	89.53	60.30	92.39	74.68	89.58	83.45
	DeLoRA	0.3M	94.10	89.00	63.60	92.80	77.10	90.90	84.60
	PERA	0.3M	94.13	89.71	64.80	92.94	77.84	91.13	85.10

Table 2: Comparisons of different methods finetuning RoBERTa-base and RoBERTa-large on GLUE benchmark. Results of all baselines are taken from ( Bini et al., 2025 ; Wu et al., 2024c ) except HiRA. The best performance within each LLM is indicated in bold, while the second best performance is highlighted in underline.

Figure 3: Accuracy

(\%)

comparison with the rank

r

increases on LLaMA3-8B model. The detailed results are provided in Appendix C.1

Placement	#Params(%)	Avg.
QKV, UD	0.3513	87.38
QKV	0.1176	87.02
UD	0.2345	86.91
QK	0.0849	85.25
QV	0.0849	86.70
Q	0.0522	84.83

Table 3: Average accuracy (%) comparison of different modules on various tasks with LLaMA3-8B model.

Figure 4: Average accuracy under different pruning ratios of Commonsense170K with LLaMA3-8B model.

Method	$\Delta W$	Avg.
LoRA	Eq. 8	82.80
LoRA + square-only	Eq. 10	87.48
LoRA + cross-only	Eq. 11	86.83
PERA	Eq. 9	87.38

Table 4: Ablation of PERA innovations on the commonsense reasoning tasks. We show how different components affect performance from LoRA ( Hu et al., 2022 ) with the LLaMA3-8B model.

Panel of Figure 5 — Figure 5: The interaction strength matrix of LoRA and PERA. We quantify the combined influence of each feature dimension on the output, not just the individual influence.

Method	Training		Inference
Method	Memory	Speed	Memory	Speed
LoRA	18.70GB	12h 27m	19.50GB	9m30s
DoRA	19.58GB	22h 07m	19.94GB	23m40s
HiRA	18.72GB	14h 02m	19.58GB	20m45s
PERA	19.12GB	13h 30m	19.70GB	14m53s

Table 5: Comparison of training and inference efficiency across methods.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

HyperParameters	LLaMA2-7B		LLaMA3-8B
Rank $r$	16	32	16	32
$\alpha$	16	32	16	32
Dropout	0.05
Optimizer	AdamW
LR	1e-4
LR Scheduler	Linear

Appendix

Table 6: The hyperparameters for PERA on the commonsense reasoning tasks.

Splits Sizes	SST-2	MRPC	CoLA	QNLI	RTE	STS-B
Training Set	67K	3.7K	8.5K	105K	2.5K	5.7K
New Validation Set	436	204	522	1K	139	750
New Test Set	436	204	521	4.5K	138	750

Appendix

Table 7: GLUE dataset sizes, with new validation and test splits following ( Wu et al., 2024c ; Bini et al., 2025 ) setup

Model	Hyperparameters	SST-2	MRPC	CoLA	QNLI	RTE	STS-B
RoBERTa-base	Learning Rate	1e-4	1e-3	1e-3	3e-4	5e-4	1e-3
	Batch Size	32	32	32	32	32	32
	Num. Epochs	50	50	40	30	75	80
	Dropout	0	0.25	0	0.25	0	0.25
	Warmup Ratio	6e-2
	Maximum Sequence Length	512

Appendix

Table 8: GLUE benchmark hyperparameters.

Figure 6: Loss Comparision between Different PEFT Methods

Rank	#Param (%)	BoolQ	PIQA	SIQA	ARC-c	ARC-e	OBQA	HellaS	WinoG	Avg.
2	0.0440	72.39	89.39	83.32	83.45	93.43	88.00	96.14	89.19	86.91
4	0.0880	74.25	89.93	81.88	83.28	93.56	87.60	96.48	89.11	87.01
8	0.1760	75.72	88.68	81.78	83.53	93.64	88.00	96.15	88.95	87.05
12	0.2638	75.11	87.49	82.60	83.79	93.56	88.40	95.87	89.74	87.07
16	0.3515	76.15	88.57	83.16	83.62	93.73	88.60	95.94	89.27	87.38
20	0.4391	74.62	89.83	81.93	83.28	93.86	90.20	95.77	88.56	87.25

Appendix

Table 9: Accuracy (%) comparison with the rank

r

increases on LLaMA3-8B model.

Placement	#Params (%)	BoolQ	PIQA	SIQA	ARC-c	ARC-e	OBQA	HellaS	WinoG	Avg.
QKV, UD	0.3513	76.15	88.57	83.16	83.62	93.73	88.60	95.94	89.27	87.38
QKV	0.1176	74.71	89.39	82.50	83.62	93.81	87.60	95.92	88.63	87.02
UD	0.2345	72.66	89.61	82.45	83.53	93.39	89.00	96.25	88.40	86.91
QK	0.0849	71.96	88.30	80.96	80.12	93.01	86.80	95.01	85.87	85.25
QV	0.0849	74.80	89.12	81.73	83.19	93.01	87.00	96.03	88.71	86.70
Q	0.0522	71.80	88.63	80.71	79.78	92.26	84.60	94.99	85.87	84.83

Appendix

Table 10: Accuracy (%) comparison of different placement on commonsense reasoning tasks with LLaMA3-8B model.

Pruning Ratio	BoolQ	PIQA	SIQA	ARC-c	ARC-e	OBQA	HellaS	WinoG	Average
0.1	68.78	86.67	77.38	80.46	92.26	82.60	92.03	84.37	83.07
0.2	68.84	84.93	77.79	79.44	91.75	81.60	91.04	82.40	82.22
0.5	73.52	88.41	80.91	82.94	93.22	85.60	95.30	89.19	86.14
0.8	75.60	89.06	82.04	84.56	93.77	87.60	96.19	88.32	87.14
1.0	76.15	88.57	83.16	83.62	93.73	88.60	95.94	89.27	87.38

Appendix

Table 11: Accuracy (%) comparision of different training samples pruning ratio fine-tuning LLaMA3-8B model on the commonsense reasoning tasks.

Component	BoolQ	PIQA	SIQA	ARC-c	ARC-e	OBQA	HellaS	WinoG	Average
LoRA	72.30	86.70	79.30	75.70	87.70	82.80	93.50	84.80	82.80
LoRA + cross features	75.41	88.52	81.78	82.76	93.05	88.80	95.75	88.55	86.83
LoRA + square features	76.17	89.28	82.34	84.72	92.80	89.20	95.83	89.50	87.48
PERA	76.15	88.57	83.16	83.62	93.73	88.60	95.94	89.27	87.38

Appendix

Table 12: Accuracy (%) comparision of different high-order components from LoRA with LLaMA3-8B model on the commonsense reasoning tasks.

Nonlinear LoRA

See all 10 papers

Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization
Sep 24, 2026Zebang Xie, Chuanyang Zheng, Yik-Chung Wu +1Low-Rank Adaptation Adaptive Rank Allocation
National University of Singapore, Singapore · Huawei, Hong Kong · The University of Hong Kong, Hong Kong +1
Post-Optimization Adaptive Rank Allocation for LoRA
Apr 30, 2026Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. SrijithLow-Rank Adaptation Low-Rank Compression
1Indian Institute of Technology Hyderabad · 2Deakin University
Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functions
Feb 5, 2026Yihao Ouyang, Shiwei Li, Haozhao Wang +5Nonlinear LoRA Low-Rank Adaptation
Huazhong University of Science and Technology, Wuhan, China

Sep 24, 2026J/K move · Enter open · S save

Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization

Zebang Xie, Chuanyang Zheng, Yik-Chung Wu, Yihang Gao

National University of Singapore, Singapore · Huawei, Hong Kong · The University of Hong Kong, Hong Kong+1

Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-rank adaptation (LoRA) has become a popular parameter-efficient fine-tuning method for large language models. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency. Existing adaptive-rank methods typically allocate ranks according to manually designed importance scores, which are not directly derived from an optimization objective. In this work, we propose

\ell_p

-LoRA, a principled rank-allocation method based on

\ell_p

regularization with

0<p<1

, which is a classical sparsity-inducing technique in signal processing and statistics. Specifically, we regularize the energy of each rank-one LoRA component, encouraging redundant components to vanish while preserving important ones. We derive the corresponding proximal subproblem and reduce the matrix optimization to a two-dimensional problem, leading to an implicit thresholding criterion for identifying redundant components. Experiments on natural language understanding and question-answering tasks demonstrate that the proposed method achieves competitive performance with existing LoRA baselines.

Abstract

Nonlinear LoRA

Explore similar work

Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization

Post-Optimization Adaptive Rank Allocation for LoRA

Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functions