Organizations: Institute of Information Engineering, CAS · School of Cyber Security, University of Chinese Academy of Sciences · State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · Beijing Academy of Artificial Intelligence · School of Computer Science and Tech., University of Chinese Academy of Sciences · Alibaba Group · School of Cyber Science and Tech., Shenzhen Campus of Sun Yat-sen University
Parameter-Efficient Fine-Tuning (PEFT) has emerged as a prevalent strategy for adapting large pre-trained models. Among PEFT techniques, reparameterization-based methods, particularly those that employ low-rank or sparsity structures, have attracted significant attention. However, these approaches often underperform full fine-tuning. Specifically, low-rank methods constrain the update rank, which is misaligned with the inherently high-rank nature of full fine-tuning. In contrast, sparsity-based techniques limit the flexibility of the parameter space and compromise connectivity across weights. To overcome these limitations, we explore the use of Toeplitz matrices, whose entries are constant along each diagonal, therefore providing a compact parameterization without imposing explicit rank or sparsity constraints. A straightforward approach is to update weights via a product of Toeplitz matrices. However, similar to RNNs, long multiplicative chains often lead to gradient instability. To address this issue, we propose ToPA, a more refined version of this idea that replaces standard Toeplitz matrices with block-wise Toeplitz, greatly increasing each factor's capacity and shortening the chain, thereby stabilizing training and improving expressiveness. We theoretically show that products of block-wise Toeplitz matrices can approximate arbitrary matrices under shorter chain length, justifying the structural design of ToPA. Furthermore, we demonstrate that ToPA offers greater expressivity than existing low-rank and sparse parameterizations. Empirical evaluations across 21 NLP and CV datasets, spanning 5 model architectures, consistently validate the effectiveness of ToPA.
Figures & tables
Figure 1: Low-rank methods (e.g., LoRA) adapt pretrained weights via additive low-rank updates, while sparsity-structured methods (e.g., OFT/BOFT) apply multiplicative sparse adapters. In contrast, ToPA adapts the model based on the block-wise Toeplitz matrices with identity residual design, enabling dense and high-rank adaptation while maintaining parameter efficiency.
Figure 2: Fine-tuning LLaMA-2-7B for mathematical reasoning under different parameter budgets. (a) Average rank of the weight update matrix ΔW . (b) Average density of the adapter R across different methods, measured by the non-zero ratio (NNZ).
Figure 3: Training dynamics over the first 700 steps of fine-tuning LLaMA-3-8B on commonsense reasoning tasks using the naive method ( t=16 ) and ToPA p=8t=2 with matched parameters. As training proceeds, the naive method collapses, so only the first 700 steps are shown.
Model
Method
# Param
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Avg
LLaMA2-7B
LoRA r=32
16.8M
70.5
81.1
80.1
71.5
85.4
78.6
84.6
78.5
78.8
DoRA r=32
17.0M
70.7
81.3
79.5
71.2
85.6
78.2
84.8
78.4
78.7
HiRA r=32
16.8M
70.6
82.2
79.3
69.8
85.9
79.4
86.5
79.8
79.2
RidgeLoRA r=32
17.0M
71.2
83.5
81.1
71.5
87.0
79.8
87.3
82.6
80.5
OFT b=64
16.8M
70.5
83.1
80.2
69.9
86.6
80.4
86.5
82.0
79.9
BOFT b=32m=2
17.0M
70.6
83.2
79.2
70.1
87.4
77.6
82.4
78.6
78.6
Table 1: Performance comparison on commonsense reasoning tasks with LLaMA-2-7B and LLaMA-3-8B backbones.
Table 5
Figure 4: Qualitative comparison of results on subject-driven generation tasks.
Method
# Param
DINO ↑
CLIP-I ↑
CLIP-T ↑
LPIPS ↑
LoRA r=64
12.8M
0.654
0.793
0.230
0.743
DoRA r=64
12.8M
0.654
0.791
0.231
0.740
PiSSA r=64
12.8M
0.634
0.782
0.228
0.739
HiRA r=64
12.8M
0.668
0.794
0.226
0.736
RidgeLoRA r=64
12.8M
0.657
0.791
0.222
0.737
OFT b=64
23.3M
0.653
0.793
0.225
0.735
Table 4: Performance comparison on subject-driven generation.
Method
# Param
MNLI
SST-2
CoLA
QQP
QNLI
RTE
MRPC
STS-B
Avg
LoRA r=8
0.44M
89.92
94.72
69.31
91.68
93.99
84.48
89.46
91.12
88.09
DoRA r=8
0.47M
89.67
94.50
68.86
91.55
94.23
85.92
89.46
91.56
88.22
PiSSA r=8
0.44M
89.90
95.07
68.52
91.78
93.98
86.64
89.22
91.11
88.28
HiRA r=8
0.44M
90.20
95.64
69.80
91.08
93.26
87.00
89.22
91.45
88.46
RidgeLoRA r=8
0.47M
89.74
95.18
68.82
91.57
93.98
85.20
88.97
91.12
88.07
OFT b=16
0.44M
89.55
94.38
68.97
91.25
93.56
84.48
88.73
91.24
87.77
Table 5: Performance comparison on the GLUE benchmark using the DeBERTaV3-base model. We report Matthew’s correlation for CoLA, Pearson correlation for STS-B, matched accuracy for MNLI, and accuracy for the other datasets.
Figure 9
Figure 7: Performance across parameter scales. We compare ToPA and LoRA on math reasoning tasks using LLaMA-3-8B.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Method
#Param (M)
Time (s)
Memory (MB)
LoRA r=16
5.9
1.1
13178
DoRA r=16
5.9
1.4
13558
HiRA r=16
5.9
1.1
13188
OFT b=32
6.1
2.8
16415
BOFT b=16m=2
6.1
3.5
19786
ToPA p=5t=3
5.5
1.8
15861
Appendix
Table 6: Efficiency comparison on Qwen3-4B for math reasoning. Time denotes the average wall-clock time per training step, and memory denotes the peak GPU memory consumption.
Method
#Param (M)
Time (s)
Memory (MB)
LoRA r=32
16.8
0.9
16494
DoRA r=32
17.0
1.3
17070
HiRA r=32
16.8
0.9
16083
OFT b=32
16.8
3.3
21814
BOFT b=16m=2
17.0
–
OOM
ToPA p=16t=2
16.7
1.6
18040
Appendix
Table 7: Efficiency comparison on LLaMA-2-7B for commonsense reasoning. Time denotes the average wall-clock time per training step, and memory denotes the peak GPU memory consumption. BOFT runs out of memory under the same settings without gradient checkpointing.
Dataset
MNLI
SST-2
CoLA
QQP
QNLI
RTE
MRPC
STS-B
# Epochs
8
15
30
15
5
40
40
25
Learning Rate
3e-4
1e-4
4e-4
1e-4
1e-4
2e-5
3e-4
8e-5
Max Seq. Len.
256
128
64
320
512
320
320
128
Appendix
Table 8: The hyperparameter settings on the GLUE benchmarks
Method
# Param
GSM8K
MATH
HumanEval
Mbpp
w/o Identity
12.6M
74.3
23.2
51.0
66.6
w/ Identity
12.6M
74.7
23.5
51.8
67.2
Appendix
Table 9: Performance comparison of LLaMA-3-8B fine-tuned with ToPA p=8t=3 on math & code generation tasks, with and without the identity residual.
Figure 8: Training dynamics over the first 300 steps of fine-tuning LLaMA-2-7B on commonsense reasoning tasks using the naive method ( t=32 ) and ToPA p=16t=2 with matched parameters. As training proceeds, the naive method collapses, so only the first 300 steps are shown.
Method
# Param
GSM8K
MATH
HumanEval
MBPP
LoRA r=32
13.6M
70.5±0.4
22.5±0.3
50.2±0.3
67.2±0.3
DoRA r=32
13.8M
71.1±0.3
22.8±0.2
51.0±0.3
67.0±0.2
OFT b=64
17.0M
73.4±0.3
22.9±0.2
45.7±0.5
66.5±0.2
BOFT b=16m=3
12.8M
69.4±0.4
22.3±0.3
45.5±0.3
67.1±0.2
ToPA p=8t=3
12.6M
74.9±0.3
23.5±0.2
51.6±0.3
67.2±0.3
Appendix
Table 10: Multi-seed performance comparison on math reasoning and code generation tasks with the LLaMA-3-8B backbone. Results are averaged over three random seeds.
Figure 9: Additional qualitative comparison of results on subject-driven generation tasks.
Figure 10: Additional qualitative comparison of results on subject-driven generation tasks.
Figure 11: Additional qualitative comparison of results on subject-driven generation tasks.
Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method that restricts weight updates to low-rank adapters, introducing a fixed low-rank inductive bias by optimizing in a low-dimensional subspace. In this work, we question whether a fixed-rank constraint is the most effective inductive bias for parameter-efficient fine-tuning. We introduce Learnable Rank LoRA (LR-LoRA), a PEFT method in which the adapter rank is learned during the training process. Instead of prescribing a uniform rank for all adapter layers, LR-LoRA allows the optimizer to determine the appropriate rank for each layer. Using this approach, we find substantial layer-wise variation in the learned ranks, with the attention and MLP layers in the transformer models exhibiting systematically different rank preferences. Across a range of language understanding and commonsense reasoning benchmarks, LR-LoRA achieves state-of-the-art performance in most settings and consistently outperforms strong PEFT baselines, demonstrating that a learnable rank provides a more flexible and effective inductive bias than fixed-rank adaptations.
Arpit Garg, Simon Lucey, Hemanth Saratchandran
Australian Institute for Machine Learning Adelaide University
Fine-tuning large language models (LLMs) aims to adapt pre-trained models to specific tasks using relatively small and domain-specific datasets. Among Parameter-Efficient Fine-Tuning (PEFT) methods, Low-Rank Adaptation (LoRA) stands out by matching the performance of full fine-tuning while avoiding additional inference latency. In this paper, we propose a novel PEFT method that incorporates the TLoRA+ optimizer into the weight matrices of pre-trained models. The proposed approach not only preserves the efficiency of low-rank adaptation but also further enhances performance without significantly increasing computational cost. We conduct experiments on the GLUE benchmark across diverse model architectures. Numerical experiments consistently demonstrate the effectiveness and robustness of our proposed method.
Yarui Cao, Kai Liu
Department of Computer Science, Clemson University, Clemson, SC 29634, USA
Parameter-efficient fine-tuning (PEFT) has emerged as an critical technique for adapting large-scale foundation models across natural language processing and computer vision. While existing methods such as low-rank adaptations achieve parameter efficiency via low-rank weight updates, they are limited in their ability to preserve the geometric structure of pretrained representations. We introduce Low-rank Compositional Orthogonal fine-tuning (LoCO), a novel PEFT method that constructs orthogonal transformations through low-rank skew-symmetric matrices and compositional rotation chains. We propose an approximation scheme that enables fully parallel computation of compositional rotations, making the approach practical for high-dimensional feature spaces. Our method maintains low computational complexity while maintaining orthogonality with controlled approximation error. We validate LoCO across diverse domains, including diffusion transformer fine-tuning, vision transformer adaptation, and language model adaptation. Our method demonstrates superior or competitive performance compared to both existing orthogonal and non-orthogonal methods.