ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning

Sicong Li; Qianqian Xu; Zhiyong Yang; Zitai Wang; Longtao Huang; Xiaochun Cao; Qingming Huang

cs.AIOct 16, 2026

ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning

Authors: Sicong Li, Qianqian Xu, Zhiyong Yang, Zitai Wang, Longtao Huang, Xiaochun Cao, Qingming Huang

Organizations: Institute of Information Engineering, CAS · School of Cyber Security, University of Chinese Academy of Sciences · State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · Beijing Academy of Artificial Intelligence ·

Abstract

Parameter-Efficient Fine-Tuning (PEFT) has emerged as a prevalent strategy for adapting large pre-trained models. Among PEFT techniques, reparameterization-based methods, particularly those that employ low-rank or sparsity structures, have attracted significant attention. However, these approaches often underperform full fine-tuning. Specifically, low-rank methods constrain the update rank, which is misaligned with the inherently high-rank nature of full fine-tuning. In contrast, sparsity-based techniques limit the flexibility of the parameter space and compromise connectivity across weights. To overcome these limitations, we explore the use of Toeplitz matrices, whose entries are constant along each diagonal, therefore providing a compact parameterization without imposing explicit rank or sparsity constraints. A straightforward approach is to update weights via a product of Toeplitz matrices. However, similar to RNNs, long multiplicative chains often lead to gradient instability. To address this issue, we propose ToPA, a more refined version of this idea that replaces standard Toeplitz matrices with block-wise Toeplitz, greatly increasing each factor's capacity and shortening the chain, thereby stabilizing training and improving expressiveness. We theoretically show that products of block-wise Toeplitz matrices can approximate arbitrary matrices under shorter chain length, justifying the structural design of ToPA. Furthermore, we demonstrate that ToPA offers greater expressivity than existing low-rank and sparse parameterizations. Empirical evaluations across 21 NLP and CV datasets, spanning 5 model architectures, consistently validate the effectiveness of ToPA.

Figures & tables

Figure 1: Low-rank methods (e.g., LoRA) adapt pretrained weights via additive low-rank updates, while sparsity-structured methods (e.g., OFT/BOFT) apply multiplicative sparse adapters. In contrast, ToPA adapts the model based on the block-wise Toeplitz matrices with identity residual design, enabling dense and high-rank adaptation while maintaining parameter efficiency.

Panel of Figure 2 — Figure 2: Fine-tuning LLaMA-2-7B for mathematical reasoning under different parameter budgets. (a) Average rank of the weight update matrix $\Delta\bm{W}$ . (b) Average density of the adapter $\bm{R}$ across different methods, measured by the non-zero ratio (NNZ).

Panel of Figure 3 — Figure 3: Training dynamics over the first 700 steps of fine-tuning LLaMA-3-8B on commonsense reasoning tasks using the naive method ( $t=16$ ) and ToPA ${}^{t=2}_{p=8}$ with matched parameters. As training proceeds, the naive method collapses, so only the first 700 steps are shown.

Model	Method	# Param	BoolQ	PIQA	SIQA	ARC-c	ARC-e	OBQA	HellaS	WinoG	Avg
LLaMA2-7B	LoRA r=32	16.8M	70.5	81.1	80.1	71.5	85.4	78.6	84.6	78.5	78.8
	DoRA r=32	17.0M	70.7	81.3	79.5	71.2	85.6	78.2	84.8	78.4	78.7
	HiRA r=32	16.8M	70.6	82.2	79.3	69.8	85.9	79.4	86.5	79.8	79.2
	RidgeLoRA r=32	17.0M	71.2	83.5	81.1	71.5	87.0	79.8	87.3	82.6	80.5
	OFT b=64	16.8M	70.5	83.1	80.2	69.9	86.6	80.4	86.5	82.0	79.9
	BOFT ${}^{m=2}_{b=32}$	17.0M	70.6	83.2	79.2	70.1	87.4	77.6	82.4	78.6	78.6

Table 1: Performance comparison on commonsense reasoning tasks with LLaMA-2-7B and LLaMA-3-8B backbones.

Table 5

Figure 4: Qualitative comparison of results on subject-driven generation tasks.

Method	# Param	DINO $\uparrow$	CLIP-I $\uparrow$	CLIP-T $\uparrow$	LPIPS $\uparrow$
LoRA r=64	12.8M	0.654	0.793	0.230	0.743
DoRA r=64	12.8M	0.654	0.791	0.231	0.740
PiSSA r=64	12.8M	0.634	0.782	0.228	0.739
HiRA r=64	12.8M	0.668	0.794	0.226	0.736
RidgeLoRA r=64	12.8M	0.657	0.791	0.222	0.737
OFT b=64	23.3M	0.653	0.793	0.225	0.735

Table 4: Performance comparison on subject-driven generation.

Method	# Param	MNLI	SST-2	CoLA	QQP	QNLI	RTE	MRPC	STS-B	Avg
LoRA r=8	0.44M	89.92	94.72	69.31	91.68	93.99	84.48	89.46	91.12	88.09
DoRA r=8	0.47M	89.67	94.50	68.86	91.55	94.23	85.92	89.46	91.56	88.22
PiSSA r=8	0.44M	89.90	95.07	68.52	91.78	93.98	86.64	89.22	91.11	88.28
HiRA r=8	0.44M	90.20	95.64	69.80	91.08	93.26	87.00	89.22	91.45	88.46
RidgeLoRA r=8	0.47M	89.74	95.18	68.82	91.57	93.98	85.20	88.97	91.12	88.07
OFT b=16	0.44M	89.55	94.38	68.97	91.25	93.56	84.48	88.73	91.24	87.77

Table 5: Performance comparison on the GLUE benchmark using the DeBERTaV3-base model. We report Matthew’s correlation for CoLA, Pearson correlation for STS-B, matched accuracy for MNLI, and accuracy for the other datasets.

Panel of Figure 7 — Figure 7: Performance across parameter scales. We compare ToPA and LoRA on math reasoning tasks using LLaMA-3-8B.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	#Param (M)	Time (s)	Memory (MB)
LoRA r=16	5.9	1.1	13178
DoRA r=16	5.9	1.4	13558
HiRA r=16	5.9	1.1	13188
OFT b=32	6.1	2.8	16415
BOFT ${}^{m=2}_{b=16}$	6.1	3.5	19786
ToPA ${}^{t=3}_{p=5}$	5.5	1.8	15861

Appendix

Table 6: Efficiency comparison on Qwen3-4B for math reasoning. Time denotes the average wall-clock time per training step, and memory denotes the peak GPU memory consumption.

Method	#Param (M)	Time (s)	Memory (MB)
LoRA r=32	16.8	0.9	16494
DoRA r=32	17.0	1.3	17070
HiRA r=32	16.8	0.9	16083
OFT b=32	16.8	3.3	21814
BOFT ${}^{m=2}_{b=16}$	17.0	–	OOM
ToPA ${}^{t=2}_{p=16}$	16.7	1.6	18040

Appendix

Table 7: Efficiency comparison on LLaMA-2-7B for commonsense reasoning. Time denotes the average wall-clock time per training step, and memory denotes the peak GPU memory consumption. BOFT runs out of memory under the same settings without gradient checkpointing.

Dataset	MNLI	SST-2	CoLA	QQP	QNLI	RTE	MRPC	STS-B
# Epochs	8	15	30	15	5	40	40	25
Learning Rate	3e-4	1e-4	4e-4	1e-4	1e-4	2e-5	3e-4	8e-5
Max Seq. Len.	256	128	64	320	512	320	320	128

Appendix

Table 8: The hyperparameter settings on the GLUE benchmarks

Method	# Param	GSM8K	MATH	HumanEval	Mbpp
w/o Identity	12.6M	74.3	23.2	51.0	66.6
w/ Identity	12.6M	74.7	23.5	51.8	67.2

Appendix

Table 9: Performance comparison of LLaMA-3-8B fine-tuned with ToPA

{}_{p=8}^{t=3}

on math & code generation tasks, with and without the identity residual.

Panel of Figure 8 — Figure 8: Training dynamics over the first 300 steps of fine-tuning LLaMA-2-7B on commonsense reasoning tasks using the naive method ( $t=32$ ) and ToPA ${}^{t=2}_{p=16}$ with matched parameters. As training proceeds, the naive method collapses, so only the first 300 steps are shown.

Method	# Param	GSM8K	MATH	HumanEval	MBPP
LoRA r=32	13.6M	$70.5{\pm}0.4$	$22.5{\pm}0.3$	$50.2{\pm}0.3$	$\textbf{67.2}{\pm}0.3$
DoRA r=32	13.8M	$71.1{\pm}0.3$	$22.8{\pm}0.2$	$51.0{\pm}0.3$	$67.0{\pm}0.2$
OFT b=64	17.0M	$73.4{\pm}0.3$	$22.9{\pm}0.2$	$45.7{\pm}0.5$	$66.5{\pm}0.2$
BOFT ${}^{m=3}_{b=16}$	12.8M	$69.4{\pm}0.4$	$22.3{\pm}0.3$	$45.5{\pm}0.3$	$67.1{\pm}0.2$
ToPA ${}^{t=3}_{p=8}$	12.6M	$\textbf{74.9}{\pm}0.3$	$\textbf{23.5}{\pm}0.2$	$\textbf{51.6}{\pm}0.3$	$\textbf{67.2}{\pm}0.3$

Appendix

Table 10: Multi-seed performance comparison on math reasoning and code generation tasks with the LLaMA-3-8B backbone. Results are averaged over three random seeds.

Figure 9: Additional qualitative comparison of results on subject-driven generation tasks.

Figure 10: Additional qualitative comparison of results on subject-driven generation tasks.

Figure 11: Additional qualitative comparison of results on subject-driven generation tasks.

Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning adapts pretrained models by updating small parameter subsets or lightweight modules instead of all weights.

See all 610 papers

Parameter-Efficient Fine-Tuning with Learnable Rank
Jun 3, 2026Arpit Garg, Simon Lucey, Hemanth SaratchandranNonlinear LoRA Low-Rank Adaptation
Australian Institute for Machine Learning Adelaide University
TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models
Apr 15, 2026Yarui Cao, Kai LiuLLM Fine-Tuning Parameter-Efficient Fine-Tuning
Department of Computer Science, Clemson University, Clemson, SC 29634, USA
LoCO: Low-rank Compositional Rotation Fine-tuning
May 15, 2026An Nguyen, Jaesik Choi, Anh TongFine-Tuning LLM Fine-Tuning
1Korea University · 2KAIST · 3INEEJI

Jun 3, 2026J/K move · Enter open · S save

Parameter-Efficient Fine-Tuning with Learnable Rank

Arpit Garg, Simon Lucey, Hemanth Saratchandran

Australian Institute for Machine Learning Adelaide University

Nonlinear LoRA Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method that restricts weight updates to low-rank adapters, introducing a fixed low-rank inductive bias by optimizing in a low-dimensional subspace. In this work, we question whether a fixed-rank constraint is the most effective inductive bias for parameter-efficient fine-tuning. We introduce Learnable Rank LoRA (LR-LoRA), a PEFT method in which the adapter rank is learned during the training process. Instead of prescribing a uniform rank for all adapter layers, LR-LoRA allows the optimizer to determine the appropriate rank for each layer. Using this approach, we find substantial layer-wise variation in the learned ranks, with the attention and MLP layers in the transformer models exhibiting systematically different rank preferences. Across a range of language understanding and commonsense reasoning benchmarks, LR-LoRA achieves state-of-the-art performance in most settings and consistently outperforms strong PEFT baselines, demonstrating that a learnable rank provides a more flexible and effective inductive bias than fixed-rank adaptations.

Abstract

Parameter-Efficient Fine-Tuning

Explore similar work

Parameter-Efficient Fine-Tuning with Learnable Rank

TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models

LoCO: Low-rank Compositional Rotation Fine-tuning