Fine-Tuning of Transformer models with Frames

Harshavardhan Adepu; Li Zhang; Sanjiv Kumar; Vikas Singh

cs.AIAug 26, 2026

Fine-Tuning of Transformer models with Frames

Authors: Harshavardhan Adepu, Li Zhang, Sanjiv Kumar, Vikas Singh

Organizations: University of Wisconsin-Madison · Google DeepMind

Abstract

Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale with the size of the model, $\mathcal{O}(dr)$ , where $d$ is the model's hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $ΔW$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling very efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, reducing the memory footprint. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames give large compute benefits, and our analysis provides formal convergence results. We evaluate the idea across a suite of supervised fine-tuning benchmarks, focusing on language tasks, but also report application to vision models. Our experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.

Figures & tables

Figure 1: Examples of tight frames with

k=5

vectors in

\mathbb{R}^{3}

Figure 2: ( left ) How parameter update $\delta W$ is modeled by LoRA and by FrameFT. ( right ) The parameter update rule in FrameFT. We freeze the Fusion Frame projection matrices and train only the coefficients $C_{l}$ .

Figure 3: An example of the coefficient matrix $C_{l}$ from a Query layer (RoBERTa-L) with different subspace dimensions $\rho_{m}=\rho_{n}=\rho$ .

Model	Method	Params	SST-2	MRPC	CoLA	QNLI	RTE	STS-B	Avg.
RoBERTa Base	FF*	$125$ M	$94.8$	$90.2$	$63.6$	$\bm{92.8}$	$78.7$	$91.2$	$85.2$
	SMT	$55$ K	$94.2$	$89.5$	$63.9$	$91.8$	$78.1$	$90.4$	$84.7$
	LoRA*	$0.3$ M	$\bm{95.1}$	$89.7$	$63.4$	$93.3$	$78.4$	$\bm{91.5}$	$85.2$
	$\text{SVFT}^{R}_{d=2}$	$92$ K	$94.3$	$89.4$	$62.4$	$92.0$	$78.3$	$91.1$	$84.6$
	AdaLoRA*	$0.3$ M	$94.5$	$88.7$	$62.0$	$93.1$	$\bm{81.0}$	$90.5$	$85.0$
	FourierFT*	$24$ K	$94.2$	$90.0$	$63.8$	$92.2$	$79.1$	$90.8$	$85.0$

Table 1: Fine-Tuning RoBERTa Base and Large models on GLUE benchmark. * indicates the results reported in prior work. FrameFT performs better than full fine-tuning and LoRA, using

10\times

fewer parameters.

	Method	#params	#toks/sec
Llama-2-7b	LoRA	262k	23.6k
	DoRA	262k	14.1k
	SVFit	1k	9.0k
	FourierFT	1k	1.8k
	FrameFT	1k	$\bm{39.4k}$
Gemma-2-9b	LoRA	245k	22.1k

Table 2: Tokens per second for different PEFT methods.

Figure 4: Performance of RoBERTa base model fine-tuned with FrameFT versus the number of non-zero coefficients. The red line indicates the performance of full fine-tuning.

Model	Method	#Params	ARC-c	ARC-e	BoolQ	HellaSwag	OBQA	PIQA	RTE	WinoGrande	Avg.
Llama-2-7b	LoRA	$16.7$ M	$45.82$	$77.02$	$78.81$	$58.08$	$35.2$	$78.83$	$61.01$	$70.72$	$63.18$
	$\text{(IA)}^{3}$	614K	$45.38$	$76.60$	$77.89$	$57.74$	$34.6$	$78.78$	$64.26$	$68.82$	$63.01$
	DoRA	$16.7$ M	$45.14$	$76.39$	$78.35$	$57.78$	$34.0$	$78.73$	$67.15$	$67.17$	$63.09$
	$\text{S}^{2}{FT}$	$56.6$ M	$46.16$	$77.44$	$78.38$	$58.10$	$32.8$	$78.84$	$61.01$	$68.58$	$62.66$
	$\text{SVFT}^{B}_{d=8}$	$4.5$ M	$45.79$	$77.07$	$78.83$	$57.98$	$34.6$	$78.56$	$61.52$	$70.64$	$63.12$
	FourierFT	$320$ K	$44.96$	$77.14$	$79.05$	$58.21$	$34.6$	$78.89$	$62.45$	$70.48$	$63.22$

Table 3: Performance of LLMs fine-tuned on the Alpaca dataset by various methods and then evaluated on the LM-evaluation-harness. FrameFT performs competitively with all the baseline methods under comparison, using a

30\times

fewer number of parameters than LoRA. FrameFT needs minimal hyperparameter tuning.

Method	SST-2	MRPC	CoLA	QNLI	RTE	STS-B	Avg.
Shared random mask	$94.40{\pm}0.17$	$92.31{\pm}0.46$	$66.58{\pm}0.77$	$92.35{\pm}0.12$	$80.21{\pm}0.82$	$90.96{\pm}0.11$	$86.18{\pm}0.18$
Per-layer random mask	$94.10{\pm}0.16$	$93.53{\pm}0.32$	$66.44{\pm}0.61$	$92.31{\pm}0.14$	$80.34{\pm}0.39$	$90.86{\pm}0.09$	$86.26{\pm}0.17$
Magnitude-based mask	$94.19$	$92.49$	$65.82$	$91.61$	$78.40$	$91.01$	$85.60$
Gradient-based mask	$89.39$	$74.59$	$60.30$	$84.80$	$63.49$	$76.40$	$74.83$

Table 4: Sparsity mask ablation on RoBERTa-Base/GLUE (averaged over five seeds). Shared and per-layer random masks perform within noise of each other while adaptive heuristics perform strictly worse.

Metric	Method	Llama-2-7B	Llama-2-13B	Gemma-2-9B	Gemma-2-2B	Llama-3.1-8B
Training time (min)	LoRA	238	363	323	126	231
Training time (min)	FrameFT	225	345	301	115	212
GPU memory (GiB)	LoRA	30.1	34.8	37.8	26.9	38.2
GPU memory (GiB)	FrameFT	29.5	34.7	37.7	26.8	37.6

Table 5: Training time (minutes) and peak GPU memory (GiB) for LoRA and FrameFT on the Alpaca instruction tuning task.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Performance of FrameFT on CoLA and MRPC for different values of subspace dimension $\rho$

	Method	Params	Pets	Cars	CIFAR10	DTD	EuroSAT	FGVC	RESISC45	CIFAR100	Avg.
ViT-B	Full finetune*	$85.8$ M	$93.14$	$79.78$	$98.92$	$77.68$	$99.05$	$54.84$	$96.13$	$92.38$	$86.49$
	LinearProbe*	$-$	$90.28$	$25.76$	$96.41$	$69.77$	$88.72$	$17.44$	$74.22$	$84.28$	$68.36$
	LoRA*	$581$ K	$93.19$	$45.38$	$\bm{98.78}$	$74.95$	$98.44$	$25.16$	$92.70$	$\bm{92.02}$	$77.58$
	FourierFT	$72$ K	$93.21$	$46.11$	$98.58$	$75.09$	$98.29$	$27.51$	$91.97$	$91.20$	$77.75$
	FourierFT	$239$ K	$93.05$	$56.36$	$98.69$	$77.30$	$98.78$	$32.44$	$\bm{94.26}$	$91.45$	$80.29$
	FrameFT ( ours )	$24$ K	$\bm{93.71}$	$70.84$	$98.50$	$77.80$	$98.52$	$44.45$	$92.30$	$90.80$	$83.36$

Appendix

Table 6: Performance of different methods for Fine-Tuning ViT Base and Large models on different datasets. * indicates the results reported in previous work. We highlight non PEFT methods in gray . FrameFT performs better than all the baseline methods on average, using a 30x lower number of parameters than LoRA.

Figure 6: Performance of RoBERTa base model fine-tuned with FrameFT versus the number of non-zero coefficients on the GLUE benchmark. The red line indicates the performance of full fine-tuning.

Optimizer	AdamW
LR scheduler	Linear schedule with warmup
Batch size	32
Head learning rate	3E-3
Adapter learning rate	0.12
Warmup ratio	0.06
Max. Seq. length	512

Appendix

Table 7: Hyperparameters for the GLUE benchmark

Optimizer	AdamW
LR scheduler	Linear schedule with warmup
Batch size	50
Head learning rate	3E-2
Adapter learning rate	0.33
Warmup ratio	0.06
$\alpha$	250.0

Appendix

Table 8: Hyperparameters for the Image classification task

Optimizer	AdamW
LR scheduler	Linear schedule with warmup
Batch size	128
Learning rate	0.1
Warmup ratio	0.03
$\alpha$	200.0

Appendix

Table 9: Hyperparameters for the Instruction tuning task

\begin{bmatrix}[l]1&1&1&\frac{1}{\sqrt{3}}&\frac{1}{\sqrt{3}}&0&0&0&0&0&0\\ 1&\omega&\omega^{2}&\frac{\sqrt{1}}{\sqrt{3}}\omega^{3}&\frac{\sqrt{1}}{\sqrt{3}}\omega^{4}&0&0&0&0&0&0\\ 1&\omega^{2}&\omega^{4}&\frac{\sqrt{1}}{\sqrt{3}}&\frac{\sqrt{1}}{\sqrt{3}}\omega^{2}&0&0&0&0&0&0\\ 1&\omega^{3}&1&\frac{\sqrt{1}}{\sqrt{3}}\omega^{3}&\frac{\sqrt{1}}{\sqrt{3}}&0&0&0&0&0&0\\ 1&\omega^{4}&\omega^{2}&\frac{\sqrt{1}}{\sqrt{3}}&\frac{\sqrt{1}}{\sqrt{3}}\omega^{4}&0&0&0&0&0&0\\ 1&\omega^{5}&\omega^{4}&\frac{\sqrt{1}}{\sqrt{3}}\omega^{3}&\frac{\sqrt{1}}{\sqrt{3}}\omega^{2}&0&0&0&0&0&0\\ 0&0&0&\frac{\sqrt{2}}{\sqrt{3}}&-\frac{\sqrt{2}}{\sqrt{3}}&1&1&\frac{1}{\sqrt{6}}&\frac{1}{\sqrt{6}}&0&0\\ 0&0&0&\frac{\sqrt{2}}{\sqrt{3}}\omega^{3}&-\frac{\sqrt{2}}{\sqrt{3}}\omega^{4}&\omega^{5}&1&\frac{1}{\sqrt{6}}\omega&\frac{1}{\sqrt{6}}\omega^{2}&0&0\\ 0&0&0&\frac{\sqrt{2}}{\sqrt{3}}&-\frac{\sqrt{2}}{\sqrt{3}}\omega^{2}&\omega^{4}&1&\frac{1}{\sqrt{6}}\omega^{2}&\frac{1}{\sqrt{6}}\omega^{4}&0&0\\ 0&0&0&\frac{\sqrt{2}}{\sqrt{3}}\omega^{3}&-\frac{\sqrt{2}}{\sqrt{3}}&\omega^{3}&1&\frac{1}{\sqrt{6}}\omega^{3}&\frac{1}{\sqrt{6}}&0&0\\ 0&0&0&\frac{\sqrt{2}}{\sqrt{3}}&-\frac{\sqrt{2}}{\sqrt{3}}\omega^{4}&\omega^{2}&1&\frac{1}{\sqrt{6}}\omega^{4}&\frac{1}{\sqrt{6}}\omega^{2}&0&0\\ 0&0&0&\frac{\sqrt{2}}{\sqrt{3}}\omega^{3}&-\frac{\sqrt{2}}{\sqrt{3}}\omega^{2}&\omega^{1}&1&\frac{1}{\sqrt{6}}\omega^{5}&\frac{1}{\sqrt{6}}\omega^{4}&0&0\\ 0&0&0&0&0&0&0&\frac{5}{\sqrt{6}}&-\frac{5}{\sqrt{6}}&1&1\\ 0&0&0&0&0&0&0&\frac{5}{\sqrt{6}}\omega&-\frac{5}{\sqrt{6}}\omega^{2}&\omega^{3}&\omega^{4}\\ 0&0&0&0&0&0&0&\frac{5}{\sqrt{6}}\omega^{2}&-\frac{5}{\sqrt{6}}\omega^{4}&1&\omega^{2}\\ 0&0&0&0&0&0&0&\frac{5}{\sqrt{6}}\omega^{3}&-\frac{5}{\sqrt{6}}&\omega^{3}&1\\ 0&0&0&0&0&0&0&\frac{5}{\sqrt{6}}\omega^{4}&-\frac{5}{\sqrt{6}}\omega^{2}&1&\omega^{4}\\ 0&0&0&0&0&0&0&\frac{5}{\sqrt{6}}\omega^{5}&-\frac{5}{\sqrt{6}}\omega^{4}&\omega^{3}&\omega^{2}\\ \end{bmatrix}

Appendix

Table 10:

\bm{(6,3,11)}

-TFF for

\mathbb{C}^{11}

. Here,

\omega=e^{i\pi/3}

. A pair of rows belongs to the same subspace if their indices differ by a multiple of 6

Model	Method	Bases construction time (s)
Llama-2-7B	Gram-Schmidt	112.1
	QR decomposition	0.31
	Spectral Tetris	0.09
Gemma-2-9B	Gram-Schmidt	228.8
	QR decomposition	0.64
	Spectral Tetris	0.21

Appendix

Table 11: Basis generation time. Spectral Tetris takes a fraction of time to construct the Basis on an NVDIA A100 gpu

$\alpha$	ARC-c	ARC-e	BoolQ	HellaSwag	OBQA	PIQA	RTE	WinoGrande	Avg.
10	45.9	77.4	78.75	58.01	34.0	78.67	61.01	70.0	62.97
100	45.47	77.1	78.53	58.28	34.4	78.78	63.89	70.71	63.40
200 (default)	45.22	76.93	78.62	58.08	34.2	78.62	66.06	71.19	63.62
400	45.47	77.31	79.2	58.32	34.0	78.62	62.45	70.32	63.21
600	45.56	77.02	78.99	58.31	34.6	78.73	62.82	70.56	63.32
LoRA	45.82	77.02	78.81	58.08	35.2	78.83	61.01	70.72	63.18

Appendix

Table 12: Scaling factor

\alpha

sensitivity on Llama-2-7B/Alpaca. FrameFT outperforms LoRA (63.18) across the full sweep.

Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning adapts pretrained models by updating small parameter subsets or lightweight modules instead of all weights.

See all 610 papers

ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning
Apr 21, 2026Xianming Li, Zongxi Li, Tsz-fung Andrew Lee +3Fine-Tuning Adapter Tuning
PolyU · School of Data Science, Lingnan University, Hong Kong SAR
RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning
Oct 5, 2026Muhammad Azeem Lodhi, Chao Zhou, Rebekka BurkholzMemory-Efficient Fine-Tuning LLM Fine-Tuning
Saarland University, Saarbrücken, Germany · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
Circuit Fine-Tuning for Compute-Efficient Transformer Adaptation
Aug 8, 2026Uri Z. Kialy, Gil Ben-ArtziFine-Tuning Vision Foundation Model Adaptation
School of Computer Science, Ariel University, Israel

Apr 21, 2026J/K move · Enter open · S save

ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning

Xianming Li, Zongxi Li, Tsz-fung Andrew Lee, Jing Li+2

PolyU · School of Data Science, Lingnan University, Hong Kong SAR

Fine-Tuning Adapter Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Popular low-rank parameter-efficient fine-tuning (PEFT) methods represent adaptation as separate updates to selected backbone weights, without maintaining an explicit task-specific state that is updated and reused across depth. These updates also require the backbone at inference and therefore cannot operate as standalone predictors. We propose ShadowPEFT, which consolidates trainable adaptation into a modular shadow component centered on a compact shadow model and lightweight Transformer layer-specific coupling modules. A persistent shadow state refines the frozen backbone representations and is updated from them in an interactive manner. Because the shadow model is trained as a complete predictor, it can be detached for shadow-only inference without executing the base model and can be initialized from a pretrained model. Experiments on text and image generation and understanding benchmarks show that ShadowPEFT matches or outperforms LoRA and DoRA under comparable trainable-parameter budgets. Additional analyses on shadow pretraining, cross-dataset transfer, parameter scaling, inference latency, and system-level evaluation suggest that centralized layer-space adaptation is a competitive and flexible alternative to conventional low-rank PEFT.

Abstract

Parameter-Efficient Fine-Tuning

Explore similar work

ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning

RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning

Circuit Fine-Tuning for Compute-Efficient Transformer Adaptation