Fine-Tuning of Transformer models with Frames

Harshavardhan Adepu; Li Zhang; Sanjiv Kumar; Vikas Singh

cs.AIAug 26, 2026

Fine-Tuning of Transformer models with Frames

Authors: Harshavardhan Adepu, Li Zhang, Sanjiv Kumar, Vikas Singh

Organizations: University of Wisconsin-Madison · Google DeepMind

Abstract

Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale with the size of the model, $\mathcal{O}(dr)$ , where $d$ is the model's hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $ΔW$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling very efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, reducing the memory footprint. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames give large compute benefits, and our analysis provides formal convergence results. We evaluate the idea across a suite of supervised fine-tuning benchmarks, focusing on language tasks, but also report application to vision models. Our experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.

Figures & tables

Figure 1: Examples of tight frames with

k=5

vectors in

\mathbb{R}^{3}

Figure 2: ( left ) How parameter update $\delta W$ is modeled by LoRA and by FrameFT. ( right ) The parameter update rule in FrameFT. We freeze the Fusion Frame projection matrices and train only the coefficients $C_{l}$ .

Figure 3: An example of the coefficient matrix $C_{l}$ from a Query layer (RoBERTa-L) with different subspace dimensions $\rho_{m}=\rho_{n}=\rho$ .

Model	Method	Params	SST-2	MRPC	CoLA	QNLI	RTE	STS-B	Avg.
RoBERTa Base	FF*	$125$ M	$94.8$	$90.2$	$63.6$	$\bm{92.8}$	$78.7$	$91.2$	$85.2$
	SMT	$55$ K	$94.2$	$89.5$	$63.9$	$91.8$	$78.1$	$90.4$	$84.7$
	LoRA*	$0.3$ M	$\bm{95.1}$	$89.7$	$63.4$	$93.3$	$78.4$	$\bm{91.5}$	$85.2$
	$\text{SVFT}^{R}_{d=2}$	$92$ K	$94.3$	$89.4$	$62.4$	$92.0$	$78.3$	$91.1$	$84.6$
	AdaLoRA*	$0.3$ M	$94.5$	$88.7$	$62.0$	$93.1$	$\bm{81.0}$	$90.5$	$85.0$
	FourierFT*	$24$ K	$94.2$	$90.0$	$63.8$	$92.2$	$79.1$	$90.8$	$85.0$

Table 1: Fine-Tuning RoBERTa Base and Large models on GLUE benchmark. * indicates the results reported in prior work. FrameFT performs better than full fine-tuning and LoRA, using

10\times

fewer parameters.

	Method	#params	#toks/sec
Llama-2-7b	LoRA	262k	23.6k
	DoRA	262k	14.1k
	SVFit	1k	9.0k
	FourierFT	1k	1.8k
	FrameFT	1k	$\bm{39.4k}$
Gemma-2-9b	LoRA	245k	22.1k

Table 2: Tokens per second for different PEFT methods.

Figure 4: Performance of RoBERTa base model fine-tuned with FrameFT versus the number of non-zero coefficients. The red line indicates the performance of full fine-tuning.

Model	Method	#Params	ARC-c	ARC-e	BoolQ	HellaSwag	OBQA	PIQA	RTE	WinoGrande	Avg.
Llama-2-7b	LoRA	$16.7$ M	$45.82$	$77.02$	$78.81$	$58.08$	$35.2$	$78.83$	$61.01$	$70.72$	$63.18$
	$\text{(IA)}^{3}$	614K	$45.38$	$76.60$	$77.89$	$57.74$	$34.6$	$78.78$	$64.26$	$68.82$	$63.01$
	DoRA	$16.7$ M	$45.14$	$76.39$	$78.35$	$57.78$	$34.0$	$78.73$	$67.15$	$67.17$	$63.09$
	$\text{S}^{2}{FT}$	$56.6$ M	$46.16$	$77.44$	$78.38$	$58.10$	$32.8$	$78.84$	$61.01$	$68.58$	$62.66$
	$\text{SVFT}^{B}_{d=8}$	$4.5$ M	$45.79$	$77.07$	$78.83$	$57.98$	$34.6$	$78.56$	$61.52$	$70.64$	$63.12$
	FourierFT	$320$ K	$44.96$	$77.14$	$79.05$	$58.21$	$34.6$	$78.89$	$62.45$	$70.48$	$63.22$

Table 3: Performance of LLMs fine-tuned on the Alpaca dataset by various methods and then evaluated on the LM-evaluation-harness. FrameFT performs competitively with all the baseline methods under comparison, using a

30\times

fewer number of parameters than LoRA. FrameFT needs minimal hyperparameter tuning.

Method	SST-2	MRPC	CoLA	QNLI	RTE	STS-B	Avg.
Shared random mask	$94.40{\pm}0.17$	$92.31{\pm}0.46$	$66.58{\pm}0.77$	$92.35{\pm}0.12$	$80.21{\pm}0.82$	$90.96{\pm}0.11$	$86.18{\pm}0.18$
Per-layer random mask	$94.10{\pm}0.16$	$93.53{\pm}0.32$	$66.44{\pm}0.61$	$92.31{\pm}0.14$	$80.34{\pm}0.39$	$90.86{\pm}0.09$	$86.26{\pm}0.17$
Magnitude-based mask	$94.19$	$92.49$	$65.82$	$91.61$	$78.40$	$91.01$	$85.60$
Gradient-based mask	$89.39$	$74.59$	$60.30$	$84.80$	$63.49$	$76.40$	$74.83$

Table 4: Sparsity mask ablation on RoBERTa-Base/GLUE (averaged over five seeds). Shared and per-layer random masks perform within noise of each other while adaptive heuristics perform strictly worse.

Metric	Method	Llama-2-7B	Llama-2-13B	Gemma-2-9B	Gemma-2-2B	Llama-3.1-8B
Training time (min)	LoRA	238	363	323	126	231
Training time (min)	FrameFT	225	345	301	115	212
GPU memory (GiB)	LoRA	30.1	34.8	37.8	26.9	38.2
GPU memory (GiB)	FrameFT	29.5	34.7	37.7	26.8	37.6

Table 5: Training time (minutes) and peak GPU memory (GiB) for LoRA and FrameFT on the Alpaca instruction tuning task.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Performance of FrameFT on CoLA and MRPC for different values of subspace dimension $\rho$

	Method	Params	Pets	Cars	CIFAR10	DTD	EuroSAT	FGVC	RESISC45	CIFAR100	Avg.
ViT-B	Full finetune*	$85.8$ M	$93.14$	$79.78$	$98.92$	$77.68$	$99.05$	$54.84$	$96.13$	$92.38$	$86.49$
	LinearProbe*	$-$	$90.28$	$25.76$	$96.41$	$69.77$	$88.72$	$17.44$	$74.22$	$84.28$	$68.36$
	LoRA*	$581$ K	$93.19$	$45.38$	$\bm{98.78}$	$74.95$	$98.44$	$25.16$	$92.70$	$\bm{92.02}$	$77.58$
	FourierFT	$72$ K	$93.21$	$46.11$	$98.58$	$75.09$	$98.29$	$27.51$	$91.97$	$91.20$	$77.75$
	FourierFT	$239$ K	$93.05$	$56.36$	$98.69$	$77.30$	$98.78$	$32.44$	$\bm{94.26}$	$91.45$	$80.29$
	FrameFT ( ours )	$24$ K	$\bm{93.71}$	$70.84$	$98.50$	$77.80$	$98.52$	$44.45$	$92.30$	$90.80$	$83.36$

Appendix

Table 6: Performance of different methods for Fine-Tuning ViT Base and Large models on different datasets. * indicates the results reported in previous work. We highlight non PEFT methods in gray . FrameFT performs better than all the baseline methods on average, using a 30x lower number of parameters than LoRA.

Figure 6: Performance of RoBERTa base model fine-tuned with FrameFT versus the number of non-zero coefficients on the GLUE benchmark. The red line indicates the performance of full fine-tuning.

Optimizer	AdamW
LR scheduler	Linear schedule with warmup
Batch size	32
Head learning rate	3E-3
Adapter learning rate	0.12
Warmup ratio	0.06
Max. Seq. length	512

Appendix

Table 7: Hyperparameters for the GLUE benchmark

Optimizer	AdamW
LR scheduler	Linear schedule with warmup
Batch size	50
Head learning rate	3E-2
Adapter learning rate	0.33
Warmup ratio	0.06
$\alpha$	250.0

Appendix

Table 8: Hyperparameters for the Image classification task

Optimizer	AdamW
LR scheduler	Linear schedule with warmup
Batch size	128
Learning rate	0.1
Warmup ratio	0.03
$\alpha$	200.0

Appendix

Table 9: Hyperparameters for the Instruction tuning task

\begin{bmatrix}[l]1&1&1&\frac{1}{\sqrt{3}}&\frac{1}{\sqrt{3}}&0&0&0&0&0&0\\ 1&\omega&\omega^{2}&\frac{\sqrt{1}}{\sqrt{3}}\omega^{3}&\frac{\sqrt{1}}{\sqrt{3}}\omega^{4}&0&0&0&0&0&0\\ 1&\omega^{2}&\omega^{4}&\frac{\sqrt{1}}{\sqrt{3}}&\frac{\sqrt{1}}{\sqrt{3}}\omega^{2}&0&0&0&0&0&0\\ 1&\omega^{3}&1&\frac{\sqrt{1}}{\sqrt{3}}\omega^{3}&\frac{\sqrt{1}}{\sqrt{3}}&0&0&0&0&0&0\\ 1&\omega^{4}&\omega^{2}&\frac{\sqrt{1}}{\sqrt{3}}&\frac{\sqrt{1}}{\sqrt{3}}\omega^{4}&0&0&0&0&0&0\\ 1&\omega^{5}&\omega^{4}&\frac{\sqrt{1}}{\sqrt{3}}\omega^{3}&\frac{\sqrt{1}}{\sqrt{3}}\omega^{2}&0&0&0&0&0&0\\ 0&0&0&\frac{\sqrt{2}}{\sqrt{3}}&-\frac{\sqrt{2}}{\sqrt{3}}&1&1&\frac{1}{\sqrt{6}}&\frac{1}{\sqrt{6}}&0&0\\ 0&0&0&\frac{\sqrt{2}}{\sqrt{3}}\omega^{3}&-\frac{\sqrt{2}}{\sqrt{3}}\omega^{4}&\omega^{5}&1&\frac{1}{\sqrt{6}}\omega&\frac{1}{\sqrt{6}}\omega^{2}&0&0\\ 0&0&0&\frac{\sqrt{2}}{\sqrt{3}}&-\frac{\sqrt{2}}{\sqrt{3}}\omega^{2}&\omega^{4}&1&\frac{1}{\sqrt{6}}\omega^{2}&\frac{1}{\sqrt{6}}\omega^{4}&0&0\\ 0&0&0&\frac{\sqrt{2}}{\sqrt{3}}\omega^{3}&-\frac{\sqrt{2}}{\sqrt{3}}&\omega^{3}&1&\frac{1}{\sqrt{6}}\omega^{3}&\frac{1}{\sqrt{6}}&0&0\\ 0&0&0&\frac{\sqrt{2}}{\sqrt{3}}&-\frac{\sqrt{2}}{\sqrt{3}}\omega^{4}&\omega^{2}&1&\frac{1}{\sqrt{6}}\omega^{4}&\frac{1}{\sqrt{6}}\omega^{2}&0&0\\ 0&0&0&\frac{\sqrt{2}}{\sqrt{3}}\omega^{3}&-\frac{\sqrt{2}}{\sqrt{3}}\omega^{2}&\omega^{1}&1&\frac{1}{\sqrt{6}}\omega^{5}&\frac{1}{\sqrt{6}}\omega^{4}&0&0\\ 0&0&0&0&0&0&0&\frac{5}{\sqrt{6}}&-\frac{5}{\sqrt{6}}&1&1\\ 0&0&0&0&0&0&0&\frac{5}{\sqrt{6}}\omega&-\frac{5}{\sqrt{6}}\omega^{2}&\omega^{3}&\omega^{4}\\ 0&0&0&0&0&0&0&\frac{5}{\sqrt{6}}\omega^{2}&-\frac{5}{\sqrt{6}}\omega^{4}&1&\omega^{2}\\ 0&0&0&0&0&0&0&\frac{5}{\sqrt{6}}\omega^{3}&-\frac{5}{\sqrt{6}}&\omega^{3}&1\\ 0&0&0&0&0&0&0&\frac{5}{\sqrt{6}}\omega^{4}&-\frac{5}{\sqrt{6}}\omega^{2}&1&\omega^{4}\\ 0&0&0&0&0&0&0&\frac{5}{\sqrt{6}}\omega^{5}&-\frac{5}{\sqrt{6}}\omega^{4}&\omega^{3}&\omega^{2}\\ \end{bmatrix}

Appendix

Table 10:

\bm{(6,3,11)}

-TFF for

\mathbb{C}^{11}

. Here,

\omega=e^{i\pi/3}

. A pair of rows belongs to the same subspace if their indices differ by a multiple of 6

Model	Method	Bases construction time (s)
Llama-2-7B	Gram-Schmidt	112.1
	QR decomposition	0.31
	Spectral Tetris	0.09
Gemma-2-9B	Gram-Schmidt	228.8
	QR decomposition	0.64
	Spectral Tetris	0.21

Appendix

Table 11: Basis generation time. Spectral Tetris takes a fraction of time to construct the Basis on an NVDIA A100 gpu

$\alpha$	ARC-c	ARC-e	BoolQ	HellaSwag	OBQA	PIQA	RTE	WinoGrande	Avg.
10	45.9	77.4	78.75	58.01	34.0	78.67	61.01	70.0	62.97
100	45.47	77.1	78.53	58.28	34.4	78.78	63.89	70.71	63.40
200 (default)	45.22	76.93	78.62	58.08	34.2	78.62	66.06	71.19	63.62
400	45.47	77.31	79.2	58.32	34.0	78.62	62.45	70.32	63.21
600	45.56	77.02	78.99	58.31	34.6	78.73	62.82	70.56	63.32
LoRA	45.82	77.02	78.81	58.08	35.2	78.83	61.01	70.72	63.18

Appendix

Table 12: Scaling factor

\alpha

sensitivity on Llama-2-7B/Alpaca. FrameFT outperforms LoRA (63.18) across the full sweep.

Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning adapts pretrained models by updating small parameter subsets or lightweight modules instead of all weights.

See all 610 papers

Apr 21, 2026

ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning

Popular low-rank parameter-efficient fine-tuning (PEFT) methods represent adaptation as separate updates to selected backbone weights, without maintaining an explicit task-specific state that is updated and reused across depth. These updates also require the backbone at inference and therefore cannot operate as standalone predictors. We propose ShadowPEFT, which consolidates trainable adaptation into a modular shadow component centered on a compact shadow model and lightweight Transformer layer-specific coupling modules. A persistent shadow state refines the frozen backbone representations and is updated from them in an interactive manner. Because the shadow model is trained as a complete predictor, it can be detached for shadow-only inference without executing the base model and can be initialized from a pretrained model. Experiments on text and image generation and understanding benchmarks show that ShadowPEFT matches or outperforms LoRA and DoRA under comparable trainable-parameter budgets. Additional analyses on shadow pretraining, cross-dataset transfer, parameter scaling, inference latency, and system-level evaluation suggest that centralized layer-space adaptation is a competitive and flexible alternative to conventional low-rank PEFT.

Xianming Li, Zongxi Li, Tsz-fung Andrew Lee +3

PolyU · School of Data Science, Lingnan University, Hong Kong SAR

Fine-Tuning Adapter Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Oct 5, 2026

RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning

Parameter-efficient fine-tuning (PEFT) reduces the cost of adapting foundation models by focusing training on a small parameter subset. Complementary to this idea, we introduce RoSA (Rotational Sparse Adaptation), which narrows adaptation to a subset of layers at a time. RoSA freezes lower layers close to the input throughout training and rotates a trainable block over later layers, progressively increasing the number of frozen layers close to the input. This design reduces optimizer-state memory, shortens backpropagation, and even forward propagation if activations at the last frozen layer are cached. Because RoSA is orthogonal to the choice of trainable parameterization, it can be combined with PEFT methods or sparse optimizers within each active block. Experiments across multiple LLM architectures and tasks show that RoSA reduces peak memory while maintaining strong fine-tuning performance.

Muhammad Azeem Lodhi, Chao Zhou, Rebekka Burkholz

Saarland University, Saarbrücken, Germany · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Memory-Efficient Fine-Tuning LLM Fine-Tuning Sparse Fine-Tuning Parameter-Efficient Fine-Tuning

Ask PDF

Aug 8, 2026

Circuit Fine-Tuning for Compute-Efficient Transformer Adaptation

Parameter-Efficient Fine-Tuning (PEFT) has become the de facto standard for adapting Vision Transformers (ViTs) to downstream tasks. While parameter count has been the dominant efficiency metric in PEFT, it does not imply compute efficiency: parameter-sparse methods can still incur full-model training cost per step, and typically need long schedules to reach peak accuracy. We introduce Circuit Fine-Tuning (CFT), a compute-efficient framework that uses circuit discovery---conventionally used to explain trained models---to select modules for fine-tuning before training. Whereas attribution is conventionally formulated against a trained task head, we formulate it against a near-zero-initialized probe head, which isolates the response of the backbone to the target distribution rather than the preferences of a particular classifier. CFT then fine-tunes only the recovered subgraph. CFT needs no learning-rate warmup and reaches peak accuracy in

{\sim}20

epochs on average---versus

44

96

for strong PEFT baselines---yielding

2.3

6.6\times

fewer training FLOPs and up to

16\times

less wall-clock time, while adding zero parameters and no inference operations. Experiments across a standard visual transfer benchmark (VTAB-1k), hierarchical backbones (Swin), domain-shifted medical imaging (CBIS-DDSM), and a vision-language model (Gemma-3 on CUB-200) demonstrate the effectiveness of CFT. Code is available at https://github.com/UriKialy/CFT

Uri Z. Kialy, Gil Ben-Artzi

School of Computer Science, Ariel University, Israel

Fine-Tuning Vision Foundation Model Adaptation Efficient ViTs Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Parameter-Efficient Fine-Tuning

Explore similar work

ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning

RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning

Circuit Fine-Tuning for Compute-Efficient Transformer Adaptation