Are Parameter-Efficient Fine-tuning Methods Really Different?

Yikuan Li; Pinyan Lu; Fanghui Liu

cs.LGOct 6, 2026

Are Parameter-Efficient Fine-tuning Methods Really Different?

Authors: Yikuan Li, Pinyan Lu, Fanghui Liu

Organizations: Department of Statistics, The University of Warwick, UK. · Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics, China. · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China.

Abstract

Parameter-efficient fine-tuning (PEFT) offers many parameterizations, yet their methodological and functional differences remain unclear. We compare six methods in language and diffusion models to examine how their parameterizations relate to task performance, forgetting, and changes in pretrained weight geometry. Motivated by the spectrum-preserving design of orthogonal fine-tuning (OFT), we first ask whether spectral preservation is itself important for adaptation and retention. We find that the selected LoRA-family methods also approximately preserve pretrained geometry, and that restoring their slightly drifted singular-value spectra largely preserves task performance, questioning the necessity of explicit geometric preservation. Beyond this, we observe that some methods exhibit distinct adaptation--retention trade-offs that vary across settings: LoRA most consistently limits forgetting at competitive performance, DoRA achieves higher mean task scores than LoRA in most comparisons, while PiSSA often incurs greater retention costs. Further intervention experiments suggest that while performance gains from different PEFT methods can be attributed to modifications in different groups of spectral components, we consistently find that restoring dominant rather than intermediate or trailing components produces the largest mean reduction in general-text NLL or base-image drift. Together, these results motivate evaluating geometric constraints through their functional consequences rather than preservation alone. Code is available at https://github.com/Kuaaannn/PEFT_methods.

Figures & tables

Figure 1: Overview of the questions studied in this work. (a) Six representative PEFT parameterizations. (b) Changes in pretrained singular spectra and the singular-value restoration intervention. (c) Performance–retention comparisons across PEFT methods. (d) Spectral interventions that selectively retain or restore dominant, intermediate, and trailing components. Singular-value decay curves and bar heights are schematic.

Figure 2: Experimental controls documented in PEFT method papers. (a) Four controls in the 64 eligible papers. (b) Components of comparable conditions, excluding inapplicable cases. Each criterion counts a paper once. Denominators appear in parentheses. Bar lengths and labels show paper counts. A control is documented when one eligible main comparison meets all applicable requirements. An explicit departure requires evidence that every eligible comparison fails the definition. Unclear covers missing or conflicting evidence and unresolved comparison coverage. Component positives may come from different comparisons. Definitions and coverage summaries appear in Appendix F .

Method	Adapted matrix $\bm{W}_{*}$	Initialization or structure
LoRA	$\bm{W}_{0}+s\bm{B}\bm{A}$	Zero initial update
DoRA	$\operatorname{diag}(\bm{m})\mathcal{N}(\bm{W}_{0}+s\bm{B}\bm{A})$	Learned magnitudes and directions
PiSSA	$\bm{W}_{\mathrm{res}}+s\bm{B}\bm{A}$	Leading singular components
MiLoRA	$\bm{W}_{\mathrm{res}}+s\bm{B}\bm{A}$	Trailing singular components
OFT	$\bm{W}_{0}\bm{R}$	$\bm{R}=\operatorname{blkdiag}(\bm{R}_{j})$
HRA	$\bm{W}_{0}\bm{R}$	$\bm{R}=\prod_{i=1}^{r_{H}}(\bm{I}-2\bm{u}_{i}\bm{u}_{i}^{\!\top})$

Table 1: Selected PEFT methods and their parameterizations.

Figure 3: Task performance largely survives singular-value restoration. At the largest capacities and development/validation-selected LRs, pale and saturated bars show trained and restored scores with sample SD error bars. LLM panels report development accuracy, and FLUX reports same-run test DINO for all six methods.

	Qwen2.5-7B			Llama-3.1-8B
Method	S	M	L	S	M	L
LoRA	$83.07_{\pm 0.54}$	$83.22_{\pm 0.16}$	$83.65_{\pm 0.42}$	$70.84_{\pm 1.09}$	$\bm{71.65}_{\pm 0.35}$	$71.90_{\pm 0.12}$
OFT	$83.35_{\pm 0.36}$	$82.92_{\pm 0.46}$	$83.50_{\pm 0.27}$	$\bm{71.42}_{\pm 1.45}$	$71.34_{\pm 0.13}$	$71.75_{\pm 1.18}$
DoRA	$83.27_{\pm 0.04}$	$82.64_{\pm 0.59}$	$83.12_{\pm 0.53}$	$69.95_{\pm 0.12}$	$70.91_{\pm 0.44}$	$\bm{71.97}_{\pm 1.16}$
PiSSA	$\bm{83.85}_{\pm 0.47}$	$\bm{83.67}_{\pm 0.23}$	$83.17_{\pm 0.20}$	$70.33_{\pm 1.49}$	$69.98_{\pm 0.27}$	$70.26_{\pm 0.98}$
MiLoRA	$83.35_{\pm 0.24}$	$\bm{83.67}_{\pm 0.46}$	$\bm{83.73}_{\pm 0.12}$	$70.43_{\pm 0.42}$	$71.06_{\pm 0.61}$	$70.68_{\pm 0.23}$

Table 2: Development-selected LLM test scores (%) and FineWiki forgetting (

10^{3}\Delta\mathrm{NLL}

). Budgets are approximately matched: S/M/L for GSM8K; one for CodeFeedback. HE+ denotes HumanEval+. Base is one unadapted evaluation. Shading marks each column's best adapted mean, including ties.

Figure 4: Development performance–retention envelopes across LRs and capacities. Shapes denote increasing capacity: circle, square, triangle, then diamond (FLUX only). Faint points are dominated within their method. Line styles identify method frontiers. Black borders mark the shared frontier across methods. Each icon counts one configuration in the full grid (89 Qwen, 90 Llama, 192 FLUX). Insets magnify boxed regions with uniform point styling. (a,b) Dev accuracy versus FineWiki

\Delta

NLL. (c) Cat validation DINO versus general-image drift.

Budget	Metric	LoRA	OFT	DoRA	PiSSA	MiLoRA	HRA
r4/b32	DINO $\uparrow$	$68.90_{\pm 0.87}$	$67.43_{\pm 4.54}$	$\bm{69.92}_{\pm 2.05}$	$68.29_{\pm 3.41}$	$66.51_{\pm 1.07}$	$68.40_{\pm 1.09}$
	Drift $\downarrow$	$\bm{14.23}_{\pm 1.01}$	$27.66_{\pm 4.85}$	$16.93_{\pm 3.89}$	$18.63_{\pm 4.99}$	$15.89_{\pm 3.13}$	$14.83_{\pm 4.11}$
	CLIP $\uparrow$	$27.00_{\pm 1.24}$	$24.88_{\pm 1.90}$	$26.81_{\pm 0.62}$	$\bm{27.80}_{\pm 1.04}$	$26.94_{\pm 1.39}$	$26.95_{\pm 2.05}$
r8/b64	DINO $\uparrow$	$68.48_{\pm 1.29}$	$68.90_{\pm 1.03}$	$68.77_{\pm 2.44}$	$69.41_{\pm 1.43}$	$65.87_{\pm 2.53}$	$\bm{70.10}_{\pm 2.95}$
	Drift $\downarrow$	$14.38_{\pm 0.68}$	$15.03_{\pm 3.55}$	$14.77_{\pm 0.57}$	$18.32_{\pm 2.22}$	$14.58_{\pm 4.24}$	$\bm{13.47}_{\pm 4.26}$
	CLIP $\uparrow$	$28.09_{\pm 1.48}$	$27.88_{\pm 2.43}$	$27.40_{\pm 2.09}$	$28.04_{\pm 0.55}$	$\bm{28.25}_{\pm 0.21}$	$26.63_{\pm 1.47}$

Table 3: FLUX-4B test/retention scores. DINO and general drift/CLIP are

\times 100

; COCO loss is unscaled. Panel (b) averages objects per seed. Shaded bold means mark the best method per budget/metric in (a), or capacity pair in (b). Budgets list rank/block sizes; HRA uses 16/32/62/124 vectors.

Figure 5: Dominant-component restoration consistently reduces forgetting, with varying effects on task performance. Cells show mean

\pm

SD changes from trained models at the largest capacities and selected LRs. Higher task scores and lower NLL/drift are better. FLUX DINO/drift are scaled by 100. Bold marks the largest absolute mean change across component groups per method.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Adaptation	Primary retention view
Qwen2.5-7B	MetaMath / six methods	General-text NLL
Llama-3.1-8B	MetaMath / six methods	General-text NLL
Qwen2.5-7B	CodeFeedback / six methods	General-text NLL
Llama-3.1-8B	CodeFeedback / six methods	General-text NLL
FLUX-4B	Cat / six methods	Image drift / CLIP
FLUX-4B	28 objects	Flow loss / drift / CLIP

Appendix

Table 4: Study overview. Each comparison uses a shared pretrained backbone. Divergent runs are reported with the corresponding results.

Adapter	Params (M)	LR	Dev $\uparrow$	Test $\uparrow$	$\Delta$ NLL $\downarrow$
LoRA r7	17.66	$1\times 10^{-3}$	$90.73_{\pm 0.35}$	$83.07_{\pm 0.54}$	$0.0237_{\pm 0.0012}$
LoRA r14	35.32	$1\times 10^{-5}$	$91.40_{\pm 0.44}$	$83.22_{\pm 0.16}$	$0.0020_{\pm 0.0001}$
LoRA r28	70.65	$3\times 10^{-5}$	$92.30_{\pm 0.46}$	$83.65_{\pm 0.42}$	$0.0038_{\pm 0.0002}$
OFT b32	17.55	$3\times 10^{-4}$	$91.37_{\pm 0.25}$	$83.35_{\pm 0.36}$	$0.0105_{\pm 0.0009}$
OFT b64	35.68	$3\times 10^{-4}$	$89.97_{\pm 0.38}$	$82.92_{\pm 0.46}$	$0.0182_{\pm 0.0008}$
OFT b128	71.92	$1\times 10^{-4}$	$90.90_{\pm 0.69}$	$83.50_{\pm 0.27}$	$0.0088_{\pm 0.0007}$

Appendix

Table 5: Qwen2.5-7B results for all development-selected configurations from the original five-LR grid. Parameters are in millions. Ranks are training ranks (HRA counts Householder vectors). Accuracy is in percent, and

\Delta

NLL is the recorded FineWiki-200 NLL minus its matching pretrained value. LRs were selected using development accuracy only.

Adapter	Params (M)	LR	Dev $\uparrow$	Test $\uparrow$	$\Delta$ NLL $\downarrow$
LoRA r7	18.35	$3\times 10^{-4}$	$71.17_{\pm 0.29}$	$70.84_{\pm 1.09}$	$0.0285_{\pm 0.0018}$
LoRA r15	39.32	$3\times 10^{-4}$	$72.30_{\pm 1.04}$	$71.65_{\pm 0.35}$	$0.0372_{\pm 0.0023}$
LoRA r30	78.64	$3\times 10^{-4}$	$71.93_{\pm 1.12}$	$71.90_{\pm 0.12}$	$0.0486_{\pm 0.0024}$
OFT b32	19.30	$3\times 10^{-4}$	$72.10_{\pm 1.31}$	$71.42_{\pm 1.45}$	$0.0371_{\pm 0.0015}$
OFT b64	39.22	$3\times 10^{-4}$	$73.10_{\pm 0.26}$	$71.34_{\pm 0.13}$	$0.0740_{\pm 0.0017}$
OFT b128	79.07	$1\times 10^{-4}$	$72.57_{\pm 0.71}$	$71.75_{\pm 1.18}$	$0.0366_{\pm 0.0011}$

Appendix

Table 6: Llama-3.1-8B results for all development-selected configurations from the original five-LR grid. Parameters are in millions. Ranks are training ranks (HRA counts Householder vectors). Accuracy is in percent, and

\Delta

NLL is the recorded FineWiki-200 NLL minus its matching pretrained value. LRs were selected using development accuracy only.

Figure 6: Six-method mathematics trade-offs, focused on the frontier regions. Each point is a complete capacity/LR configuration, with shapes denoting increasing capacity. Dashed lines and solid points mark within-method frontiers, and black borders mark the pooled frontier. Dominated points are faint. Frontiers use all 89 Qwen and 90 Llama configurations before axis clipping. The later HRA extension and diverged PiSSA configuration are excluded.

Adapter	LR	$n$	Dev (%) $\uparrow$	Test (%) $\uparrow$	$\Delta$ NLL $\downarrow$
HRA r16	$3\times 10^{-3}$	3	$70.47_{\pm 0.42}$	$70.15_{\pm 0.27}$	$0.0553_{\pm 0.0028}$
HRA r32	$3\times 10^{-3}$	3	$70.87_{\pm 0.90}$	$71.37_{\pm 1.20}$	$0.1132_{\pm 0.0060}$
HRA r64	$1\times 10^{-3}$	3	$71.47_{\pm 1.01}$	$71.04_{\pm 0.20}$	$0.0988_{\pm 0.0031}$

Appendix

Table 7: Llama-3.1-8B HRA development-selected outcomes after the separate seven-LR follow-up. The r64 selection is unchanged from the original grid.

n

is the successful seed count. LR selection uses development scores.

Model	Method	Selected LR	HumanEval	HE+	MBPP	MBPP+
Qwen7B	Base	—	$58.54$	$51.22$	$76.72$	$64.02$
	LoRA	$1\times 10^{-4}$	$59.96_{\pm 0.70}$	$52.24_{\pm 0.35}$	$76.90_{\pm 0.40}$	$64.29_{\pm 0.53}$
	OFT	$1\times 10^{-4}$	$58.94_{\pm 0.70}$	$51.83_{\pm 0.61}$	$77.43_{\pm 0.55}$	$64.55_{\pm 0.53}$
	DoRA	$1\times 10^{-4}$	$59.96_{\pm 0.93}$	$52.64_{\pm 0.70}$	$76.63_{\pm 1.00}$	$64.37_{\pm 0.61}$
	PiSSA	$1\times 10^{-5}$	$59.15_{\pm 1.06}$	$52.44_{\pm 1.61}$	$76.63_{\pm 0.40}$	$63.93_{\pm 0.61}$
	MiLoRA	$1\times 10^{-4}$	$58.94_{\pm 0.35}$	$51.02_{\pm 0.93}$	$76.63_{\pm 0.55}$	$64.02_{\pm 0.53}$

Appendix

Table 8: CodeFeedback results for all original and strengthened-test pass@1 scores (%). A single greedy completion is graded under both test suites. LR minimizes mean held-out response NLL. Only selected checkpoints have test evaluations. The Base row has no training-seed SD.

Method	Capacity	Params	LR	Validation	Test DINO	Gen. drift	Gen. CLIP
LoRA	r4	4.79	$3\times 10^{-4}$	$73.48_{\pm 2.90}$	$68.90_{\pm 0.87}$	$14.23_{\pm 1.01}$	$27.00_{\pm 1.24}$
OFT	b32	4.76	$3\times 10^{-4}$	$74.58_{\pm 2.56}$	$67.43_{\pm 4.54}$	$27.66_{\pm 4.85}$	$24.88_{\pm 1.90}$
DoRA	r4	5.68	$5\times 10^{-4}$	$71.87_{\pm 2.31}$	$69.92_{\pm 2.05}$	$16.93_{\pm 3.89}$	$26.81_{\pm 0.62}$
PiSSA	r4	4.79	$1\times 10^{-4}$	$73.93_{\pm 1.62}$	$68.29_{\pm 3.41}$	$18.63_{\pm 4.99}$	$27.80_{\pm 1.04}$
MiLoRA	r4	4.79	$1\times 10^{-4}$	$71.79_{\pm 5.90}$	$66.51_{\pm 1.07}$	$15.89_{\pm 3.13}$	$26.94_{\pm 1.39}$
HRA	h16	4.92	$5\times 10^{-4}$	$72.99_{\pm 1.85}$	$68.40_{\pm 1.09}$	$14.83_{\pm 4.11}$	$26.95_{\pm 2.05}$

Appendix

Table 9: Complete validation-selected FLUX cat results. Budgets are approximately matched, with HRA using 16/32/62/124 vectors and DoRA adding magnitude parameters. Parameters are in millions. Validation DINO at step 700 selects LR separately for each method/capacity. Step 750 supplies test DINO, general drift and general CLIP. Scores are

\times 100

Figure 7: Six-method FLUX cat test trade-offs, focused on the frontier regions. Shapes denote increasing capacity. Dashed lines and solid points mark within-method frontiers, and black borders mark the pooled frontier. Dominated points are faint. Frontiers use all 192 configurations before axis clipping, with separate y ranges for each panel. Test DINO describes final-step outcomes, while validation DINO selects LR.

	Original selection		LR chosen on other seeds
Method	Lowest forgetting	Average rank	Lowest forgetting	Average rank
LoRA	9/12	1.58	8/12	2.08
OFT	1/12	4.58	1/12	4.33
DoRA	0/12	2.75	1/12	2.50
PiSSA	0/12	5.25	0/12	5.42
MiLoRA	1/12	3.42	1/12	3.25

Appendix

Table 10: Forgetting across all six methods. The lowest-forgetting columns count settings in which a method has the lowest mean forgetting. A rank of 1 means least forgetting, and lower average rank is better. The right columns choose LR using the other two seeds, evaluate the remaining seed, then average those evaluations. LoRA has the lowest forgetting in the most settings and the lowest average rank under both selections.

	Original selection		LR chosen on other seeds
Check	Frequency	Average rank	Frequency	Average rank
Equal model/task weights	1/1	1/1	1/1	1/1
Remove one backbone	3/3	3/3	3/3	2/3
Remove one setting	12/12	12/12	12/12	12/12
Use individual seeds	9/9	9/9	9/9	6/9

Appendix

Table 11: Robustness of the retention comparison. Frequency columns count versions in which LoRA has the lowest forgetting in more settings than any other method. Average-rank columns count versions in which LoRA has the lowest average rank. Equal weighting averages within the five model/task groups before averaging groups. The nine seed combinations cross all language seeds with all image seeds.

Method	Capacity	LR	$100d_{W}$	$100d_{S}$	$100d_{S}/d_{W}$
LoRA	7	$1\times 10^{-3}$	$8.154_{\pm 0.010}$	$1.67437_{\pm 0.01900}$	$20.53_{\pm 0.25}$
LoRA	14	$1\times 10^{-5}$	$0.151_{\pm 0.000}$	$0.00161_{\pm 0.00003}$	$1.07_{\pm 0.02}$
LoRA	28	$3\times 10^{-5}$	$0.307_{\pm 0.001}$	$0.00321_{\pm 0.00014}$	$1.05_{\pm 0.04}$
OFT	32	$3\times 10^{-4}$	$4.213_{\pm 0.006}$	$0.00250_{\pm 0.00062}$	$0.06_{\pm 0.01}$
OFT	64	$3\times 10^{-4}$	$6.368_{\pm 0.022}$	$0.00340_{\pm 0.00085}$	$0.05_{\pm 0.01}$
OFT	128	$1\times 10^{-4}$	$3.090_{\pm 0.009}$	$0.00183_{\pm 0.00011}$	$0.06_{\pm 0.00}$

Appendix

Table 12: Qwen2.5-7B spectral displacement at every selected capacity and LR.

Method	Capacity	LR	Trained acc.	Rebuilt acc.	Restored acc.	$\Delta$ NLL vs. rebuilt
LoRA	7	$1\times 10^{-3}$	$90.87_{\pm 0.90}$	$90.60_{\pm 0.85}$	$90.73_{\pm 0.55}$	$0.00756_{\pm 0.00546}$
LoRA	14	$1\times 10^{-5}$	$91.73_{\pm 0.29}$	$91.53_{\pm 0.29}$	$91.73_{\pm 0.15}$	$-0.00002_{\pm 0.00007}$
LoRA	28	$3\times 10^{-5}$	$92.20_{\pm 0.53}$	$92.23_{\pm 0.47}$	$92.07_{\pm 0.49}$	$0.00005_{\pm 0.00005}$
OFT	32	$3\times 10^{-4}$	$91.77_{\pm 0.71}$	$91.67_{\pm 0.75}$	$91.73_{\pm 0.68}$	$0.00001_{\pm 0.00009}$
OFT	64	$3\times 10^{-4}$	$89.87_{\pm 0.42}$	$90.17_{\pm 0.21}$	$89.63_{\pm 0.42}$	$-0.00014_{\pm 0.00023}$
OFT	128	$1\times 10^{-4}$	$90.70_{\pm 0.36}$	$91.07_{\pm 0.61}$	$90.80_{\pm 0.53}$	$0.00001_{\pm 0.00030}$

Appendix

Table 13: Qwen2.5-7B singular-value restoration and reconstruction controls, with accuracy in percent.

Method	Capacity	LR	Top $\Delta$ acc.	Middle $\Delta$ acc.	Tail $\Delta$ acc.	Top $\Delta$ NLL
LoRA	7	$1\times 10^{-3}$	$-82.00_{\pm 2.40}$	$-0.70_{\pm 0.44}$	$0.07_{\pm 0.93}$	$-0.01099_{\pm 0.00343}$
LoRA	14	$1\times 10^{-5}$	$-49.67_{\pm 1.91}$	$-1.17_{\pm 0.12}$	$-0.33_{\pm 0.47}$	$-0.00139_{\pm 0.00026}$
LoRA	28	$3\times 10^{-5}$	$-1.03_{\pm 0.06}$	$0.20_{\pm 0.17}$	$-0.10_{\pm 0.20}$	$-0.00259_{\pm 0.00023}$
OFT	32	$3\times 10^{-4}$	$0.13_{\pm 0.75}$	$0.23_{\pm 0.84}$	$-0.17_{\pm 0.06}$	$-0.00772_{\pm 0.00056}$
OFT	64	$3\times 10^{-4}$	$2.27_{\pm 0.81}$	$1.37_{\pm 0.55}$	$0.30_{\pm 0.36}$	$-0.01298_{\pm 0.00104}$
OFT	128	$1\times 10^{-4}$	$1.03_{\pm 0.45}$	$1.07_{\pm 0.91}$	$0.60_{\pm 0.70}$	$-0.00669_{\pm 0.00015}$

Appendix

Table 14: Qwen2.5-7B component-restoration effects relative to the trained model, with accuracy changes in percentage points.

Method	Capacity	LR	$100d_{W}$	$100d_{S}$	$100d_{S}/d_{W}$
LoRA	7	$3\times 10^{-4}$	$2.167_{\pm 0.020}$	$0.04985_{\pm 0.00207}$	$2.30_{\pm 0.08}$
LoRA	15	$3\times 10^{-4}$	$2.845_{\pm 0.026}$	$0.08243_{\pm 0.00717}$	$2.90_{\pm 0.23}$
LoRA	30	$3\times 10^{-4}$	$3.707_{\pm 0.024}$	$0.12641_{\pm 0.00536}$	$3.41_{\pm 0.13}$
OFT	32	$3\times 10^{-4}$	$5.023_{\pm 0.011}$	$0.00190_{\pm 0.00000}$	$0.04_{\pm 0.00}$
OFT	64	$3\times 10^{-4}$	$6.626_{\pm 0.015}$	$0.00193_{\pm 0.00000}$	$0.03_{\pm 0.00}$
OFT	128	$1\times 10^{-4}$	$3.488_{\pm 0.007}$	$0.00188_{\pm 0.00000}$	$0.05_{\pm 0.00}$

Appendix

Table 15: Llama-3.1-8B spectral displacement at every selected capacity and LR.

Method	Capacity	LR	Trained acc.	Rebuilt acc.	Restored acc.	$\Delta$ NLL vs. rebuilt
LoRA	7	$3\times 10^{-4}$	$71.00_{\pm 0.35}$	$71.30_{\pm 0.46}$	$71.07_{\pm 0.51}$	$0.00006_{\pm 0.00016}$
LoRA	15	$3\times 10^{-4}$	$72.33_{\pm 0.93}$	$72.50_{\pm 0.26}$	$72.23_{\pm 0.90}$	$-0.00023_{\pm 0.00025}$
LoRA	30	$3\times 10^{-4}$	$72.20_{\pm 1.28}$	$72.40_{\pm 0.89}$	$71.97_{\pm 1.43}$	$-0.00254_{\pm 0.00068}$
OFT	32	$3\times 10^{-4}$	$71.80_{\pm 1.37}$	$71.77_{\pm 0.81}$	$71.73_{\pm 1.29}$	$-0.00007_{\pm 0.00012}$
OFT	64	$3\times 10^{-4}$	$73.43_{\pm 0.50}$	$73.03_{\pm 0.29}$	$73.57_{\pm 0.49}$	$0.00003_{\pm 0.00010}$
OFT	128	$1\times 10^{-4}$	$72.47_{\pm 1.31}$	$72.40_{\pm 0.87}$	$72.33_{\pm 0.85}$	$0.00004_{\pm 0.00014}$

Appendix

Table 16: Llama-3.1-8B singular-value restoration and reconstruction controls, with accuracy in percent.

Method	Capacity	LR	Top $\Delta$ acc.	Middle $\Delta$ acc.	Tail $\Delta$ acc.	Top $\Delta$ NLL
LoRA	7	$3\times 10^{-4}$	$-48.23_{\pm 11.06}$	$-0.67_{\pm 0.59}$	$-0.03_{\pm 0.71}$	$-0.02150_{\pm 0.00224}$
LoRA	15	$3\times 10^{-4}$	$-34.73_{\pm 6.28}$	$-1.57_{\pm 1.36}$	$-0.17_{\pm 0.61}$	$-0.02192_{\pm 0.00220}$
LoRA	30	$3\times 10^{-4}$	$-27.43_{\pm 1.07}$	$-0.60_{\pm 1.31}$	$-0.60_{\pm 1.14}$	$-0.02506_{\pm 0.00329}$
OFT	32	$3\times 10^{-4}$	$-23.30_{\pm 3.12}$	$-0.40_{\pm 1.56}$	$-0.03_{\pm 0.38}$	$-0.03143_{\pm 0.00099}$
OFT	64	$3\times 10^{-4}$	$-17.63_{\pm 3.27}$	$-1.43_{\pm 1.22}$	$-0.63_{\pm 1.42}$	$-0.06295_{\pm 0.00097}$
OFT	128	$1\times 10^{-4}$	$-18.97_{\pm 0.61}$	$-0.83_{\pm 1.12}$	$-0.07_{\pm 0.58}$	$-0.03398_{\pm 0.00105}$

Appendix

Table 17: Llama-3.1-8B component-restoration effects relative to the trained model, with accuracy changes in percentage points.

Figure 8: Singular-value restoration across capacities and seeds. Trained, reconstructed and restored models are shown separately, extending the main accuracy comparison with reconstruction controls and retention measurements.

Figure 9: Measured weight and singular-value displacement across all six methods and both models at each selected capacity and LR. Ratios express spectral displacement relative to weight displacement.

Figure 10: Singular-value changes across spectral components at largest capacity. PiSSA’s displacement concentrates near the leading end, whereas MiLoRA’s concentrates near the tail, consistent with their initialization choices. The pretrained reference and method colors are retained.

(a) Absolute relative HE change ( $\times 10^{-4}\%$ )
Method	Qwen2.5-7B	Llama-3.1-8B	FLUX-4B cat
LoRA	$14.8_{\pm 0.96}$	$1.72_{\pm 0.17}$	$0.281_{\pm 0.025}$
OFT	$1.5_{\pm 0.2}$	$0.0215_{\pm 0.0014}$	$0.533_{\pm 0.065}$
DoRA	$80.3_{\pm 15}$	$1.7_{\pm 0.069}$	$0.714_{\pm 0.072}$
PiSSA	$91.9_{\pm 4.6}$	$1.53_{\pm 0.099}$	$3.02_{\pm 0.24}$
MiLoRA	$1.35_{\pm 0.37}$	$4.36_{\pm 0.17}$	$0.987_{\pm 0.032}$

Appendix

Table 18: Geometric preservation at largest selected capacities. Each checkpoint averages matrices equally, followed by the seed mean and sample SD. HE changes are in units of

10^{-4}\%

, and cosine changes are in units of

10^{-4}

Figure 11: Dominant-component restoration at every selected configuration and seed. The retention response persists across the grid, while task-accuracy responses vary. Recovery is normalized by each checkpoint’s trained-model NLL increase.

Figure 12: Effects of adapting each component group under all combinations of the other groups. Dominant-component adaptation consistently increases general NLL, while intermediate and trailing effects can change sign. Both NLL evaluations agree on the dominant-component direction.

Figure 13: Task performance when retaining only some adapted components. Cells show mean

\pm

SD changes from

\bm{W}_{R}

(all singular values restored) at the largest capacities. Columns retain the adapted dominant components alone or the intermediate and trailing components together. Bold means mark losses exceeding 1 pp for LLMs or 1 DINO

\times 100

unit for FLUX.

Model	Method	Small		Medium		Large
		Top	Complement	Top	Complement	Top	Complement
Qwen2.5-7B	LoRA	3/3	0/3	0/3	0/3	3/3	3/3
	OFT	3/3	3/3	3/3	3/3	3/3	3/3
	DoRA	3/3	3/3	3/3	1/3	3/3	3/3
	PiSSA	3/3	0/3	3/3	0/3	3/3	0/3
	MiLoRA	0/3	0/3	0/3	3/3	0/3	3/3

Appendix

Table 19: Task sufficiency across all selected language capacities. Entries count seeds with accuracy loss

\leq 1

pp relative to full value restoration, out of three seeds per configuration. Top and complement retain those adapted components on a pretrained background. Tables 12 and 15 give capacities and selected LRs.

Model	Checkpoints	Max. $\\|\bm{R}\bm{R}^{\!\top}-\bm{I}\\|_{\mathrm{F}}$	Max. entry	Max. polar residual
FLUX-4B	33	$8.04$	$0.136$	$0.11$
Llama-8B	21	$0.118$	$0.00528$	$0.000936$
Qwen-7B	21	$0.42$	$0.0233$	$0.00154$

Appendix

Table 20: Native OFT-transform diagnostics. Maxima are taken over all matrices and checkpoints in each row. Frobenius and entrywise maxima describe different norms.

Method	Capacity	LR	Params. (M)	$100d_{W}$	$100d_{S}$	$100d_{S}/d_{W}$
LoRA	32	$5\times 10^{-5}$	$38.34$	$0.783_{\pm 0.029}$	$0.0088_{\pm 0.0005}$	$1.12_{\pm 0.09}$
OFT	256	$1\times 10^{-4}$	$39.17$	$10.587_{\pm 0.085}$	$0.0071_{\pm 0.0013}$	$0.07_{\pm 0.01}$
DoRA	32	$1\times 10^{-4}$	$39.23$	$1.586_{\pm 0.062}$	$0.0204_{\pm 0.0007}$	$1.28_{\pm 0.02}$
PiSSA	32	$5\times 10^{-5}$	$38.34$	$2.477_{\pm 0.027}$	$0.1434_{\pm 0.0021}$	$5.79_{\pm 0.04}$
MiLoRA	32	$1\times 10^{-4}$	$38.34$	$2.019_{\pm 0.046}$	$0.1763_{\pm 0.0029}$	$8.73_{\pm 0.06}$
HRA	124	$3\times 10^{-4}$	$38.09$	$11.744_{\pm 0.260}$	$0.0017_{\pm 0.0000}$	$0.01_{\pm 0.00}$

Appendix

Table 21: FLUX intervention operating points selected by mean validation DINO. Parameters are in millions. Displacements pool all adapted matrices before normalization. All edits act jointly on 80 matrices.

Method	Trained	Reconstructed	Restored	Values only	$\Delta$ vs. recon.
LoRA	$68.06_{\pm 0.78}$	$67.84_{\pm 0.47}$	$68.12_{\pm 0.45}$	$36.24_{\pm 1.13}$	$0.28_{\pm 0.04}$
OFT	$59.04_{\pm 5.51}$	$59.33_{\pm 5.91}$	$59.18_{\pm 5.42}$	$36.17_{\pm 1.09}$	$-0.15_{\pm 0.57}$
DoRA	$70.47_{\pm 3.96}$	$70.95_{\pm 4.05}$	$70.59_{\pm 3.75}$	$36.24_{\pm 1.11}$	$-0.36_{\pm 0.36}$
PiSSA	$68.88_{\pm 3.45}$	$69.07_{\pm 2.91}$	$69.03_{\pm 2.96}$	$36.21_{\pm 1.24}$	$-0.05_{\pm 0.83}$
MiLoRA	$66.13_{\pm 2.10}$	$66.03_{\pm 2.17}$	$66.06_{\pm 2.15}$	$36.34_{\pm 1.35}$	$0.03_{\pm 0.60}$
HRA	$63.75_{\pm 5.76}$	$63.69_{\pm 5.78}$	$63.54_{\pm 5.99}$	$36.19_{\pm 1.06}$	$-0.15_{\pm 0.27}$

Appendix

Table 22: FLUX value-restoration controls. DINO scores and paired differences are multiplied by 100. The last column compares restoration with reconstruction through the same SVD/storage path.

Figure 14: FLUX numerical controls. The left panel compares value restoration with reconstruction. The right panel compares the same-run merged trained model with the earlier native-adapter evaluation. Diamonds and bars show means and sample SDs, with faint points for individual checkpoints. Differences are in DINO scores

\times 100

Figure 15: FLUX spectral displacement. The left panel shows the pooled spectral-to-weight displacement ratio with sample SDs. The right panel shows mean shares of squared singular-value displacement across component groups, with boundaries adjusted to pretrained spectral gaps. Shares are computed from all measured spectral vectors.

Figure 16: FLUX component-restoration responses. The top row restores the named component group, and the bottom row keeps only that adapted group. All changes use the same-run trained model. Lines show method means and faint points show individual checkpoints.

Control	Required evidence
Training runs and uncertainty	At least two independent training runs per method with a defined SD, SE or CI attributable to training variation.
LR selection	At least three learning rates evaluated per method, with separate selection using a stated validation criterion.
LR candidates	Numerical candidate values or a search range stated for both methods.
Comparable conditions	Shared checkpoint and data, comparable exposure, and adapted modules and trainable budgets matched or addressed through an appropriate controlled comparison.

Appendix

Table 23: Definitions used in the review. Evidence must cover both methods and the stated main comparison. These are operational criteria for this review.

Control	Documented	Explicit departure	Unclear	All main comparisons
Training runs and uncertainty	9	5	50	2
LR selection	7	4	53	1
LR candidates	13	0	51	4
Comparable conditions	25	10	29	1

Appendix

Table 24: Paper-level coverage of experimental controls. The first three columns partition the 64 eligible papers using the at least one comparison rule. The final column requires established coverage of every applicable main comparison. Unresolved coverage prevents a universal positive.

Domain	Training uncertainty	LR selection	LR candidates	Conditions
Decoder language	4/51	4/51	7/51	18/51
Other language	5/40	3/40	6/40	7/40
Vision and multimodal	3/20	1/20	3/20	2/20
Diffusion	0/11	0/11	1/11	3/11
Other	0/1	0/1	0/1	0/1
Decoder subset	3/40	2/40	4/40	12/40

Appendix

Table 25: Documented controls within each domain. Entries give qualifying papers over applicable papers. A paper can occur in several domains, so rows cannot be summed. The decoder subset restricts comparisons to standard decoder adaptation against vanilla LoRA, with the exclusions described by Lee et al. Its definitions and sampling frame still differ from theirs.

Explore similar work

May 27, 2026cs.LG

PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective

Parameter-efficient finetuning (PEFT) has become the standard approach for adapting large language models, yet evaluations largely emphasize downstream accuracy while overlooking the retention of pretrained capabilities. We argue that PEFT should be assessed through the stability-plasticity dilemma: the trade-off between target-task adaptation and resistance to forgetting. We introduce PEFT-Arena, a benchmark that jointly measures downstream performance and general capability retention. Across methods, we find distinct stability-plasticity profiles; under comparable parameter budgets, orthogonal finetuning achieves the most favorable Pareto frontier. To explain these differences, we analyze PEFT updates from two geometric perspectives. In weight space, spectral analysis reveals how parameterizations interact with the pretrained singular-value structure. In activation space, retention metrics show whether finetuning preserves or distorts general-capability representations, with forgetting linked to non-isometric representation distortion. Finally, an analysis shows that final SFT checkpoints often overshoot a better target-retention operating point. Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.

Yangyi Huang, Ruotian Peng, Zeju Qiu +4

The Chinese University of Hong Kong · Westlake University · MPI for Intelligent Systems

Parameter-Efficient Fine-Tuning Methods Model Fine-Tuning Efficient Task Adaptation Plasticity Domain Adaptation

Ask PDF

May 14, 2026cs.LG

GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioning

Low-rank adaptation (LoRA) has become a dominant paradigm for parameter-efficient fine-tuning (PEFT) of large-scale deep learning models. However, its bilinear parameterization induces a parameter-dependent geometry: the mapping from trainable parameters to weight updates is not generally distance-preserving. Related methods that project a low-dimensional vector into LoRA's parameter space, such as Uni-LoRA, improve parameter efficiency, but the subsequent bilinear map breaks end-to-end isometry. We propose GPart (Global Partition fine-tuning), a highly parameter-efficient fine-tuning method that maps a

d

-dimensional trainable vector directly into the full weight space through a sparse, isometric partition matrix. GPart retains a fixed global parameter-sharing prior while removing the additional low-rank reconstruction used by LoRA-based methods. This yields a simple parameterization with a single main hyperparameter (

d

), exact end-to-end isometry, and a minimal checkpoint representation consisting of the trainable vector and a random seed. GPart builds on the premise of effective fine-tuning within random low-dimensional subspaces of the full weight space without requiring a low-rank matrix factorization. Across natural language understanding, computer vision, and mathematical reasoning benchmarks, GPart matches or improves over existing PEFT methods at ultra-low parameter budgets. Beyond offering mathematical tractability and memory efficiency, the direct linear parameterization of GPart streamlines model selection and paves the way for compact adapter composition. Overall, GPart provides an elegant and competitive alternative for fine-tuning under small parameter budgets, with a fixed and predictable geometry between trainable coordinates and weight-space updates.

Paolo Mandica, Michał Brzozowski, Zuzanna Dubanowska +1

Samsung AI Center, Warsaw, Poland · University of Warsaw, Poland

Parameter-Efficient Fine-Tuning Methods Training-Side Stage-Aware Low-Rank Adaptation Model Fine-Tuning Partitioning

Ask PDF

May 12, 2026cs.LG

LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection

Orthogonal parameter-efficient fine-tuning (PEFT) adapts pretrained weights through structure-preserving multiplicative transformations, but existing methods often conflate two distinct design choices: the subspace in which adaptation occurs and the transformation applied within that subspace. This paper introduces LOFT, a low-rank orthogonal fine-tuning framework that explicitly separates these two components. By viewing orthogonal adaptation as a multiplicative subspace rotation, LOFT provides a unified formulation that recovers representative orthogonal PEFT methods, including coordinate-, butterfly-, Householder-, and principal-subspace-based variants. More importantly, this perspective exposes support selection as a central design axis rather than a byproduct of a particular parameterization. We develop a first-order analysis showing that useful adaptation supports should be informed by the downstream training signal, motivating practical task-aware support selection strategies. Across language understanding, visual transfer, mathematical reasoning, and multilingual out-of-distribution adaptation, LOFT recovers principal-subspace orthogonal adaptation while gradient-informed supports improve the efficiency-performance trade-off under matched parameter, memory, and compute budgets. These results suggest that principled support selection is an important direction for improving orthogonal PEFT.

Lanxin Zhao, Bamdev Mishra, Pratik Jawanpuria +4

The University of Sydney, Australia. · Microsoft India. · Indian Institute of Technology Bombay, India. +1

Parameter-Efficient Fine-Tuning Methods Model Fine-Tuning Rapid Adaptation Floor

Ask PDF