Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning

Zichen Tian; Yaoyao Liu; Qianru Sun

cs.LGMar 2, 2026

Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning

Authors: Zichen Tian, Yaoyao Liu, Qianru Sun

Organizations: Singapore Management University · University of Illinois Urbana-Champaign

Abstract

Training large foundation models from scratch for domain-specific applications is almost impossible due to data limits and long-tailed distributions -- taking remote sensing (RS) as an example. Fine-tuning natural image pre-trained models on RS images is a straightforward solution. To reduce computational costs and improve performance on tail classes, existing methods apply parameter-efficient fine-tuning (PEFT) techniques, such as LoRA and AdaptFormer. However, we observe that fixed hyperparameters -- such as intra-layer positions, layer depth, and scaling factors, can considerably hinder PEFT performance, as fine-tuning on RS images proves highly sensitive to these settings. To address this, we propose MetaPEFT, a method incorporating adaptive scalers that dynamically adjust module influence during fine-tuning. MetaPEFT dynamically adjusts three key factors of PEFT on RS images: module insertion, layer selection, and module-wise learning rates, which collectively control the influence of PEFT modules across the network. We conduct extensive experiments on three transfer-learning scenarios and five datasets in both RS and natural image domains. The results show that MetaPEFT achieves state-of-the-art performance in cross-spectral adaptation, requiring only a small amount of trainable parameters and improving tail-class accuracy significantly.

Figures & tables

Figure 1: Comparing PEFT methods for the model adaptation of IN21K

\to

DOTA. (a) Bubble plot of overall accuracy, tail-class accuracy, and performance variance (in standard deviation) of 5 PEFT methods in 6 total versions. Additive methods exhibit consistently higher accuracy and lower variance than non-additive methods. (b) Inter-class feature distances of the PEFT methods measured by cosine similarity. Additive methods achieve 13% further feature distances (which means better discrimination among tail classes) with comparable head-class distances. (c) Accuracy heatmap for applying PEFT on different positions of ViT: on different intra-block layers v.s. among different attention blocks (depth). Deeper blocks yield better performance (86.5% to 90.4%), but the combination of optimal block and intra-block position shows unexpected degradation (0.6% drop for FFN layer with depth 10

\to

11). The marks optimal combination. (d) Accuracy heatmap of intra-block positions v.s. scaling factors. Different positions show distinct sensitivity to scaling factors, with sharp accuracy drops observed ( e.g . , applying PEFT on the attention output layer (denoted as Out) drops from 87% to 6.7% when its PEFT scaling factor increases from 2 to 4). These highlight the non-monotonic complexity of PEFT hyperparameters.

Figure 2: Architecture and optimization framework for MetaPEFT . In Figures (a)-(c) , we illustrate how our proposed modulator

\gamma

is integrated with three representative additive PEFT methods: (a) AdaptFormer with modulated up/down projections (

W_{\mathrm{up}}

W_{\mathrm{down}}

), (b) LoRA with modulated low-rank decomposition matrices (

A

B

), and (c) Adapter with modulated projection layers (

W_{\mathrm{up}}

W_{\mathrm{down}}

). The

\sigma

denotes the non-linear activation function ( e.g . , ReLU). In Figure (d) , we show our bi-level optimization framework. The inner loop optimizes PEFT parameters

\phi

on training data

\mathcal{D}_{\mathrm{train}}

, and the outer loop updates modulator

\gamma

on validation data

\mathcal{D}_{\mathrm{val}}

sampled from the training set. The symbol

N\times

indicates the operation is repeated for

N

attention blocks ( e.g . ,

N

=12 for ViT-B/16 ).

Position	CIFAR100			iNat2018			Places-LT			DOTA			Avg	Avg tail
Position	Head	Med	Tail	Head	Med	Tail	Head	Med	Tail	Head	Med	Tail	Avg	Avg tail
ATTN	92.2	87.7	86.3	67.9	76.3	76.7	48.4	48.3	45.7	94.1	94.7	91.1	75.8	74.9
FFN	92.4	87.9	86.8	66.7	75.6	76.9	48.6	47.9	45.6	94.6	94.8	91.4	75.8	75.2
ATTN-FFN	92.3	88.1	86.6	67.7	76.8	77.5	48.2	48.3	45.5	94.6	95.4	92.4	76.1	75.5

Table 1: Applying PEFT on Different Intra-Block Layer Positions. We compare three LoRA insertion positions within the attention block: attention-projection-layers-only (ATTN), feedforward-network-only (FFN), and combined (ATTN-FFN) across four adaptation settings: IN21K pre-trained ViT-B/16

\to

{CIFAR100, iNaturalist2018, Places-LT, DOTA}. Results reported in Accuracy (

\%

). Results show that combined positions yield slightly better overall performance and tail-class accuracy.

Module	Position	Accuracy (%)
Module	Position	Head	Med	Tail	Avg
ATTN	K	91.6	93.0	87.7	90.6
	Q	91.4	93.1	90.0	91.5
	V	94.0	94.5	90.2	92.7
	Out	93.6	94.1	90.0	92.3
FFN	MLP 1	94.6	94.6	91.6	93.4

Table 2: Applying PEFT on Layer-Wise Intra-Block Positions for IN21K

\to

DOTA. We compare LoRA’s different layer-wise insertion positions within the attention block. ATTN in Table 1 now has

4

versions due to

4

kinds of projection layers (Q/K/V/Out) in the attention computation and output of the attention block. Results reported in Accuracy (

\%

). LoRA on the first FFN layer (MLP 1) yields the best performance with an average accuracy of

93.4\%

Block Group	Many	Med	Few	Avg	Drop (%)
L9-11	88.1	89.8	88.8	89.0	3.2
L6-8	92.3	93.5	89.3	91.6	0.3
L3-5	92.4	93.7	89.8	91.9	-
L0-2	90.5	92.8	88.4	90.6	1.4

Table 3: Ablate Different Block Depths for IN21K

\to

DOTA (

\%

). We analyze the impact of applying LoRA across different transformer blocks. Results reported in Accuracy (

\%

). Results indicate that using LoRA on the middle-lower blocks (L3-5) achieves the best performance (

91.9\%

overall accuracy), while the deepest blocks (L9-11) show an unexpected performance drop. This suggests that we should apply a higher scaling factor for LoRA in intermediate blocks.

Category	Method	IN21K $\to$ iNat2018			IN21K $\to$ DOTA			SatMAE $\to$ SAR		Avg ${}_{\text{tail}}$	Avg
Category	Method	Head	Med	Tail	Head	Med	Tail	Head	Tail	Avg ${}_{\text{tail}}$	Avg
Non-additive	VPT-Shallow	57.5	65.5	65.9	85.4	89.0	82.4	39.8	68.4	72.23	69.24
	BitFit	57.6	67.3	68.4	92.0	93.7	89.1	32.1	74.7	77.40	71.86
	VPT-Deep	64.9	74.2	75.9	92.1	93.5	90.2	30.6	50.0	72.03	71.43
Additive	Adapter	67.9	76.8	77.7	93.2	94.9	90.6	37.9	75.8	81.37	76.85
	w/ Ours	68.2	76.9	78.1	93.2	95.1	90.7	37.7	76.0	81.60	76.99

Table 4: Comparing different PEFT methods w/ or w/o ours. We evaluate three transfer scenarios (IN21K

\to

iNat2018, IN21K

\to

DOTA, SatMAE

\to

SAR) using non-additive methods (VPT, BitFit) and additive methods (Adapter, LoRA, AdaptFormer). Performance is measured across head, medium, and tail classes for each dataset (the SAR dataset has only head and tail) and reported in Accuracy (

\%

). “Avg

{}_{\text{tail}}

” shows mean performance on tail classes, and “Avg” is the macro-average across all class splits. Results show that our method consistently improves additive methods, with LoRA achieving the highest tail-class performance (

81.63\%

). Ours are marked in gray .

Category	Method	Inter-class Distance
Category	Method	Head	Tail
Non-additive	VPT-Shallow	0.65	0.58
	BitFit	0.81	0.75
	VPT-Deep	0.81	0.72
Additive	Adapter	0.83	0.77
	w/ Ours	0.84	0.79

Table 5: Inter-Class Feature Distance. We measure the inter-class cosine distance between head classes, and between tail classes. Results are reported on transfer scenario IN21K

\to

DOTA using average cosine distance, where higher values indicate better class separation. Our method (marked in gray ) achieves the best feature separation when combined with LoRA.

Ratio (%)	Head	Med	Tail	Avg	$\Delta$ Tail (%)
5	91.2	92.1	88.2	90.5	-5.2
10	93.4	94.2	90.8	92.8	-2.6
30	95.0	95.8	93.4	94.7	-
50	94.8	95.6	93.0	94.5	-0.4

Table 6: Impact of Sampling Strategies (

\%

). We evaluate different sampling ratios in the outer loop of MetaPEFT . Results reported in Accuracy (

\%

). While 30% sampling achieves the best performance (94.7% average accuracy), we adopt 20% in our default setting due to time constraints.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Strategy	Head	Med	Tail	Avg
Random (20%)	93.2	94.6	93.1	93.6
Class-Balanced	93.0 $\downarrow$ 0.2	94.5 $\downarrow$ 0.1	93.3 $\uparrow$ 0.2	93.6 -
Tail-Heavy	92.0 $\downarrow$ 1.2	93.4 $\downarrow$ 1.2	94.8 $\uparrow$ 1.7	93.4 $\downarrow$ 0.2
Random (30%)	93.5 $\uparrow$ 0.3	94.8 $\uparrow$ 0.2	93.9 $\uparrow$ 0.8	94.1 $\uparrow$ 0.5

Appendix

Table S1: Accuracy (

\%

) comparison of different sampling strategies. We compare different sampling strategies in the outer loop of MetaPEFT . While tail-heavy sampling achieves the highest tail-class accuracy, it comes at the cost of reduced head-class performance. Random sampling with 30% ratio provides the best overall performance while maintaining balanced improvements across all classes. The arrows indicate changes compared to the 20% random sampling baseline ( green for decrease, red for increase).

Dataset	Dim	LoRA		Adapter		AdaptFormer
Dataset	Dim	Tuner (M)	Head (M)	Tuner (M)	Head (M)	Tuner (M)	Head (M)
CIFAR100-IR100	4	0.249	0.076	0.101	0.076	0.101	0.076
FUSRS	4	0.197	0.005	0.136	0.005	0.136	0.005
DOTA	4	0.249	0.011	0.101	0.011	0.101	0.011
Places-LT	8	0.470	0.280	0.175	0.280	0.175	0.280
iNat2018	256	9.437	6.253	4.749	6.253	4.749	6.253

Appendix

Table S2: Comparison of learnable parameters across different PEFT methods and datasets. “Dim” refers to the model dimension ( i.e . , rank for LoRA, hidden dimension for Adapter/AdaptFormer). For each method, “Tuner (M)” refers to the number of learnable parameters within the PEFT modules, while “Head (M)” refers to those in the classifier head.

Position	CIFAR100			iNat2018			Places-LT			Avg tail	Avg
Position	Head	Med	Tail	Head	Med	Tail	Head	Med	Tail	Avg tail	Avg
K	87.3	84.6	84.0	63.2	72.3	73.3	46.5	47.0	46.4	67.9	67.9
Q	87.4	84.3	84.1	63.3	72.7	73.7	46.7	47.2	46.6	68.1	68.1
V	91.8	88.1	85.6	65.9	74.8	75.8	47.9	47.7	45.7	69.0	70.1
Out	91.8	88.2	85.4	65.8	75.3	75.3	47.5	47.8	46.0	68.9	70.1
MLP1	92.6	87.9	85.6	66.5	75.6	76.1	48.3	47.9	45.9	69.2	70.5

Appendix

Table S3: Accuracy (%) comparison of intra-block positions in natural vision domain. Results are reported on three transfer scenarios IN21K

\to

{CIFAR100, iNaturalist-2018, and Places-LT}. “Out” denotes the attention output module, and “MLP1” and “MLP2” denote the first and second FFN linear layers. MLP1 achieves the highest overall accuracy (70.5%), while both MLP positions show strong tail-class performance (69.2-69.5%).

Intra-Attn Positions				CIFAR100			iNat2018			Places-LT			Avg tail	Avg
Q	V	K	Out	Head	Med	Tail	Head	Med	Tail	Head	Med	Tail	Avg tail	Avg
✓	✓			91.2	87.9	86.8	66.5	75.4	76.9	48.4	47.6	45.4	69.70	69.57
✓	✓	✓		91.6	87.7	86.3	66.0	75.7	76.7	48.3	47.7	46.3	69.77	69.59
✓	✓	✓	✓	92.2	87.7	86.3	66.7	75.6	76.9	48.6	47.9	45.6	69.60	69.72

Appendix

Table S4: Accuracy (%) comparison of intra-attention positions’ combinations.

Intra-FFN Positions		CIFAR100			iNat2018			Places-LT			Avg tail	Avg
MLP1	MLP2	Head	Med	Tail	Head	Med	Tail	Head	Med	Tail	Avg tail	Avg
✓		92.6	87.9	85.6	66.5	75.6	76.1	48.3	47.9	45.9	69.20	69.60
	✓	92.3	88.1	86.6	65.9	75.1	76.1	47.9	47.9	45.7	69.47	69.51
✓	✓	92.4	87.9	86.8	67.9	76.3	76.7	48.4	48.3	45.7	69.73	70.04

Appendix

Table S5: Accuracy (%) comparison of intra-FFN positions’ combinations.

Method	CIFAR100			iNat2018			Places-LT			Avg
Method	Head	Med	Tail	Head	Med	Tail	Head	Med	Tail	Avg
LoRA	91.2	87.9	86.8	55.8	64.0	64.7	47.5	47.8	46.1	66.5
AdaptFormer	92.3	88.3	86.8	68.7	76.7	78.0	49.1	47.9	45.1	71.1
LoRA+AdaptFormer	92.5	88.4	87.0	67.9	77.2	77.8	48.6	48.0	45.3	71.2

Appendix

Table S6: Accuracy (%) of combining different additive PEFT methods.

Figure S1 : Accuracy heatmap for head/med/tail/overall classes. Visualization of accuracy heatmaps on different intra-block layers vs among different blocks (depth), where the block stands for the attention block of ViT. Results are reported on transfer scenarios IN21K $\to$ DOTA. The optima in each heatmap is highlighted by a yellow box. Results show that: 1) the (c) Tail classes exhibit non-monotonic accuracy changes across positions, while (a) Head and (b) Medium classes show monotonic trends; 2) the optimal configuration in (c) determines the overall optimal configuration in (d), indicating that tail-class performance dominates the model’s overall performance. These findings validate our position-aware optimization strategy for long-tailed datasets.

Figure S2 : Linear relationship between rank and optimal scaling factor. The heatmaps show accuracy distributions across different ranks and scaling factors for (a) tail classes and (b) all classes. Results reported on IN21K $\to$ CIFAR100. Yellow highlights indicate the optimal scaling factor for each rank (the rank 32 in Figure (b) has two equally optima). We can observe a consistent linear relationship, i.e . , optimal scaling factor $\propto$ rank. This pattern suggests that rank optimization can be decoupled from remaining hyperparameters.

Figure S3 : t-SNE Visualization of (a) LoRA and (b) LoRA + Ours.

Method	IN21K $\to$ DOTA
Method	Head	Med	Tail
LIFT [ 49 ]	93.1	94.9	90.2
w/ Ours	93.2	94.7	90.6
LoTR [ 4 ]	93.0	93.1	90.3
w/ Ours	93.1	92.9	91.1

Appendix

Table S7: Apply our method to LIFT and LoTR.

Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning adapts pretrained models by updating small parameter subsets or lightweight modules instead of all weights.

See all 596 papers

Apr 21, 2026

ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning

Popular low-rank parameter-efficient fine-tuning (PEFT) methods represent adaptation as separate updates to selected backbone weights, without maintaining an explicit task-specific state that is updated and reused across depth. These updates also require the backbone at inference and therefore cannot operate as standalone predictors. We propose ShadowPEFT, which consolidates trainable adaptation into a modular shadow component centered on a compact shadow model and lightweight Transformer layer-specific coupling modules. A persistent shadow state refines the frozen backbone representations and is updated from them in an interactive manner. Because the shadow model is trained as a complete predictor, it can be detached for shadow-only inference without executing the base model and can be initialized from a pretrained model. Experiments on text and image generation and understanding benchmarks show that ShadowPEFT matches or outperforms LoRA and DoRA under comparable trainable-parameter budgets. Additional analyses on shadow pretraining, cross-dataset transfer, parameter scaling, inference latency, and system-level evaluation suggest that centralized layer-space adaptation is a competitive and flexible alternative to conventional low-rank PEFT.

Xianming Li, Zongxi Li, Tsz-fung Andrew Lee +3

PolyU · School of Data Science, Lingnan University, Hong Kong SAR

Fine-Tuning Adapter Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Jul 23, 2026

Three-Pronged Spectral Control for Federated Parameter Efficient Fine Tuning

Federated parameter-efficient fine-tuning (PEFT) enables communication-efficient adaptation of large pretrained models on decentralized edge data, but it remains fragile under non-IID client heterogeneity. In low-rank adaptation (LoRA), different clients may learn locally useful but spectrally misaligned update subspaces, causing high-variance aggregation and poor global transfer. We propose TRISHUL, a spectral-control framework for robust federated PEFT. TRISHUL follows the FL no-raw-data-sharing setting but does not itself provide formal privacy guarantees. TRISHUL uses shared frozen multi-head low-rank bases to obtain algebraically exact aggregation of compact core updates, applies nuclear norm proximal shrinkage to suppress client-specific high-rank spectral components before upload, and allocates adaptation heads non-uniformly across layers using a concave water filling budget rule derived from pretrained layer capacity. Because shrinkage is performed only on small core matrices, TRISHUL adds negligible computation and no extra per-round communication over the underlying multi-head PEFT protocol. Across vision and language benchmarks, including CIFAR-100, SVHN, 20 Newsgroups, MRQA, and GLUE with LLaMA3.2-1B, TRISHUL improves convergence, stability, and final performance over federated LoRA baselines, with greater gains under stronger heterogeneity.

Shiva Raj Pokhrel, Dipsan Bhattarai, Anwar Walid

School of IT, Deakin University, Australia · Columbia University, USA

Federated Learning Aggregation Nuclear Norm Minimization Non-IID Federated Learning Low-Rank Adaptation Federated Fine-Tuning Federated Learning

Ask PDF

Oct 5, 2026

RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning

Parameter-efficient fine-tuning (PEFT) reduces the cost of adapting foundation models by focusing training on a small parameter subset. Complementary to this idea, we introduce RoSA (Rotational Sparse Adaptation), which narrows adaptation to a subset of layers at a time. RoSA freezes lower layers close to the input throughout training and rotates a trainable block over later layers, progressively increasing the number of frozen layers close to the input. This design reduces optimizer-state memory, shortens backpropagation, and even forward propagation if activations at the last frozen layer are cached. Because RoSA is orthogonal to the choice of trainable parameterization, it can be combined with PEFT methods or sparse optimizers within each active block. Experiments across multiple LLM architectures and tasks show that RoSA reduces peak memory while maintaining strong fine-tuning performance.

Muhammad Azeem Lodhi, Chao Zhou, Rebekka Burkholz

Saarland University, Saarbrücken, Germany · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Memory-Efficient Fine-Tuning LLM Fine-Tuning Sparse Fine-Tuning Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Parameter-Efficient Fine-Tuning

Explore similar work

ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning

Three-Pronged Spectral Control for Federated Parameter Efficient Fine Tuning

RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning