Frozen-backbone transfer with Vision Transformers faces two under-addressed issues: optimization instability when adapters are naively inserted into a fixed feature extractor, and the absence of principled guidance for setting adapter capacity. We introduce AdapterTune, which augments each transformer block with a residual low-rank bottleneck whose up-projection is zero-initialized, guaranteeing that the adapted network starts exactly at the pretrained function and eliminates early-epoch representation drift. On the analytical side, we formalize adapter rank as a capacity budget for approximating downstream task shifts in feature space. The resulting excess-risk decomposition predicts monotonic but diminishing accuracy gains with increasing rank, an ``elbow'' behavior we confirm through controlled sweeps. We evaluate on 9 datasets and 3 backbone scales with multi-seed reporting throughout. On a core 5 dataset transfer suite, AdapterTune improves top-1 accuracy over head-only transfer by +14.9 points on average while training only 0.92 of the parameters required by full fine-tuning, and outperforms full fine-tuning on 10 of 15 dataset-backbone pairs. Across the full benchmark, AdapterTune improves over head-only transfer on every dataset-backbone pair tested. Ablations on rank, placement, and initialization isolate each design choice. The code is available at: https://github.com/salimkhazem/adaptertune
Figures & tables
Figure 1: AdapterTune architecture. (Left) Trainable residual adapters (orange) are inserted into the strictly frozen Vision Transformer backbone (blue). (Right) The adapter uses a low-rank bottleneck where the up-projection is zero-initialized. This guarantees an initial zero output ( Aℓ(hℓ)=0 ), acting as an exact identity mapping to prevent early-epoch optimization drift.
Backbone
d
L
Adapter params
%FT
DeiT-T/16
192
12
76 K
0.67%
ViT-S/16
384
12
303 K
0.70%
ViT-B/16
768
12
1.2 M
1.40%
Table 1: Trainable parameters for each backbone at adapter rank r=16 , every block. “%FT” is the fraction relative to full fine-tuning parameters (all backbone weights plus head).
Backbone
Method
C-10
C-100
SVHN
Pets
Food
Δ Head
ViT-S/16
Head (0.1%)
89.7
72.0
54.5
90.5
68.8
—
Ours (0.9%)
97.5
84.9
96.2
93.5
85.0
+14.7
Full (100%)
97.2
79.6
97.4
89.5
86.5
+12.7
ViT-B/16
Head (0.1%)
94.8
81.5
65.5
93.3
84.3
—
Ours (0.9%)
98.9
91.2
97.5
94.3
90.9
+14.9
Full (100%)
95.3
80.7
97.5
86.6
84.4
+10.0
Table 2: Core benchmark (top-1 %, 3 seeds). † AdapterTune wins 10/15 pairs vs. full FT while training <1% of its parameters.
Figure 2: Per-dataset accuracy comparison. Each row corresponds to one dataset. Gray circles: Head-Only; blue squares: Full Fine-Tune; red stars: AdapterTune. Connecting lines show the performance gap bridged by each method. AdapterTune (red stars) reaches or surpasses full fine-tuning on most datasets, while using only 0.92% of its parameters.
Figure 3: Accuracy versus trainable parameter count (Pareto frontier). AdapterTune (red stars) achieves comparable or higher accuracy than full fine-tuning (blue squares) at 1-2 orders of magnitude fewer trainable parameters, demonstrating a clearly favourable position on the accuracy-efficiency frontier.
Adapter Rank r
Dataset / Backbone
r=8
r=16
r=32
r=64
CIFAR-10 / ViT-S/16
97.56±0.11
97.61±0.02
97.75±0.05
97.85±0.09
Gain vs. r=8
—
+0.05
+0.20
+0.29
Table 3: Rank sweep on CIFAR-10 / ViT-S/16 (adapter, every=1 , zero init, 3 seeds). Accuracy improves steadily from r=8 to r=64 , with smaller increments beyond r=32 , consistent with Corollary 1 .
Figure 4: (Left) Rank sweep across all core datasets and backbone scales. The diminishing-returns elbow (predicted by Corollary 1 ) appears consistently across every dataset-backbone pair, not just CIFAR-10/ViT-S. Accuracy gains from r=8→32 uniformly exceed gains from r=32→64 , validating the O(r1/2−p) decay law broadly. (Right) Rank × adapter scale ( α ) joint sensitivity on CIFAR-10/ViT-S. Accuracy is robust across the full r∈[8,64] range for α≤1 . Only α=2 at low rank causes a visible drop, confirming α=1 as a safe default that need not be tuned.
Design Axis
Setting
Top-1 (%)
Δ
Placement
Every block ( every=1 )
97.61±0.02
+0.00
Every 2 blocks ( every=2 )
97.51±0.06
-0.10
Initialization
Zero (default)
97.61±0.02
+0.00
Small random ( σ0=10−4 )
97.59±0.10
-0.01
Table 4: Placement and initialization ablations on CIFAR-10 / ViT-S/16 ( r=16 , 3 seeds). Δ is relative to the default configuration (every=1, zero init). Both axes stay within 0.10 points in mean accuracy, confirming design robustness.
Learning rate
Weight decay
Scale α
2.5e−4
5e−4
1e−3
0.01
0.05
0.1
0.5
1.0
2.0
97.56
97.68
97.61
97.54
97.61
97.79
97.70
97.61
97.47
Table 5: Hyperparameter sensitivity on CIFAR-10 / ViT-S/16 ( r=16 , 3 seeds). All configurations stay within 0.3 pp of the best.
Dataset
Backbone
Head-Only
AdapterTune (Ours)
Full FT
Top-1 (%)
Top-1 (%)
Top-1 (%)
Flowers102
ViT-S/16
87.73±0.53
94.43±0.04
92.92±0.73
ViT-B/16
98.71±0.11
99.43±0.04
93.23±2.75
DeiT-T
85.87±0.21
94.19±0.11
93.18±0.18
ImageNet-R
ViT-S/16
46.94±0.66
69.26±0.41
59.59±0.28
ViT-B/16
62.17±0.66
80.10±0.41
57.66±0.22
Table 6: Extended benchmark results (top-1 accuracy, mean ± std, 3 seeds). AdapterTune consistently outperforms head-only tuning across all 4 extended datasets and 3 backbones.
Figure 5: Backbone scaling trends. (Left) Average gain of AdapterTune over Head-Only across all datasets as backbone parameter count grows from DeiT-T/16 (5M) to ViT-B/16 (86M). The gains are consistent across backbone scales, with larger backbones showing slightly higher gains on fine-grained tasks. (Right) Average gain of AdapterTune over Full Fine-Tuning: the adapter advantage over full fine-tuning increases with backbone size, attributed to the stronger implicit regularization of the low-rank constraint as model capacity grows.
Figure 6: Generalisation gap analysis. Each point plots training accuracy ( x -axis) versus test accuracy ( y -axis) for a dataset/method combination. Full fine-tuning (blue squares) exhibits large train-test gaps (11–13%), indicating overfitting on smaller datasets under the fixed training protocol. AdapterTune (red stars) clusters near the diagonal with average gaps of only 1.7-2.7%, while Head-Only (gray circles) shows near-zero gaps reflecting underfitting. AdapterTune occupies a favorable bias-variance operating point between the two extremes.
Figure 7: Full hyperparameter sensitivity grid on CIFAR-10/ViT-S/16 ( r=16 , 3 seeds each). The grid jointly sweeps three hyperparameters: learning rate ∈{2.5×10−4,5×10−4,10−3} , weight decay ∈{0.01,0.05,0.1} , and adapter scale α∈{0.5,1.0,2.0} . Every configuration achieves >97.4 % top-1 accuracy; the total spread across all 27 cells is less than 0.4 pp. This confirms that AdapterTune does not require careful per-dataset hyperparameter tuning: practitioners may use the recommended defaults (LR =10−3 , WD =0.05 , α=1 ) across a wide range of tasks without a dedicated sweep.