A note on data-efficient finetuning of vision foundation models

Niclas Hergenröther; Antonio Orvieto

cs.CVOct 27, 2026

A note on data-efficient finetuning of vision foundation models

Authors: Niclas Hergenröther, Antonio Orvieto

Organizations: ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center

Abstract

Adapting foundation models to a new task within or beyond the training domain is the de facto standard for building specialized machine learning models. Recently, Visual Prompting (VP) has emerged as a promising technique for efficient knowledge transfer. As new foundation model families~(like Mamba) get introduced and VP pipelines such as AutoVP reach greater maturity, we find a growing need for a systematic evaluation of current approaches. In this work, we assess the performance of the latest models, comparing them to earlier architectures and alternative fine-tuning methods, to better understand the progress, challenges and opportunities in the field of efficient fine-tuning under resource limitations. Towards this goal, this paper provides a concise empirical overview of the interactions among foundation model families (Attention-, Convolution-, and Mamba-based) and transfer paradigms: VP, Linear Probing (LP), Full Fine-tuning (FFT) and Low Rank Adaptation (LoRA). Our work builds up on previous findings by broadening the selection of evaluated models and techniques while tuning hyperparameters. In the interest of delivering practical guidelines for the user, we also explore application of prevalent regularization techniques to boost performance in the context of VP.

Figures & tables

Figure 1: ImageNet1k-pretrain and average VP top-1 accuracy over the tasks described in Section 3 . We follow the recipe of Tsao et al. (2024) . All models are tuned to best. Dotted line is linear regression, full results in Table 1 .

	CIFAR10	CIFAR100	Flowers102	GTSRB	DTD	EuroSAT	Food101	OxfordPets	$\mu$	pretrain acc.
VMamba-T	94.89	80.53	91.67	86.75	67.17	95.72	69.25	88.07	84.25	82.60
MambaOut-T	95.07	81.36	89.58	88.27	67.73	95.70	71.26	89.14	84.76	82.70
MambaVision-T	95.44	80.89	92.55	87.74	66.26	96.56	71.56	90.91	85.24	82.30
Swin-T	94.28	80.32	92.34	88.95	66.37	96.31	72.99	88.83	85.05	81.50
MambaVision-S	95.97	82.37	90.71	86.93	66.58	96.62	74.18	91.57	85.61	83.30
MambaVision-B	96.54	83.88	91.79	88.79	65.64	96.63	75.65	91.22	86.27	84.20

Table 1: Top-1 accuracy after training with AutoVP using fully connected label mapping.

\mu

denotes the mean fine-tuning performance. The last column represents the ImageNet pretraining accuracy. The bottom row reports the correlation coefficient between the corresponding column and the ImageNet pretraining accuracy.

	VP	LP	FFT	LoRA
MambaVision	65.64	70.21	75.16	73.88
ConvNext	70.77	70.90	73.18	76.60
VMamba	67.87	67.45	73.30	75.00
DeiT	66.54	68.56	72.98	74.68
ViT	62.13	71.38	75.32	77.02
Swin	68.47	77.07	79.63	81.33

Table 2: Top 1 test-performance in % of various base-models on DTD using different Transfer Learning paradigms. Learning rate, rank (LoRA) and weight decay tuned. Base-size used for all models.

Figure 2: Mean prompt size of the best performing run, with the initial prompt scaling being 1. Mean and standard deviation calculated over the tasks introduced in Sec. 3 .

Figure 3: Flowers102 performance of VMamba-T for combinations of ILM and FCLM with and without enabled resizing.

	# param.	GPU-Mem (GB)	Epochtime (s)
VP	197,576	12.5	39.8
LP	48,175	11.2	36.1
FFT	87,601,967	72.3	49.2
LoRA	740,911	67.5	45.0

Table 3: VP refers to AutoVP with fully connected label mapping and adjustable prompt size. GPU-Mem describes the peak allocated GPU Memory. Reported Runtime is the median over all runs. All reported numbers refer to the use of VMamba-B on the DTD dataset.

rank $r$	1	2	4	8	16
M-Vision	73.09	73.78	72.77	73.88	73.94
ConvNext	76.22	76.39	76.49	76.54	76.12
VMamba	73.78	75.00	74.41	74.57	74.15
DeiT	73.09	73.94	74.57	73.46	73.67
ViT	76.70	76.44	76.60	75.85	77.02
Swin	80.00	81.33	81.17	80.90	80.00

Table 4: Top 1 test-performance in % for different rank configurations.

Reference	67.87
Dropout	68.73
L1 Prompt	69.42
L1 Mapping	69.37
L2 Prompt	68.84
L2 Mapping	70.31
Autoaugment	68.65

Table 5: AutoVP performances of VMamba-B using different regularization mechanics. First row reports the unregularized VP performance.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	CIFAR10	CIFAR100	Flowers102	GTSRB	DTD	EuroSAT	Food101	OxfordPets	$\mu$	$\sigma$
VMamba-T	0.990	0.999	0.996	1.047	0.999	1.083	1.000	0.999	1.014	0.031
MambaOut-T	1.000	0.999	0.999	3.674	0.997	0.691	1.254	2.275	1.486	0.938
MambaVision-T	0.995	0.998	1.401	3.111	2.195	0.927	1.182	2.342	1.644	0.756
SwinT-T	0.986	1.001	0.999	0.995	0.999	0.950	1.000	1.196	1.016	0.070
MambaVision-S	0.927	1.006	2.930	1.105	0.997	1.006	1.238	1.942	1.394	0.656
MambaVision-B	1.003	1.006	1.002	3.912	2.362	0.994	1.386	1.574	1.655	0.960

Appendix

Table 6: Prompt scaling of the best run for the respective model-task combination

Vision Foundation Model Adaptation

Vision foundation model adaptation tailors pretrained image and video models to new tasks and domains while balancing accuracy, efficiency, and generalization.

See all 326 papers

Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation
Jul 24, 2026Yuqi Li, Xi Xiao, Yunbei Zhang +6Information Bottleneck Vision Foundation Model Adaptation
City College of New York, CUNY · New York, NY, USA · University of Alabama at Birmingham +11
Low-Rank Adaptation Redux for Large Models
Apr 23, 2026Bingcong Li, Yilang Zhang, Georgios B. GiannakisAdapter Tuning Low-Rank Adaptation
Dept. of CS, ETH Zürich, Switzerland · Dept. of ECE, U. of Minnesota, USA · Morgan Stanley, USA
Efficient PEFT Methods with Adaptive Checkpointing for Vision Models and VLMs on Resource Constrained Consumer-GPUs
Jul 2, 2026Altay Toktassyn, Jurn-Gyu ParkFine-Tuning Memory-Efficient Fine-Tuning
Department of Computer Science, Nazarbayev University, Astana 010000, Kazakhstan

Jul 24, 2026J/K move · Enter open · S save

Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation

Yuqi Li, Xi Xiao, Yunbei Zhang, Lin Zhao+5

City College of New York, CUNY · New York, NY, USA · University of Alabama at Birmingham+11

Information Bottleneck Vision Foundation Model Adaptation Visual Prompt Learning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Vision foundation models are increasingly reused as frozen backbones for downstream visual recognition, making parameter-efficient adaptation a central problem. Prompt-based adaptation, including Visual Prompt Tuning (VPT), provides a lightweight way to specialize these models, but its layer-wise behavior remains poorly understood: performance is sensitive to prompt depth, placement, and task distribution, and gains on standard in-domain benchmarks do not always translate into robust generalization. We argue that this limitation is not solely an optimization issue, but a layer-wise information allocation issue: existing prompt-based methods lack principled control over what prompt-conditioned representations should preserve, suppress, and propagate across depth. Inspired by the Information Bottleneck principle, we introduce Prompted Information Bottlenecks (PIB), a framework that regularizes layer-wise compression-sufficiency trade-offs and promotes a more coherent cross-layer information path. The key idea is that effective adaptation should be minimal yet sufficient, retaining task-relevant local evidence in earlier layers while progressively discarding nuisance factors and redundant details in deeper layers. Extensive experiments show that PIB achieves strong performance across 34 datasets, reaching 92.1% on FGVC, 93.01% on HTA, and 77.33% on VTAB-1k, while tuning only 0.35% parameters on average across the main settings. Beyond benchmark accuracy, PIB helps explain the non-monotonic behavior of prompt capacity scaling, reduces shortcut reliance, and improves robustness under distribution shift and fine-grained recognition settings. These results position PIB as both a practical method and an information-allocation perspective for adapting frozen vision foundation models. Our code is available at https://github.com/itsnotacie/MM-26-PIB

Abstract

Vision Foundation Model Adaptation

Explore similar work

Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation

Low-Rank Adaptation Redux for Large Models

Efficient PEFT Methods with Adaptive Checkpointing for Vision Models and VLMs on Resource Constrained Consumer-GPUs