Weight Space Learning for Certifiable Few-shot Transfer Learning

Fady Rezk; Royson Lee; Henry Gouk; Timothy Hospedales; Minyoung Kim

cs.LGFeb 10, 2025

Weight Space Learning for Certifiable Few-shot Transfer Learning

Authors: Fady Rezk, Royson Lee, Henry Gouk, Timothy Hospedales, Minyoung Kim

Organizations: School of Informatics, University of Edinburgh · Samsung AI Center, Cambridge

Abstract

In contemporary deep learning, a prevalent and effective workflow for solving low-data problems is adapting powerful pre-trained foundation models (FMs) to new tasks via parameter-efficient fine-tuning (PEFT). However, while empirically effective, the resulting solutions lack generalisation guarantees to certify their accuracy - which may be required for ethical or legal reasons prior to deployment in high-importance applications. In this paper we develop a novel transfer learning approach that is designed to facilitate non-vacuous learning theoretic generalisation guarantees for downstream tasks, even in the low-shot regime. Specifically, we first use upstream tasks to train a distribution over PEFT parameters. We then learn the downstream task by a sample-and-evaluate procedure -- sampling plausible PEFTs from the trained diffusion model and selecting the one with the highest likelihood on the downstream data. Crucially, this confines our model hypothesis to a finite set of PEFT samples. In contrast to the typical continuous hypothesis spaces of neural network weights, this facilitates tighter risk certificates. We instantiate our bound and show non-trivial generalization guarantees compared to existing learning approaches which lead to vacuous bounds in the low-shot regime.

Figures & tables

Figure 1: Contrasting meta-learning paradigms. Top row (MAML): Traditional gradient-based meta-learning. During meta-training, MAML learns an initialization point

\theta^{0}

positioned close to all task-specific optima. At meta-test, a few gradient descent steps adapt

\theta^{0}

to the new task’s optimum

\theta^{*}

\theta^{0}

is the transferred meta-knowledge. Bottom row (STEEL, Ours): Manifold-based meta-learning. During meta-training, we collect adapters

\{\theta_{i}\}

from upstream tasks and fit a diffusion model to capture their distribution—the task solution manifold. At meta-test, we sample candidates from the learned manifold and select

\theta^{*}

by evaluating support set loss

r(\theta)

, replacing continuous optimization with discrete selection over a finite hypothesis space. The diffusion model is the transferred meta-knowledge.

Figure 2: Generalization bounds for adapting CLIP to novel tasks (5-way classification with 1–16 examples per class). Plots show classification error (y-axis) versus the complexity term (x-axis, log scale; square root terms from Equations 3 , and 7 ). Top/Bottom: Mean support/query (train/test) error on new tasks. Shaded regions indicate vacuous bounds, where (support error + complexity)

\geq 1

. Non-vacuous guarantees lie in the unshaded region. Competing methods (SGD, BBPT) fail to achieve non-vacuous bounds. In contrast, our method yields non-vacuous guarantees without significantly compromising training fit (top) or test accuracy (bottom).

Figure 3: Distribution of generalisation guarantees (x-axis, log scale) obtained over few-shot LLM adaptation episodes. Vertical lines indicate the vacuous bound threshold. STEEL provides a dramatically better distribution of provable generalisation outcomes compared to alternatives.

		SGD	LoRA-Hub	MeZO	STEEL
LaMP-2	% Non-Vacuous Tasks	0.00%	32.13%	0.00%	65.12%
	Average Gap	9.03	0.38	9.27	0.36
	Min Bound	3.32	0.59	3.60	0.47
	Median Bound	8.52	1.12	8.85	0.80
	Max Bound	20.86	2.90	21.36	1.99
	Accuracy $\uparrow$	63.25%	57.51%	63.30%	63.74%

Table 1: LaMP LLM adaptation benchmark results.

Figure 4: Dependence of generalisation guarantee on training set size. Our finite-hypothesis class learner STEEL achieves non-vacuous guarantees from 4-shot onward. Standard approaches provide no guarantees anywhere in this low-shot range.

Figure 5: “Learning curves” illustrating empirical and certified learning dynamics of STEEL with respect to samples/iterations, which is equivalent to hypothesis space size. More samples improves the training (support) error, while increasing the complexity penalty. The sum of these two terms instantiates the generalisation guarantee (Eq. 3 ) achieved for a given number of samples.

CUBirds
Method	SGD	BBPT	LoRA-Hub	MetaPB	STEEL
Non-Vacuous Ratio	0.00%	0.00%	100.00%	100.00%	100.00%
Average Gap	2.49	2.48	0.39	0.29	0.24
Min Bound	2.55	2.55	0.44	0.31	0.30
Median Bound	2.58	2.59	0.50	0.39	0.36
Max Bound	2.64	2.68	0.61	0.54	0.47

Table 2: Aggregate over 16-Shots 5-way visual recognition episodes. Standard Errors are reported in Appendix C.5

Figure 6: Support/query error and certified risk vs. support set size on iNaturalist birds. As the number of shots increases, support and query errors converge, while the bound continues to tighten. The gap between the certified risk and empirical query error drops to 6% at 128 shots, demonstrating the ability to produce tight certificates.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Median Bound	Flowers	CUB	DTD	FGVVCA
STEEL	0.40 (+0.03)	0.36 (-0.06)	0.42 (+0.11)	0.61 (+0.10)
ZS CLIP	0.43	0.30	0.53	0.71

Appendix

Table 3: Comparing median bounds between STEELand zero-shot CLIP given a 16-shots support set.

Dataset	Zoo (10K)	STEEL (10K)	STEEL (20K)
CUBirds	87.97	88.30	88.40
DescribableTextures	79.25	81.42	81.50
FGVCAircrafts	61.02	61.50	61.37
Flowers-102	82.92	84.22	84.90
Average	77.79	78.86	79.04

Appendix

Table 4: Comparison of candidate pool size and source. STEEL at 10K samples already outperforms selecting from the 10K zoo, indicating gains come from the learned distribution rather than pool size. Increasing to 20K provides marginal additional improvement.

1-Shots
Dataset	Zero Shot	SGD	BBPT	Lora-Hub	Meta-PB	Model Zoo	STEEL
CUBirds	83.82%	81.77%	85.80%	86.45%	88.25%	86.00%	86.72%
DescribableTextures	67.29%	69.97%	74.10%	69.75%	78.55%	72.12%	73.97%
FGVCAircraft	47.44%	53.00%	55.05%	54.05%	56.46%	54.37%	55.40%
Flowers-101	81.14%	83.55%	80.40%	73.02%	54.20%	74.25%	76.75%
Avg	69.92%	72.07%	73.84%	70.82%	69.37%	71.69%	73.21%

Appendix

Table 5: CLIP+CoOp few-shot learning. Accuracies over different number of shots.

CUBirds
Method	SGD	BBPT	LoRA-Hub	MetaPB	STEEL
Non-Vacuous Ratio	0.00	0.00	100.00	100.00	100.00
Average Gap	2.49 $\pm$ 0.01	2.48 $\pm$ 0.01	0.39 $\pm$ 0.01	0.29 $\pm$ 0.01	0.24 $\pm$ 0.01
Min Bound	2.55	2.55	0.44	0.31	0.30
Median Bound	2.58 $\pm$ 0.01	2.59 $\pm$ 0.01	0.50 $\pm$ 0.01	0.39 $\pm$ 0.01	0.36 $\pm$ 0.01
Max Bound	2.64	2.68	0.61	0.54	0.47

Appendix

Table 6: 16-shot 5-way: bound statistics with standard errors. Standard errors for the non-vacuous ratio, average gap and median bound are bootstrap estimates (2000 resamples with replacement); the accuracy standard error is the standard error of the mean. We report no standard error for a non-vacuous ratio of 0% or 100%, since the bootstrap estimate is zero by construction.

Dataset	Baseline	$\Delta$ Median Bound	$\Delta$ Accuracy
CUBirds	LoRA-Hub	$-0.142$ $[-0.167,-0.117]$	$+0.58$ $[-1.65,2.73]$
	MetaPB	$-0.032$ $[-0.060,-0.009]$	$-1.16$ $[-2.97,0.77]$
Describable Textures	LoRA-Hub	$-0.192$ $[-0.217,-0.154]$	$+3.90$ $[1.37,6.38]$
	MetaPB	$-0.086$ $[-0.111,-0.061]$	$+1.35$ $[-0.90,3.59]$
FGVCAircrafts	LoRA-Hub	$-0.223$ $[-0.273,-0.179]$	$+3.68$ $[-0.43,7.55]$
	MetaPB	$-0.124$ $[-0.173,-0.087]$	$+1.75$ $[-1.79,5.21]$

Appendix

Table 7: Bootstrap over episodes at 16 shots (episodes resampled independently): STEEL minus baseline, with 95% intervals. Negative bound differences favour STEEL; positive accuracy differences (percentage points) favour STEEL.

Method	SGD	BBPT	Meta-PB	STEEL
CUBirds
Non-Vacuous Ratio	0.00 ± 0.00	0.00 ± 0.00	97.50 ± 2.47	100.00 ± 0.00
Average Gap	2.49 ± 0.01	2.48 ± 0.01	0.45 ± 0.01	0.24 ± 0.01
Min Bound	2.55	2.55	0.34	0.30
Median Bound	2.58 ± 0.01	2.59 ± 0.01	0.55 ± 0.01	0.36 ± 0.01
Max Bound	2.64	2.68	0.85	0.47

Appendix

Table 8: Aggregate over 16-Shots 5-way visual recognition episodes with Standard Errors

Figure 7: Mean support (top) and query (bottom) error versus complexity term for all methods, 5-way, 1–16 shots per class. Shaded regions indicate vacuous bounds. Moving right to left, the number of shots increases.

Method	Zero-Shot	SGD	Random	GMM	Diffusion
Accuracy	61.68%	63.25%	0.00%	60.29%	63.74%

Appendix

Table 9: Comparison of generative models for sampling parameter candidates on LaMP-2. The diffusion model is the only method that improves upon both the zero-shot baseline and SGD fine-tuning.

	Clean	Mild	Moderate	Severe
Non-Vacuous Ratio	100%	100%	100%	100%
Min Bound	0.28	0.30	0.30	0.30
Median Bound	0.35	0.37	0.38	0.45
Max Bound	0.43	0.52	0.61	0.80
Avg. Accuracy	88.60%	86.70%	84.30%	76.95%

Appendix

Table 10: STEEL on CUB-C: certificate metrics across severity levels (16-shot, 5-way; 200 episodes). The non-vacuous ratio is preserved at 100% across all severities, while the bound widens gracefully in proportion to empirical-risk degradation.

Severity	Metric	MAML	STEEL
Mild	Non-Vacuous Ratio	0%	100%
	Median Bound	2.63	0.37
	Avg. Accuracy	87.82%	86.70%
Moderate	Non-Vacuous Ratio	0%	100%
	Median Bound	2.65	0.38
	Avg. Accuracy	84.72%	84.30%

Appendix

Table 11: MAML vs. STEEL on CUB-C (16-shot, 5-way). MAML provides zero non-vacuous certificates at every severity. Its accuracy advantage over STEEL is at most

\sim

1pp under mild shift, vanishes under moderate shift, and reverses under severe shift.

Transfer Learning

Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.

See all 417 papers

Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols
Feb 28, 2026Xu Luo, Ji Zhang, Lianli Gao +2Few-Shot Learning Transfer Learning
University of Electronic Science and Technology of China, Chengdu, China. · Southwest Jiaotong University, Chengdu, China. · Tongji University, Shanghai, China.
Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform
Mar 8, 2026Jianlu Shen, Fu Feng, Yucheng Xie +2Transfer Learning
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
Zero-Shot Quantization via Weight-Space Arithmetic
Apr 3, 2026Daniele Solombrino, Antonio Andrea Gargiulo, Alessandro Zirilli +3Quantization Robustness Quantization
Sapienza University of Rome · Sapienza University of Rome / Paradigma

Feb 28, 2026J/K move · Enter open · S save

Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols

Xu Luo, Ji Zhang, Lianli Gao, Heng Tao Shen+1

University of Electronic Science and Technology of China, Chengdu, China. · Southwest Jiaotong University, Chengdu, China. · Tongji University, Shanghai, China.

Few-Shot Learning Transfer Learning Fine-Tuning

Open paper AskPDF arXiv

Few-shot transfer has been revolutionized by stronger pre-trained models and improved adaptation algorithms.However, there lacks a unified, rigorous evaluation protocol that is both challenging and realistic for real-world usage. In this work, we establish FEWTRANS, a comprehensive benchmark containing 10 diverse datasets, and propose the Hyperparameter Ensemble (HPE) protocol to overcome the "validation set illusion" in data-scarce regimes. Our empirical findings demonstrate that the choice of pre-trained model is the dominant factor for performance, while many sophisticated transfer methods offer negligible practical advantages over a simple full-parameter fine-tuning baseline. To explain this surprising effectiveness, we provide an in-depth mechanistic analysis showing that full fine-tuning succeeds via distributed micro-adjustments and more flexible reshaping of high-level semantic presentations without suffering from overfitting. Additionally, we quantify the performance collapse of multimodal models in specialized domains as a result of linguistic rarity using adjusted Zipf frequency scores. By releasing FEWTRANS, we aim to provide a rigorous "ruler" to streamline reproducible advances in few-shot transfer learning research. We make the FEWTRANS benchmark publicly available at https://github.com/Frankluox/FewTrans.

Abstract

Transfer Learning

Explore similar work

Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols

Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform

Zero-Shot Quantization via Weight-Space Arithmetic