$σ$Transfer: Uncertainty Transfer from Small to Large Networks under $μ\mathrm{P}$

Richard Bergna; Fernando Ruiz Mazo; Nicolò Felicioni; José Miguel Hernández-Lobato; Kamil Ciosek

stat.MLOct 8, 2026

$σ$ Transfer: Uncertainty Transfer from Small to Large Networks under $μ\mathrm{P}$

Authors: Richard Bergna, Fernando Ruiz Mazo, Nicolò Felicioni, José Miguel Hernández-Lobato, Kamil Ciosek

Organizations: University of Cambridge · Spotify

Abstract

Reliable predictive uncertainty in Laplace approximations depends critically on the prior precision, yet selecting it requires a posterior sweep that is prohibitively expensive for neural networks with billions of parameters. Under the Maximal Update Parametrization ( $μ\mathrm{P}$ ), we derive a rescaling of the prior covariance that makes the selected precision stable as model width grows. This leads to $σ\mathrm{Transfer}$ : we select the precision on a smaller model and zero-shot transfer it to the much larger model, i.e., without searching for the precision on the larger model at all. We show convergence of the prior kernel, posterior covariance, selected precision, and posterior-derived decisions under explicit conditions, and verify $σ\mathrm{Transfer}$ across regression, image classification, and Transformer readouts. For example, measured precision-sweep speedups reach $\sim 5000\times$ when transferring from width 128 to 4096 on MNIST, at a target-NLL degradation of $0.002$ ; transferring from a public 1B to 7B model gives a median search speedup of $\sim 2.3\times$ (up to $\sim 330\times$ ), with a mean measured target-NLL increase below $10^{-4}$ across ten tasks. The same posterior stability also enables transfer of acquisition, OOD-detection, and abstention decisions without constructing a target posterior.

Figures & tables

Figure 1: Why prior precision transfers under

\sigma\mathrm{Transfer}

. The same regression architecture at widths

128

512

, and

4096

, under standard parametrization ( SP, red ) and

\sigma\mathrm{Transfer}

(blue) . (a) The prior kernel, in a common two-dimensional projection. (b) Predictive posteriors using the precision selected at width

128

. (c) Plug-in evidence curves used to select it; the dashed line marks the width-

128

optimum. (d) Predictive NLL per example, width

128

versus wider networks; points on the diagonal indicate agreement. Overall,

\sigma\mathrm{Transfer}

remains stable across width, while SP can drift.

Figure 2: Evidence as a function of prior precision

\lambda

on FreeSolv. Light and dark curves show proxy (width

512

) and target (width

4096

); bands are mean

\pm

SEM over five seeds. Under SP the landscape shifts with width, so the proxy’s optimum is far from the target’s. Under

\sigma\mathrm{Transfer}

the two curves nearly overlap. Table 1 reports numerical results for all three datasets.

Figure 3: Sequential active learning on MNIST with proxy acquisition (EPIG, proxy width

128

, target width

4096

, 20 seeds, bands: 95% CI). Left and centre: target test accuracy under proxy, target-reference, and random acquisition. Under

\sigma\mathrm{Transfer}

the proxy and target curves nearly overlap; under SP + isotropic the target’s own acquisition falls below random. Right: per-round Spearman

\rho

between proxy and target rankings. See Appendix F for Fashion-MNIST and AULC contrasts.

Figure 4: A prior precision chosen on a proxy transfers under

\sigma\mathrm{Transfer}

, not under SP. Left: evidence-based selection on in-distribution training data, per task (thin) and task mean (thick). Middle and right: target NLL at the proxy’s

\lambda

minus the target’s own (lower is better), same ten tasks.

Figure 5: Sequential active learning on AG News, starting with

50

500

5{,}000

labelled examples and acquiring

250

additional labels using EPIG. Top: Spearman correlation between the proxy’s and target’s rankings of unlabelled data by informativeness (EPIG), averaged over acquisition steps. Bottom: mean absolute test-NLL gap when the target is trained on proxy-selected rather than target-selected labels (ticks

\times 10^{-3}

; lower is better). All methods use

\lambda=1

. Means and

95\%

CIs over ten seeds; the starred setting uses five.

Task	Dataset	Transfer	Approx.	Proxy (s)	Target (s)	Speedup	Degradation
$\lambda$ sweep	MNIST	$128\!\to\!4096$	LL full	$1.5$	$7{,}560$	${\color[rgb]{0,0.3125,0.6289}{\sim}5000\times}$	$0.002$ NLL
OOD-aware $\lambda$	MNIST	$128\!\to\!4096$	all $+$ diag	$4.1$	$384$	${\color[rgb]{0,0.3125,0.6289}94\times}$	$0.001$ AUROC
evidence search	LLM (ours)	$128\!\to\!2048$	LL full	$3.5$	$25$	${\color[rgb]{0,0.3125,0.6289}{\sim}7\times}$	$0.0014$ NLL
evidence search	LLM (public)	1B $\to$ 7B	LL full	$4$	$9$	${\color[rgb]{0,0.3125,0.6289}{\sim}2.3\times}$	$0$ NLL
EPIG acquisition	AG News	$128\!\to\!2048$	LL full	$64.9$	$187.6$	${\color[rgb]{0,0.3125,0.6289}{\sim}3\times}$	$0.0014$ ACC
evidence search	PenDigits	$32\!\to\!128$	all $+$ full	$0.68$	$14.95$	${\color[rgb]{0,0.3125,0.6289}{\sim}22\times}$	$-0.007$ NLL

Table 3:

\sigma\mathrm{Transfer}

does the expensive selection on the proxy rather than the target. Speedup is target

\div

proxy selection time; deployment still fits one target posterior, so the end-to-end saving is capped by the

61

-point grid. Degradation is the resulting change in the stated metric, positive meaning worse. EPIG times average 10 seeds; all times exclude common MAP training and one-off frozen-trunk feature extraction. For the public 1B

\to

7B pair, the row reports AG News, whose speedup is the median across nine timed tasks, calculated from the rounded times in Table 15 . LL = last-layer.

Location	Contents
Appendix A	Limitations and directions for future work.
Appendix B	Metrics for prior-precision and decision transfer.
Appendix C	Curvature approximations, the geometric mechanism, regression setup, and additional normalization baselines.
Appendix D	Evidence-selected precision transfer on MNIST and Fashion-MNIST.
Appendix E	Additional results under distribution shift.
Appendix F	Sequential active-learning setup and results.

Table 4: Roadmap to the appendix.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Time-to-quality tradeoff for OOD precision transfer (

128\!\to\!4096

, all-parameter diagonal GGN). Filled markers use the proxy-selected precision; open markers use target-side selection. Arrows point toward the cheaper proxy search. The reported A100 times compare proxy-side and target-side selection procedures (

4.1

seconds versus

6.4

minutes,

94\times

). Zero-shot transfer removes the target sweep but retains one target posterior at the transferred precision. The right panel reports paired transfer regret over 10 seeds. Only

\sigma\mathrm{Transfer}

removes the target search without measurable OOD-quality loss.

Method	AUROC using $\lambda^{\star}_{128}$	AUROC using $\lambda^{\star}_{4096}$	Transfer regret $\downarrow$
SP	$0.741\pm 0.029$	$0.813\pm 0.027$	$0.072\pm 0.046$
$\mu\mathrm{P}$ + isotropic	$0.418\pm 0.032$	$0.745\pm 0.020$	$0.327\pm 0.038$
$\sigma\mathrm{Transfer}$ (ours)	$\mathbf{0.950\pm 0.003}$	$\mathbf{0.950\pm 0.004}$	$\mathbf{0.001\pm 0.004}$

Appendix

Table 8: OOD-aware precision transfer at width

4096

(10 seeds; mean

\pm

sample s.d.). The precision is selected on MNIST-validation versus FMNIST and evaluated on disjoint MNIST-test versus EMNIST

+

KMNIST. Transfer regret is

\mathrm{AUROC}(\lambda^{\star}_{4096})-\mathrm{AUROC}(\lambda^{\star}_{128})

; lower is better.

Method	Target width $n$	AUROC using $\lambda^{\star}_{128}$	AUROC using $\lambda^{\star}_{n}$	Regret $\downarrow$
SP	128	$0.957\pm 0.005$	$0.957\pm 0.005$	$0.0000\pm 0.0000$
	512	$0.909\pm 0.008$	$0.909\pm 0.008$	$0.0000\pm 0.0000$
	2048	$0.807\pm 0.012$	$0.802\pm 0.013$	$-0.0044\pm 0.0193$
	4096	$0.741\pm 0.029$	$0.813\pm 0.027$	$0.0716\pm 0.0455$
$\mu\mathrm{P}$ + isotropic	128	$0.957\pm 0.004$	$0.957\pm 0.004$	$0.0000\pm 0.0000$
	512	$0.868\pm 0.024$	$0.868\pm 0.024$	$0.0000\pm 0.0000$

Appendix

Table 9: OOD-aware precision transfer across width (10 seeds; mean

\pm

sample s.d.). Both AUROC columns evaluate target width

n

;

\lambda^{\star}_{128}

is selected on the proxy and

\lambda^{\star}_{n}

on that target. Regret is

\mathrm{AUROC}(\lambda^{\star}_{n})-\mathrm{AUROC}(\lambda^{\star}_{128})

on the held-out evaluation distributions. The main text reports

n=4096

Figure 7: Sequential active learning on Fashion-MNIST (20 seeds; panels and encoding as in Figure 3 ). The agreement ordering replicates with a compressed baseline spread (

\bar{\rho}=0.73

for

\sigma\mathrm{Transfer}

against

0.36

–

0.43

). On this task the acquisition curves do not separate from random, so we claim decision transfer on both datasets and downstream utility on MNIST only.

Figure 8: Sequential active learning: decisions and outcomes (10-class MNIST, last-layer diagonal posterior at fixed

\log_{2}\lambda=0

, width

128\!\to\!4096

, 20 seeds). Left: per-round Spearman correlation between the width-

128

and width-

4096

acquisition scores (band: 95% CI of the mean).

\sigma\mathrm{Transfer}

holds

\rho\!\approx\!0.6

–

0.8

for the whole trajectory while every baseline decays toward zero, and the trajectory means are

\bar{\rho}=0.75\pm 0.01

(

\sigma\mathrm{Transfer}

0.38\pm 0.02

(

\mu\mathrm{P}

+ isotropic),

0.24\pm 0.01

(SP + metric) and

0.07\pm 0.03

(SP + isotropic). Right: paired area-under-the-learning-curve contrasts (per seed, then averaged; bars: 95% CI). Filled: proxy

-

target reference, so

0

means the proxy acquires as well as the target’s own posterior; open: proxy

-

random. Only

\sigma\mathrm{Transfer}

is simultaneously faithful (its fidelity interval is the only one covering zero,

-0.0001

[-0.005,+0.005]

) and useful (

+0.018

[+0.014,+0.022]

); SP + isotropic’s apparent

+0.039

fidelity reflects a degenerate target reference that itself falls

0.050

below random. Full learning curves: Figure 10 ; Fashion-MNIST: Figures 9 and 11 .

Figure 9: Sequential active learning on Fashion-MNIST: decisions and outcomes (20 seeds; same protocol and panels as Figure 8 ). The agreement ordering replicates (

\bar{\rho}=0.73\pm 0.02

for

\sigma\mathrm{Transfer}

against

0.36

–

0.43

; see also Figure 7 ), with a compressed baseline spread. The outcome panel is negative for every method: on this task EPIG acquisition does not beat random over the trajectory (a cold-start effect that endpoint accuracy alone conceals). At

20

seeds every method’s utility interval excludes zero;

\sigma\mathrm{Transfer}

is the least affected (

-0.011

[-0.018,-0.003]

) against

-0.047

[-0.055,-0.039]

for SP + isotropic. We therefore claim decision transfer on both datasets and downstream utility on MNIST only.

Figure 10: Sequential active learning on 10-class MNIST (last-layer diagonal posterior, 20 seeds; bands: 95% CI of the mean). Every curve is averaged over the same complete-trajectory seed set. One panel per method, titled above each panel. Orange selects each acquisition batch with the width-

128

posterior and blue with the width-

4096

posterior; both are then evaluated on the same width-

4096

model, and the grey band between them is the acquisition-transfer gap. The thin orange line is drawn over the thicker blue one, so a method in which the proxy reproduces the target’s decisions shows orange tracing blue with no blue visible beneath. Markers (circles for width

128

, squares for width

4096

) are stamped every seventh acquisition round and carry no information beyond series identity.

\bar{\rho}

is the mean rank correlation between the two posteriors’ acquisition scores over the trajectory, the direct measure of how closely the small model reproduces the large model’s decisions. Accuracy on this task is largely insensitive to acquisition quality, but the decisions are not:

\bar{\rho}

rises from

0.07

(SP + isotropic) through

0.24

(SP + metric) and

0.38

(

\mu\mathrm{P}

+ isotropic) to

0.75

under

\sigma\mathrm{Transfer}

, so neither the parametrization nor the prior geometry alone suffices.

Figure 11: Sequential active learning on Fashion-MNIST (last-layer diagonal posterior,

20

seeds; bands: 95% CI of the mean; same protocol and encoding as Figure 10 ). The decision-agreement ordering matches MNIST, with

\bar{\rho}=0.73

under

\sigma\mathrm{Transfer}

, compared with

0.36

–

0.43

for the three baselines. SP + isotropic has higher agreement on Fashion-MNIST than on MNIST (

0.36

versus

0.07

Figure 12: Direct acquisition transfer (EPIG, exact last-layer posterior, validation-NLL-selected transferred precision, 10 seeds; bands 95% CI of the mean). (a) Proxy–target acquisition-rank agreement. (b) Top-10 Jaccard overlap. (c) Symmetric predictive KL. The target posterior is used only to measure agreement; the proxy choices do not require it.

		Test acc.	Test NLL	Regret
SP	random	$0.901\pm 0.016$	$0.976\pm 0.141$	$+0.009$
SP	proxy EPIG	$0.920\pm 0.012$	$0.774\pm 0.171$	$-0.011$
SP	oracle EPIG	$0.910\pm 0.008$	$0.931\pm 0.111$	—
$\mu\mathrm{P}$ + isotropic	random	$0.890\pm 0.009$	$0.920\pm 0.120$	$+0.028$
$\mu\mathrm{P}$ + isotropic	proxy EPIG	$0.889\pm 0.021$	$0.909\pm 0.298$	$+0.029$
$\mu\mathrm{P}$ + isotropic	oracle EPIG	$0.918\pm 0.004$	$0.694\pm 0.107$	—

Appendix

Table 11: Sequential active-learning utility at

n=1000

(10-class MNIST, last-layer diagonal posterior at fixed

\log_{2}\lambda=0

, width

128\!\to\!4096

; mean

\pm

sample s.d.). Regret

=

oracle-EPIG accuracy

-

method accuracy. Round wall-clock is dominated by retraining the width-

4096

model and is the same for every strategy, so this experiment is a decision-transfer result, not a speed result.

ID tasks (test-set sizes)	Source $A$ (pool size)	Source $B$ (pool size)
AG News (7600), DBpedia (8000)	SST-2 (872)	TREC (500)
IMDB (8000), Rotten Tomatoes (1066), Yelp Polarity (8000), SST-2 (872)	AG News (7600)	TREC (500)
Emotion (2000), SNLI (8000), Yahoo (8000), TREC (500)	AG News (7600)	SST-2 (872)

Appendix

Table 12: OOD source assignments and test-set sizes.

Figure 13: Figure 4 with the geometry ablation added: the same

\mu\mathrm{P}

checkpoints with an isotropic prior drift faster than SP (

5.5\to 10.2

bits mean optimum,

4.7\pm 1.3

bits at

128\to 2048

) and produce the largest mean target-NLL degradation (

0.028

, reaching

0.1

on TREC).

	$\sigma\mathrm{Transfer}$			SP $+$ isotropic
Dataset	$\lambda^{\star}_{128}$	$\lambda^{\star}_{2048}$	$\Delta$ NLL	$\lambda^{\star}_{128}$	$\lambda^{\star}_{2048}$	$\Delta$ NLL
AG News	$-3.5$	$-2.5$	$+0.0017$	$+4.5$	$+7.0$	$+0.0146$
DBpedia	$-4.5$	$-3.5$	$+0.0082$	$+5.0$	$+7.5$	$+0.0458$
Emotion	$-3.0$	$-2.5$	$-0.0001$	$+6.0$	$+8.5$	$+0.0012$
IMDB	$-2.0$	$-1.5$	$-0.0000$	$+6.5$	$+9.5$	$+0.0006$
Rotten Tomatoes	$-2.0$	$-1.5$	$+0.0002$	$+7.0$	$+10.5$	$+0.0020$

Appendix

Table 14: Our language models, width

128\to 2048

(Section 6.3 ).

\lambda

values are

\log_{2}

Dataset	gap	$\Delta$ NLL	gap	$\Delta$ NLL	search 1B	search 7B	speedup
	u- $\mu\mathrm{P}$ $+$ $\sigma$ -metric		SP $+$ isotropic
AG News	$0.0$	$+0.0000$	$0.5$	$+0.0030$	$4$ s	$9$ s	${\sim}2.3\times$
DBpedia	$0.0$	$+0.0000$	$1.0$	$+0.0247$	$40$ s	$3.7$ h	$330\times$
Emotion	$0.5$	$+0.0002$	$1.0$	$+0.0030$	$5$ s	$24$ s	$5\times$
IMDB	$0.0$	$+0.0000$	$0.5$	$-0.0016$	$2$ s	$4$ s	$2\times$
Rotten Tomatoes	$0.5$	$+0.0002$	$0.5$	$+0.0011$	$2$ s	$3$ s	$2\times$

Appendix

Table 15: Public u-

\mu\mathrm{P}

/SP pairs, 1B

\to

7B, all ten tasks (primary cells; extends Table 17 ). Each model’s search time is its curvature eigendecomposition plus grid sweep (u-

\mu\mathrm{P}

family; SP within

10\%

); Yahoo and DBpedia run on CPU because their

41

57

k-dimensional eigenproblems exceed the GPU solver. The reported ratios compare proxy- and target-side search procedures, not end-to-end Regime-A deployment; the 7B model still constructs one posterior at the transferred precision. Timing results are available for nine tasks; SST-2 is excluded from the timing summary. Speedups are approximate ratios of the displayed, rounded times.

Parametrization	$\|\Delta\log_{2}\alpha\|$	$\Delta\mathrm{NLL}$	Evidence: interior	Val. NLL: interior
SP	$3.95$	$+0.0367$	$10/10$	$4/10$
$\mu\mathrm{P}$	$3.50$	$+0.0229$	$10/10$	$1/10$

Appendix

Table 16: Diagonal neural g-prior on our language models (

128\to 2048

, ten tasks, one checkpoint seed).

|\Delta\log_{2}\alpha|

is the mean absolute proxy–target selection gap and

\Delta\mathrm{NLL}

is mean transferred-minus-target-local test NLL, both under the fixed-checkpoint plug-in evidence score. Counts give tasks with both proxy and target selections strictly inside the search grid. Validation-NLL gaps are omitted because only five of twenty pairs satisfy this condition.

Cell	gap, plug-in evid. (bits)	gap, corrected evid.	$\Delta$ NLL
u- $\mu\mathrm{P}$ $+$ $\sigma$ -metric	$0.19\pm 0.26$	$0.56\pm 0.32$	$+4\times 10^{-5}$
SP $+$ $\sigma$ -metric	$0.06\pm 0.18$	$0.38\pm 0.23$	$-1\times 10^{-4}$
u- $\mu\mathrm{P}$ $+$ isotropic	$0.88\pm 0.23$	$1.19\pm 0.65$	$+1.1\times 10^{-3}$
SP $+$ isotropic (standard)	$0.69\pm 0.46$	$0.56\pm 0.42$	$+8.6\times 10^{-4}$

Appendix

Table 17: Public 1B

\to

7B prior-precision transfer on eight tasks excluding Yahoo and DBpedia. Precision gaps are in bits (mean

\pm

s.d. across tasks); the

61

-point grid has

0.5

-bit spacing.

\Delta

NLL is mean target test NLL at the proxy’s precision minus that at the target’s own precision, both selected by fixed-checkpoint plug-in evidence. The two primary comparisons cover all ten tasks in Table 15 . Sweep arrays use

200

MC samples in fp16.

Parameter block	Stored-coordinate $s_{g,n}$
Input weight	$\mathrm{fan\_out}$
Hidden weight	$\mathrm{fan\_out}/\mathrm{fan\_in}$
Hidden bias	$\mathrm{fan\_out}$
MuReadout weight	$1/(\mathrm{fan\_in}\,c_{\rm out}^{2})$
Output bias	$1$

Appendix

Table 18: Prior-covariance scales for Linear/MLP parameters, with

c_{\rm out}=\texttt{output\_mult}/\texttt{width\_mult}

Explore similar work

Scalable AI Uncertainty Quantification via Generalized Laplace Active Subspaces
Oct 8, 2026Wouter N. Edeling, Peter V. CoveneyUncertainty Quantification Uncertainty Calibration
Scientific Computing Group, CWI, Science park 123, Amsterdam, 1098XG, the Netherlands. · Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente, Hallenweg 15, Enschede, 7500AE, the Netherlands. · Centre for Computational Science, University College London, 20 Gordon Street, London, WC1H 0AJ, UK. +1
Optimality of Sub-network Laplace Approximations: New Results and Methods
May 9, 2026Swarnali Raha, Kshitij Khare, Rohit K PatraBayesian Neural Networks Uncertainty Quantification
Department of Statistics University of Florida Gainesville, FL · Linkedin Inc. New York, NY
$μ$ pscaling small models: Principled warm starts and hyperparameter transfer
Feb 11, 2026Yuxin Ma, Nan Chen, Mateo Díaz +3Neural Network Optimization Hyperparameter Transfer
Department of Applied Mathematics and Statistics Johns Hopkins University

Oct 8, 2026, cs.AIJ/K move · Enter open · S save

Scalable AI Uncertainty Quantification via Generalized Laplace Active Subspaces

Wouter N. Edeling, Peter V. Coveney

Scientific Computing Group, CWI, Science park 123, Amsterdam, 1098XG, the Netherlands. · Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente, Hallenweg 15, Enschede, 7500AE, the Netherlands. · Centre for Computational Science, University College London, 20 Gordon Street, London, WC1H 0AJ, UK.+1

Uncertainty Quantification Uncertainty Calibration Bayesian Inference

Open paper AskPDF arXiv

Reliable uncertainty quantification (UQ) is essential for deploying neural networks in scientific and high-stakes applications, but full Bayesian inference over the network parameters is computationally infeasible. We propose a low-rank generalized Laplace approximation for neural-network UQ based on a small number of data-informed curvature directions. Starting from a generalized Bayesian posterior defined through an empirical loss, we construct a local Gaussian approximation around a pretrained set of weights in this active curvature subspace. The posterior variances in the retained subspace are available in closed form, and the prior variance is calibrated by an empirical Bayes procedure. The generalized Bayesian formulation allows us to compare two posterior scalings: the standard Bayesian scaling associated with the summed negative log likelihood, and a mean-loss scaling in which the empirical loss is normalized by the number of data. A central finding is that the standard scaling induces a data-size dependent contraction of the posterior variance in the leading active directions. In regression problems, this can force the low-rank framework to retain additional weak-curvature directions in order to achieve nominal coverage of calibration data. When posterior samples are propagated through the non-linear network, these additional directions can degrade the coherence of the predictive intervals and shift the posterior predictive mean away from the pretrained model. In contrast, the generalized mean-loss scaling yields a more stable, lower dimensional active subspace and produces calibrated, coherent predictive confidence intervals. These results indicate that generalized Laplace active subspaces provide a practical and scalable route to calibrated uncertainty quantification in neural networks.

Abstract

Explore similar work

Scalable AI Uncertainty Quantification via Generalized Laplace Active Subspaces

Optimality of Sub-network Laplace Approximations: New Results and Methods

μμμpscaling small models: Principled warm starts and hyperparameter transfer

$μ$ pscaling small models: Principled warm starts and hyperparameter transfer