How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

Manuel Hülskamp; Julius Kobialka; Emanuel Sommer; Thomas Nagler; David Rügamer

stat.MLOct 9, 2026

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

Authors: Manuel Hülskamp, Julius Kobialka, Emanuel Sommer, Thomas Nagler, David Rügamer

Organizations: LMU Munich and Munich Center for Machine Learning

Abstract

Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in rare settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models.

Figures & tables

Panel of Figure 1 — Figure 1: Decomposition of the deployment error of a PFN ( Proposition 2 ). This paper studies the amortization error, the only term controlled by pretraining, which splits into optimization, estimation, and approximation error ( 8 ). The context and prior error support this analysis: the context error is incurred even by the exact PPD, and the prior error arises when the deployment tasks differ from the prior (see Section 3.3 ).

Figure 2: Comparison of PPD approximation quality to MCMC reference on a classification task across both Gaussian and TabICL prior feature distributions (SCM prior). The labeling mechanism is a Bayesian neural network with two hidden layers with 16 neurons each, and the large TabICL variant is trained on these priors from scratch. The PFN approximates the ground-truth PPD provided by the MCMC reference very well across different context sizes (Figure inspired by Müller et al., 2021 ). Each point is the risk on datasets of context size

n

;

\Psi

is the size distribution of pretraining.

Figure 3: Estimation error

\widehat{EE}(M)

(log-log) on synthetic data and member disagreement on TabArena (

\widehat{AU}_{\mathrm{JSD}}

\widehat{AU}_{\mathrm{dis}}

) versus the number of pretraining datasets

M

for the small models; the tiny models are shown in Fig. 10 .

	context size 1000				context size 10000
Model	DE	IVON $S{=}8$	IVON $S{=}64$	SG-MCMC	DE	IVON $S{=}8$	IVON $S{=}64$	SG-MCMC
tiny	$+3.10$	$+2.38^{\ast}$	$+2.48^{\ast}$	$+0.27^{\ast}$	$+2.65$	$-0.13$	$-0.24$	$+0.35^{\ast}$
small	$+2.89$	$+0.52^{\ast}$	$+0.65^{\ast}$	$+0.37^{\ast}$	$+2.56$	$-0.14$	$-0.04$	$+0.42^{\ast}$
medium	$+2.67$	$+0.42^{\ast}$	$+0.44^{\ast}$	$+0.07$	$+2.82$	$-0.31$	$-0.14$	$+0.08$
large	$+1.69$	$+0.23^{\ast}$	$+0.18^{\ast}$	$+0.27^{\ast}$	$+1.81$	$-0.13$	$-0.06$	$+0.32^{\ast}$

Table 1: Relative LPPD gain (%) of Bayesian model averages of TabICLv2 on 30 TabArena classification datasets at context sizes 1000 and 10000. DE : deep ensemble of 8 members (4 for the large model) over its average member. IVON : average over

S

posterior samples over the posterior mean of the same model. SG-MCMC : Bayesian deep ensemble over the DE it is warm-started from, sampled with pSMILE. ∗ : significant change, paired Wilcoxon over datasets,

p<0.05

Figure 4: Classification test performance measured by accuracy, ROC AUC and LPPD on the TabArena dataset suite across a sweep of the maximal number of context rows for deep ensembles of TabICLv2 across a range of model sizes. The corresponding results for nanoTabPFN are shown in Fig. 13 .

Figure 5: Deep ensembles of full-size TabICLv2. (a, b) LPPD and nRMSE on TALENT datasets after each training stage: ensemble of 8 members, average member and best member (selected after the fact). The gap between ensemble and average member persists while all members improve. (c) TabArena classification Elo (95% CI) of the single members (dots), the 8-member ensemble, the official TabICLv2 checkpoint and further published methods. Note that the individual DE members are trained for only 27.5% of the steps that official TabICLv2 uses.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Error decompositions used in this paper. (a) The deployment error of a PFN splits into prior, amortization and context error ( Proposition 2 ); the latter two enter with the expressible share $\lambda$ , and the prior error comprises the coverage, transport and off-support terms. For the trained network $q_{\hat{\bm{\theta}}}$ , the amortization error splits further into optimization, estimation and approximation error ( 8 ). (b) For $K$ PFNs, the error of the average member splits into the error of their mixture $\bar{q}$ , whose amortization error is the part shared by all members ( Corollary 1 ), and the gain of averaging, which splits into the removable amortization error and the transport and off-support terms of Corollary 2 .

Figure 7: Holdout NLL of Bayesian model average of

k

pooled MCMC chains for classification. The NLL saturates as more chains are added, which means a small number of chains seems sufficient to capture the functional diversity of the posterior distribution of the weights.

Figure 8: PFN to MCLMC gap under covariate shift. Test features are

\mathbf{x}=s\bm{z}

with

\bm{z}\sim\mathcal{N}(0,\bm{I}_{4})

and labels come from the BNN prior (

2\times 8

hidden neurons). The PFN was pretrained at

s=1

. Each point is the mean difference over 256 test tasks. Bands are 95% task-bootstrap intervals of each mean. Curves show the large PFN pretrained with uniform or log-uniform table size

\Psi

, and the uniform-

\Psi

PFN with context-standardized features. Dotted grey (

n=4096

only): excess NLL of the MCLMC reference over the oracle.

Figure 9: BNN regression with hidden width

8\times 8

d=4

, 512 test tasks: prior-data NLL (left) and Continuous Ranked Probability Score (CRPS; right) against the number of context points

n

for the small and large PFN, the MILE (MCLMC) reference (24 chains), and the oracle. The PFN’s NLL gap to MILE stems mostly from the density implied by its 999 predicted quantiles and on CRPS, which needs no density, the PFN nearly matches MILE.

	tiny	small	medium	large
Maximum epochs	30	40	55	70
Maximum steps	18,000	24,000	33,000	42,000
Peak learning rate, nanoTabPFN	$4\cdot 10^{-4}$	$4\cdot 10^{-4}$	–	–
Peak learning rate, TabICLv2	$4\cdot 10^{-4}$	$4\cdot 10^{-4}$	$4\cdot 10^{-4}$	$4\cdot 10^{-4}$
Ensemble members $K$	8	8	8	4

Appendix

Table 2: Size-dependent training settings of the deep ensembles in Section 5 . Maximum steps are reached only if early stopping does not trigger.

	adaSGHMC	pSMILE
Step size	$2\cdot 10^{-4}$	$10^{-4}$
Warm-up steps	500	250
Sampling steps	2500 / 3000 / 2500 / 3500	1250 / 1500 / 1250 / 1750
Thinning	every $10\%$ of the sampling steps (10 samples per chain)
Gradient evaluations per chain	3000 / 3500 / 3000 / 4000
Chains	8 / 8 / 8 / 4

Appendix

Table 3: SG-MCMC settings. Sizes vary as tiny / small / medium / large. Both samplers use the same number of gradient evaluations per chain; pSMILE takes two per step.

	tiny	small	medium	large
Effective sample size $\beta$	$1.92\cdot 10^{6}$	$3.84\cdot 10^{6}$	$6.4\cdot 10^{6}$	$1.92\cdot 10^{7}$
Learning rate	1.0	1.0	0.3	0.3
Training steps	300,000	200,000	200,000	200,000
$\beta_{1}$	0.9
$\beta_{2}$	0.99999
Weight decay	$10^{-4}$

Appendix

Table 4: IVON settings. All sizes additionally share gradient clipping at

10

, a cosine schedule with

1\%

linear warm-up decaying to

10^{-6}

, batches of 64 datasets and bfloat16 precision.

	tiny	small	medium	large	full
NanoTabPFN
Parameters (rounded)	1M	2M	4M	–	–
Transformer layers	4	4	5	–	–
Embedding dimension	128	192	256	–	–
Attention heads	4	6	8	–	–
MLP hidden dimension	256	384	512	–	–

Appendix

Table 5: Architecture hyperparameters. NanoTabPFN large is omitted due to its training cost; TabICLv2 full, used in Section F.7 , follows the architecture of Qu et al. (2026) with added QK-normalization and bounded QASSMax scaling. In TabICLv2, the ICL stage operates on the concatenated CLS column tokens (ICL width

=

embedding dimension

\times

CLS column tokens), and every block’s MLP hidden dimension is twice its width. All models predict up to 10 classes.

Figure 10: As Fig. 3 , for the tiny models: estimation error

\widehat{EE}(M)

(log-log) on synthetic data and member disagreement on TabArena versus the number of pretraining datasets

M

. Bands show

\pm 1

standard error over the

K{=}4

members (

\widehat{EE}

) or over datasets (disagreement).

Figure 11: Member disagreement (

\widehat{AU}_{\mathrm{JSD}}

\widehat{AU}_{\mathrm{dis}}

) on the synthetic test set versus the number of pretraining datasets

M

, complementing the TabArena results of Fig. 3 . Top: tiny, bottom: small models.

Model	$\widehat{AU}_{\mathrm{JSD}}$	$\widehat{AU}_{\mathrm{dis}}$	$\widehat{AU}_{\mathrm{JSD}}$	$\widehat{AU}_{\mathrm{dis}}$	$\widehat{AU}_{\mathrm{JSD}}$	$\widehat{AU}_{\mathrm{dis}}$
	synthetic		TabArena, context 1000		TabArena, context 10000
tiny (1.5M)	16.1	9.4	$12.5\pm 2.0$	$7.5\pm 1.4$	$11.8\pm 1.6$	$8.0\pm 1.5$
small (2M)	15.5	9.2	$11.4\pm 1.6$	$7.5\pm 1.5$	$11.0\pm 1.4$	$7.8\pm 1.5$
medium (7.5M)	–	–	$9.7\pm 1.5$	$6.7\pm 1.4$	$10.5\pm 1.3$	$7.0\pm 1.4$
large (9M)	13.6	8.2	$7.4\pm 1.4$	$5.9\pm 1.2$	$8.5\pm 1.4$	$6.5\pm 1.2$

Appendix

Table 6: Member disagreement (

K{=}4

) across TabICLv2 sizes on the synthetic test set and on the 30 TabArena classification datasets (mean

\pm

standard error over datasets).

\widehat{AU}_{\mathrm{JSD}}

10^{-3}

nats,

\widehat{AU}_{\mathrm{dis}}

in %.

Figure 12: Share of the total pointwise gain carried by the test points with the largest gain, for tiny (left) and large (right) TabICLv2 with K = 4 members. Dashed: gain spread evenly over all points.

Figure 13: As Fig. 4 , for nanoTabPFN: accuracy, ROC AUC and LPPD on the 30 TabArena classification datasets across the maximal number of context rows, for deep ensembles and their average single member at the tiny and small model sizes.

Model	$S$	Accuracy $\uparrow$	ROC AUC $\uparrow$	LPPD $\uparrow$
tiny	0	0.8073	0.8737	$-0.4464$
	1	0.8035 -	0.8714 -	$-0.4558^{-}$
	8	0.8068	0.8738	$-0.4493^{-}$
	16	0.8063	0.8742	$-0.4488^{-}$
	32	0.8077	0.8752	$-0.4484$
	64	0.8072	0.8749	$-0.4489$

Appendix

Table 7: IVON variational sampling on the 170 TALENT classification datasets.

S=0

is the posterior mean;

S\geq 1

averages

S

posterior samples without the mean. LPPD is the mean log predictive density per test point. + / - : significantly better/worse than the same model at

S=0

(paired Wilcoxon over datasets,

p<0.05

	context size 1000		context size 10000
Model	adaSGHMC	pSMILE	adaSGHMC	pSMILE
tiny	$-0.05$ (11/30)	$+0.27^{\ast}$ (26/30)	$-0.05$ (11/30)	$+0.35^{\ast}$ (26/30)
small	$-0.64^{\ast}$ (4/30)	$+0.37^{\ast}$ (22/30)	$-1.12^{\ast}$ (5/30)	$+0.42^{\ast}$ (25/30)
medium	–	$+0.07$ (12/30)	–	$+0.08$ (13/30)
large	$+0.12$ (18/30)	$+0.27^{\ast}$ (26/30)	$-0.52^{\ast}$ (9/30)	$+0.32^{\ast}$ (26/30)

Appendix

Table 8: Relative LPPD gain (%) of the Bayesian deep ensemble over the deep ensemble it is warm-started from, for both samplers, on the 30 TabArena classification datasets at context sizes 1000 and 10000. In parentheses: number of datasets on which the Bayesian deep ensemble improves on the deep ensemble. ∗ : significant change, paired Wilcoxon over datasets,

p<0.05

	steps			wall-clock (h)
Stage (max. rows)	official	ours	steps/h	official	ours
1 (1024)	$500{,}000$	$130{,}000$	$6{,}033$	$82.9$	$21.5$
2 (10,240)	$40{,}000$	$10{,}400$	$2{,}150$	$18.6$	$4.8$
3 (60,000)	$10{,}000$	$11{,}000$	$650$	$15.4$	$16.9$
Total, one model	$550{,}000$	$151{,}400$		$116.9$	$43.3$
GPU hours, one model				$467$	$173$

Appendix

Table 9: Training budget of one full-size member against the official TabICLv2 recipe ( Qu et al., 2026 ) . Throughput was measured for our setup (bfloat16, four H200 GPUs, batch size 64); the official durations apply our throughput to its step counts and are therefore an estimate. GPU hours count four GPUs.

Classification (170 datasets)
Stage		LPPD $\uparrow$	Brier $\downarrow$	ROC AUC $\uparrow$
1	average member	$-0.3713$	$0.2132$	$0.9021$
	best member	$-0.3693$	$0.2122$	$0.9024$
	deep ensemble	$-0.3684^{\ast}$	$0.2116^{\ast}$	$0.9032^{\ast}$
2	average member	$-0.3634$	$0.2089$	$0.9039$
	best member	$-0.3619$	$0.2082$	$0.9045$

Appendix

Table 10: Full-size TabICLv2 on TALENT after each training stage. Average member averages each dataset’s metric over the 8 members; best member is the member with the best mean at that stage, selected after the fact. ∗ : deep ensemble significantly better than the best member (paired Wilcoxon over datasets,

p<0.05

	$\leq$ 1024	1024–10240	$>$ 10240
Classification	$n=33$	$n=100$	$n=37$
Stage 1	$+0.85$	$+0.70$	$+0.86$
Stage 2	$+0.85$	$+0.77$	$+0.72$
Stage 3	$+0.85$	$+0.78$	$+0.75$
Regression	$n=21$	$n=43$	$n=21$
Stage 1	$+0.69$	$+2.16$	$+3.24$

Appendix

Table 11: Relative improvement (%) of the deep ensemble over its average member per context-size bin (training rows available as context) on TALENT. Classification uses LPPD, regression nRMSE; positive values are improvements.

Figure 14: Full-size TabICLv2 deep ensembles on TALENT: LPPD on the 170 classification datasets (left) and nRMSE on the 85 regression datasets (right) versus ensemble size

K

. Lines show the mean, bands

\pm 1

standard deviation over all

\binom{8}{K}

sub-ensembles of the eight stage-3 members.

Figure 15: Empirical CDFs over datasets of the disagreement between full-size TabICLv2 classification members,

\widehat{AU}_{\mathrm{JSD}}

(left) and the pairwise disagreement rate

\widehat{AU}_{\mathrm{dis}}

(right, log scale), on TabArena datasets within the prior’s support (iid splits, at most 100 features, 10 classes and 48,000 training rows) and on BeyondArena datasets with a temporal or grouped split or outside that task space. Group summaries in Table 12 .

	TabArena, within prior support	BeyondArena OOD
Datasets	30	38
Split	iid	20 iid, 7 temporal, 11 grouped
Features, median (range)	20 (4–94)	152 (3–22,215)
Training rows, median (range)	3,637 (499–47,679)	30,625 (114–1,000,350)
Classes, max	5	20
Beyond prior support	0	33 (23 $>$ 100 features, 17 $>$ 48,000 rows, 3 $>$ 10 classes)

Appendix

Table 12: Member disagreement of full-size TabICLv2 classification members on TabArena datasets within the prior’s support and on BeyondArena datasets beyond it. Prior support: at most 100 features, 10 classes and 48,000 training rows; BeyondArena OOD: a temporal or grouped split, or outside that task space. Text datasets are excluded. Each dataset’s value is the mean over its splits;

p

: one-sided Mann–Whitney

U

test, datasets are listed in Tables 13 and 14 .

Dataset	Features	Train rows	Classes
Amazon_employee_access	9	21,846	2
anneal	38	599	5
bank-marketing	13	30,141	2
Bank_Customer_Churn	10	6,667	2
blood-transfusion-service-center	4	499	2
churn	19	3,333	2

Appendix

Table 13: TabArena classification datasets within the prior’s support (all iid splits): number of features, mean number of training rows per split and number of classes. The eight remaining TabArena classification datasets lie beyond the support (more than 100 features: APSFailure, Bioresponse, hiva_agnostic, kddcup09_appetency, MIC; more than 48,000 training rows: customer_satisfaction_in_airline, GiveMeSomeCredit, SDSS17) and are not used for the experiments resulting in Table 12 .

Dataset	Split	Feat.	Train rows	Cls	Outside the prior
amex_non_iid_1m	grouped	190	1,000,350	2	shift, $>$ 100 feat., $>$ 48k rows
asp_potassco_classification	grouped	137	808	11	shift, $>$ 100 feat., $>$ 10 cls
cardiotocography	grouped	23	1,417	3	shift
covertype	grouped	14	341,750	3	shift, $>$ 48k rows
dementia_prediction	grouped	9	247	3	shift
mice_protein_trisomy_discriminant	grouped	77	720	8	shift

Appendix

Table 14: BeyondArena classification datasets outside the prior’s support or with a shifted (temporal or grouped) split, with the reason for inclusion. Training rows: mean per split. Datasets with more than 700,000 training rows are evaluated with a single preprocessing configuration instead of eight.

Bayesian Model Averaging

See all 8 papers

Debiased Prior-Data Fitted Networks for Amortized Bayesian Semi-Parametric Inference
Oct 20, 2026Valentyn Melnychuk, Emil Javurek, Stefan FeuerriegelPrior-Data Fitted Networks Amortized Inference
LMU Munich & Munich Center for Machine Learning, Munich, Germany
Characterizing the Separability Gap in Prior-Data Fitted Networks
Oct 10, 2026Julius Kobialka, Manuel Hülskamp, Thomas Nagler +1Prior-Data Fitted Networks Bayesian Inference
Department of Statistics, LMU Munich · Munich Center for Machine Learning
Amortized Factor Inference Networks for Posterior Inference
May 26, 2026Joohwan Ko, Justin DomkeAmortized Inference Bayesian Inference
Manning College of Information and Computer Sciences University of Massachusetts Amherst

Oct 20, 2026J/K move · Enter open · S save

Debiased Prior-Data Fitted Networks for Amortized Bayesian Semi-Parametric Inference

Valentyn Melnychuk, Emil Javurek, Stefan Feuerriegel

LMU Munich & Munich Center for Machine Learning, Munich, Germany

Prior-Data Fitted Networks Amortized Inference Semiparametric Inference

Open paper AskPDF arXiv

Prior-data fitted networks (PFNs) have recently gained popularity as an end-to-end approach to amortized Bayesian inference. Yet, their statistical behavior for semi-parametric targets, such as means under missingness or causal effects, remains poorly understood. In this work, we develop theoretical foundations for debiased PFNs to perform inference on finite-dimensional targets in general semi-parametric settings. First, we show that, despite being Bayes-optimal with respect to the target, an idealized PFN can exhibit a posterior concentration rate that implicitly depends on the concentration rates of (latent) nuisance functions and can become slower as the prior becomes more diverse. Then, to speed up the target posterior concentration rates, we introduce two generic debiasing strategies based on efficient influence functions: (i) a label correction and (ii) a prior correction. Finally, we show that the debiased PFNs (i) & (ii) possess concentration rates that asymptotically match a convergence rate given by traditional frequentist bias-corrected estimators. In this way, we establish a novel connection between amortized Bayesian inference with PFNs and frequentist semi-parametric efficiency theory. Under additional assumptions, this allows us to guarantee local and global optimality of PFN-based estimators (namely, the Bernstein-von Mises property and minimax optimality, respectively). Therefore, our work provides guidelines for designing debiased PFNs for semi-parametric inference with principled and provable asymptotic guarantees.

Abstract

Bayesian Model Averaging

Explore similar work

Debiased Prior-Data Fitted Networks for Amortized Bayesian Semi-Parametric Inference

Characterizing the Separability Gap in Prior-Data Fitted Networks

Amortized Factor Inference Networks for Posterior Inference