Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in rare settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models.
Figures & tables
Figure 1: Decomposition of the deployment error of a PFN ( Proposition 2 ). This paper studies the amortization error, the only term controlled by pretraining, which splits into optimization, estimation, and approximation error ( 8 ). The context and prior error support this analysis: the context error is incurred even by the exact PPD, and the prior error arises when the deployment tasks differ from the prior (see Section 3.3 ).
Figure 2: Comparison of PPD approximation quality to MCMC reference on a classification task across both Gaussian and TabICL prior feature distributions (SCM prior). The labeling mechanism is a Bayesian neural network with two hidden layers with 16 neurons each, and the large TabICL variant is trained on these priors from scratch. The PFN approximates the ground-truth PPD provided by the MCMC reference very well across different context sizes (Figure inspired by Müller et al., 2021 ). Each point is the risk on datasets of context size n ; Ψ is the size distribution of pretraining.
Figure 3: Estimation error EE(M) (log-log) on synthetic data and member disagreement on TabArena ( AUJSD , AUdis ) versus the number of pretraining datasets M for the small models; the tiny models are shown in Fig. 10 .
context size 1000
context size 10000
Model
DE
IVON S=8
IVON S=64
SG-MCMC
DE
IVON S=8
IVON S=64
SG-MCMC
tiny
+3.10
+2.38∗
+2.48∗
+0.27∗
+2.65
−0.13
−0.24
+0.35∗
small
+2.89
+0.52∗
+0.65∗
+0.37∗
+2.56
−0.14
−0.04
+0.42∗
medium
+2.67
+0.42∗
+0.44∗
+0.07
+2.82
−0.31
−0.14
+0.08
large
+1.69
+0.23∗
+0.18∗
+0.27∗
+1.81
−0.13
−0.06
+0.32∗
Table 1: Relative LPPD gain (%) of Bayesian model averages of TabICLv2 on 30 TabArena classification datasets at context sizes 1000 and 10000. DE : deep ensemble of 8 members (4 for the large model) over its average member. IVON : average over S posterior samples over the posterior mean of the same model. SG-MCMC : Bayesian deep ensemble over the DE it is warm-started from, sampled with pSMILE. ∗ : significant change, paired Wilcoxon over datasets, p<0.05 .
Figure 4: Classification test performance measured by accuracy, ROC AUC and LPPD on the TabArena dataset suite across a sweep of the maximal number of context rows for deep ensembles of TabICLv2 across a range of model sizes. The corresponding results for nanoTabPFN are shown in Fig. 13 .
Figure 5: Deep ensembles of full-size TabICLv2. (a, b) LPPD and nRMSE on TALENT datasets after each training stage: ensemble of 8 members, average member and best member (selected after the fact). The gap between ensemble and average member persists while all members improve. (c) TabArena classification Elo (95% CI) of the single members (dots), the 8-member ensemble, the official TabICLv2 checkpoint and further published methods. Note that the individual DE members are trained for only 27.5% of the steps that official TabICLv2 uses.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Error decompositions used in this paper. (a) The deployment error of a PFN splits into prior, amortization and context error ( Proposition 2 ); the latter two enter with the expressible share λ , and the prior error comprises the coverage, transport and off-support terms. For the trained network qθ^ , the amortization error splits further into optimization, estimation and approximation error ( 8 ). (b) For K PFNs, the error of the average member splits into the error of their mixture qˉ , whose amortization error is the part shared by all members ( Corollary 1 ), and the gain of averaging, which splits into the removable amortization error and the transport and off-support terms of Corollary 2 .
Figure 7: Holdout NLL of Bayesian model average of k pooled MCMC chains for classification. The NLL saturates as more chains are added, which means a small number of chains seems sufficient to capture the functional diversity of the posterior distribution of the weights.
Figure 8: PFN to MCLMC gap under covariate shift. Test features are x=sz with z∼N(0,I4) and labels come from the BNN prior ( 2×8 hidden neurons). The PFN was pretrained at s=1 . Each point is the mean difference over 256 test tasks. Bands are 95% task-bootstrap intervals of each mean. Curves show the large PFN pretrained with uniform or log-uniform table size Ψ , and the uniform- Ψ PFN with context-standardized features. Dotted grey ( n=4096 only): excess NLL of the MCLMC reference over the oracle.
Figure 9: BNN regression with hidden width 8×8 , d=4 , 512 test tasks: prior-data NLL (left) and Continuous Ranked Probability Score (CRPS; right) against the number of context points n for the small and large PFN, the MILE (MCLMC) reference (24 chains), and the oracle. The PFN’s NLL gap to MILE stems mostly from the density implied by its 999 predicted quantiles and on CRPS, which needs no density, the PFN nearly matches MILE.
tiny
small
medium
large
Maximum epochs
30
40
55
70
Maximum steps
18,000
24,000
33,000
42,000
Peak learning rate, nanoTabPFN
4⋅10−4
4⋅10−4
–
–
Peak learning rate, TabICLv2
4⋅10−4
4⋅10−4
4⋅10−4
4⋅10−4
Ensemble members K
8
8
8
4
Appendix
Table 2: Size-dependent training settings of the deep ensembles in Section 5 . Maximum steps are reached only if early stopping does not trigger.
adaSGHMC
pSMILE
Step size
2⋅10−4
10−4
Warm-up steps
500
250
Sampling steps
2500 / 3000 / 2500 / 3500
1250 / 1500 / 1250 / 1750
Thinning
every 10% of the sampling steps (10 samples per chain)
Gradient evaluations per chain
3000 / 3500 / 3000 / 4000
Chains
8 / 8 / 8 / 4
Appendix
Table 3: SG-MCMC settings. Sizes vary as tiny / small / medium / large. Both samplers use the same number of gradient evaluations per chain; pSMILE takes two per step.
tiny
small
medium
large
Effective sample size β
1.92⋅106
3.84⋅106
6.4⋅106
1.92⋅107
Learning rate
1.0
1.0
0.3
0.3
Training steps
300,000
200,000
200,000
200,000
β1
0.9
β2
0.99999
Weight decay
10−4
Appendix
Table 4: IVON settings. All sizes additionally share gradient clipping at 10 , a cosine schedule with 1% linear warm-up decaying to 10−6 , batches of 64 datasets and bfloat16 precision.
tiny
small
medium
large
full
NanoTabPFN
Parameters (rounded)
1M
2M
4M
–
–
Transformer layers
4
4
5
–
–
Embedding dimension
128
192
256
–
–
Attention heads
4
6
8
–
–
MLP hidden dimension
256
384
512
–
–
Appendix
Table 5: Architecture hyperparameters. NanoTabPFN large is omitted due to its training cost; TabICLv2 full, used in Section F.7 , follows the architecture of Qu et al. (2026) with added QK-normalization and bounded QASSMax scaling. In TabICLv2, the ICL stage operates on the concatenated CLS column tokens (ICL width = embedding dimension × CLS column tokens), and every block’s MLP hidden dimension is twice its width. All models predict up to 10 classes.
Figure 10: As Fig. 3 , for the tiny models: estimation error EE(M) (log-log) on synthetic data and member disagreement on TabArena versus the number of pretraining datasets M . Bands show ±1 standard error over the K=4 members ( EE ) or over datasets (disagreement).
Figure 11: Member disagreement ( AUJSD , AUdis ) on the synthetic test set versus the number of pretraining datasets M , complementing the TabArena results of Fig. 3 . Top: tiny, bottom: small models.
synthetic
TabArena, context 1000
TabArena, context 10000
Model
AUJSD
AUdis
AUJSD
AUdis
AUJSD
AUdis
tiny (1.5M)
16.1
9.4
12.5±2.0
7.5±1.4
11.8±1.6
8.0±1.5
small (2M)
15.5
9.2
11.4±1.6
7.5±1.5
11.0±1.4
7.8±1.5
medium (7.5M)
–
–
9.7±1.5
6.7±1.4
10.5±1.3
7.0±1.4
large (9M)
13.6
8.2
7.4±1.4
5.9±1.2
8.5±1.4
6.5±1.2
Appendix
Table 6: Member disagreement ( K=4 ) across TabICLv2 sizes on the synthetic test set and on the 30 TabArena classification datasets (mean ± standard error over datasets). AUJSD in 10−3 nats, AUdis in %.
Figure 12: Share of the total pointwise gain carried by the test points with the largest gain, for tiny (left) and large (right) TabICLv2 with K = 4 members. Dashed: gain spread evenly over all points.
Figure 13: As Fig. 4 , for nanoTabPFN: accuracy, ROC AUC and LPPD on the 30 TabArena classification datasets across the maximal number of context rows, for deep ensembles and their average single member at the tiny and small model sizes.
Model
S
Accuracy ↑
ROC AUC ↑
LPPD ↑
tiny
0
0.8073
0.8737
−0.4464
1
0.8035 -
0.8714 -
−0.4558−
8
0.8068
0.8738
−0.4493−
16
0.8063
0.8742
−0.4488−
32
0.8077
0.8752
−0.4484
64
0.8072
0.8749
−0.4489
Appendix
Table 7: IVON variational sampling on the 170 TALENT classification datasets. S=0 is the posterior mean; S≥1 averages S posterior samples without the mean. LPPD is the mean log predictive density per test point. + / - : significantly better/worse than the same model at S=0 (paired Wilcoxon over datasets, p<0.05 ).
context size 1000
context size 10000
Model
adaSGHMC
pSMILE
adaSGHMC
pSMILE
tiny
−0.05 (11/30)
+0.27∗ (26/30)
−0.05 (11/30)
+0.35∗ (26/30)
small
−0.64∗ (4/30)
+0.37∗ (22/30)
−1.12∗ (5/30)
+0.42∗ (25/30)
medium
–
+0.07 (12/30)
–
+0.08 (13/30)
large
+0.12 (18/30)
+0.27∗ (26/30)
−0.52∗ (9/30)
+0.32∗ (26/30)
Appendix
Table 8: Relative LPPD gain (%) of the Bayesian deep ensemble over the deep ensemble it is warm-started from, for both samplers, on the 30 TabArena classification datasets at context sizes 1000 and 10000. In parentheses: number of datasets on which the Bayesian deep ensemble improves on the deep ensemble. ∗ : significant change, paired Wilcoxon over datasets, p<0.05 .
steps
wall-clock (h)
Stage (max. rows)
official
ours
steps/h
official
ours
1 (1024)
500,000
130,000
6,033
82.9
21.5
2 (10,240)
40,000
10,400
2,150
18.6
4.8
3 (60,000)
10,000
11,000
650
15.4
16.9
Total, one model
550,000
151,400
116.9
43.3
GPU hours, one model
467
173
Appendix
Table 9: Training budget of one full-size member against the official TabICLv2 recipe ( Qu et al., 2026 ) . Throughput was measured for our setup (bfloat16, four H200 GPUs, batch size 64); the official durations apply our throughput to its step counts and are therefore an estimate. GPU hours count four GPUs.
Classification (170 datasets)
Stage
LPPD ↑
Brier ↓
ROC AUC ↑
1
average member
−0.3713
0.2132
0.9021
best member
−0.3693
0.2122
0.9024
deep ensemble
−0.3684∗
0.2116∗
0.9032∗
2
average member
−0.3634
0.2089
0.9039
best member
−0.3619
0.2082
0.9045
Appendix
Table 10: Full-size TabICLv2 on TALENT after each training stage. Average member averages each dataset’s metric over the 8 members; best member is the member with the best mean at that stage, selected after the fact. ∗ : deep ensemble significantly better than the best member (paired Wilcoxon over datasets, p<0.05 ).
≤ 1024
1024–10240
> 10240
Classification
n=33
n=100
n=37
Stage 1
+0.85
+0.70
+0.86
Stage 2
+0.85
+0.77
+0.72
Stage 3
+0.85
+0.78
+0.75
Regression
n=21
n=43
n=21
Stage 1
+0.69
+2.16
+3.24
Appendix
Table 11: Relative improvement (%) of the deep ensemble over its average member per context-size bin (training rows available as context) on TALENT. Classification uses LPPD, regression nRMSE; positive values are improvements.
Figure 14: Full-size TabICLv2 deep ensembles on TALENT: LPPD on the 170 classification datasets (left) and nRMSE on the 85 regression datasets (right) versus ensemble size K . Lines show the mean, bands ±1 standard deviation over all (K8) sub-ensembles of the eight stage-3 members.
Figure 15: Empirical CDFs over datasets of the disagreement between full-size TabICLv2 classification members, AUJSD (left) and the pairwise disagreement rate AUdis (right, log scale), on TabArena datasets within the prior’s support (iid splits, at most 100 features, 10 classes and 48,000 training rows) and on BeyondArena datasets with a temporal or grouped split or outside that task space. Group summaries in Table 12 .
Table 12: Member disagreement of full-size TabICLv2 classification members on TabArena datasets within the prior’s support and on BeyondArena datasets beyond it. Prior support: at most 100 features, 10 classes and 48,000 training rows; BeyondArena OOD: a temporal or grouped split, or outside that task space. Text datasets are excluded. Each dataset’s value is the mean over its splits; p : one-sided Mann–Whitney U test, datasets are listed in Tables 13 and 14 .
Dataset
Features
Train rows
Classes
Amazon_employee_access
9
21,846
2
anneal
38
599
5
bank-marketing
13
30,141
2
Bank_Customer_Churn
10
6,667
2
blood-transfusion-service-center
4
499
2
churn
19
3,333
2
Appendix
Table 13: TabArena classification datasets within the prior’s support (all iid splits): number of features, mean number of training rows per split and number of classes. The eight remaining TabArena classification datasets lie beyond the support (more than 100 features: APSFailure, Bioresponse, hiva_agnostic, kddcup09_appetency, MIC; more than 48,000 training rows: customer_satisfaction_in_airline, GiveMeSomeCredit, SDSS17) and are not used for the experiments resulting in Table 12 .
Dataset
Split
Feat.
Train rows
Cls
Outside the prior
amex_non_iid_1m
grouped
190
1,000,350
2
shift, > 100 feat., > 48k rows
asp_potassco_classification
grouped
137
808
11
shift, > 100 feat., > 10 cls
cardiotocography
grouped
23
1,417
3
shift
covertype
grouped
14
341,750
3
shift, > 48k rows
dementia_prediction
grouped
9
247
3
shift
mice_protein_trisomy_discriminant
grouped
77
720
8
shift
Appendix
Table 14: BeyondArena classification datasets outside the prior’s support or with a shifted (temporal or grouped) split, with the reason for inclusion. Training rows: mean per split. Datasets with more than 700,000 training rows are evaluated with a single preprocessing configuration instead of eight.
Prior-data fitted networks (PFNs) have recently gained popularity as an end-to-end approach to amortized Bayesian inference. Yet, their statistical behavior for semi-parametric targets, such as means under missingness or causal effects, remains poorly understood. In this work, we develop theoretical foundations for debiased PFNs to perform inference on finite-dimensional targets in general semi-parametric settings. First, we show that, despite being Bayes-optimal with respect to the target, an idealized PFN can exhibit a posterior concentration rate that implicitly depends on the concentration rates of (latent) nuisance functions and can become slower as the prior becomes more diverse. Then, to speed up the target posterior concentration rates, we introduce two generic debiasing strategies based on efficient influence functions: (i) a label correction and (ii) a prior correction. Finally, we show that the debiased PFNs (i) & (ii) possess concentration rates that asymptotically match a convergence rate given by traditional frequentist bias-corrected estimators. In this way, we establish a novel connection between amortized Bayesian inference with PFNs and frequentist semi-parametric efficiency theory. Under additional assumptions, this allows us to guarantee local and global optimality of PFN-based estimators (namely, the Bernstein-von Mises property and minimax optimality, respectively). Therefore, our work provides guidelines for designing debiased PFNs for semi-parametric inference with principled and provable asymptotic guarantees.
Valentyn Melnychuk, Emil Javurek, Stefan Feuerriegel
LMU Munich & Munich Center for Machine Learning, Munich, Germany
Prior-data fitted networks (PFNs) are transformers pretrained on simulated datasets to approximate Bayesian posterior predictive distributions. Standard PFN theory assumes a separable task prior, under which the feature distribution and the mechanism generating labels from features are independent. We show that common PFN priors are non-separable, even in simple settings. Consequently, standard pretraining targets a predictive distribution based on a task posterior updated using both the labeled context and the query features. We characterize its difference from the predictive based on the context posterior alone and show that this difference vanishes under posterior concentration. Under misspecification (i.e., when the true joint distribution lies outside the prior's support), however, prior non-separability can favor better feature fit at the expense of conditional label fit. We illustrate the consequences for feature attribution, Bayesian optimization, and martingale checks of Bayesian behavior. The same dependence also enables PFNs to learn from additional unlabeled observations. We demonstrate this with SemiTabICL, a PFN explicitly pretrained to use these observations, which improves accuracy over a matched supervised baseline. Our findings highlight the need for explicit rules on unlabeled data use in PFN evaluation and establish the dependence between feature and label distributions as a consequential prior design choice.
Julius Kobialka, Manuel Hülskamp, Thomas Nagler +1
Department of Statistics, LMU Munich · Munich Center for Machine Learning
Amortized inference promises fast test-time Bayesian inference, but existing methods typically target a fixed model or a restricted model family. Extending amortization to new model families often requires retraining or test-time fine-tuning. In this paper, we ask: is it possible to build a single inference network capable of generalizing across varying priors, likelihoods, latent dimensions, and observation counts? We introduce Amortized Factor Inference Networks (AFINs), which describe a model as a list of typed prior and likelihood factors and map this description, together with the observations, to a Gaussian posterior approximation through an encode--merge--decode network whose learned parameters do not depend on the latent dimension or the number of observations. Experiments on synthetic and real-data benchmarks show that a single trained AFIN provides competitive posterior approximations at substantially lower test-time cost than NUTS and per-task variational inference. Optional importance sampling can further improve accuracy.
Joohwan Ko, Justin Domke
Manning College of Information and Computer Sciences University of Massachusetts Amherst