Prior-data fitted networks (PFNs) are transformers pretrained on simulated datasets to approximate Bayesian posterior predictive distributions. Standard PFN theory assumes a separable task prior, under which the feature distribution and the mechanism generating labels from features are independent. We show that common PFN priors are non-separable, even in simple settings. Consequently, standard pretraining targets a predictive distribution based on a task posterior updated using both the labeled context and the query features. We characterize its difference from the predictive based on the context posterior alone and show that this difference vanishes under posterior concentration. Under misspecification (i.e., when the true joint distribution lies outside the prior's support), however, prior non-separability can favor better feature fit at the expense of conditional label fit. We illustrate the consequences for feature attribution, Bayesian optimization, and martingale checks of Bayesian behavior. The same dependence also enables PFNs to learn from additional unlabeled observations. We demonstrate this with SemiTabICL, a PFN explicitly pretrained to use these observations, which improves accuracy over a matched supervised baseline. Our findings highlight the need for explicit rules on unlabeled data use in PFN evaluation and establish the dependence between feature and label distributions as a consequential prior design choice.
Figures & tables
Figure 1: Illustration of the anticausal setting from Schölkopf et al. (2012) : Two Gaussian class centers μ0,μ1 generate the features from the label, such that the location of an unlabeled point is informative about the labeling mechanism. The mechanism in the left figure does not allow the query’s location to update the belief about the centers; the right figure does use the point’s position.
Figure 2: Recovery of the query-updated predictive on the non-separable ordered-cluster prior with context length n=2 . Standard pretraining follows the identity line, indicating recovery of π , whereas the difference for the decoupled PFN remains near zero, indicating recovery of πˉ . In single-class contexts, both labeled rows come from the same cluster, so the query is evidence about the unobserved center and the difference between π and πˉ is larger (note the different axis ranges).
Figure 3: Feature effects under a prior in which every task has a constant class-1 probability cp that is associated with its feature distribution. There are two tasks, a low- cp and a high- cp one. X1 and X2 are discrete features shifted toward negative values in the former and toward positive values in the latter, so they indicate the task, while X3 is independent noise with the same distribution in both tasks. Left: PD of X1 . Middle: ICE curves of X1 for fixed X2 . Right: mean absolute exact SHAP attributions of π , conditional and marginal. Since no labeling mechanism depends on x , πˉ is constant and all its attributions are zero (dashed). π assigns importance to X1 and X2 because they identify the task, and none to the dummy feature X3 .
Figure 4: Non-separable priors can induce drift in exact Bayesian predictives. Curves show the RMS of the cumulative expected drift across 2000 contexts of size n=2 , with eight queries each, under the ordered-cluster prior. Every future feature is fixed at the query (left) or at one selected context feature (right), and labels are sampled from the predictive being evaluated with absolute conditional one-step drifts in the right figure. No trained network is involved.
Figure 5: Accuracy gains (percentage points) for semi-supervised classification on the semi-TALENT benchmark for a range of labeled context dataset sizes ( n ) and multiples thereof of unlabeled context rows. Left: SemiTabICL vs. identical baseline trained and evaluated without unlabeled context rows. Right: Semi-supervised extension of the baseline model via self-training vs. no additional self-training
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
semi-TALENT
semi-UCI
Comparison
Δ acc
win
p
Δ NLL
win
p
Δ acc
Δ NLL
SemiTabICL vs. TabICL
+0.63
0.81
5e−16
+0.015
0.85
6e−19
+0.48
+0.014
SemiTabICL vs. shuffled
+0.66
0.67
1e−10
+0.017
0.65
2e−7
+0.47
+0.017
Shuffled vs. TabICL
−0.03
0.59
0.56
−0.001
0.63
0.06
+0.01
−0.003
TabICL ST vs. TabICL
−0.33
0.41
3e−4
−0.440
0.03
6e−26
−0.53
−0.393
GBDT ST vs. GBDT
+0.45
0.77
3e−14
−0.577
0.01
2e−26
+0.46
−0.452
Appendix
Table 1: Paired comparisons of SemiTabICL and its controls. Δ acc is the difference in accuracy (percentage points) and Δ NLL the reduction in log loss (nats), both positive when the first method is better. For each dataset, differences are averaged over all its splits; for semi-TALENT, these pool the full 4×4 grid of n∈{16,32,64,96} and r∈{2,4,8,10} with five repetitions each. Reported values are means of these per-dataset differences. For semi-TALENT, win is the share of the 150 datasets with a positive difference, and p is a two-sided Wilcoxon signed-rank test of whether the per-dataset differences are centred at zero.
Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in rare settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models.
Manuel Hülskamp, Julius Kobialka, Emanuel Sommer +2
Prior-Fitted Networks (PFNs) amortize Bayesian prediction by meta-learning over a synthetic task prior, but their standard output is a posterior predictive distribution over noisy observations. For sequential decision-making, such as active learning and Bayesian optimization, acquisition should prioritize epistemic uncertainty about the latent signal rather than irreducible aleatoric observation noise. We show that this epistemic--aleatoric split is not identifiable in general from the posterior predictive distribution alone, even when that distribution is known exactly. We then exploit a distinctive advantage of PFNs: because the synthetic data-generating process is under our control, each task can contain an explicit latent signal and noise function, and the generator can provide query-level labels for both the noiseless target and the observation-noise variance. We use these labels to train a decoupled PFN with separate latent-signal and aleatoric heads. The observation-level predictive is induced by convolving the latent signal distribution with the learned noise model. Empirically, epistemic-only acquisition mitigates the failure mode of total-variance exploration in noisy and heteroscedastic settings. In matched comparisons, decoupled models usually improve over tuned observation-level baselines, with the clearest gains in HPO; in broader sweeps, a decoupled model obtains the best average rank in both HPO and synthetic BO.
Richard Bergna, Stefan Depeweg, José Miguel Hernández-Lobato
Prior-data fitted networks (PFNs) have recently gained popularity as an end-to-end approach to amortized Bayesian inference. Yet, their statistical behavior for semi-parametric targets, such as means under missingness or causal effects, remains poorly understood. In this work, we develop theoretical foundations for debiased PFNs to perform inference on finite-dimensional targets in general semi-parametric settings. First, we show that, despite being Bayes-optimal with respect to the target, an idealized PFN can exhibit a posterior concentration rate that implicitly depends on the concentration rates of (latent) nuisance functions and can become slower as the prior becomes more diverse. Then, to speed up the target posterior concentration rates, we introduce two generic debiasing strategies based on efficient influence functions: (i) a label correction and (ii) a prior correction. Finally, we show that the debiased PFNs (i) & (ii) possess concentration rates that asymptotically match a convergence rate given by traditional frequentist bias-corrected estimators. In this way, we establish a novel connection between amortized Bayesian inference with PFNs and frequentist semi-parametric efficiency theory. Under additional assumptions, this allows us to guarantee local and global optimality of PFN-based estimators (namely, the Bernstein-von Mises property and minimax optimality, respectively). Therefore, our work provides guidelines for designing debiased PFNs for semi-parametric inference with principled and provable asymptotic guarantees.
Valentyn Melnychuk, Emil Javurek, Stefan Feuerriegel
LMU Munich & Munich Center for Machine Learning, Munich, Germany