Characterizing the Separability Gap in Prior-Data Fitted Networks

Julius Kobialka; Manuel Hülskamp; Thomas Nagler; David Rügamer

cs.LGOct 10, 2026

Characterizing the Separability Gap in Prior-Data Fitted Networks

Authors: Julius Kobialka, Manuel Hülskamp, Thomas Nagler, David Rügamer

Organizations: Department of Statistics, LMU Munich · Munich Center for Machine Learning

Abstract

Prior-data fitted networks (PFNs) are transformers pretrained on simulated datasets to approximate Bayesian posterior predictive distributions. Standard PFN theory assumes a separable task prior, under which the feature distribution and the mechanism generating labels from features are independent. We show that common PFN priors are non-separable, even in simple settings. Consequently, standard pretraining targets a predictive distribution based on a task posterior updated using both the labeled context and the query features. We characterize its difference from the predictive based on the context posterior alone and show that this difference vanishes under posterior concentration. Under misspecification (i.e., when the true joint distribution lies outside the prior's support), however, prior non-separability can favor better feature fit at the expense of conditional label fit. We illustrate the consequences for feature attribution, Bayesian optimization, and martingale checks of Bayesian behavior. The same dependence also enables PFNs to learn from additional unlabeled observations. We demonstrate this with SemiTabICL, a PFN explicitly pretrained to use these observations, which improves accuracy over a matched supervised baseline. Our findings highlight the need for explicit rules on unlabeled data use in PFN evaluation and establish the dependence between feature and label distributions as a consequential prior design choice.

Figures & tables

Figure 1: Illustration of the anticausal setting from Schölkopf et al. (2012) : Two Gaussian class centers $\mu_{0},\mu_{1}$ generate the features from the label, such that the location of an unlabeled point is informative about the labeling mechanism. The mechanism in the left figure does not allow the query’s location to update the belief about the centers; the right figure does use the point’s position.

Figure 2: Recovery of the query-updated predictive on the non-separable ordered-cluster prior with context length

n=2

. Standard pretraining follows the identity line, indicating recovery of

\pi

, whereas the difference for the decoupled PFN remains near zero, indicating recovery of

\bar{\pi}

. In single-class contexts, both labeled rows come from the same cluster, so the query is evidence about the unobserved center and the difference between

\pi

and

\bar{\pi}

is larger (note the different axis ranges).

Figure 3: Feature effects under a prior in which every task has a constant class-1 probability

c_{p}

that is associated with its feature distribution. There are two tasks, a low-

c_{p}

and a high-

c_{p}

one.

X_{1}

and

X_{2}

are discrete features shifted toward negative values in the former and toward positive values in the latter, so they indicate the task, while

X_{3}

is independent noise with the same distribution in both tasks. Left: PD of

X_{1}

. Middle: ICE curves of

X_{1}

for fixed

X_{2}

. Right: mean absolute exact SHAP attributions of

\pi

, conditional and marginal. Since no labeling mechanism depends on

\bm{x}

\bar{\pi}

is constant and all its attributions are zero (dashed).

\pi

assigns importance to

X_{1}

and

X_{2}

because they identify the task, and none to the dummy feature

X_{3}

Panel of Figure 4 — Figure 4: Non-separable priors can induce drift in exact Bayesian predictives. Curves show the RMS of the cumulative expected drift across $2000$ contexts of size $n=2$ , with eight queries each, under the ordered-cluster prior. Every future feature is fixed at the query (left) or at one selected context feature (right), and labels are sampled from the predictive being evaluated with absolute conditional one-step drifts in the right figure. No trained network is involved.

Figure 5: Accuracy gains (percentage points) for semi-supervised classification on the semi-TALENT benchmark for a range of labeled context dataset sizes (

n

) and multiples thereof of unlabeled context rows. Left: SemiTabICL vs. identical baseline trained and evaluated without unlabeled context rows. Right: Semi-supervised extension of the baseline model via self-training vs. no additional self-training

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	semi-TALENT						semi-UCI
Comparison	$\Delta$ acc	win	$p$	$\Delta$ NLL	win	$p$	$\Delta$ acc	$\Delta$ NLL
SemiTabICL vs. TabICL	$+0.63$	$0.81$	$5\mathrm{e}{-16}$	$+0.015$	$0.85$	$6\mathrm{e}{-19}$	$+0.48$	$+0.014$
SemiTabICL vs. shuffled	$+0.66$	$0.67$	$1\mathrm{e}{-10}$	$+0.017$	$0.65$	$2\mathrm{e}{-7}$	$+0.47$	$+0.017$
Shuffled vs. TabICL	$-0.03$	$0.59$	$0.56$	$-0.001$	$0.63$	$0.06$	$+0.01$	$-0.003$
TabICL ST vs. TabICL	$-0.33$	$0.41$	$3\mathrm{e}{-4}$	$-0.440$	$0.03$	$6\mathrm{e}{-26}$	$-0.53$	$-0.393$
GBDT ST vs. GBDT	$+0.45$	$0.77$	$3\mathrm{e}{-14}$	$-0.577$	$0.01$	$2\mathrm{e}{-26}$	$+0.46$	$-0.452$

Appendix

Table 1: Paired comparisons of SemiTabICL and its controls.

\Delta

acc is the difference in accuracy (percentage points) and

\Delta

NLL the reduction in log loss (nats), both positive when the first method is better. For each dataset, differences are averaged over all its splits; for semi-TALENT, these pool the full

4\times 4

grid of

n\in\{16,32,64,96\}

and

r\in\{2,4,8,10\}

with five repetitions each. Reported values are means of these per-dataset differences. For semi-TALENT, win is the share of the 150 datasets with a positive difference, and

p

is a two-sided Wilcoxon signed-rank test of whether the per-dataset differences are centred at zero.

Prior-Data Fitted Networks

Prior-Data Fitted Networks (PFNs) learn in-context prediction by training on synthetic datasets sampled from designed priors.

See all 45 papers

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?
Oct 9, 2026Manuel Hülskamp, Julius Kobialka, Emanuel Sommer +2Bayesian Model Averaging Prior-Data Fitted Networks
LMU Munich and Munich Center for Machine Learning
Decoupled PFNs: Identifiable Epistemic-Aleatoric Decomposition via Structured Synthetic Priors
May 7, 2026Richard Bergna, Stefan Depeweg, José Miguel Hernández-LobatoUncertainty Quantification Bayesian Optimization
University of Cambridge · 2Siemens AG
Debiased Prior-Data Fitted Networks for Amortized Bayesian Semi-Parametric Inference
Oct 20, 2026Valentyn Melnychuk, Emil Javurek, Stefan FeuerriegelPrior-Data Fitted Networks Amortized Inference
LMU Munich & Munich Center for Machine Learning, Munich, Germany

Oct 9, 2026J/K move · Enter open · S save

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

Manuel Hülskamp, Julius Kobialka, Emanuel Sommer, Thomas Nagler+1

LMU Munich and Munich Center for Machine Learning

Bayesian Model Averaging Prior-Data Fitted Networks Amortized Inference Bayesian Inference

Open paper AskPDF arXiv

Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in rare settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models.

Abstract

Prior-Data Fitted Networks

Explore similar work

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

Decoupled PFNs: Identifiable Epistemic-Aleatoric Decomposition via Structured Synthetic Priors

Debiased Prior-Data Fitted Networks for Amortized Bayesian Semi-Parametric Inference