SelectPFN: Amortized Bayesian Variable Selection for Sparse Nonlinear Regression with Prior-Data Fitted Networks

Bin Luo

stat.MLOct 15, 2026

SelectPFN: Amortized Bayesian Variable Selection for Sparse Nonlinear Regression with Prior-Data Fitted Networks

Authors: Bin Luo

Organizations: School of Data Science and Analytics, Kennesaw State University

Abstract

Tabular foundation models based on prior-data fitted networks (PFNs) predict on a new data set in a single forward pass, but they cannot report which inputs their predictions depend on. We propose SelectPFN, a PFN that performs Bayesian variable selection and prediction jointly for sparse nonlinear regression. In one forward pass it returns posterior inclusion probabilities for every variable, and its predictions are computed from sampled subsets only, so excluded variables provably cannot affect them; we show this is exactly what the Bayesian model prescribes. The network is pretrained once on synthetic tasks from a meta-prior built around a group spike-and-slab sparse-input neural network. Column-permutation equivariance combined with random interleaving of original and knockoff columns yields knockoff statistics with exact finite-sample false discovery rate control, even when the meta-prior is wrong. We develop theory for amortized selection: a Bayes-regret bound for the amortized median probability model, a characterization showing that the information needed to detect a variable grows like $(2/λ)\log(1/ω)$ in the prior weight $ω$ of structures that can use it, where $λ\le1$ is an efficiency, and a negative result showing that average-case training does not transfer to frequentist guarantees at a fixed truth. On out-of-prior nonlinear designs with p up to 500, SelectPFN improves the area under the precision-recall curve by 0.23-0.28 over the lasso, BART, DART, LassoNet and a group-penalized sparse neural network, at a false discovery proportion of about 0.07 and 0.04 seconds per data set. The advantage persists on real covariate matrices, and, as the theory predicts, detection of interaction-only variables rises monotonically with their prior weight.

Figures & tables

Figure 1: Mean AUPRC by design with 95% bootstrap intervals; most intervals are narrower than the markers. The two SelectPFN models are shades of one hue. Numerical values are in Table 16 .

Figure 2: Design C: how often the MPM includes the interaction-only variables

x_{3},x_{4}

(average of the two; band: 95% bootstrap interval over data sets). Without the interaction-only family the meta-prior gives such structures no weight, and the variables are essentially never found, as Corollary 1 and Theorem 2 predict.

Figure 3: Models that differ only in the prior weight

\omega

of the interaction-only family. Left: detection of

x_{3},x_{4}

(acting only through

\sin(x_{3}x_{4})

in design C) against the information index

t_{j}

, seeds pooled. Right: detection at two fixed

(n,\mathrm{SNR})

; dots are training seeds, lines their mean.

Figure 4: Reliability on held-out meta-prior tasks: observed frequency of activity minus mean

q

in bins of width 0.1, with 95% binomial intervals. The shaded band is

\pm 0.05

Figure 5: Knockoff mode: FDR (left; whiskers

\pm 2

s.e.; dashed: nominal level) and power (right) of knockoff+ with exact knockoffs on the

|S_{0}|=20

designs L20 and B20 (

n=300

p\in\{100,200\}

; 1,600 data sets).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Distribution
Sample size	$N\sim\mathrm{LU}(50,1000)$ rows per task; a fraction $n/N\sim U(0.5,0.9)$ are training (context) rows, the remaining $m=N-n$ are query rows
Dimension	$p\sim\mathrm{LU}(5,p_{\max})$ with curriculum $p_{\max}=50\to 200\to 500$ in thirds of training
Sparsity	Beta–Bernoulli, $w\sim\mathrm{Beta}(\bar{w}\nu,(1-\bar{w})\nu)$ , $\mathbb{E}\|S\|=\bar{w}\,p\sim\mathrm{LU}(1,\min(20,p/3))$ , $\nu\sim\mathrm{LU}(2,50)$ ; $\|S\|\leq 40$ ; an empty $\gamma$ is redrawn w.p. 0.9
Dependence	independent; AR(1); block equicorrelation; latent factor (equal weights). Active variables at random positions
Marginals	Gaussian copula with per-column transform: identity, uniform, log-normal skew, Tukey- $h$ heavy tails ( $h\sim U(0.1,0.4)$ ), discretized; columns standardized
Function family	sparse-input MLP with group spike-and-slab first layer, eq. 3 (0.30); additive (0.12); additive + pairwise, with no main effect for a variable w.p. 0.3 (0.12); trees/step functions (0.10); GP with ARD kernel, length-scales $\mathrm{LU}(0.3,2.5)$ (0.10); linear (0.14); interaction-only pairs/triples through functions odd in each argument (a single active variable gets a main effect), plus weak main effects w.p. 0.3 (0.12)

Appendix

Table 14: Meta-prior of SelectPFN.

\mathrm{LU}(a,b)

is log-uniform on

[a,b]

. The ablation model of Section 5.4.4 uses the same meta-prior without the interaction-only family, the weak-effect regime and interaction-only variables in the pairwise family, with family weights MLP 0.40, additive 0.15, pairwise 0.15, tree 0.10, GP 0.10 and linear 0.10.

Figure 6: Exact-posterior toy problem: detection threshold

t_{50}

against

\log(1/\omega)

, with the line of slope 2 (the limit in Corollary 2 ) and a heuristic fit. This is a numerical verification and is not used in any proof.

Feature Selection

Feature selection identifies compact, informative variables or input dimensions to improve prediction, efficiency, interpretability, and statistical inference.

See all 194 papers

Oct 5, 2026

LinearPFN: Amortized Variable Selection for Linear Models with Interactions

Spike-and-slab regression is a standard Bayesian formulation of variable selection: it returns a posterior distribution over which candidate effects are active rather than a single selected subset, so that every candidate effect carries an inclusion probability. Its cost grows exponentially with the number of candidate effects, so the posterior can be enumerated exactly only when the number of predictors is small. Beyond that reach, the posterior has to be approximated, typically by Markov chain Monte Carlo over the model space, which requires a fresh run for every dataset and, within a fixed budget of steps, may fail to converge. We present LinearPFN, a prior-data fitted transformer network that amortizes spike-and-slab inference for linear models with main effects and pairwise interactions. The network is pretrained once on synthetic datasets, drawn from an explicitly specified prior, and a single forward pass over a new dataset returns posterior inclusion probabilities, posterior-mean coefficients and posterior predictive distributions with no per-dataset fitting. The prior is conjugate by design, so that the posterior for each fixed set of active effects has a closed form, and wherever the exact posterior is still computable by enumeration we verify the network's outputs against it. On real predictor matrices from published social-science datasets, with outcomes drawn from the prior so that the true active set is known, LinearPFN attains a higher per-dataset selection AUC and a higher F1 under the median probability model rule than five classical baselines. The lead holds when the coefficients, the interactions or the noise depart from the prior. Code: https://github.com/schiekiera/LinearPFN. Trained model: https://huggingface.co/schiekiera/LinearPFN.

Louis Schiekiera, Max Zimmer, Christophe Roux +3

Institute of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany · Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany · Department for AI in Society, Science, and Technology, Zuse Institute Berlin, Germany +1

Model Selection Prior-Data Fitted Networks Linear Regression Feature Selection

Ask PDF

Oct 9, 2026

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in rare settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models.

Manuel Hülskamp, Julius Kobialka, Emanuel Sommer +2

LMU Munich and Munich Center for Machine Learning

Bayesian Model Averaging Prior-Data Fitted Networks Amortized Inference Bayesian Inference

Ask PDF

Oct 10, 2026

Characterizing the Separability Gap in Prior-Data Fitted Networks

Prior-data fitted networks (PFNs) are transformers pretrained on simulated datasets to approximate Bayesian posterior predictive distributions. Standard PFN theory assumes a separable task prior, under which the feature distribution and the mechanism generating labels from features are independent. We show that common PFN priors are non-separable, even in simple settings. Consequently, standard pretraining targets a predictive distribution based on a task posterior updated using both the labeled context and the query features. We characterize its difference from the predictive based on the context posterior alone and show that this difference vanishes under posterior concentration. Under misspecification (i.e., when the true joint distribution lies outside the prior's support), however, prior non-separability can favor better feature fit at the expense of conditional label fit. We illustrate the consequences for feature attribution, Bayesian optimization, and martingale checks of Bayesian behavior. The same dependence also enables PFNs to learn from additional unlabeled observations. We demonstrate this with SemiTabICL, a PFN explicitly pretrained to use these observations, which improves accuracy over a matched supervised baseline. Our findings highlight the need for explicit rules on unlabeled data use in PFN evaluation and establish the dependence between feature and label distributions as a consequential prior design choice.

Julius Kobialka, Manuel Hülskamp, Thomas Nagler +1

Department of Statistics, LMU Munich · Munich Center for Machine Learning

Prior-Data Fitted Networks Bayesian Inference

Ask PDF

Abstract

Feature Selection

Explore similar work

LinearPFN: Amortized Variable Selection for Linear Models with Interactions

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

Characterizing the Separability Gap in Prior-Data Fitted Networks