SelectPFN: Amortized Bayesian Variable Selection for Sparse Nonlinear Regression with Prior-Data Fitted Networks

Bin Luo

stat.MLOct 15, 2026

SelectPFN: Amortized Bayesian Variable Selection for Sparse Nonlinear Regression with Prior-Data Fitted Networks

Authors: Bin Luo

Organizations: School of Data Science and Analytics, Kennesaw State University

Abstract

Tabular foundation models based on prior-data fitted networks (PFNs) predict on a new data set in a single forward pass, but they cannot report which inputs their predictions depend on. We propose SelectPFN, a PFN that performs Bayesian variable selection and prediction jointly for sparse nonlinear regression. In one forward pass it returns posterior inclusion probabilities for every variable, and its predictions are computed from sampled subsets only, so excluded variables provably cannot affect them; we show this is exactly what the Bayesian model prescribes. The network is pretrained once on synthetic tasks from a meta-prior built around a group spike-and-slab sparse-input neural network. Column-permutation equivariance combined with random interleaving of original and knockoff columns yields knockoff statistics with exact finite-sample false discovery rate control, even when the meta-prior is wrong. We develop theory for amortized selection: a Bayes-regret bound for the amortized median probability model, a characterization showing that the information needed to detect a variable grows like $(2/λ)\log(1/ω)$ in the prior weight $ω$ of structures that can use it, where $λ\le1$ is an efficiency, and a negative result showing that average-case training does not transfer to frequentist guarantees at a fixed truth. On out-of-prior nonlinear designs with p up to 500, SelectPFN improves the area under the precision-recall curve by 0.23-0.28 over the lasso, BART, DART, LassoNet and a group-penalized sparse neural network, at a false discovery proportion of about 0.07 and 0.04 seconds per data set. The advantage persists on real covariate matrices, and, as the theory predicts, detection of interaction-only variables rises monotonically with their prior weight.

Figures & tables

Figure 1: Mean AUPRC by design with 95% bootstrap intervals; most intervals are narrower than the markers. The two SelectPFN models are shades of one hue. Numerical values are in Table 16 .

Figure 2: Design C: how often the MPM includes the interaction-only variables

x_{3},x_{4}

(average of the two; band: 95% bootstrap interval over data sets). Without the interaction-only family the meta-prior gives such structures no weight, and the variables are essentially never found, as Corollary 1 and Theorem 2 predict.

Figure 3: Models that differ only in the prior weight

\omega

of the interaction-only family. Left: detection of

x_{3},x_{4}

(acting only through

\sin(x_{3}x_{4})

in design C) against the information index

t_{j}

, seeds pooled. Right: detection at two fixed

(n,\mathrm{SNR})

; dots are training seeds, lines their mean.

Figure 4: Reliability on held-out meta-prior tasks: observed frequency of activity minus mean

q

in bins of width 0.1, with 95% binomial intervals. The shaded band is

\pm 0.05

Figure 5: Knockoff mode: FDR (left; whiskers

\pm 2

s.e.; dashed: nominal level) and power (right) of knockoff+ with exact knockoffs on the

|S_{0}|=20

designs L20 and B20 (

n=300

p\in\{100,200\}

; 1,600 data sets).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Distribution
Sample size	$N\sim\mathrm{LU}(50,1000)$ rows per task; a fraction $n/N\sim U(0.5,0.9)$ are training (context) rows, the remaining $m=N-n$ are query rows
Dimension	$p\sim\mathrm{LU}(5,p_{\max})$ with curriculum $p_{\max}=50\to 200\to 500$ in thirds of training
Sparsity	Beta–Bernoulli, $w\sim\mathrm{Beta}(\bar{w}\nu,(1-\bar{w})\nu)$ , $\mathbb{E}\|S\|=\bar{w}\,p\sim\mathrm{LU}(1,\min(20,p/3))$ , $\nu\sim\mathrm{LU}(2,50)$ ; $\|S\|\leq 40$ ; an empty $\gamma$ is redrawn w.p. 0.9
Dependence	independent; AR(1); block equicorrelation; latent factor (equal weights). Active variables at random positions
Marginals	Gaussian copula with per-column transform: identity, uniform, log-normal skew, Tukey- $h$ heavy tails ( $h\sim U(0.1,0.4)$ ), discretized; columns standardized
Function family	sparse-input MLP with group spike-and-slab first layer, eq. 3 (0.30); additive (0.12); additive + pairwise, with no main effect for a variable w.p. 0.3 (0.12); trees/step functions (0.10); GP with ARD kernel, length-scales $\mathrm{LU}(0.3,2.5)$ (0.10); linear (0.14); interaction-only pairs/triples through functions odd in each argument (a single active variable gets a main effect), plus weak main effects w.p. 0.3 (0.12)

Appendix

Table 14: Meta-prior of SelectPFN.

\mathrm{LU}(a,b)

is log-uniform on

[a,b]

. The ablation model of Section 5.4.4 uses the same meta-prior without the interaction-only family, the weak-effect regime and interaction-only variables in the pairwise family, with family weights MLP 0.40, additive 0.15, pairwise 0.15, tree 0.10, GP 0.10 and linear 0.10.

Figure 6: Exact-posterior toy problem: detection threshold

t_{50}

against

\log(1/\omega)

, with the line of slope 2 (the limit in Corollary 2 ) and a heuristic fit. This is a numerical verification and is not used in any proof.

Feature Selection

Feature selection identifies compact, informative variables or input dimensions to improve prediction, efficiency, interpretability, and statistical inference.

See all 196 papers

LinearPFN: Amortized Variable Selection for Linear Models with Interactions
Oct 5, 2026Louis Schiekiera, Max Zimmer, Christophe Roux +3Model Selection Prior-Data Fitted Networks
Institute of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany · Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany · Department for AI in Society, Science, and Technology, Zuse Institute Berlin, Germany +1
How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?
Oct 9, 2026Manuel Hülskamp, Julius Kobialka, Emanuel Sommer +2Bayesian Model Averaging Prior-Data Fitted Networks
LMU Munich and Munich Center for Machine Learning
Characterizing the Separability Gap in Prior-Data Fitted Networks
Oct 10, 2026Julius Kobialka, Manuel Hülskamp, Thomas Nagler +1Prior-Data Fitted Networks Bayesian Inference
Department of Statistics, LMU Munich · Munich Center for Machine Learning

Oct 5, 2026J/K move · Enter open · S save

LinearPFN: Amortized Variable Selection for Linear Models with Interactions

Louis Schiekiera, Max Zimmer, Christophe Roux, Manuel Arnold+2

Institute of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany · Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany · Department for AI in Society, Science, and Technology, Zuse Institute Berlin, Germany+1

Model Selection Prior-Data Fitted Networks Linear Regression Feature Selection

Open paper AskPDF arXiv

Spike-and-slab regression is a standard Bayesian formulation of variable selection: it returns a posterior distribution over which candidate effects are active rather than a single selected subset, so that every candidate effect carries an inclusion probability. Its cost grows exponentially with the number of candidate effects, so the posterior can be enumerated exactly only when the number of predictors is small. Beyond that reach, the posterior has to be approximated, typically by Markov chain Monte Carlo over the model space, which requires a fresh run for every dataset and, within a fixed budget of steps, may fail to converge. We present LinearPFN, a prior-data fitted transformer network that amortizes spike-and-slab inference for linear models with main effects and pairwise interactions. The network is pretrained once on synthetic datasets, drawn from an explicitly specified prior, and a single forward pass over a new dataset returns posterior inclusion probabilities, posterior-mean coefficients and posterior predictive distributions with no per-dataset fitting. The prior is conjugate by design, so that the posterior for each fixed set of active effects has a closed form, and wherever the exact posterior is still computable by enumeration we verify the network's outputs against it. On real predictor matrices from published social-science datasets, with outcomes drawn from the prior so that the true active set is known, LinearPFN attains a higher per-dataset selection AUC and a higher F1 under the median probability model rule than five classical baselines. The lead holds when the coefficients, the interactions or the noise depart from the prior. Code: https://github.com/schiekiera/LinearPFN. Trained model: https://huggingface.co/schiekiera/LinearPFN.

Abstract

Feature Selection

Explore similar work

LinearPFN: Amortized Variable Selection for Linear Models with Interactions

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

Characterizing the Separability Gap in Prior-Data Fitted Networks