Architecture Alignment With Sparse Priors in Tabular Foundation Models

Tianqi Zhao; Tianyi Zhuang; Shuo Duan; Guanyang Wang; Yan Shuo Tan; Qiong Zhang

41021	Moneyball	44971	White Wine	44989	King County
44956	Abalone	44972	Red Wine	44990	Brazilian Houses
44957	Airfoil Self Noise	44973	Grid Stability	44992	FPS Benchmark
44958	Auction Verification	44974	Video Transcoding	44993	Health Insurance
44959	Concrete Strength	44975	Wave Energy	44994	Cars
44960	Energy Efficiency	44976	SARCOs	45012	FIFA
44962	Forest Fires	44977	California Housing	45402	Space GA

Appendix

Table 1: OpenML regression datasets used in the irrelevant-feature experiments of Section 2 . Numbers are OpenML dataset IDs.

System	Package	Official checkpoint	Departure from default
TabDPT (retrieval off)	tabdpt 1.2.0	tabdpt1_2.safetensors	retrieval disabled
TabPFN v2	tabpfn 8.0.8	tabpfn-v2-regressor.ckpt	width-limit override
TabPFN v2.5	tabpfn 8.0.8	tabpfn-v2.5-regressor-v2.5_default.ckpt	none
TabICL v2	tabicl 2.1.1	tabicl-regressor-v2-20260212.ckpt	none
TabPFN v3	tabpfn 8.0.8	tabpfn-v3-regressor-v3_default.ckpt	none

Appendix

Table 2: Official released regression checkpoints and departures from their package-default inference settings.

System	Friedman	OpenML
TabDPT (retrieval off)	0.0370 [0.0246, 0.0562]	0.0713 [0.0291, 0.1584]
TabPFN v2	0.0008 [ $-$ 0.00003, 0.0018]	0.0014 [0.0001, 0.0096]
TabPFN v2.5	$-$ 0.0002 [ $-$ 0.0012, 0.0002]	0.0011 [ $-$ 0.0010, 0.0063]
TabICL v2	0.0021 [0.0011, 0.0055]	0.0026 [0.0000, 0.0084]
TabPFN v3	0.0014 [0.0005, 0.0034]	0.0017 [ $-$ 0.0001, 0.0057]

Appendix

Table 3: Median paired

R^{2}

drop at the maximum dose

\rho=7

. Friedman pools seven configurations with 50 tasks each; OpenML contains 42 datasets. Brackets contain the 25th and 75th percentiles.

$\rho$	Friedman	OpenML
0.2	0.0001 [ $-$ 0.0004, 0.0007]	0.0003 [ $-$ 0.0011, 0.0040]
0.4	0.0002 [ $-$ 0.0005, 0.0011]	0.0005 [ $-$ 0.0015, 0.0083]
1	0.0006 [ $-$ 0.0002, 0.0020]	0.0022 [ $-$ 0.0004, 0.0214]
2	0.0020 [0.0008, 0.0051]	0.0117 [0.0020, 0.0476]
4	0.0107 [0.0049, 0.0213]	0.0270 [0.0120, 0.1143]
7	0.0363 [0.0238, 0.0550]	0.0698 [0.0280, 0.1525]

Appendix

Table 4: Paired

R^{2}

-drop difference

\Delta_{t}(\rho)

between TabDPT (retrieval off) and TabPFN v2. Entries are medians [25th, 75th percentiles] across 350 Friedman task-settings or 42 OpenML datasets, after averaging five construction repeats within each task. Positive values mean greater degradation for TabDPT.

Figure 9: Raw query-set

R^{2}

without subtracting the

\rho=0

baseline. Points show medians across tasks or datasets after averaging construction repeats. The five models have comparable clean performance within each benchmark setting.

Figure 10: Friedman noise and sample-size sweeps for the

R^{2}

drop. Panels (a) and (b) vary observation noise and total sample size, respectively; shade and line width encode the setting, and ribbons span the IQR across 50 tasks. Panel (c) reports the median and IQR across 42 OpenML datasets.

Figure 11: OpenML

R^{2}

drop stratified by the number of retained features before augmentation,

d_{\mathrm{base}}

. Points and ribbons show medians and IQRs across datasets in each stratum.

Figure 12: Normalized prediction shift from Eq. 10 . Panels (a,b) show relevant Friedman features and original OpenML features; panels (c,d) show added null features in the same order. The two feature-role pairs use separate vertical scales. Points show medians and ribbons span the IQR after averaging features and five construction repeats within each task. Friedman panels pool the seven configurations equally; OpenML panels summarize 42 datasets. Panels (c,d) show the null-feature shift plotted in Figure 2 (b).

Figure 13: Normalized one-way PDP variation. Panels (a,b) show relevant Friedman features and original OpenML features; panels (c,d) show added null features in the same order, with a separate vertical scale for each feature-role pair. Points show medians across tasks or datasets and ribbons span the IQR after averaging eligible features and five construction repeats within each task. The ordering agrees with the permutation intervention in Figure 12 .

Figure 14: Column-resolved PDPs for a fixed Friedman task with

N=1000

and

\sigma=1

. Both models respond to the relevant features, while TabDPT shows a larger response to the added null features.

Figure 15: Column-resolved PDPs for Grid Stability, a medium-dimensional OpenML dataset with 12 original features.

Figure 16: Friedman permutation sensitivity across noise and sample-size settings. Shade and line width encode the setting; ribbons show IQRs across 50 tasks. Columns vary noise (left) and total sample size (right), while rows show relevant (top) and null (bottom) features.

Figure 17: OpenML permutation sensitivity stratified by pre-augmentation feature count. Rows show original and null features; points and ribbons show medians and IQRs across datasets.

Figure 18: OpenML PDP variation stratified by pre-augmentation feature count, using the same aggregation as Figure 13 .

Release	Package	Official checkpoint	Default ensemble	Departure from default
TabDPT v1.0	tabdpt 0.1.0	tabdpt_76M.ckpt	single estimator	full context instead of 128-row retrieval
TabDPT v1.1	tabdpt 1.1.14	tabdpt1_1.safetensors	8	none in effect
TabDPT v1.2	tabdpt 1.2.0	tabdpt1_2.safetensors	8	none
TabDPT v1.3	tabdpt 1.3.0	tabdpt1_3.safetensors	8	none

Appendix

Table 5: TabDPT releases used in the replication. The default context of tabdpt 1.1.14 already exceeds every context in our tasks, and tabdpt 1.2.0 and 1.3.0 use the full context by default, so only v1.0 departs from its package default.

Figure 19: Null-feature dose response of four TabDPT releases with TabPFN v2 as reference. (a, b) Median

R^{2}

drop with interquartile bands. (c, d) Median absolute

R^{2}

at each dose. Aggregation matches Figure 10 : five construction repeats are averaged within each task before taking medians across 350 Friedman task-settings or 42 OpenML datasets.

Figure 20: Normalized prediction shift under single-feature permutation for four TabDPT releases, with TabPFN v2 as reference. (a, b) Relevant Friedman features and original OpenML features as positive controls. (c, d) Added null features. Aggregation matches Figure 12 .

	$R^{2}$ drop		Null-feature shift		Exceeds TabPFN v2
Release	Friedman	OpenML	Friedman	OpenML	(% of tasks)
TabDPT v1.0	0.1466 [0.1213, 0.1780]	0.1205 [0.0547, 0.1661]	0.0344 [0.0324, 0.0367]	0.0276 [0.0194, 0.0431]	100.0 / 97.6
TabDPT v1.1	0.0727 [0.0533, 0.1095]	0.0999 [0.0406, 0.1709]	0.0300 [0.0278, 0.0333]	0.0229 [0.0143, 0.0304]	100.0 / 97.6
TabDPT v1.2	0.0370 [0.0246, 0.0562]	0.0713 [0.0291, 0.1584]	0.0188 [0.0177, 0.0202]	0.0180 [0.0127, 0.0330]	98.9 / 95.2
TabDPT v1.3	0.0089 [0.0039, 0.0233]	0.0619 [0.0162, 0.1130]	0.0115 [0.0096, 0.0135]	0.0156 [0.0088, 0.0272]	94.0 / 92.9
TabPFN v2 (reference)	0.0008 [ $-$ 0.00003, 0.0018]	0.0014 [0.0001, 0.0096]	0.0029 [0.0020, 0.0043]	0.0021 [0.0011, 0.0051]	—

Appendix

Table 6: TabDPT releases at the maximum dose

\rho=7

. Entries are medians [25th, 75th percentiles] across 350 Friedman task-settings or 42 OpenML datasets after averaging five construction repeats within each task. The last column is the percentage of paired tasks on which the release exceeds TabPFN v2 in

R^{2}

drop, Friedman / OpenML.

$d$	$k$	Model	Selected LR	Max. steps	Checkpoint step (0/1/2)
10	1	Row-token	0.0010	10000	10000 / 10000 / 10000
10	1	Alternating-axis	0.0010	10000	9500 / 9500 / 9500
10	5	Row-token	0.0010	10000	9500 / 10000 / 10000
10	5	Alternating-axis	0.0010	10000	9500 / 10000 / 10000
10	10	Row-token	0.0010	10000	10000 / 10000 / 7000
10	10	Alternating-axis	0.0010	10000	10000 / 10000 / 6000

Appendix

Table 7: Selected learning rates for schedule-free AdamW and checkpoints from final training. Maximum step budgets are fixed by dimension; checkpoint steps are chosen by validation MSE and listed in training-seed order 0/1/2.

Figure 21: Dense-prior and architecture gaps across support sizes. (a) Mean dense-prior prediction gap

\Delta_{k}^{\mathrm{dense}}(D)

(Eq. 50 ) over 32,000 test datasets per condition. The dense endpoint is exactly zero. (b) Per-dataset architecture gaps

\widehat{G}_{k}^{(t)}

(Eq. 51 ), averaged equally over the same 32,000 test datasets for each seed. Positive values favor the alternating-axis model. Large markers show means over three seeds, small points show individual seed averages, and error bars show seed SDs (ddof=0). The axis label omits the dataset index. Circles/dashed lines denote

d=10

; squares/solid lines denote

d=20

. Panel (a) uses unnormalized errors on a linear axis; panel (b) uses task-normalized errors on a log axis. Support sizes are categorical.

Figure 22: Task-level distributions of the architecture gap. Rows correspond to

d=10

and

d=20

; columns to

k=1

,

k=d/2

, and

k=d

. Positive gaps favor the alternating-axis model. Gray shading shows the distribution of per-dataset gaps

\widehat{G}_{k}^{(t)}

after averaging over three training seeds; colored curves show the distributions for individual seeds. All distributions use the same histogram bins within each panel; the dashed line marks zero. Annotations report the positive fraction, mean, and median of the seed-averaged gaps, computed over all 32,000 tasks per condition. Axis ranges focus on the distribution bodies; the fraction of seed-averaged gaps outside each view is noted below the panel. Densities are normalized by the full task count.

Figure 23: Task-normalized error components at

\bm{d=20}

. Bars stack coefficient, squared-intercept, and residual errors for row-token and alternating-axis predictors at

k=1,10,20

. For each seed, components are averaged equally over the same 32,000 test datasets. Bars show means over three seeds; points mark each seed’s cumulative component means at the stack boundaries. Colors match Figure 5 .

Figure 24: The sparse architecture gap lies mainly in coefficients. Rows correspond to

d=10,20

; columns to

k=1,d/2,d

. Bars show row-token minus alternating-axis gaps in task-normalized prediction, coefficient, squared-intercept, and residual error. For each seed, gaps are averaged equally over the same 32,000 test datasets. Bars show means over three seeds; points show individual seed averages; error bars show seed SDs (ddof=0). Positive values favor the alternating-axis model. Hollow diamonds show component sums; insets magnify the two small non-coefficient gaps. Main panels share vertical ranges within each column. Labels

G

,

G^{C}

,

G^{\zeta}

, and

G^{N}

suppress hats and the indices

k,t

(Eqs. 51 and 55 ).

Figure 25: Active and inactive coefficient errors by architecture. Stacked bars show task-normalized group errors (Eq. 57 ); each bar sums to the model’s coefficient error. Rows correspond to

d=10,20

and columns to

k=1,d/2,d

. For each seed, group errors are averaged equally over the same 32,000 test datasets. Bars show means over three seeds; points mark each seed’s mean active and total coefficient errors at the corresponding stack boundaries. At

k=d

, the inactive contribution is zero. Vertical ranges are shared across dimensions within each column.

Figure 26: Coefficient errors per coordinate under sparse priors. Each model’s task-normalized active and inactive errors (Eq. 57 ) are divided by

k

and

d-k

, respectively. Rows correspond to

d=10,20

and columns to

k=1,d/2

. Colors match Figure 25 . Per-coordinate errors are averaged equally over the same 32,000 test datasets for each seed. Diamonds show means over three seeds, circles show individual seed averages, and error bars show seed SDs (ddof=0). All panels share a logarithmic vertical axis.

Figure 27: OLS and learned-model coefficient errors relative to Bayes. Rows correspond to

d=10,20

and columns to

k=1,d/2,d

. Bars stack task-normalized active and inactive coefficient errors; annotations give their sum. Every bar uses the same 32,000 test datasets per condition. OLS is deterministic given the context; learned-model bars average three training seeds and reproduce Figure 25 . At

k=d

, the inactive sum is zero. Linear vertical ranges are shared within each column.

Figure 28: OLS coefficients and posterior-mean coefficients. Each panel plots all coordinates from 150 fixed, randomly sampled test tasks. Dark red circles denote active coordinates and dark gray crosses denote inactive coordinates in the realized support. Dashed lines indicate equal coefficients. At

k=d

, variation in context length produces different shrinkage slopes across tasks. Error summaries use the complete 32,000-task test set.

Figure 29: Fidelity of coefficient changes to prediction changes. All layers,

k=1

,

\delta=0.1

, joint context-and-query attenuation. Panels compare active and inactive sources within (a) the alternating-axis model and (b) frozen TabPFN v2, each using the same 128 tasks per dimension. Colors distinguish

d=10

and

d=20

. Solid and dashed lines show active- and inactive-source medians of within-context role-averaged

\widehat{R}^{2}_{\Delta}

, respectively, with equal context weights. Bands use the pointwise 95% bootstrap procedure in Appendix F.1 . Higher values indicate better fidelity (

\uparrow

); the maximum is one, and values can be negative.

Figure 30: Coefficient responses to context-message attenuation. Alternating-axis model,

d=10

,

k=1

,

\delta=0.1

; panels show layers 1–3. Each matrix column attenuates one source on context rows. The task, matrix axes, per-layer color scales, and gold outlines match Figure 7 , allowing direct comparison with query interventions.

Figure 31: Coefficient responses within a

\bm{d=20}

task. Alternating-axis model,

k=1

,

\delta=0.1

. Top: context interventions; bottom: query interventions. Columns show layers 1–3. The same task is used throughout; context and query panels share a symmetric color scale within each layer. Matrix axes and gold outlines follow Figure 7 .

Figure 32: Predictive consequences of context- and query-message attenuation. Panel (c): alternating-axis model; panel (f): frozen TabPFN v2.

d=10

,

k=1

,

\delta=0.1

; the same 128 tasks as in Figure 8 . Curves show median prediction-error changes after averaging sources within each role and task; bands are pointwise 95% generator-batch bootstrap intervals. Panels (c) and (f) complete Figure 8 , whose panels (a, b, d, e) appear in the main text.

Figure 33: Context and query message effects at

\bm{d=20}

.

k=1

,

\delta=0.1

, 128 shared tasks. Top: alternating-axis model; bottom: frozen TabPFN v2. Layout, source-role summaries, and confidence intervals follow Figure 8 .

Figure 34: Direction and magnitude of active-source control. Final-layer empirical distributions of signed sensitivity

s_{g}

, with

k=1

,

\delta=0.1

, and the same 128 tasks per dimension and model. Top: alternating-axis model; bottom: TabPFN v2. Left:

d=10

; right:

d=20

. Colors distinguish context and query attenuation, as in Figure 8 . All tasks are shown. The horizontal axis is linear for

|s_{g}|\leq 0.05

and logarithmic outside that interval; the dotted line marks zero.

Figure 35: Attenuation-strength comparison at

d=10

.

k=1

, 128 tasks, joint context-and-query attenuation. Panels (a–d): alternating-axis model; (e–h): TabPFN v2. For each model, rows show sensitivity and error change; columns show active and inactive sources. Curves show task medians of role averages; bands are 95% batch-bootstrap intervals (Appendix F.1 ). Dose comparisons share bootstrap draws. Sensitivity axes match across dimensions; error axes are panel-specific.

Figure 36: Attenuation-strength comparison at

d=20

.

k=1

, 128 tasks, joint context-and-query attenuation. Panels (a–d): alternating-axis model; (e–h): TabPFN v2. For each model, rows show sensitivity and error change; columns show active and inactive sources. Curves show task medians of role averages; bands are 95% batch-bootstrap intervals (Appendix F.1 ). Dose comparisons share bootstrap draws. Sensitivity axes match across dimensions; error axes are panel-specific.

Figure 37: Coefficient control across support sizes. Panels (a–f): alternating-axis model; (g–l): frozen TabPFN v2. Within each model, rows show

k=d/2

and

k=d

; columns show specificity, sensitivity, and prediction-error change. Joint attenuation,

\delta=0.1

, 128 shared tasks per dimension and prior. Colors distinguish dimensions; solid and dashed curves denote active and inactive sources. All sources are active at

k=d

. Bands are pointwise 95% batch-bootstrap intervals. TabPFN v2 sources are native two-feature groups.

Symbol	Meaning
$D=(X,y)$ ; $n,d$	Context, context size, and feature dimension; $X\in\mathbb{R}^{n\times d}$ , $y\in\mathbb{R}^{n}$ .
$x\in\mathbb{R}^{d}$ ; $x_{i}^{\top}$	Independent query vector; row $i$ of the context design. Within a query, $x_{j}$ denotes coordinate $j$ .
$\beta$ ; $S$ ; $k$ ; $P_{k}$	Data-generating regression coefficients, their active support, its size, and the corresponding coefficient prior.
$v^{2}$ ; $\sigma^{2}$ ; $\tau^{2}$	Total prior signal energy, response-noise variance, and $\sigma^{2}/n$ .
$c=X^{\top}y/n$	Sufficient statistic for the coefficient posterior.
$\pi_{k}(A\mid c)$ ; $q_{k,j}(c)$	Posterior probability of support $A$ ; posterior inclusion probability of coordinate $j$ .

Appendix

Table 8: Notation for the controlled model, affine projection, and interventions.

Architecture Alignment With Sparse Priors in Tabular Foundation Models

Abstract

Figures & tables

Appendix

Tabular Foundation Models

FTFM: Factorized Tabular Foundation Models

Tabular Foundation Models Are Effectively Shallow

LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models

Abstract

Tabular Foundation Models

Explore similar work

FTFM: Factorized Tabular Foundation Models

Tabular Foundation Models Are Effectively Shallow

LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models