Architecture Alignment With Sparse Priors in Tabular Foundation Models

Tianqi Zhao; Tianyi Zhuang; Shuo Duan; Guanyang Wang; Yan Shuo Tan; Qiong Zhang

cs.LGSep 29, 2026

Architecture Alignment With Sparse Priors in Tabular Foundation Models

Authors: Tianqi Zhao, Tianyi Zhuang, Shuo Duan, Guanyang Wang, Yan Shuo Tan, Qiong Zhang

Organizations: National University of Singapore · Rutgers University

Abstract

Tabular foundation models (TFMs) are increasingly popular because they deliver strong predictions on new datasets through in-context learning, without task-specific training or extensive tuning. Yet released TFMs differ simultaneously in their pretraining priors, architectures, and objectives, obscuring their respective inductive biases. We therefore examine one concrete capability: irrelevant-feature suppression. Across synthetic tasks and real-world datasets, adding null features causes substantially greater predictive degradation in the row-token model TabDPT, whereas the cell-token alternating-axis model TabPFN v2 and other TFMs remain comparatively stable. This gap motivates us to ask whether architecture contributes to irrelevant-feature suppression. Because released TFMs remain confounded by other design choices, we train streamlined row-token and alternating-axis transformers under identical sparse-to-dense linear priors. Exact Bayes analysis shows that sparse prediction requires context-dependent feature gating, whereas the dense endpoint requires only uniform feature weighting. Consistent with this distinction, the alternating-axis model is substantially closer to the Bayesian optimal predictor on sparse tasks, while the architecture gap becomes negligible on dense tasks; almost all of the sparse gap arises from linear coefficient-estimation error. Finally, in both the controlled model and frozen TabPFN v2, we examine the effect of interventions on the feature-attention outputs on the linear coefficients, finding evidence of task-dependent selective routing of computation through feature-indexed pathways. Together, these results support architecture-prior alignment: preserving an addressable feature axis provides an inductive bias for task-adaptive relevance inference. Code is available at https://github.com/Tianqi-Zhao/ArchitecturePriorTFMs.

Figures & tables

Figure 1: Adding null features degrades TabDPT v1.2 more than the other released TFMs . Median query-set

R^{2}

drop on (a) synthetic and (b) OpenML datasets as the ratio

\rho

increases.

Figure 2: Null features affect TabDPT more often and more strongly than TabPFN. (a) Fraction of paired tasks on which TabDPT v1.2 has a larger

R^{2}

drop or null-feature shift than TabPFN v2. (b) Median null-feature shift for both models across values of

\rho

. Solid: OpenML; dashed: Friedman.

Figure 3: Architectures and an idealized one-sparse Bayes posterior predictive mean. (a–b) The alternating-axis and row-token models used in the controlled comparison. (c) For

k=1

, response routing and within-feature aggregation form

c_{j}

, and feature-axis softmax combines

\lambda_{1}c_{j}x_{j}

using logits proportional to

c_{j}^{2}

. Panel (c) is a mathematical construction, not an identified circuit in the trained models (App. D.5 ). Colors denote features; dashed and heavy outlines mark query and readout tokens, respectively; shading denotes pointwise maps.

Figure 4: The sparse advantage contracts at the dense endpoint. Task-normalized Bayes approximation error (log scale) for the row-token and alternating-axis models. denotes

d=10

; denotes

d=20

. Markers and error bars show the mean

\pm

SD across three training seeds, each evaluated on 32,000 shared test tasks.

Figure 5: Task-normalized error components at

\bm{d=10}

. coefficient; squared intercept; residual. Bars stack these components for one architecture. Each component is averaged equally over 32,000 shared test datasets per seed. Bars show three-seed means, and points show seed averages. The total bar height equals the sum of the three components.

Figure 6: Feature-message attenuation in the simplified alternating-axis model. Source feature

j

’s contribution is scaled by

1-\delta

at every receiver and head, either in context rows or in query rows. Other sources’ contributions at that operation and the residual pathway are unchanged. Bar sizes are schematic; App. F.1 gives implementation details.

Panel of Figure 7 — Figure 7: Coefficient responses to query-message attenuation. Alternating-axis model, $d=10$ , $k=1$ , $\delta=0.1$ . The task is nearest the median Bayes coefficient norm among the 128 tasks. Each matrix column attenuates one source on query rows; matrix rows index affected coefficients. Color shows signed $\widehat{J}_{rj}^{(\ell)}$ with a symmetric scale per layer. Gold outlines mark the active source and its own coefficient.

Panel of Figure 8 — Figure 8: Support-selective message effects in context and query rows. Panels (a,b): alternating-axis model; (d,e): frozen TabPFN v2. $d=10$ , $k=1$ , $\delta=0.1$ ; the same 128 tasks are used throughout. denotes context-row interventions; denotes query-row interventions. and denote active and inactive sources, respectively. For each model, panels show mean specificity $P_{R}$ and median sensitivity $w$ , after averaging sources within each role and task. Prediction-error panels (c,f) appear in Fig. 32 . Bands are pointwise 95% intervals from 2,000 generator-batch bootstrap resamples.

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

41021	Moneyball	44971	White Wine	44989	King County
44956	Abalone	44972	Red Wine	44990	Brazilian Houses
44957	Airfoil Self Noise	44973	Grid Stability	44992	FPS Benchmark
44958	Auction Verification	44974	Video Transcoding	44993	Health Insurance
44959	Concrete Strength	44975	Wave Energy	44994	Cars
44960	Energy Efficiency	44976	SARCOs	45012	FIFA
44962	Forest Fires	44977	California Housing	45402	Space GA

Appendix

Table 1: OpenML regression datasets used in the irrelevant-feature experiments of Section 2 . Numbers are OpenML dataset IDs.

System	Package	Official checkpoint	Departure from default
TabDPT (retrieval off)	tabdpt 1.2.0	tabdpt1_2.safetensors	retrieval disabled
TabPFN v2	tabpfn 8.0.8	tabpfn-v2-regressor.ckpt	width-limit override
TabPFN v2.5	tabpfn 8.0.8	tabpfn-v2.5-regressor-v2.5_default.ckpt	none
TabICL v2	tabicl 2.1.1	tabicl-regressor-v2-20260212.ckpt	none
TabPFN v3	tabpfn 8.0.8	tabpfn-v3-regressor-v3_default.ckpt	none

Appendix

Table 2: Official released regression checkpoints and departures from their package-default inference settings.

System	Friedman	OpenML
TabDPT (retrieval off)	0.0370 [0.0246, 0.0562]	0.0713 [0.0291, 0.1584]
TabPFN v2	0.0008 [ $-$ 0.00003, 0.0018]	0.0014 [0.0001, 0.0096]
TabPFN v2.5	$-$ 0.0002 [ $-$ 0.0012, 0.0002]	0.0011 [ $-$ 0.0010, 0.0063]
TabICL v2	0.0021 [0.0011, 0.0055]	0.0026 [0.0000, 0.0084]
TabPFN v3	0.0014 [0.0005, 0.0034]	0.0017 [ $-$ 0.0001, 0.0057]

Appendix

Table 3: Median paired

R^{2}

drop at the maximum dose

\rho=7

. Friedman pools seven configurations with 50 tasks each; OpenML contains 42 datasets. Brackets contain the 25th and 75th percentiles.

$\rho$	Friedman	OpenML
0.2	0.0001 [ $-$ 0.0004, 0.0007]	0.0003 [ $-$ 0.0011, 0.0040]
0.4	0.0002 [ $-$ 0.0005, 0.0011]	0.0005 [ $-$ 0.0015, 0.0083]
1	0.0006 [ $-$ 0.0002, 0.0020]	0.0022 [ $-$ 0.0004, 0.0214]
2	0.0020 [0.0008, 0.0051]	0.0117 [0.0020, 0.0476]
4	0.0107 [0.0049, 0.0213]	0.0270 [0.0120, 0.1143]
7	0.0363 [0.0238, 0.0550]	0.0698 [0.0280, 0.1525]

Appendix

Table 4: Paired

R^{2}

-drop difference

\Delta_{t}(\rho)

between TabDPT (retrieval off) and TabPFN v2. Entries are medians [25th, 75th percentiles] across 350 Friedman task-settings or 42 OpenML datasets, after averaging five construction repeats within each task. Positive values mean greater degradation for TabDPT.

Figure 9: Raw query-set

R^{2}

without subtracting the

\rho=0

baseline. Points show medians across tasks or datasets after averaging construction repeats. The five models have comparable clean performance within each benchmark setting.

Figure 10: Friedman noise and sample-size sweeps for the

R^{2}

drop. Panels (a) and (b) vary observation noise and total sample size, respectively; shade and line width encode the setting, and ribbons span the IQR across 50 tasks. Panel (c) reports the median and IQR across 42 OpenML datasets.

Figure 11: OpenML

R^{2}

drop stratified by the number of retained features before augmentation,

d_{\mathrm{base}}

. Points and ribbons show medians and IQRs across datasets in each stratum.

Figure 12: Normalized prediction shift from Eq. 10 . Panels (a,b) show relevant Friedman features and original OpenML features; panels (c,d) show added null features in the same order. The two feature-role pairs use separate vertical scales. Points show medians and ribbons span the IQR after averaging features and five construction repeats within each task. Friedman panels pool the seven configurations equally; OpenML panels summarize 42 datasets. Panels (c,d) show the null-feature shift plotted in Figure 2 (b).

Figure 13: Normalized one-way PDP variation. Panels (a,b) show relevant Friedman features and original OpenML features; panels (c,d) show added null features in the same order, with a separate vertical scale for each feature-role pair. Points show medians across tasks or datasets and ribbons span the IQR after averaging eligible features and five construction repeats within each task. The ordering agrees with the permutation intervention in Figure 12 .

Figure 14: Column-resolved PDPs for a fixed Friedman task with

N=1000

and

\sigma=1

. Both models respond to the relevant features, while TabDPT shows a larger response to the added null features.

Figure 15: Column-resolved PDPs for Grid Stability, a medium-dimensional OpenML dataset with 12 original features.

Figure 16: Friedman permutation sensitivity across noise and sample-size settings. Shade and line width encode the setting; ribbons show IQRs across 50 tasks. Columns vary noise (left) and total sample size (right), while rows show relevant (top) and null (bottom) features.

Figure 17: OpenML permutation sensitivity stratified by pre-augmentation feature count. Rows show original and null features; points and ribbons show medians and IQRs across datasets.

Figure 18: OpenML PDP variation stratified by pre-augmentation feature count, using the same aggregation as Figure 13 .

Release	Package	Official checkpoint	Default ensemble	Departure from default
TabDPT v1.0	tabdpt 0.1.0	tabdpt_76M.ckpt	single estimator	full context instead of 128-row retrieval
TabDPT v1.1	tabdpt 1.1.14	tabdpt1_1.safetensors	8	none in effect
TabDPT v1.2	tabdpt 1.2.0	tabdpt1_2.safetensors	8	none
TabDPT v1.3	tabdpt 1.3.0	tabdpt1_3.safetensors	8	none

Appendix

Table 5: TabDPT releases used in the replication. The default context of tabdpt 1.1.14 already exceeds every context in our tasks, and tabdpt 1.2.0 and 1.3.0 use the full context by default, so only v1.0 departs from its package default.

Figure 19: Null-feature dose response of four TabDPT releases with TabPFN v2 as reference. (a, b) Median

R^{2}

drop with interquartile bands. (c, d) Median absolute

R^{2}

at each dose. Aggregation matches Figure 10 : five construction repeats are averaged within each task before taking medians across 350 Friedman task-settings or 42 OpenML datasets.

Figure 20: Normalized prediction shift under single-feature permutation for four TabDPT releases, with TabPFN v2 as reference. (a, b) Relevant Friedman features and original OpenML features as positive controls. (c, d) Added null features. Aggregation matches Figure 12 .

	$R^{2}$ drop		Null-feature shift		Exceeds TabPFN v2
Release	Friedman	OpenML	Friedman	OpenML	(% of tasks)
TabDPT v1.0	0.1466 [0.1213, 0.1780]	0.1205 [0.0547, 0.1661]	0.0344 [0.0324, 0.0367]	0.0276 [0.0194, 0.0431]	100.0 / 97.6
TabDPT v1.1	0.0727 [0.0533, 0.1095]	0.0999 [0.0406, 0.1709]	0.0300 [0.0278, 0.0333]	0.0229 [0.0143, 0.0304]	100.0 / 97.6
TabDPT v1.2	0.0370 [0.0246, 0.0562]	0.0713 [0.0291, 0.1584]	0.0188 [0.0177, 0.0202]	0.0180 [0.0127, 0.0330]	98.9 / 95.2
TabDPT v1.3	0.0089 [0.0039, 0.0233]	0.0619 [0.0162, 0.1130]	0.0115 [0.0096, 0.0135]	0.0156 [0.0088, 0.0272]	94.0 / 92.9
TabPFN v2 (reference)	0.0008 [ $-$ 0.00003, 0.0018]	0.0014 [0.0001, 0.0096]	0.0029 [0.0020, 0.0043]	0.0021 [0.0011, 0.0051]	—

Appendix

Table 6: TabDPT releases at the maximum dose

\rho=7

. Entries are medians [25th, 75th percentiles] across 350 Friedman task-settings or 42 OpenML datasets after averaging five construction repeats within each task. The last column is the percentage of paired tasks on which the release exceeds TabPFN v2 in

R^{2}

drop, Friedman / OpenML.

$d$	$k$	Model	Selected LR	Max. steps	Checkpoint step (0/1/2)
10	1	Row-token	0.0010	10000	10000 / 10000 / 10000
10	1	Alternating-axis	0.0010	10000	9500 / 9500 / 9500
10	5	Row-token	0.0010	10000	9500 / 10000 / 10000
10	5	Alternating-axis	0.0010	10000	9500 / 10000 / 10000
10	10	Row-token	0.0010	10000	10000 / 10000 / 7000
10	10	Alternating-axis	0.0010	10000	10000 / 10000 / 6000

Appendix

Table 7: Selected learning rates for schedule-free AdamW and checkpoints from final training. Maximum step budgets are fixed by dimension; checkpoint steps are chosen by validation MSE and listed in training-seed order 0/1/2.

Panel of Figure 21 — Figure 21: Dense-prior and architecture gaps across support sizes. (a) Mean dense-prior prediction gap $\Delta_{k}^{\mathrm{dense}}(D)$ (Eq. 50 ) over 32,000 test datasets per condition. The dense endpoint is exactly zero. (b) Per-dataset architecture gaps $\widehat{G}_{k}^{(t)}$ (Eq. 51 ), averaged equally over the same 32,000 test datasets for each seed. Positive values favor the alternating-axis model. Large markers show means over three seeds, small points show individual seed averages, and error bars show seed SDs (ddof=0). The axis label omits the dataset index. Circles/dashed lines denote $d=10$ ; squares/solid lines denote $d=20$ . Panel (a) uses unnormalized errors on a linear axis; panel (b) uses task-normalized errors on a log axis. Support sizes are categorical.

Panel of Figure 22 — Figure 22: Task-level distributions of the architecture gap. Rows correspond to $d=10$ and $d=20$ ; columns to $k=1$ , $k=d/2$ , and $k=d$ . Positive gaps favor the alternating-axis model. Gray shading shows the distribution of per-dataset gaps $\widehat{G}_{k}^{(t)}$ after averaging over three training seeds; colored curves show the distributions for individual seeds. All distributions use the same histogram bins within each panel; the dashed line marks zero. Annotations report the positive fraction, mean, and median of the seed-averaged gaps, computed over all 32,000 tasks per condition. Axis ranges focus on the distribution bodies; the fraction of seed-averaged gaps outside each view is noted below the panel. Densities are normalized by the full task count.

Panel of Figure 23 — Figure 23: Task-normalized error components at $\bm{d=20}$ . Bars stack coefficient, squared-intercept, and residual errors for row-token and alternating-axis predictors at $k=1,10,20$ . For each seed, components are averaged equally over the same 32,000 test datasets. Bars show means over three seeds; points mark each seed’s cumulative component means at the stack boundaries. Colors match Figure 5 .

Panel of Figure 24 — Figure 24: The sparse architecture gap lies mainly in coefficients. Rows correspond to $d=10,20$ ; columns to $k=1,d/2,d$ . Bars show row-token minus alternating-axis gaps in task-normalized prediction, coefficient, squared-intercept, and residual error. For each seed, gaps are averaged equally over the same 32,000 test datasets. Bars show means over three seeds; points show individual seed averages; error bars show seed SDs (ddof=0). Positive values favor the alternating-axis model. Hollow diamonds show component sums; insets magnify the two small non-coefficient gaps. Main panels share vertical ranges within each column. Labels $G$ , $G^{C}$ , $G^{\zeta}$ , and $G^{N}$ suppress hats and the indices $k,t$ (Eqs. 51 and 55 ).

Panel of Figure 25 — Figure 25: Active and inactive coefficient errors by architecture. Stacked bars show task-normalized group errors (Eq. 57 ); each bar sums to the model’s coefficient error. Rows correspond to $d=10,20$ and columns to $k=1,d/2,d$ . For each seed, group errors are averaged equally over the same 32,000 test datasets. Bars show means over three seeds; points mark each seed’s mean active and total coefficient errors at the corresponding stack boundaries. At $k=d$ , the inactive contribution is zero. Vertical ranges are shared across dimensions within each column.

Panel of Figure 26 — Figure 26: Coefficient errors per coordinate under sparse priors. Each model’s task-normalized active and inactive errors (Eq. 57 ) are divided by $k$ and $d-k$ , respectively. Rows correspond to $d=10,20$ and columns to $k=1,d/2$ . Colors match Figure 25 . Per-coordinate errors are averaged equally over the same 32,000 test datasets for each seed. Diamonds show means over three seeds, circles show individual seed averages, and error bars show seed SDs (ddof=0). All panels share a logarithmic vertical axis.

Figure 27: OLS and learned-model coefficient errors relative to Bayes. Rows correspond to

d=10,20

and columns to

k=1,d/2,d

. Bars stack task-normalized active and inactive coefficient errors; annotations give their sum. Every bar uses the same 32,000 test datasets per condition. OLS is deterministic given the context; learned-model bars average three training seeds and reproduce Figure 25 . At

k=d

, the inactive sum is zero. Linear vertical ranges are shared within each column.

Figure 28: OLS coefficients and posterior-mean coefficients. Each panel plots all coordinates from 150 fixed, randomly sampled test tasks. Dark red circles denote active coordinates and dark gray crosses denote inactive coordinates in the realized support. Dashed lines indicate equal coefficients. At $k=d$ , variation in context length produces different shrinkage slopes across tasks. Error summaries use the complete 32,000-task test set.

Panel of Figure 29 — Figure 29: Fidelity of coefficient changes to prediction changes. All layers, $k=1$ , $\delta=0.1$ , joint context-and-query attenuation. Panels compare active and inactive sources within (a) the alternating-axis model and (b) frozen TabPFN v2, each using the same 128 tasks per dimension. Colors distinguish $d=10$ and $d=20$ . Solid and dashed lines show active- and inactive-source medians of within-context role-averaged $\widehat{R}^{2}_{\Delta}$ , respectively, with equal context weights. Bands use the pointwise 95% bootstrap procedure in Appendix F.1 . Higher values indicate better fidelity ( $\uparrow$ ); the maximum is one, and values can be negative.

Panel of Figure 30 — Figure 30: Coefficient responses to context-message attenuation. Alternating-axis model, $d=10$ , $k=1$ , $\delta=0.1$ ; panels show layers 1–3. Each matrix column attenuates one source on context rows. The task, matrix axes, per-layer color scales, and gold outlines match Figure 7 , allowing direct comparison with query interventions.

Panel of Figure 31 — Figure 31: Coefficient responses within a $\bm{d=20}$ task. Alternating-axis model, $k=1$ , $\delta=0.1$ . Top: context interventions; bottom: query interventions. Columns show layers 1–3. The same task is used throughout; context and query panels share a symmetric color scale within each layer. Matrix axes and gold outlines follow Figure 7 .

Panel of Figure 32 — Figure 32: Predictive consequences of context- and query-message attenuation. Panel (c): alternating-axis model; panel (f): frozen TabPFN v2. $d=10$ , $k=1$ , $\delta=0.1$ ; the same 128 tasks as in Figure 8 . Curves show median prediction-error changes after averaging sources within each role and task; bands are pointwise 95% generator-batch bootstrap intervals. Panels (c) and (f) complete Figure 8 , whose panels (a, b, d, e) appear in the main text.

Panel of Figure 33 — Figure 33: Context and query message effects at $\bm{d=20}$ . $k=1$ , $\delta=0.1$ , 128 shared tasks. Top: alternating-axis model; bottom: frozen TabPFN v2. Layout, source-role summaries, and confidence intervals follow Figure 8 .

Panel of Figure 34 — Figure 34: Direction and magnitude of active-source control. Final-layer empirical distributions of signed sensitivity $s_{g}$ , with $k=1$ , $\delta=0.1$ , and the same 128 tasks per dimension and model. Top: alternating-axis model; bottom: TabPFN v2. Left: $d=10$ ; right: $d=20$ . Colors distinguish context and query attenuation, as in Figure 8 . All tasks are shown. The horizontal axis is linear for $|s_{g}|\leq 0.05$ and logarithmic outside that interval; the dotted line marks zero.

Panel of Figure 35 — Figure 35: Attenuation-strength comparison at $d=10$ . $k=1$ , 128 tasks, joint context-and-query attenuation. Panels (a–d): alternating-axis model; (e–h): TabPFN v2. For each model, rows show sensitivity and error change; columns show active and inactive sources. Curves show task medians of role averages; bands are 95% batch-bootstrap intervals (Appendix F.1 ). Dose comparisons share bootstrap draws. Sensitivity axes match across dimensions; error axes are panel-specific.

Panel of Figure 36 — Figure 36: Attenuation-strength comparison at $d=20$ . $k=1$ , 128 tasks, joint context-and-query attenuation. Panels (a–d): alternating-axis model; (e–h): TabPFN v2. For each model, rows show sensitivity and error change; columns show active and inactive sources. Curves show task medians of role averages; bands are 95% batch-bootstrap intervals (Appendix F.1 ). Dose comparisons share bootstrap draws. Sensitivity axes match across dimensions; error axes are panel-specific.

Panel of Figure 37 — Figure 37: Coefficient control across support sizes. Panels (a–f): alternating-axis model; (g–l): frozen TabPFN v2. Within each model, rows show $k=d/2$ and $k=d$ ; columns show specificity, sensitivity, and prediction-error change. Joint attenuation, $\delta=0.1$ , 128 shared tasks per dimension and prior. Colors distinguish dimensions; solid and dashed curves denote active and inactive sources. All sources are active at $k=d$ . Bands are pointwise 95% batch-bootstrap intervals. TabPFN v2 sources are native two-feature groups.

Symbol	Meaning
$D=(X,y)$ ; $n,d$	Context, context size, and feature dimension; $X\in\mathbb{R}^{n\times d}$ , $y\in\mathbb{R}^{n}$ .
$x\in\mathbb{R}^{d}$ ; $x_{i}^{\top}$	Independent query vector; row $i$ of the context design. Within a query, $x_{j}$ denotes coordinate $j$ .
$\beta$ ; $S$ ; $k$ ; $P_{k}$	Data-generating regression coefficients, their active support, its size, and the corresponding coefficient prior.
$v^{2}$ ; $\sigma^{2}$ ; $\tau^{2}$	Total prior signal energy, response-noise variance, and $\sigma^{2}/n$ .
$c=X^{\top}y/n$	Sufficient statistic for the coefficient posterior.
$\pi_{k}(A\mid c)$ ; $q_{k,j}(c)$	Posterior probability of support $A$ ; posterior inclusion probability of coordinate $j$ .

Appendix

Table 8: Notation for the controlled model, affine projection, and interventions.

Tabular Foundation Models

Tabular foundation models learn reusable representations or in-context prediction skills for structured data such as spreadsheets and databases.

See all 188 papers

Oct 18, 2026

FTFM: Factorized Tabular Foundation Models

Tabular foundation models (TFMs) are becoming a powerful alternative to conventional tabular learning, adapting to new datasets in context without task-specific training. Current architectures, however, face a trade-off: models that preserve cell-level representations are expensive because they repeatedly attend across rows and columns, while faster models collapse each row into a fixed embedding and lose this fine-grained state. We introduce Factorized Tabular Foundation Models (FTFM), which preserve cell representations while factorizing their computation through one contextual latent per row and one per column. Cells no longer attend to one another directly; instead, each cell is updated through the interaction of its row and column latents, reducing repeated attention while retaining feature-specific information. FTFM achieves competitive predictive performance in tabular classification on major benchmarks such as TabArena and BeyondArena.

Josif Grabocka

University of Technology Nuremberg

Tabular Foundation Models Tabular ICL Tabular Classification Transformer Attention

Ask PDF

Oct 9, 2026

Tabular Foundation Models Are Effectively Shallow

In-context-learning Tabular Foundation Models (TFMs) are uniquely suited to scenarios with limited resources or privacy constraints, as they solve new tasks without additional training. However, their architecture consists of multiple transformer blocks, resulting in high computational and memory footprints. Meanwhile, traditional alternatives like Gradient-Boosted Decision Trees (GBDTs) remain highly competitive and run efficiently on CPUs, suggesting that tabular tasks may not inherently require deep architectures. This contrast raises a fundamental question: can TFMs be simplified by removing architectural redundancies without sacrificing performance? To answer this, we introduce a closed-form, label-free affine translator that replaces a contiguous window of up to 96% of the blocks, leaving the rest unchanged. Evaluating eight frontier TFMs on 171 classification datasets, we reveal that tabular transformers are vastly overparameterized: replacing half of a model's blocks costs at most

0.002

median AUROC across all architectures, with the half-depth TabFM still outperforming every other full-depth TFMs and tuned GBDTs. Furthermore, different model families are overparameterized in different ways: TabPFN-3 relies on its encoders, maintaining performance regardless of which single block is retained. Other single-axis models spread computation thinly, needing only their first block, while dual-axis models concentrate it in one critical block. Ultimately, removing depth saves up to 1.5B parameters, speeds up GPU inference by up to

6\times

at a median cost of

0.03

AUROC, while a single-block TabPFN-3 serves cached queries

2.4\times

faster using

6\times

less memory at negligible cost.

Irene Cannistraci, Julia E. Vogt

Department of Computer Science ETH Zurich Switzerland

Transformer Pruning Tabular Foundation Models Tabular ICL Model Compression Efficient Transformer Inference Neural Network Pruning

Ask PDF

Jun 3, 2026

LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models

Tabular foundation models (TFMs) increasingly rival tree ensembles, but their performance is often compute-inefficient: with standard affine scalar tokenization, each feature injects value variation through an essentially one-dimensional channel, and feature IDs/positional signals cannot increase within-feature value degrees of freedom, yielding weak early-layer value sensitivity and redundant hidden states. We present a unified tokenize-and-route framework for strong TFMs: RaBEL expands each scalar into compact localized RBF features (optionally exponent-gated) to improve conditioning and shallow-layer effective rank, while a reordered bidirectional block S->N->F aligns computation with the readout by aggregating cross-sample context before feature mixing and using attention pooling. Together, these changes yield LimiX-2M, a 2M-parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used tabular benchmarks while reducing training and inference costs. These results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs. Model checkpoints and inference code are available at https://github.com/limix-ldm-ai/LimiX.

Yuanrui Wang, Xingxuan Zhang, Han Yu +7

Stable AI · Tsinghua University

Tabular Foundation Models Tabular Representation Learning

Ask PDF

Abstract

Tabular Foundation Models

Explore similar work

FTFM: Factorized Tabular Foundation Models

Tabular Foundation Models Are Effectively Shallow

LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models