Tabular Foundation Models Are Effectively Shallow

Irene Cannistraci; Julia E. Vogt

cs.LGOct 9, 2026

Tabular Foundation Models Are Effectively Shallow

Authors: Irene Cannistraci, Julia E. Vogt

Organizations: Department of Computer Science ETH Zurich Switzerland

Abstract

In-context-learning Tabular Foundation Models (TFMs) are uniquely suited to scenarios with limited resources or privacy constraints, as they solve new tasks without additional training. However, their architecture consists of multiple transformer blocks, resulting in high computational and memory footprints. Meanwhile, traditional alternatives like Gradient-Boosted Decision Trees (GBDTs) remain highly competitive and run efficiently on CPUs, suggesting that tabular tasks may not inherently require deep architectures. This contrast raises a fundamental question: can TFMs be simplified by removing architectural redundancies without sacrificing performance? To answer this, we introduce a closed-form, label-free affine translator that replaces a contiguous window of up to 96% of the blocks, leaving the rest unchanged. Evaluating eight frontier TFMs on 171 classification datasets, we reveal that tabular transformers are vastly overparameterized: replacing half of a model's blocks costs at most $0.002$ median AUROC across all architectures, with the half-depth TabFM still outperforming every other full-depth TFMs and tuned GBDTs. Furthermore, different model families are overparameterized in different ways: TabPFN-3 relies on its encoders, maintaining performance regardless of which single block is retained. Other single-axis models spread computation thinly, needing only their first block, while dual-axis models concentrate it in one critical block. Ultimately, removing depth saves up to 1.5B parameters, speeds up GPU inference by up to $6\times$ at a median cost of $0.03$ AUROC, while a single-block TabPFN-3 serves cached queries $2.4\times$ faster using $6\times$ less memory at negligible cost.

Figures & tables

Figure 1: TabPFN-3 at different levels of compression. Change in the classification AUROC using MIMIC-III vs. (left) % of replaced blocks, (middle) cached-query GPU speed-up, and (right) serving-memory reduction. Keeping only one transformer block (96% of blocks replaced) costs

-0.009

AUROC for a

2.4\times

speed-up and

6.1\times

less memory.

Figure 2: Pairwise Cosine Similarity. Layerwise cosine similarity between all block representations using MIMIC-III across different TFMs . Except for TabFM, models show architecture-specific structure that does not reliably predict which blocks are replaceable.

Figure 3: Per-block Attention Entropy. Normalized attention entropy (percentage of maximum

\ln S

) for three representative TFMs (on MIMIC-III and Higgs) vs. Qwen3-4B. While tabular models attend near-uniformly to all tokens, Qwen3-4B displays more peaked attention.

Figure 4: Per-block Sensitivity.

\Delta

AUROC when a single block is replaced by the translator. Mean across 8 clinical datasets and 3 seeds. Full results in Figure 11 .

Figure 5: Classification Performance at Varying Compression Levels. Per-dataset

\Delta

AUROC (compressed

-

full) over 163 datasets at half-depth (left), reduced to a single block using our translator (center), and a single-block baseline using an identity mapping (right). Additional results in Tables 9 and 10 .

Figure 6: Regression Performance at Varying Compression Levels. Per-dataset

\Delta R^{2}

(compressed

-

full) over 41 regression datasets at half-depth (left), reduced to a single block using our translator (center), and a single-block baseline using an identity mapping (right).

	Blocks params (M)			$\Delta$ FLOPs		Re-encode speed-up		Cached query speed-up		Cached memory reduction
Model	Full	Half	One-block	Half	One-block	Half	One-block	Half	One-block	Half	One-block
TabPFNv2	7.1	5.0	2.0	$-$ 50%	$-$ 91%	1.7 $\times$	4.6 $\times$	1.9 $\times$	5.1 $\times$	1.9 $\times$	10.0 $\times$
TabPFN-3	51.4	25.9	2.4	$-$ 26%	$-$ 51%	1.1 $\times$	1.2 $\times$	1.5 $\times$	2.4 $\times$	1.8 $\times$	6.1 $\times$
TabICL	25.7	13.1	2.4	$-$ 23%	$-$ 42%	1.0 $\times$	1.1 $\times$	1.7 $\times$	4.0 $\times$	1.7 $\times$	4.3 $\times$
TabDPT	75.6	38.4	5.3	$-$ 49%	$-$ 92%	1.5 $\times$	3.3 $\times$	—	—	—	—
TabFM	1,611	810	71.3	$-$ 31%	$-$ 59%	1.2 $\times$	1.4 $\times$	—	—	—	—

Table 1: Hardware Impact. MIMIC-III, single RTX 5090. Blocks params: parameters of the transformer blocks kept, translator included.

\Delta

FLOPs: forward-pass FLOPs removed. Re-encode: batch of 512 queries, context re-encoded on every call. Cached: one query against a cached context (three models); memory = weights + cache.

Figure 7: Pre-training Redundancies. nanoTabPFN test AUROC, averaged over four clinical datasets. Compressing 12 blocks to

k

matches native training at depth

k

, but only when the translator is fitted on the target’s (unlabelled) rows rather than on prior tasks (right). Exact values in Table 25 .

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark suites (classification)
Name	#Datasets	#Instances	#Features	#Classes
OpenML-CC18 ( Bischl et al., 2021 )	56	500–96K	4–3,072	2–46
TabZilla ( McElfresh et al., 2023 )	44	148–1.0M	4–4,296	2–100
Grinsztajn ( Grinsztajn et al., 2022 )	25	3.4K–940K	7–419	2
TabArena ( Erickson et al., 2025 )	38	748–150K	4–1,776	2–8
Clinical (classification)

Appendix

Table 2: Dataset summary. Suite rows aggregate benchmarks deduplicated by OpenML ID, each dataset counted once under one suite; clinical rows list each dataset individually.

Model	Tokenisation	Encoder	Params (M)	Blocks $\times$ $d$	Params (share)	Cached mode	Context
		Encoders (never replaced)		Transformer stack (probed)
TabPFNv2	cell, dual-axis	linear ( $x$ and $y$ )	0.01	12 $\times$ 192	7.1 M (98%)	yes	3,000
TabPFN-3	row	feature-distribution embedder + column aggregator	1.3	24 $\times$ 512	51.4 M (97%)	yes	3,000
TabICL	row	set transformer per column + 3-layer row transformer	1.3	12 $\times$ 512	25.7 M (93%)	yes	3,000
TabDPT	row	linear	0.08	16 $\times$ 768	75.6 M (98%)	no (retrieval per query)	1,500
TabFM	row	cell embedder + 2 column set transformers + 2 row interactors	19.5	24 $\times$ 2048	1,611 M (98%)	no	3,000

Appendix

Table 3: Architecture of the eight evaluated models. The translator replaces blocks of the transformer stack only; encoders and head run unchanged. Params are the encoder and stack parameters (stack share of total), while context is the support-set cap in our experiments, not the model’s limit.

Figure 8: Calibration budget:

\Delta

AUROC vs. number of fitting positions

N

(seed 0, five clinical datasets, three draws, KL-selected windows). Solid: width 3; dotted: one block kept. Mild compression needs at most 100 positions; with one block kept the knee moves to 250–500.

Dataset	columns	tokens	params (M)	per-feat.	shared	per-feat.	shared
			translator	width 4		one block kept
cc18_1485	500	252	9.3	-0.018	-0.152	-0.321	-0.427
grin_45019	419	211	7.8	+0.003	-0.047	-0.080	-0.289
philippine	308	156	5.8	-0.009	-0.091	-0.154	-0.261
cc18_1501	256	184	6.8	-0.001	-0.021	-0.085	-0.416
cc18_40979	240	302	11.2	+0.000	-0.007	-0.056	-0.408

Appendix

Table 4: Per-feature vs. shared translator for TabPFNv2 on 14 suite datasets.

Figure 9: Per-block attention entropy as a percentage of the maximum possible entropy (

\ln S

) for all six tabular TFMs on MIMIC-III (solid) and Higgs (dotted) vs. Qwen3-4B (dashed). All models maintain near-uniform attention (70–99% outside TabFM’s read-out block) compared to Qwen3-4B’s sharply peaked attention (16–62%). TabFM’s read-out block drops to 2–8%.

Figure 10: TabICL per-block attention entropy before vs. after replacing blocks 4–8 with the affine translator (shaded). Surviving blocks do not reorganise their attention to compensate.

Dataset	Skip Window	$\varepsilon$	Mean $\|\Delta\mathcal{H}\|$	Max $\|\Delta\mathcal{H}\|$	Pre $\overline{\mathcal{H}}$	Post $\overline{\mathcal{H}}$
Diabetes	4–8	.197	.019	.114	5.80	5.82
Heart	4–8	.149	.004	.017	4.90	4.90
Br. Cancer	4–8	.122	.029	.097	5.48	5.45
SUPPORT2	4–8	.246	.027	.068	6.95	6.94
Diabetes (early)	2–6	.190	.028	.110	5.75	5.76
Heart (late)	6–10	.097	.001	.005	4.89	4.89

Appendix

Table 5: TabICL attention stability: per-block entropy on surviving blocks before vs. after replacing a contiguous window.

\varepsilon

= translator residual;

|\Delta\mathcal{H}|

= absolute entropy change (nats). Shifts are

<0.03

nats on average.

Figure 11: Per-block sensitivity of every reported model:

\Delta

AUROC when a single block is replaced by the ridge translator (mean over the eight clinical datasets and three seeds).

Model	KL (ours)	Labelled	$\varepsilon$	Oracle	KL (ours)	Labelled	$\varepsilon$	Oracle	$n$
	Half: mean $\Delta$ AUROC				One-block: mean $\Delta$ AUROC
TabPFN-3	$-$ .001	$-$ .001	$-$ .003	$+$ .001	$-$ .006	$-$ .006	$-$ .007	$-$ .005	171
TabICL	$-$ .003	$-$ .003	$-$ .004	$-$ .001	$-$ .015	$-$ .015	$-$ .016	$-$ .014	171
TabDPT	$-$ .003	$-$ .004	$-$ .005	$-$ .000	$-$ .024	$-$ .025	$-$ .025	$-$ .023	171
TabFM	$-$ .004	$-$ .005	$-$ .007	$-$ .002	$-$ .025	$-$ .026	$-$ .025	$-$ .025	165
Orion-MSP	$-$ .002	$-$ .003	$-$ .002	$+$ .001	$-$ .018	$-$ .018	$-$ .018	$-$ .016	160

Appendix

Table 6: Window selection on the full sweep. Mean over datasets of the test

\Delta

AUROC (averaged over seeds) (compressed

-

full) of the window chosen by each criterion, per (model, dataset, seed): the smallest calibration KL

\mathrm{KL}(p_{\text{full}}\,\|\,p_{\text{c}})

(ours, no labels; every result in the paper), the highest calibration AUROC (labelled), the smallest translator fit residual

\varepsilon

, and the highest test AUROC (oracle, an upper bound). All suite and clinical datasets the model runs on (

n

), three seeds, ridge translators.

Model	KL-selected	Fixed: last half	Fixed: keep block 0, replace next $L/2$	Test oracle	$n$
TabPFN-3	$-$ .001 (4%)	$-$ .003 (13%)	$-$ .007 (26%)	$+$ .001	163
TabICL	$-$ .003 (10%)	$-$ .004 (17%)	$-$ .008 (27%)	$-$ .001	163
TabDPT	$-$ .003 (9%)	$-$ .006 (23%)	$-$ .009 (31%)	$-$ .000	163
TabFM	$-$ .005 (13%)	$-$ .006 (13%)	$-$ .007 (23%)	$-$ .002	157
Orion-MSP	$-$ .002 (7%)	$-$ .002 (7%)	$-$ .012 (33%)	$+$ .000	152
Orion-Bix	$-$ .001 (3%)	$-$ .001 (4%)	$-$ .005 (20%)	$+$ .004	157

Appendix

Table 7: Half depth needs no window selection for the single-axis models. Mean over the suite datasets of the

\Delta

AUROC (averaged over seeds) at Half (width

L/2

) with the window chosen by the calibration KL (no labels), with two fixed windows that need no selection at all (replace the last half of the stack; keep block 0 and replace the next

L/2

blocks), and with the test-set oracle as the bound; in parentheses the share of datasets losing more than one point. Three seeds, ridge translators. TabPFNv2 (per-feature map) needs the selection.

	All datasets (5 widths)			Clinical (8 datasets, greedy)
Model	removed	$\Delta$ AUROC	$\geq-0.01$	removed	$\Delta$ AUROC	$\geq-0.01$	greedy = min-KL
TabPFN-3	63%	-0.001	95%	70%	-0.001	100%	22%
TabICL	61%	-0.003	94%	81%	-0.002	100%	64%
TabDPT	52%	-0.003	91%	74%	-0.000	100%	46%
TabFM	53%	-0.002	95%	79%	-0.003	75%	81%
Orion-MSP	60%	-0.002	93%	73%	-0.001	100%	92%

Appendix

Table 8: Label-free depth from a drift tolerance. For each dataset and seed, the deepest compression whose calibration KL to the full model stays at most 0.01: on the 171 suite and clinical datasets among the swept widths

\{3,4,L/2,L{-}2,L{-}1\}

(KL-selected window per width), and on the eight clinical datasets along the greedy path that grows one window block by block. Removed = mean share of blocks replaced;

\Delta

AUROC = mean test cost (compressed

-

full);

\geq-0.01

= share of datasets within one point; greedy = min-KL = share of path steps where the greedy window is the lowest-KL window of that width. No label is used at any step. Three seeds.

Figure 12:

\Delta

AUROC vs. share of blocks replaced on MIMIC-III for every model, at every swept width (three seeds, KL-selected windows, ridge translator).

Model		Clinical (8)	TabArena	TabZilla	CC-18	Grinsztajn
XGBoost		.911 ±.084	.860 ±.100	.907 ±.086	.911 ±.123	.856 ±.096
LightGBM		.909 ±.086	.859 ±.100	.907 ±.086	.911 ±.124	.857 ±.097
TabPFNv2	Full	.904 ±.099	.852 ±.112	.901 ±.091	.913 ±.122	.834 ±.099
	Half (6/12)	.904 ±.097	.850 ±.113	.896 ±.093	.911 ±.122	.829 ±.099
	One-block (11/12)	.862 ±.103	.792 ±.113	.793 ±.137	.777 ±.158	.758 ±.091
	Id (11/12)	.481 ±.083	.503 ±.097	.489 ±.080	.521 ±.070	.498 ±.065

Appendix

Table 9: Classification results (AUROC). Clinical and benchmark suites, mean

\pm

std over datasets and 3 seeds. Half:

L/2

blocks replaced. One-block:

L{-}1

blocks replaced. Ridge translators; windows selected by calibration KL (no labels). Id: the One-block window with the translator replaced by the identity (naive dropping). Per-dataset clinical results in Table 10 .

Model		Diabetes	Heart	Br. Cancer	SUPPORT2	MIMIC	eICU	CTG †	Thyroid †
XGBoost		.828 ±.022	.885 ±.041	.988 ±.008	.981 ±.001	.819 ±.005	.792 ±.002	.997 ±.001	1.00 ±.000
LightGBM		.819 ±.012	.880 ±.036	.990 ±.007	.981 ±.001	.817 ±.009	.789 ±.004	.997 ±.001	1.00 ±.000
TabPFNv2	Full	.844 ±.024	.900 ±.030	.995 ±.004	.980 ±.000	.773 ±.011	.741 ±.004	.998 ±.000	1.00 ±.000
	Half (6/12)	.841 ±.027	.903 ±.036	.994 ±.003	.976 ±.001	.774 ±.014	.750 ±.003	.996 ±.001	.998 ±.001
	One-block (11/12)	.826 ±.021	.842 ±.047	.995 ±.003	.931 ±.005	.716 ±.018	.708 ±.007	.979 ±.008	.901 ±.014
	Id (11/12)	.624 ±.047	.314 ±.082	.547 ±.026	.503 ±.010	.452 ±.007	.455 ±.004	.466 ±.023	.488 ±.064

Appendix

Table 10: Clinical datasets. AUROC for tuned GBDT baselines and eight tabular foundation models. Full = pre-trained model unchanged; Half =

L/2

blocks replaced by the translator; One-block =

L{-}1

L

blocks replaced; Id = the same maximal window with the translator replaced by the identity (naive layer dropping), isolating the translator’s contribution. Translators: ridge with label-free

\lambda

selection; all skip windows selected label-free by the smallest calibration-split KL to the full model; subscripts

\pm

std over seeds, where each seed is a different train/calibration/test split (patient-disjoint on MIMIC-III and eICU) and support subsample, so the Full rows vary with the seed as well. GBDTs are tuned (30-config random search on the calibration split) and share test splits and seeds with the foundation-model rows. All entries are raw AUROC.

Method	Avg. rank	$\pm$	Best	Mean AUROC
TabFM	2.26	2.48	101	0.9017
TabFM (Half)	3.53	2.54	6	0.8971
TabICL	3.88	2.35	10	0.8912
TabICL (Half)	5.29	2.26	4	0.8883
TabPFN-3	5.54	2.58	5	0.8868
TabPFN-3 (Half)	6.26	2.64	6	0.8858

Appendix

Table 11: Average rank over the shared suite datasets (lower is better,

\pm

std; Friedman

p<10^{-100}

). Left: Full and Half versions jointly ranked. Right: Half models vs. tuned GBDTs. Half-depth TabFM outranks every other full model.

Model	Win	Tie	Loss	Median $\Delta$ AUROC
TabPFNv2	14	48	95	-0.0017
TabPFN-3	19	77	67	-0.0004
TabICL	7	70	86	-0.0011
TabDPT	13	55	95	-0.0018
TabFM	10	47	100	-0.0021
Orion-MSP	22	44	86	-0.0014

Appendix

Table 12: Per-model win/tie/loss of the compressed model (Half window, ridge) against its own full version over the suite datasets, with the median

\Delta

AUROC.

			$\leq$ 3,000 rows		$>$ 3,000 rows
Model	datasets	Pearson $r$	$n$	ours $-$ TabArena	$n$	ours $-$ TabArena
TabPFN-3	30	0.951	9	-0.004	21	-0.022
TabICL	30	0.962	9	-0.005	21	-0.018
TabDPT	30	0.935	9	-0.004	21	-0.029
TabFM	30	0.968	9	-0.007	21	-0.018
LimiX-16M	30	0.933	9	-0.012	21	-0.007

Appendix

Table 13: Our Full baselines vs. the TabArena leaderboard (default configurations, ROC AUC, 30 binary TabArena datasets): per-dataset Pearson correlation and the mean difference, split at 3,000 rows.

Binary tasks
		AUROC			Accuracy			Balanced accuracy
Model	$n$	Full	Half	One	Full	Half	One	Full	Half	One
TabPFNv2	114	.858	.856	.797	.840	.837	.771	.753	.749	.626
TabPFN-3	114	.865	.864	.860	.846	.844	.839	.761	.756	.746
TabICL	114	.868	.865	.853	.849	.846	.832	.765	.760	.733
TabDPT	114	.849	.847	.828	.833	.830	.813	.741	.734	.691

Appendix

Table 14: AUROC, accuracy and balanced accuracy at both operating points. Mean over datasets of Full / Half / One-block (KL-selected windows, ridge translator, three seeds; decisions by the most probable class, no recalibration), for the binary and the multi-class datasets (suite and clinical);

n

= datasets. At Half every metric stays close to Full; with one block kept the threshold-dependent metrics fall more than AUROC, most on multi-class tasks.

		Half							One-block
Model	Cohort	$\Delta$ AUROC	Q1	Q4	$\Delta$ sens	$\Delta$ spec	$\|\Delta p\|$ died	surv.	$\Delta$ AUROC	Q1	Q4	$\Delta$ sens	$\Delta$ spec	$\|\Delta p\|$ died	surv.
TabPFNv2	MIMIC-III	$+0.001$	$+0.000$	$+0.004$	$+0.001$	$-0.011$	0.053	0.029	$-0.057$	$-0.035$	$-0.057$	$+0.243$	$-0.637$	0.132	0.093
TabPFNv2	eICU	$+0.008$	$+0.013$	$+0.007$	$-0.048$	$+0.036$	0.058	0.028	$-0.033$	$-0.015$	$-0.048$	$-0.015$	$-0.166$	0.107	0.073
TabPFN-3	MIMIC-III	$-0.003$	$+0.002$	$-0.007$	$-0.022$	$+0.019$	0.027	0.014	$-0.009$	$-0.002$	$-0.014$	$-0.048$	$+0.042$	0.043	0.023
TabPFN-3	eICU	$+0.002$	$-0.001$	$+0.003$	$-0.020$	$+0.018$	0.022	0.010	$-0.001$	$-0.006$	$-0.001$	$-0.024$	$+0.015$	0.031	0.016
TabICL	MIMIC-III	$-0.004$	$-0.003$	$-0.004$	$+0.006$	$-0.009$	0.019	0.010	$-0.012$	$-0.005$	$-0.012$	$-0.077$	$+0.038$	0.063	0.025

Appendix

Table 15: Subgroup analysis on MIMIC-III and eICU (patient-disjoint, 3 seeds, windows selected by calibration KL). Per model at Half and One-block: overall

\Delta

AUROC;

\Delta

AUROC by feature-completeness quartile (Q1 = least complete);

\Delta

sensitivity/

\Delta

specificity at the full model’s Youden-

J

threshold; mean

|\Delta p|

by outcome group.

			early ( $\ell\leq L/3$ )		mid		late ( $\ell>2L/3$ )
Model	runs	transl. $\geq$ probe	probe	transl.	probe	transl.	probe	transl.
TabPFNv2	288	67%	-0.048	-0.082	-0.035	-0.003	-0.016	+0.000
TabPFN-3	576	77%	-0.017	-0.002	-0.015	-0.002	-0.014	-0.000
TabICL	288	95%	-0.022	-0.001	-0.024	-0.000	-0.018	+0.000
TabDPT	384	86%	-0.032	-0.003	-0.032	-0.002	-0.030	-0.000
TabFM	552	91%	-0.026	-0.004	-0.018	-0.002	-0.031	-0.000

Appendix

Table 16: Early-exit probe vs. affine translator: share of configurations where the translator matches or beats the probe, and median

\Delta

AUROC vs. the full model across three depth bands (8 clinical datasets, 3 seeds, all widths).

L

= total blocks;

\ell

= exit layer / translator start.

Model	Keeps first block (suite / clinical)	Cost, first kept	Cost, last kept	$n$
TabPFNv2	14% / 8%	$-$ .281	$-$ .071	165
TabPFN-3	43% / 92%	$-$ .007	$-$ .006	171
TabICL	86% / 79%	$-$ .016	$-$ .019	171
TabDPT	97% / 100%	$-$ .025	$-$ .049	171
TabFM	93% / 96%	$-$ .025	$-$ .036	165
Orion-MSP	92% / 79%	$-$ .017	$-$ .023	160

Appendix

Table 17: Which block is kept at One-block. Share of dataset–seed pairs whose KL-selected window keeps the first block rather than the last (suite / clinical datasets), and mean

\Delta

AUROC when the first or the last block is kept (averaged over seeds;

n

: datasets with both windows).

Model	Variant	$r=1$	$r=2$	$r=3$	$r=6$	$r=12$
TabICL	kept block $\times r$ + map	$-$ .002	$-$ .002	$-$ .002	$-$ .003	$-$ .003
	kept block $\times r$ , no map	$-$ .067	$-$ .034	$-$ .034	$-$ .044	$-$ .071
TabDPT	kept block $\times r$ + map	$-$ .005	$-$ .005	$-$ .004	$-$ .005	$-$ .006
	kept block $\times r$ , no map	$-$ .417	$-$ .409	$-$ .430	$-$ .480	$-$ .404
TabFM	kept block $\times r$ + map	$-$ .006	$-$ .014	$-$ .015	$-$ .016	$-$ .016
	kept block $\times r$ , no map	$-$ .113	$-$ .047	$-$ .051	$-$ .085	$-$ .179

Appendix

Table 18: Looping the kept block

r

times, with the translator refitted on its output (+ map) or removed (no map).

\Delta

AUROC vs. the full model, eight clinical datasets, three seeds.

Figure 13: Context size vs. compression cost:

\Delta

AUROC as the support set grows from 500 to 18,000 rows (3 seeds, shaded

\pm

std; windows re-selected at every size). Solid: one block kept; dotted: Half (

L/2

). At Half the mean cost stays within

0.009

at every size (worst dataset: TabPFNv2 on Covertype,

-0.030

); with one block kept it grows with context (TabFM:

-0.005

at 500 rows to

-0.043

at 18,000).

Figure 14: Reusing the map across sites and datasets. Median

\Delta

AUROC vs. the full model on the target for the map fitted on the target itself (

\blacklozenge

), a map fitted on another dataset and applied unchanged (

\bullet

, bar: interquartile range over pairs; for the two MIMIC-III

\leftrightarrow

eICU pairs, the range of the two), and dropping the blocks with no map (

\circ

). The closer

\bullet

lies to

\blacklozenge

, the more of the local map survives transfer. Pairs are averaged over three seeds; all three maps sit at the source’s KL-selected window. TabFM, TabDPT and a synthetic source are in Table 19 .

Model	Depth	Map	MIMIC $\leftrightarrow$ eICU	Same data	Same domain	Other domain	Synthetic
TabICL	Half	local	$-0.003$	$-0.003$	$-0.001$	$-0.001$	$-0.001$
		transferred	$-0.004$	$-0.008$	$-0.006$	$-0.006$	$-0.008$
		dropped	$-0.053$	$-0.032$	$-0.016$	$-0.018$	$-0.010$
	One-block	local	$-0.014$	$-0.020$	$-0.003$	$-0.002$	$-0.002$
		transferred	$-0.032$	$-0.038$	$-0.043$	$-0.054$	$-0.536$
		dropped	$-0.086$	$-0.091$	$-0.093$	$-0.060$	$-0.059$

Appendix

Table 19: Reusing the map across datasets, all models. Median

\Delta

AUROC vs. the full model on the target, at the source’s KL-selected window, for the map fitted on the target (local), a map fitted on another dataset (transferred) and no map (dropped), by source–target similarity. Pairs are averaged over seeds first: three seeds for TabICL and TabPFN-3, seed 0 for TabFM and TabDPT.

Model	datasets	Full $R^{2}$	best of 3–4	$L/2$	$L{-}2$	$L{-}1$
TabPFNv2 (12)	41	0.784	-0.001	-0.019	-0.176	-0.324
TabPFN-3 (24)	41	0.859	-0.002	-0.012	-0.111	-0.132
TabICL (12)	41	0.863	-0.003	-0.009	-0.065	-0.105
TabDPT (16)	41	0.769	-0.006	-0.026	-0.244	-0.401
TabFM (24)	41	0.873	-0.000	-0.001	-0.016	-0.022
LimiX-16M (12)	41	0.853	-0.002	-0.020	-0.293	-0.568

Appendix

Table 20: Regression over 41 datasets (MIMIC-III length of stay, 13 TabArena tasks, OpenML-CTR23): median full-model

R^{2}

and median

\Delta R^{2}

(compressed

-

full) at the best of widths 3–4 (chosen by calibration

R^{2}

), at half depth, and with two and one blocks kept; medians over datasets of the mean over 3 seeds.

Model	Blocks replaced	$\Delta R^{2}$	recal.	$\rho$ full $\to$ compr.	scale
TabPFNv2	4/12	-.002	-.000	.92 $\to$ .91	0.99
	11/12	-.333	-.164	.92 $\to$ .73	0.48
TabPFN-3	4/24	-.001	-.002	.93 $\to$ .93	0.98
	23/24	-.139	-.093	.93 $\to$ .86	0.72
TabICL	4/12	-.004	-.005	.93 $\to$ .92	0.98
	11/12	-.095	-.060	.93 $\to$ .86	0.71

Appendix

Table 21: Scale vs. ordering decomposition (seed 0, medians over 41 datasets). Recal. =

\Delta R^{2}

after affine recalibration on the calibration split;

\rho

= Spearman rank correlation; scale = compressed/full std ratio. Most models lose ordering, not just scale.

Model	Absorbed into	Pairs	Max $\|\Delta p\|$	Parameters
TabPFNv2	encoder	22/24	$4\times 10^{-6}$	0.22–1.45M $\to$ +5–43k
TabDPT	head	24/24	$4\times 10^{-6}$	0.59M $\to$ +0

Appendix

Table 22: Absorbing the translator at One-block, eight clinical datasets

\times

3 seeds, fp32 on an RTX 5090. Pairs: dataset–seed pairs whose KL-selected window reaches the absorbing end (Heart keeps TabPFNv2’s block 0 at two seeds). Max

|\Delta p|

: largest difference in predicted probability, with vs. without absorption. Parameters : translator size before absorption

\to

extra parameters the absorbed model carries over the one-block model (TabPFNv2: per-feature copies of the encoder’s last linear layer; range over datasets).

	Blocks params (M)			$\Delta$ FLOPs		End-to-end speed-up
Model	Full	Half	One-block	Half	One-block	Half	One-block
TabPFNv2	7.1	5.0	2.0	$-$ 50%	$-$ 91%	2.2 $\times$	12.4 $\times$
TabPFN-3	51.4	25.9	2.4	$-$ 26%	$-$ 51%	1.2 $\times$	1.4 $\times$
TabICL	25.7	13.1	2.4	$-$ 23%	$-$ 42%	1.1 $\times$	1.3 $\times$
TabDPT	75.6	38.4	5.3	$-$ 49%	$-$ 92%	2.0 $\times$	11.0 $\times$
TabFM	1,611	810	71.3	$-$ 31%	$-$ 59%	1.3 $\times$	1.8 $\times$

Appendix

Table 23: Hardware Impact on a CPU. MIMIC-III on a desktop CPU (Intel i9-14900K, fp32). Blocks params: parameters of the transformer blocks kept, translator included.

\Delta

FLOPs: forward-pass FLOPs removed. End-to-end: batch of 512 queries, context re-encoded on every call.

		Width 4		Half		One-block
Model	Stack share of time	stack	end-to-end	stack	end-to-end	stack	end-to-end
TabPFNv2	75%	1.5 $\times$	1.38 $\times$	2.0 $\times$	1.73 $\times$	11.2 $\times$	4.63 $\times$
LimiX-16M	91%	1.5 $\times$	1.43 $\times$	2.0 $\times$	1.83 $\times$	11.5 $\times$	6.10 $\times$
TabDPT	80%	1.2 $\times$	1.17 $\times$	1.7 $\times$	1.52 $\times$	6.9 $\times$	3.25 $\times$
TabFM	29%	1.2 $\times$	1.05 $\times$	2.0 $\times$	1.17 $\times$	23.1 $\times$	1.38 $\times$
TabPFN-3	19%	1.2 $\times$	1.02 $\times$	1.9 $\times$	1.09 $\times$	19.7 $\times$	1.23 $\times$

Appendix

Table 24: Stack-only vs. end-to-end speed-up on GPU (MIMIC-III, 3,000-row context (1,500 for TabDPT), one RTX 5090, batch of 512 queries, re-encode regime, median of 10 runs, CUDA synchronised at the stack boundaries). Stack: speed-up of the transformer stack alone; end-to-end: of the whole forward pass. The stack alone still gets close to

L/(\text{blocks kept})

faster, but on a GPU it is a smaller share of the forward pass than on a CPU, so less of that gain reaches the end-to-end time.

Kept blocks $k$	1	2	3	6	12
Trained at depth $k$	$.850_{\pm.009}$	$.902_{\pm.008}$	$.894_{\pm.005}$	$.902_{\pm.001}$	$.903_{\pm.003}$
Trained, compressed topology	$.852_{\pm.008}$	$.888_{\pm.008}$	$.900_{\pm.004}$	$.900_{\pm.002}$	—
Compressed $12\to k$	$.854_{\pm.012}$	$.891_{\pm.001}$	$.903_{\pm.005}$	$.902_{\pm.007}$	—
Compressed $12\to k$ , prior-fitted map	$.428_{\pm.158}$	$.550_{\pm.116}$	$.832_{\pm.091}$	$.756_{\pm.202}$	—

Appendix

Table 25: Pre-training shallow vs. compressing deep , values of Figure 7 . nanoTabPFN test AUROC, averaged over four clinical datasets; mean

\pm

std over three seeds.

Figure 15: nanoTabPFN training dynamics: loss curves, evaluation AUROC and learning-rate restarts across depths

L\in\{1,2,3,6,12\}

(three seeds each). Deeper models (

L\geq 6

) collapse to uniform predictions at the default learning rate and require an automatic restart at a quarter of the rate.

Explore similar work

May 7, 2026

Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models

Transformer-based tabular foundation models (TFMs) dominate small to medium tabular predictive benchmark tasks, yet their inference mechanisms remain largely unexplored. We present the first large-scale mechanistic study of layerwise dynamics in 6 state-of-the-art tabular in-context learning models. We explore how predictions emerge across depth, identify distinct stages of inference and reveal latent-space dynamics that differ from those of language models. Our findings indicate substantial depthwise redundancy across multiple models, suggesting iterative refinement with overlapping computations during inference stages. Guided by these insights, we design a proof-of-concept, looped single-layer model that uses only 20% of the original model's parameters while achieving comparable performance. The code is available at https://github.com/amirbalef/is_one_layer_enough.

Amir Rezaei Balef, Mykhailo Koshil, Katharina Eggensperger

TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany · University of Tübingen, Tübingen, Germany

Tabular Foundation Models Transformer Interpretability Tabular ICL

Ask PDF

Oct 14, 2026

When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models

Transformer-based tabular foundation models (TFMs) achieve state-of-the-art performance on small to medium-sized tabular prediction tasks while processing diverse inputs through a shared stack of layers. Recent layerwise analyses identify strong predictive performance in early TFM representations and substantial overlap in computation across layers. However, early predictive accessibility alone does not establish whether later layers are dispensable for every input, and aggregate accuracy can obscure which inputs benefit from additional computation. We study all 24 in-context-learning layers of TabPFN-3 across 272 classification datasets, combining representation geometry, prediction trajectories, truncation, and structural interventions, while also examining related representation dynamics in four additional TFMs. We uncover a pronounced expansion-compression trajectory, with stronger late dimensional contraction associated with larger increases in class-prototype separation. A targeted analysis of early-stabilizing samples shows that substantial representational changes persist after predicted classes stabilize, while locally ambiguous queries obtain larger accuracy gains from shallow to late layers than well-separated queries. Repeating a single pretrained block or perturbing layer order reduces predictive performance on average, constraining simple post-hoc substitutions in the frozen model. These findings distinguish predictive accessibility, class stability, and the benefits of continuing computation, supporting an input-dependent assessment of depthwise redundancy in TFMs.

Wenwen Gong, Haotian Gu, Zhen Yang +6

China Agricultural University · Tsinghua University · Nankai University

Tabular Foundation Models Transformer Inference Tabular ICL Tabular Representation Learning Transformer Interpretability

Ask PDF

Jul 30, 2026

Memory Efficient Tabular Foundation Models

Tabular Foundation Models, such as TabPFN, have received a large amount of recent attention due to their performance on in-context tabular machine learning tasks, which often exceeds classical baselines. However, practical deployment considerations of these models has received less attention. In this paper we investigate the memory requirements for these models. We demonstrate that employing model compression approaches can enable memory reductions of up to 7.6 with similar levels of performance, reducing deployment requirements by nearly 87%. Our work provides insight to practitioners seeking efficient deployment of these models in practical settings.

Shuting Luo, Monika Mikhail Kanaan, Cameron Gordon +2

Commonwealth Bank of Australia, Australia · Australian Institute for Machine Learning, University of Adelaide, Adelaide, Australia

Model Compression Tabular Foundation Models Memory-Efficient Inference Tabular ICL

Ask PDF