When Repeated Measurements Hurt Tabular Foundation Models

Yixiang Liu; Ruizhe Deng; Bibhas Chakraborty; Qiong Zhang; Yan Shuo Tan

stat.MLOct 12, 2026

When Repeated Measurements Hurt Tabular Foundation Models

Authors: Yixiang Liu, Ruizhe Deng, Bibhas Chakraborty, Qiong Zhang, Yan Shuo Tan

Organizations: National University of Singapore · Duke-NUS Medical School · Mohamed bin Zayed University of Artificial Intelligence

Abstract

Tabular foundation models (TFMs) achieve state-of-the-art performance on IID benchmarks, but how do they respond to repeated measurements at the same inputs? Across six TFMs, synthetic and semi-synthetic experiments show that additional measurements can improve predictions at measured inputs while degrading them elsewhere. Harm is most frequent at low noise, and irrelevant features amplify it in several models; real repeated-measurement datasets show similar patterns. Statistically, the IID rule targeted by PFN pretraining treats each repeated input as fresh evidence about the input distribution. When the prior links this distribution to the response function, the resulting protocol mismatch can outweigh the benefit of additional measurements, even under exact inference. Computationally, interventions identify row-count dependence in TabPFN-v3's attention scaling as a contributor to harm. Resetting the count largely reverses distortion in a noiseless linear diagnostic and attenuates amplification by irrelevant features. Finally, we evaluate ways to retain the benefits of repeated measurements while limiting their harm. Averaging measurements usually outperforms supplying them separately at larger repetition counts, whereas count reset can suppress both harm and useful gains. Code, saved results and reproduction instructions are available at https://github.com/drizmiz/PFN-Repeated-Measurement-Supp

Figures & tables

Figure 1: More readings, worse predictions between the sites. Top: TabPFN-v3 on one design (quadratic, twelve sites, noise SD

0.10

) with one reading per site (blue) and

64

per site as rows (red); dots are the site means. Bottom: median over twenty seeds of each seed’s MSE

(r)/

MSE

(1)

for the six TFMs and the four conventional learners. Raw errors and all six TFMs’ fits are in Appendix Table 12 and Figure 6 .

Figure 2: Fraction of the twenty paired seeds, pooled over functions or covariate datasets, in which Expanded at $r=64$ has higher fresh-input MSE than Single (TFMs above the line, conventional learners below; RF: random forest, GBDT: gradient boosting). Synthetic tasks are scored on a grid (one dimension) or fresh draws, semi-synthetic tasks on held-out rows. The irrelevant-features panel fixes a function of two coordinates and adds ignored ones. Real data: twenty splits at the largest count $R$ , scored against held-out site means and ordered by within-site SD ratio (Section 3.3 ).

Figure 3: Full count reset on a fixed table of readings: TabPFN-v3, quadratic target,

12

sites,

64

independent readings, SD

0.05

, one ensemble member,

201

shared queries. (a) Prediction minus truth; zero means correct prediction and grey ticks mark sites. (b) Site-pooled

n_{\mathrm{eff}}

minus Single , after the median over heads/ICL operations at each query; zero matches Single ’s concentration. Single ’s query median is

11.06

(uniform:

12

). Full count reset sets all count inputs to

12

noise SD	Single	Expanded	Mean	full reset	Mean wins
0	0.0088	0.053	0.0088	0.0084	36
0.05	0.015	0.049	0.0094	0.0135	61
0.15	0.039	0.045	0.010	0.032	90
0.30	0.104	0.047	0.013	0.076	107
0.50	0.194	0.052	0.017	0.174	115

Table 1: Count reset against averaging, TabPFN-v3,

r=64

: median MSE at fresh inputs over the six one-dimensional functions and twenty seeds (

m=12

, eight ensemble members), and cells (of

120

) in which Mean beats full count reset. At SD

0

, Mean equals Single .

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Count thresholds within a specified unknown-noise Bayesian model. Markers show the IID-posterior risk minus Single risk, computed from the integrated reading benefit and cue cost for the nonlinear query model with

m=2

and fixed cue strength

t=.02

. Dashed curves show Equation ( 17 ). Colors vary the population mean variance

\nu

and the corresponding prior used by the predictor. Error bars are pointwise

1.96

Monte Carlo standard errors of the estimated benefit; cue costs are integrated deterministically. The plot verifies this reference calculation and does not report TFM predictions.

Figure 5: Coverage limit within the unknown-variance region model. Direct paired IID-minus- Single risks at population mean variance

\nu=.003

are plotted against

u^{2}(1-1/K)^{m}\delta_{r}(t)

, with

u=1

and

t=.25

. All combinations

K=2,4,8

m=2,4

, and

r=4,16

are included. The dashed line is equality; bars are pointwise

1.96

Monte Carlo standard errors over 8,192 tasks. Each predictor integrates the same

\operatorname{InvGamma}(4,3\nu)

prior within its condition.

dataset	a reading is	$d$	sites	$R$	context sites
SGEMM kernel timings	one run of a kernel configuration	14	2,994	4	150
Kernel Tuner convolution	one launch of a kernel configuration	7	3,000	16	150
Kernel Tuner dedispersion	one launch of a kernel configuration	6	3,000	32	150
FCNet slice localization	one training run with its own random seed	9	3,000	4	150
Bayes-factor surface	one Monte Carlo replicate	2	80	4	40
G. aparine bioassay	one pot at a dose and formulation	2	21	8	10

Appendix

Table 2: Real datasets with repeated observations, after retaining sites with at least two finite, non-identical responses at one covariate vector.

d

counts covariates that vary;

R

is the largest count used; the context holds half of the sites that have at least

R

readings, at most

150

	equal counts		unequal counts
learner	weighted	unweighted	weighted	unweighted
least squares	440 (440)	440 (440)	440 (440)	40 (40)
kernel ridge, fixed kernel	440 (440)	0 (0)	440 (440)	0 (0)
GP, fixed kernel and noise	440 (440)	0 (0)	440 (440)	0 (0)
regression tree	241 (440)	241 (440)	240 (440)	240 (438)
gradient boosting	240 (440)	240 (440)	240 (440)	0 (0)

Appendix

Table 3: Proposition 1 in practice. Fits (of

440

: six one-dimensional functions and five real-covariate sets, twenty seeds, two count configurations) in which the prediction from the means equals the prediction from the rows to

10^{-6}

at every query, and in parentheses at the sites only. Equal counts are

r=4

and

64

; unequal counts are the grid’s random and clustered schemes. Weighted: count-weighted means

(\bar{\mathcal{D}},\mathbf{r})

(for the Gaussian processes, each mean’s noise variance divided by its count).

	setting	Single	Expanded	Mean	full reset	embedding only	mean better
1-D, $r=4$	noise 0	0.0088	0.028	0.0088	0.0089	0.021	36
	noise 0.05	0.015	0.021	0.016	0.015	0.018	59
	noise 0.15	0.039	0.028	0.031	0.028	0.029	64
	noise 0.30	0.104	0.047	0.050	0.069	0.040	77
	noise 0.50	0.194	0.084	0.082	0.176	0.076	94
1-D, $r=64$	noise 0	0.0088	0.053	0.0088	0.0084	0.038	36

Appendix

Table 4: Count reset in TabPFN-v3: median error at fresh inputs of Single , Expanded , Mean , full count reset (all

27

count arguments set to

m

) and embedding-only reset (the three embedding modules), and cells in which Mean beats full count reset. One-dimensional synthetic: six functions,

120

cells per row; dimension axis: three families,

m=24

60

cells; semi-synthetic: five datasets,

100

cells; real data: twenty splits at the largest count.

$d$	Expanded / Single	full/ Single	full wins
2	1.600	1.045	15/20
5	3.100	1.051	19/20
10	3.146	1.151	20/20

Appendix

Table 5: TabPFN-v3 with irrelevant features: median paired fresh-input MSE ratios to Single , and cells in which full count reset beats Expanded . Each dimension has twenty seed cells; all arms share the sites, readings and queries within a cell.

	Single		Expanded		Mean
model	bias 2	var	bias 2	var	bias 2	var
TabPFN-v3	0.023	0.013	0.041	0.0005	0.007	0.0005
TabPFN-v3.5	0.100	0.016	0.240	0.0013	0.035	0.0006
TabICLv2	0.072	0.018	0.107	0.0004	0.063	0.0005
TabDPT	0.082	0.015	0.064	0.0006	0.036	0.0004
LimiX-2	0.041	0.014	0.200	0.0037	0.027	0.0013

Appendix

Table 6: What averaging removes. Squared bias and variance of the prediction at fresh inputs, estimated from eight independent banks of

64

readings at the same sites; median over

60

cells (six one-dimensional functions times ten site seeds,

m=12

, noise SD

0.15

		Single		Expanded		Mean
designs	model	bias 2	var	bias 2	var	bias 2	var
1-D, noise 0.5	TabPFN-v3	0.112	0.110	0.043	0.0033	0.012	0.0036
	TabPFN-v3.5	0.273	0.073	0.278	0.0028	0.049	0.0040
	TabICLv2	0.121	0.126	0.147	0.0031	0.062	0.0045
	TabDPT	0.273	0.098	0.138	0.0033	0.047	0.0032
	LimiX-2	0.118	0.093	0.318	0.0073	0.029	0.0046

Appendix

Table 7: Squared bias and variance at fresh inputs over eight independent banks of

64

readings at the same sites, finite-bank corrected; median over cells (one-dimensional: six functions times ten site seeds, noise SD

0.50

; semi-synthetic: five datasets times ten site seeds, noise multiplier

0.25

One-dimensional
	Coverage				CRPS: MR/comparator
model	Single	Exp.	Mean	MR	Exp.	Single	Mean
TabPFN-v3	0.87	0.91	0.43	0.92	0.926	0.810	0.906
TabPFN-v3.5	0.96	0.99	0.68	0.94	0.684	0.808	0.983
TabICLv2	0.67	0.98	0.57	0.92	0.819	0.835	0.948
TabDPT	0.93	0.90	0.65	0.96	0.966	0.853	0.988

Appendix

Table 8: Future-reading prediction at fresh inputs,

r=64

, under Gaussian, homoscedastic reading noise. Coverage entries are median probabilities for nominal

90\%

intervals; CRPS entries are median paired ratios of Gaussian MR to the named comparator, using the

19

-quantile approximation (lower is better). Each model has

120

function–seed and

100

real-covariate dataset–seed pairs, with

m=12

sites, noise SD

0.15

(synthetic) or residual-SD multiplier

0.25

(semi-synthetic), and

400

300

fresh queries, respectively. MR is defined in Equation ( 5 ); Single uses

r=1

. Exp. denotes Expanded . LimiX-2 and TabFM return point predictions only.

model	scheme	mean	copies	Expanded	worse	mean	copies	Expanded	worse
		one-dimensional				semi-synthetic
TabPFN-v3	random	0.027	0.054	0.040	89	0.444	0.490	0.489	65
	clustered	0.026	0.083	0.039	97	0.429	0.491	0.501	69
TabPFN-v3.5	random	0.078	0.169	0.163	83	0.473	0.491	0.524	61
	clustered	0.070	0.182	0.128	90	0.473	0.482	0.492	54
LimiX-2	random	0.054	0.156	0.096	108	0.410	0.477	0.483	66

Appendix

Table 9: The count encoding under unequal counts (the grid’s random and clustered schemes): median error at fresh inputs of Mean , Mean with the row of site

i

copied

r_{i}

times, and Expanded , and cells (of

120

one-dimensional,

100

semi-synthetic) in which the copies are worse than Mean .

	$n_{\mathrm{eff}}$					fresh-input MSE
noise SD	Single	Expanded	full	Exp. $<$ Single	full $>$ Exp.	Single	Expanded	full
0.05	10.91	10.56	11.12	112/120	120/120	0.017	0.051	0.010
0.15	10.89	10.70	11.19	94/120	120/120	0.042	0.051	0.029

Appendix

Table 10: Site-pooled attention in TabPFN-v3 (one ensemble member,

r=64

, six functions, twenty seeds): median

n_{\mathrm{eff}}

over

120

cells (each the median over

400

fresh queries of the median jointly over heads and layers; uniform attention gives

12

), cells in which it falls with the rows and rises under full count reset, and the error of the same fits. Full count reset’s

n_{\mathrm{eff}}

exceeds Single ’s in

114/120

cells at SD

0.05

and

120/120

at SD

0.15

	$n_{\mathrm{eff}}\downarrow$		$n_{\mathrm{eff}}\not\downarrow$
noise SD	harm	gain	harm	gain
0.05	74	38	6	2
0.15	43	51	11	15

Appendix

Table 11: Attention concentration and prediction error in the same one-member fits as Table 10 : counts over

120

function–seed cells per noise level. A down-arrow denotes Expanded ’s site-pooled

n_{\mathrm{eff}}

below Single ; harm denotes Expanded ’s fresh-input MSE above Single . Cells summarize

400

fresh queries each.

Figure 6: Fitted functions on the left design of Figure 1 (quadratic, twelve sites, noise SD

0.10

) with one reading per site (blue) and

64

readings per site entered as rows (orange); the numbers in each panel are the two errors. The points are the site means at

r=64

, which every model passes through; the changes are between them.

learner	MSE $(1)$	MSE $(256)$	worse
Gaussian process	0.014	0.0002	0/20
Gradient boosting	0.172	0.158	6/20
Random forest	0.182	0.158	4/20
MLP	0.043	0.018	1/20
TabPFN-v3	0.017	0.070	18/20
TabPFN-v3.5	0.062	0.379	20/20

Appendix

Table 12: Raw MSE at

r=1

and

r=256

on the design of Figure 1 (noise SD

0.10

), for all ten learners: median over

20

seeds, with the number of seeds in which more readings made the error worse.

Figure 7: Error at the measured sites (blue) and at fresh inputs (orange) with one reading and with

64

readings as rows; one line per synthetic function,

m=12

\sigma=0.15

, medians over twenty seeds. At the sites the error of every model falls by a factor of

35

or more.

family	LimiX-2	TabICLv2	TabPFN-v3.5	TabPFN-v3	TabFM	TabDPT	forest	GP (rows)	MLP	boosting
1-D (60)	39	26	35	14	8	4	10	0	0	0
Real covariates (45)	25	34	24	30	33	17	34	19	8	6
Multivariate (9)	9	9	4	8	7	4	9	3	0	0
All (114)	73	69	63	52	48	25	53	22	8	6

Appendix

Table 13: Designs in which Expanded at

r=64

is worse than Single at fresh inputs in at least

11

20

seeds. Parentheses give the number of tested designs in each family; designs sharing a function or seed are not independent replications.

scored at	TabPFN-v3	TabPFN-v3.5	LimiX-2	TabICLv2	TabDPT	TabFM	forest
measured sites	0	0	0	0	0	0	0
gap midpoints	37	78	91	59	16	30	111
fresh inputs	50	93	102	70	29	22	57

Appendix

Table 14: Query location. Function–seed pairs (of

120

: six one-dimensional functions, twenty seeds, default design) in which Expanded at

r=64

is worse than Single , by where the prediction is scored.

model	6	12	24	48	96	even	random
	sites $m$					spacing
TabPFN-v3	5	16	8	1	0	0.23	0.42
TabPFN-v3.5	6	18	11	9	7	0.78	0.78
LimiX-2	11	20	20	10	3	0.88	0.85
TabICLv2	9	12	6	5	2	0.32	0.58
TabDPT	1	9	8	5	2	0.07	0.24

Appendix

Table 15: Left: quadratic,

r=64

, seeds (of

20

) in which Expanded is worse than Single , by the number of sites. Right: share of function–seed pairs harmed with evenly spaced and with random sites (

m=12

, six one-dimensional functions).

model	$d$	Single	Expanded	Mean	ratio	worse
TabPFN-v3	2 / 5 / 10	0.051 / 0.069 / 0.084	0.076 / 0.204 / 0.259	0.012 / 0.022 / 0.026	1.60 / 3.10 / 3.15	15 / 18 / 19
TabPFN-v3.5	2 / 5 / 10	0.066 / 0.099 / 0.124	0.070 / 0.101 / 0.133	0.028 / 0.045 / 0.065	1.00 / 1.00 / 1.04	10 / 10 / 13
LimiX-2	2 / 5 / 10	0.040 / 0.050 / 0.046	0.045 / 0.088 / 0.175	0.014 / 0.016 / 0.015	1.11 / 1.61 / 3.06	13 / 17 / 20
TabICLv2	2 / 5 / 10	0.083 / 0.193 / 0.243	0.172 / 0.279 / 0.424	0.060 / 0.136 / 0.202	1.89 / 1.80 / 1.69	19 / 18 / 18
TabDPT	2 / 5 / 10	0.079 / 0.113 / 0.120	0.072 / 0.175 / 0.282	0.046 / 0.060 / 0.084	0.97 / 1.46 / 1.72	8 / 17 / 20
TabFM	2 / 5 / 10	0.138 / 0.182 / 0.211	0.172 / 0.256 / 0.297	0.118 / 0.145 / 0.164	1.11 / 1.31 / 1.30	15 / 18 / 17

Appendix

Table 16: Irrelevant features (

m=24

, noise

0.15

r=64

, twenty seeds): median error of Single , Expanded and Mean , median paired Expanded / Single ratio, and seeds in which Expanded is worse than Single .

	on the rows	means with variances	means only
One-dimensional synthetic	0 of 60	0 of 60	0 of 54
Real covariates	19 of 45	18 of 45	5 of 40
Multivariate synthetic ( $d=2,5,10$ )	3 of 9	0 of 9	0 of 9

Appendix

Table 17: The Gaussian process: designs (of those fitted) in which the table at

r=64

is worse than Single in at least

11

20

seeds.

dataset ( $r$ )	rows	means+var.	median
	harmed splits		relative MSE gap
SGEMM ( $4$ )	14/20	14/20	$5.0\times 10^{-7}$
KT convolution ( $16$ )	19/20	19/20	$7.2\times 10^{-7}$
KT dedispersion ( $32$ )	17/20	17/20	$1.4\times 10^{-6}$
FCNet slice ( $4$ )	11/20	11/20	$3.4\times 10^{-5}$

Appendix

Table 18: GP aggregation on the four actual datasets with the lowest within-site SD ratios, at their largest count. The row fit estimates kernel and noise parameters from Expanded ; the means+variance fit uses site means with pooled empirical within-site variance divided by

r

, and estimates only kernel parameters. Harm counts compare each fit with the same Single GP on

20

paired splits. The last column is the median of

|\mathrm{MSE}_{\rm rows}-\mathrm{MSE}_{\rm means+var.}|/\mathrm{MSE}_{\rm means+var.}

Figure 8: Single , Expanded and Mean against readings per site. Marginal median fresh-input MSE over six one-dimensional functions (top, noise SD

0.15

120

cells) and five real covariate datasets (bottom, noise multiplier

0.25

100

cells), twenty seeds each,

m=12

. Single is constant; curves start at its

r=1

value. Table 19 lists the

r=4,64,256

medians from the same fits.

			Expanded			Mean
	model	Single	4	64	256	4	64	256
one-dimensional	TabPFN-v3	0.039	0.028	0.045	0.049	0.031	0.010	0.010
synthetic	TabPFN-v3.5	0.107	0.050	0.265	0.274	0.063	0.044	0.037
	LimiX-2	0.069	0.077	0.204	0.162	0.053	0.034	0.032
	TabICLv2	0.109	0.105	0.131	0.136	0.096	0.073	0.075
	TabDPT	0.087	0.047	0.067	0.081	0.057	0.044	0.045

Appendix

Table 19: Marginal median fresh-input MSE on the default designs of Figure 8 : six one-dimensional functions at noise SD

0.15

(

120

cells) and five real covariate datasets at noise multiplier

0.25

(

100

cells), twenty seeds each,

m=12

. Single , and Expanded and Mean at

4

64

and

256

readings.

	SGEMM ( $150\times 4$ )				Kernel Tuner conv. ( $150\times 16$ )				Hydraulic rig ( $72\times 8$ )
	S	E	M	worse	S	E	M	worse	S	E	M	worse
TabPFN-v3	0.105	0.161	0.104	20	0.223	0.259	0.218	15	0.178	0.074	0.071	2
TabPFN-v3.5	0.097	0.089	0.097	2	0.211	0.216	0.203	13	0.199	0.110	0.078	6
LimiX-2	0.091	0.116	0.093	20	0.207	0.222	0.200	17	0.141	0.057	0.045	4
TabICLv2	0.112	0.232	0.111	20	0.208	0.252	0.198	17	0.102	0.051	0.047	2
TabDPT	0.086	0.162	0.086	20	0.229	0.250	0.223	16	0.185	0.097	0.102	3

Appendix

Table 20: Three real datasets at the largest count: median error over twenty splits for Single , Expanded and Mean , and splits in which Expanded is worse than Single .

Scope	Near slope	Middle slope	MSE
Native	0.688	1.354	0.001345
Embedding-only	0.803	1.141	0.000203
ICL only	0.943	1.069	0.000196
Full	0.996	0.999	0.000065

Appendix

Table 21: Linear exact-copy interventions on the same table and model weights. Twelve sites,

64

copies, six site seeds, one ensemble member; arithmetic means over six cells.

Figure 9: Inside the table of Figure 1 . The twelve sites of its left design (here at the grid’s default noise SD $0.15$ ), drawn at their mean reading and coloured by the pooled attention weight one query (star) places on each. Left: one reading per site. Right: $64$ readings. One site’s weight rises from $0.28$ to $0.51$ and the prediction at the query moves from $-0.16$ to $-0.42$ against a truth of $-0.17$ .

Figure 10: Two-input prediction surfaces with independent readings. Additive target,

24

sites, noise SD

0.5

r=64

; Single uses the first reading. Height and colour share scales on a

61\times 61

grid; fresh errors use

512

separately drawn queries.

S_{\rm near}

denotes

S_{r}(A,h)

on the near-site probes;

W_{80}

is the mean

80\%

-variation path fraction over eight neighboring-site paths. One design and one bank of readings, eight-member TabPFN-v3 (v3 checkpoint, tabpfn 9.0.0).

function	TabDPT	TabFM
quadratic	1.243	1.000
absolute value	0.712	0.999
linear	1.420	1.009
hyperbolic tangent	1.023	1.007
step	0.932	1.003
sine	0.484	0.999

Appendix

Table 22: Main-grid exact copies: median paired fresh-input MSE ratio Expanded / Single , twelve sites,

64

copies, twenty seed cells per function.

Figure 11: Effective number of attended sites and top pooled weight at a representative query as

r

grows, TabPFN-v3, Expanded table,

m=12

\sigma=0.15

; mean and

95\%

interval over twenty seeds. Selective layers are

18,19,21,22

; the all-layer curves pool all

24

ICL layers.

Figure 12: The embedding stage of TabPFN-v3 under repetition (

m=12

r=64

, four functions, six seeds, medians). Spread of one site’s rows relative to the spread between sites along the forward pass; displacement of a site between the

r=64

and

r=1

contexts under count overrides; and the resulting error relative to Single .

trial	shipped	trained	shipped	trained	wins
	Expanded		Mean
Four ICL layers + count + head	0.0648	0.0621	0.0079	0.0481	23/36
All ICL layers + count + head	0.0648	0.0730	0.0079	0.0307	17/36
Values-only adapter	0.0655	0.0727	0.0076	0.0076	17/36

Appendix

Table 23: Three fine-tuning trials on TabPFN-v3, each with one training seed. Median fresh-input MSE over

36

paired cells: six one-dimensional functions times six evaluation seeds (

0

–

5

m=12

r=64

, noise SD

0.05

. Each trial uses its own shipped baseline under the same inference settings; wins compare trained Expanded with shipped Expanded .

Tabular Foundation Models

Tabular foundation models learn reusable representations or in-context prediction skills for structured data such as spreadsheets and databases.

See all 188 papers

When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models
Oct 14, 2026Wenwen Gong, Haotian Gu, Zhen Yang +6Tabular Foundation Models Transformer Inference
China Agricultural University · Tsinghua University · Nankai University
Noise Immunity in In-Context Tabular Learning: An Empirical Robustness Analysis of TabPFN's Attention Mechanisms
Apr 6, 2026James Hu, Mahdi GhelichiTabular ML Tabular ICL
Model Development Innovation, Risk Management TD Bank, Toronto, Canada
Tabular foundation models for non-tabular tasks
Aug 23, 2026Goran Nakerst, John Brennan, Wouter Beugeling +1Handwritten Digit Recognition Tabular Foundation Models
Institut für Theoretische Physik, Technische Universität Dresden, 01062 Dresden, Germany · Department of Physics, Maynooth University, Ireland · Physikalisches Institut (EP3) and Institute for Topological Insulators, Universität Würzburg, Am Hubland, 97074 Würzburg, Germany

Oct 14, 2026J/K move · Enter open · S save

When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models

Wenwen Gong, Haotian Gu, Zhen Yang, Jiaying Ma+5

China Agricultural University · Tsinghua University · Nankai University

Tabular Foundation Models Transformer Inference Tabular ICL Tabular Representation Learning Transformer Interpretability

Open paper AskPDF arXiv

Transformer-based tabular foundation models (TFMs) achieve state-of-the-art performance on small to medium-sized tabular prediction tasks while processing diverse inputs through a shared stack of layers. Recent layerwise analyses identify strong predictive performance in early TFM representations and substantial overlap in computation across layers. However, early predictive accessibility alone does not establish whether later layers are dispensable for every input, and aggregate accuracy can obscure which inputs benefit from additional computation. We study all 24 in-context-learning layers of TabPFN-3 across 272 classification datasets, combining representation geometry, prediction trajectories, truncation, and structural interventions, while also examining related representation dynamics in four additional TFMs. We uncover a pronounced expansion-compression trajectory, with stronger late dimensional contraction associated with larger increases in class-prototype separation. A targeted analysis of early-stabilizing samples shows that substantial representational changes persist after predicted classes stabilize, while locally ambiguous queries obtain larger accuracy gains from shallow to late layers than well-separated queries. Repeating a single pretrained block or perturbing layer order reduces predictive performance on average, constraining simple post-hoc substitutions in the frozen model. These findings distinguish predictive accessibility, class stability, and the benefits of continuing computation, supporting an input-dependent assessment of depthwise redundancy in TFMs.

Abstract

Tabular Foundation Models

Explore similar work

When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models

Noise Immunity in In-Context Tabular Learning: An Empirical Robustness Analysis of TabPFN's Attention Mechanisms

Tabular foundation models for non-tabular tasks