Tabular foundation models (TFMs) achieve state-of-the-art performance on IID benchmarks, but how do they respond to repeated measurements at the same inputs? Across six TFMs, synthetic and semi-synthetic experiments show that additional measurements can improve predictions at measured inputs while degrading them elsewhere. Harm is most frequent at low noise, and irrelevant features amplify it in several models; real repeated-measurement datasets show similar patterns. Statistically, the IID rule targeted by PFN pretraining treats each repeated input as fresh evidence about the input distribution. When the prior links this distribution to the response function, the resulting protocol mismatch can outweigh the benefit of additional measurements, even under exact inference. Computationally, interventions identify row-count dependence in TabPFN-v3's attention scaling as a contributor to harm. Resetting the count largely reverses distortion in a noiseless linear diagnostic and attenuates amplification by irrelevant features. Finally, we evaluate ways to retain the benefits of repeated measurements while limiting their harm. Averaging measurements usually outperforms supplying them separately at larger repetition counts, whereas count reset can suppress both harm and useful gains. Code, saved results and reproduction instructions are available at https://github.com/drizmiz/PFN-Repeated-Measurement-Supp
Figures & tables
Figure 1: More readings, worse predictions between the sites. Top: TabPFN-v3 on one design (quadratic, twelve sites, noise SD 0.10 ) with one reading per site (blue) and 64 per site as rows (red); dots are the site means. Bottom: median over twenty seeds of each seed’s MSE (r)/ MSE (1) for the six TFMs and the four conventional learners. Raw errors and all six TFMs’ fits are in Appendix Table 12 and Figure 6 .
Figure 2: Fraction of the twenty paired seeds, pooled over functions or covariate datasets, in which Expanded at r=64 has higher fresh-input MSE than Single (TFMs above the line, conventional learners below; RF: random forest, GBDT: gradient boosting). Synthetic tasks are scored on a grid (one dimension) or fresh draws, semi-synthetic tasks on held-out rows. The irrelevant-features panel fixes a function of two coordinates and adds ignored ones. Real data: twenty splits at the largest count R , scored against held-out site means and ordered by within-site SD ratio (Section 3.3 ).
Figure 3: Full count reset on a fixed table of readings: TabPFN-v3, quadratic target, 12 sites, 64 independent readings, SD 0.05 , one ensemble member, 201 shared queries. (a) Prediction minus truth; zero means correct prediction and grey ticks mark sites. (b) Site-pooled neff minus Single , after the median over heads/ICL operations at each query; zero matches Single ’s concentration. Single ’s query median is 11.06 (uniform: 12 ). Full count reset sets all count inputs to 12 .
noise SD
Single
Expanded
Mean
full reset
Mean wins
0
0.0088
0.053
0.0088
0.0084
36
0.05
0.015
0.049
0.0094
0.0135
61
0.15
0.039
0.045
0.010
0.032
90
0.30
0.104
0.047
0.013
0.076
107
0.50
0.194
0.052
0.017
0.174
115
Table 1: Count reset against averaging, TabPFN-v3, r=64 : median MSE at fresh inputs over the six one-dimensional functions and twenty seeds ( m=12 , eight ensemble members), and cells (of 120 ) in which Mean beats full count reset. At SD 0 , Mean equals Single .
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Count thresholds within a specified unknown-noise Bayesian model. Markers show the IID-posterior risk minus Single risk, computed from the integrated reading benefit and cue cost for the nonlinear query model with m=2 and fixed cue strength t=.02 . Dashed curves show Equation ( 17 ). Colors vary the population mean variance ν and the corresponding prior used by the predictor. Error bars are pointwise 1.96 Monte Carlo standard errors of the estimated benefit; cue costs are integrated deterministically. The plot verifies this reference calculation and does not report TFM predictions.
Figure 5: Coverage limit within the unknown-variance region model. Direct paired IID-minus- Single risks at population mean variance ν=.003 are plotted against u2(1−1/K)mδr(t) , with u=1 and t=.25 . All combinations K=2,4,8 , m=2,4 , and r=4,16 are included. The dashed line is equality; bars are pointwise 1.96 Monte Carlo standard errors over 8,192 tasks. Each predictor integrates the same InvGamma(4,3ν) prior within its condition.
dataset
a reading is
d
sites
R
context sites
SGEMM kernel timings
one run of a kernel configuration
14
2,994
4
150
Kernel Tuner convolution
one launch of a kernel configuration
7
3,000
16
150
Kernel Tuner dedispersion
one launch of a kernel configuration
6
3,000
32
150
FCNet slice localization
one training run with its own random seed
9
3,000
4
150
Bayes-factor surface
one Monte Carlo replicate
2
80
4
40
G. aparine bioassay
one pot at a dose and formulation
2
21
8
10
Appendix
Table 2: Real datasets with repeated observations, after retaining sites with at least two finite, non-identical responses at one covariate vector. d counts covariates that vary; R is the largest count used; the context holds half of the sites that have at least R readings, at most 150 .
equal counts
unequal counts
learner
weighted
unweighted
weighted
unweighted
least squares
440 (440)
440 (440)
440 (440)
40 (40)
kernel ridge, fixed kernel
440 (440)
0 (0)
440 (440)
0 (0)
GP, fixed kernel and noise
440 (440)
0 (0)
440 (440)
0 (0)
regression tree
241 (440)
241 (440)
240 (440)
240 (438)
gradient boosting
240 (440)
240 (440)
240 (440)
0 (0)
Appendix
Table 3: Proposition 1 in practice. Fits (of 440 : six one-dimensional functions and five real-covariate sets, twenty seeds, two count configurations) in which the prediction from the means equals the prediction from the rows to 10−6 at every query, and in parentheses at the sites only. Equal counts are r=4 and 64 ; unequal counts are the grid’s random and clustered schemes. Weighted: count-weighted means (Dˉ,r) (for the Gaussian processes, each mean’s noise variance divided by its count).
setting
Single
Expanded
Mean
full reset
embedding only
mean better
1-D, r=4
noise 0
0.0088
0.028
0.0088
0.0089
0.021
36
noise 0.05
0.015
0.021
0.016
0.015
0.018
59
noise 0.15
0.039
0.028
0.031
0.028
0.029
64
noise 0.30
0.104
0.047
0.050
0.069
0.040
77
noise 0.50
0.194
0.084
0.082
0.176
0.076
94
1-D, r=64
noise 0
0.0088
0.053
0.0088
0.0084
0.038
36
Appendix
Table 4: Count reset in TabPFN-v3: median error at fresh inputs of Single , Expanded , Mean , full count reset (all 27 count arguments set to m ) and embedding-only reset (the three embedding modules), and cells in which Mean beats full count reset. One-dimensional synthetic: six functions, 120 cells per row; dimension axis: three families, m=24 , 60 cells; semi-synthetic: five datasets, 100 cells; real data: twenty splits at the largest count.
d
Expanded / Single
full/ Single
full wins
2
1.600
1.045
15/20
5
3.100
1.051
19/20
10
3.146
1.151
20/20
Appendix
Table 5: TabPFN-v3 with irrelevant features: median paired fresh-input MSE ratios to Single , and cells in which full count reset beats Expanded . Each dimension has twenty seed cells; all arms share the sites, readings and queries within a cell.
Single
Expanded
Mean
model
bias 2
var
bias 2
var
bias 2
var
TabPFN-v3
0.023
0.013
0.041
0.0005
0.007
0.0005
TabPFN-v3.5
0.100
0.016
0.240
0.0013
0.035
0.0006
TabICLv2
0.072
0.018
0.107
0.0004
0.063
0.0005
TabDPT
0.082
0.015
0.064
0.0006
0.036
0.0004
LimiX-2
0.041
0.014
0.200
0.0037
0.027
0.0013
Appendix
Table 6: What averaging removes. Squared bias and variance of the prediction at fresh inputs, estimated from eight independent banks of 64 readings at the same sites; median over 60 cells (six one-dimensional functions times ten site seeds, m=12 , noise SD 0.15 ).
Single
Expanded
Mean
designs
model
bias 2
var
bias 2
var
bias 2
var
1-D, noise 0.5
TabPFN-v3
0.112
0.110
0.043
0.0033
0.012
0.0036
TabPFN-v3.5
0.273
0.073
0.278
0.0028
0.049
0.0040
TabICLv2
0.121
0.126
0.147
0.0031
0.062
0.0045
TabDPT
0.273
0.098
0.138
0.0033
0.047
0.0032
LimiX-2
0.118
0.093
0.318
0.0073
0.029
0.0046
Appendix
Table 7: Squared bias and variance at fresh inputs over eight independent banks of 64 readings at the same sites, finite-bank corrected; median over cells (one-dimensional: six functions times ten site seeds, noise SD 0.50 ; semi-synthetic: five datasets times ten site seeds, noise multiplier 0.25 ).
Coverage
CRPS: MR/comparator
model
Single
Exp.
Mean
MR
Exp.
Single
Mean
One-dimensional
TabPFN-v3
0.87
0.91
0.43
0.92
0.926
0.810
0.906
TabPFN-v3.5
0.96
0.99
0.68
0.94
0.684
0.808
0.983
TabICLv2
0.67
0.98
0.57
0.92
0.819
0.835
0.948
TabDPT
0.93
0.90
0.65
0.96
0.966
0.853
0.988
Appendix
Table 8: Future-reading prediction at fresh inputs, r=64 , under Gaussian, homoscedastic reading noise. Coverage entries are median probabilities for nominal 90% intervals; CRPS entries are median paired ratios of Gaussian MR to the named comparator, using the 19 -quantile approximation (lower is better). Each model has 120 function–seed and 100 real-covariate dataset–seed pairs, with m=12 sites, noise SD 0.15 (synthetic) or residual-SD multiplier 0.25 (semi-synthetic), and 400 or 300 fresh queries, respectively. MR is defined in Equation ( 5 ); Single uses r=1 . Exp. denotes Expanded . LimiX-2 and TabFM return point predictions only.
one-dimensional
semi-synthetic
model
scheme
mean
copies
Expanded
worse
mean
copies
Expanded
worse
TabPFN-v3
random
0.027
0.054
0.040
89
0.444
0.490
0.489
65
clustered
0.026
0.083
0.039
97
0.429
0.491
0.501
69
TabPFN-v3.5
random
0.078
0.169
0.163
83
0.473
0.491
0.524
61
clustered
0.070
0.182
0.128
90
0.473
0.482
0.492
54
LimiX-2
random
0.054
0.156
0.096
108
0.410
0.477
0.483
66
Appendix
Table 9: The count encoding under unequal counts (the grid’s random and clustered schemes): median error at fresh inputs of Mean , Mean with the row of site i copied ri times, and Expanded , and cells (of 120 one-dimensional, 100 semi-synthetic) in which the copies are worse than Mean .
neff
fresh-input MSE
noise SD
Single
Expanded
full
Exp. < Single
full > Exp.
Single
Expanded
full
0.05
10.91
10.56
11.12
112/120
120/120
0.017
0.051
0.010
0.15
10.89
10.70
11.19
94/120
120/120
0.042
0.051
0.029
Appendix
Table 10: Site-pooled attention in TabPFN-v3 (one ensemble member, r=64 , six functions, twenty seeds): median neff over 120 cells (each the median over 400 fresh queries of the median jointly over heads and layers; uniform attention gives 12 ), cells in which it falls with the rows and rises under full count reset, and the error of the same fits. Full count reset’s neff exceeds Single ’s in 114/120 cells at SD 0.05 and 120/120 at SD 0.15 .
neff↓
neff↓
noise SD
harm
gain
harm
gain
0.05
74
38
6
2
0.15
43
51
11
15
Appendix
Table 11: Attention concentration and prediction error in the same one-member fits as Table 10 : counts over 120 function–seed cells per noise level. A down-arrow denotes Expanded ’s site-pooled neff below Single ; harm denotes Expanded ’s fresh-input MSE above Single . Cells summarize 400 fresh queries each.
Figure 6: Fitted functions on the left design of Figure 1 (quadratic, twelve sites, noise SD 0.10 ) with one reading per site (blue) and 64 readings per site entered as rows (orange); the numbers in each panel are the two errors. The points are the site means at r=64 , which every model passes through; the changes are between them.
learner
MSE (1)
MSE (256)
worse
Gaussian process
0.014
0.0002
0/20
Gradient boosting
0.172
0.158
6/20
Random forest
0.182
0.158
4/20
MLP
0.043
0.018
1/20
TabPFN-v3
0.017
0.070
18/20
TabPFN-v3.5
0.062
0.379
20/20
Appendix
Table 12: Raw MSE at r=1 and r=256 on the design of Figure 1 (noise SD 0.10 ), for all ten learners: median over 20 seeds, with the number of seeds in which more readings made the error worse.
Figure 7: Error at the measured sites (blue) and at fresh inputs (orange) with one reading and with 64 readings as rows; one line per synthetic function, m=12 , σ=0.15 , medians over twenty seeds. At the sites the error of every model falls by a factor of 35 or more.
family
LimiX-2
TabICLv2
TabPFN-v3.5
TabPFN-v3
TabFM
TabDPT
forest
GP (rows)
MLP
boosting
1-D (60)
39
26
35
14
8
4
10
0
0
0
Real covariates (45)
25
34
24
30
33
17
34
19
8
6
Multivariate (9)
9
9
4
8
7
4
9
3
0
0
All (114)
73
69
63
52
48
25
53
22
8
6
Appendix
Table 13: Designs in which Expanded at r=64 is worse than Single at fresh inputs in at least 11 of 20 seeds. Parentheses give the number of tested designs in each family; designs sharing a function or seed are not independent replications.
scored at
TabPFN-v3
TabPFN-v3.5
LimiX-2
TabICLv2
TabDPT
TabFM
forest
measured sites
0
0
0
0
0
0
0
gap midpoints
37
78
91
59
16
30
111
fresh inputs
50
93
102
70
29
22
57
Appendix
Table 14: Query location. Function–seed pairs (of 120 : six one-dimensional functions, twenty seeds, default design) in which Expanded at r=64 is worse than Single , by where the prediction is scored.
sites m
spacing
model
6
12
24
48
96
even
random
TabPFN-v3
5
16
8
1
0
0.23
0.42
TabPFN-v3.5
6
18
11
9
7
0.78
0.78
LimiX-2
11
20
20
10
3
0.88
0.85
TabICLv2
9
12
6
5
2
0.32
0.58
TabDPT
1
9
8
5
2
0.07
0.24
Appendix
Table 15: Left: quadratic, r=64 , seeds (of 20 ) in which Expanded is worse than Single , by the number of sites. Right: share of function–seed pairs harmed with evenly spaced and with random sites ( m=12 , six one-dimensional functions).
model
d
Single
Expanded
Mean
ratio
worse
TabPFN-v3
2 / 5 / 10
0.051 / 0.069 / 0.084
0.076 / 0.204 / 0.259
0.012 / 0.022 / 0.026
1.60 / 3.10 / 3.15
15 / 18 / 19
TabPFN-v3.5
2 / 5 / 10
0.066 / 0.099 / 0.124
0.070 / 0.101 / 0.133
0.028 / 0.045 / 0.065
1.00 / 1.00 / 1.04
10 / 10 / 13
LimiX-2
2 / 5 / 10
0.040 / 0.050 / 0.046
0.045 / 0.088 / 0.175
0.014 / 0.016 / 0.015
1.11 / 1.61 / 3.06
13 / 17 / 20
TabICLv2
2 / 5 / 10
0.083 / 0.193 / 0.243
0.172 / 0.279 / 0.424
0.060 / 0.136 / 0.202
1.89 / 1.80 / 1.69
19 / 18 / 18
TabDPT
2 / 5 / 10
0.079 / 0.113 / 0.120
0.072 / 0.175 / 0.282
0.046 / 0.060 / 0.084
0.97 / 1.46 / 1.72
8 / 17 / 20
TabFM
2 / 5 / 10
0.138 / 0.182 / 0.211
0.172 / 0.256 / 0.297
0.118 / 0.145 / 0.164
1.11 / 1.31 / 1.30
15 / 18 / 17
Appendix
Table 16: Irrelevant features ( m=24 , noise 0.15 , r=64 , twenty seeds): median error of Single , Expanded and Mean , median paired Expanded / Single ratio, and seeds in which Expanded is worse than Single .
on the rows
means with variances
means only
One-dimensional synthetic
0 of 60
0 of 60
0 of 54
Real covariates
19 of 45
18 of 45
5 of 40
Multivariate synthetic ( d=2,5,10 )
3 of 9
0 of 9
0 of 9
Appendix
Table 17: The Gaussian process: designs (of those fitted) in which the table at r=64 is worse than Single in at least 11 of 20 seeds.
harmed splits
relative MSE gap
dataset ( r )
rows
means+var.
median
SGEMM ( 4 )
14/20
14/20
5.0×10−7
KT convolution ( 16 )
19/20
19/20
7.2×10−7
KT dedispersion ( 32 )
17/20
17/20
1.4×10−6
FCNet slice ( 4 )
11/20
11/20
3.4×10−5
Appendix
Table 18: GP aggregation on the four actual datasets with the lowest within-site SD ratios, at their largest count. The row fit estimates kernel and noise parameters from Expanded ; the means+variance fit uses site means with pooled empirical within-site variance divided by r , and estimates only kernel parameters. Harm counts compare each fit with the same Single GP on 20 paired splits. The last column is the median of ∣MSErows−MSEmeans+var.∣/MSEmeans+var. .
Figure 8: Single , Expanded and Mean against readings per site. Marginal median fresh-input MSE over six one-dimensional functions (top, noise SD 0.15 , 120 cells) and five real covariate datasets (bottom, noise multiplier 0.25 , 100 cells), twenty seeds each, m=12 . Single is constant; curves start at its r=1 value. Table 19 lists the r=4,64,256 medians from the same fits.
Expanded
Mean
model
Single
4
64
256
4
64
256
one-dimensional
TabPFN-v3
0.039
0.028
0.045
0.049
0.031
0.010
0.010
synthetic
TabPFN-v3.5
0.107
0.050
0.265
0.274
0.063
0.044
0.037
LimiX-2
0.069
0.077
0.204
0.162
0.053
0.034
0.032
TabICLv2
0.109
0.105
0.131
0.136
0.096
0.073
0.075
TabDPT
0.087
0.047
0.067
0.081
0.057
0.044
0.045
Appendix
Table 19: Marginal median fresh-input MSE on the default designs of Figure 8 : six one-dimensional functions at noise SD 0.15 ( 120 cells) and five real covariate datasets at noise multiplier 0.25 ( 100 cells), twenty seeds each, m=12 . Single , and Expanded and Mean at 4 , 64 and 256 readings.
SGEMM ( 150×4 )
Kernel Tuner conv. ( 150×16 )
Hydraulic rig ( 72×8 )
S
E
M
worse
S
E
M
worse
S
E
M
worse
TabPFN-v3
0.105
0.161
0.104
20
0.223
0.259
0.218
15
0.178
0.074
0.071
2
TabPFN-v3.5
0.097
0.089
0.097
2
0.211
0.216
0.203
13
0.199
0.110
0.078
6
LimiX-2
0.091
0.116
0.093
20
0.207
0.222
0.200
17
0.141
0.057
0.045
4
TabICLv2
0.112
0.232
0.111
20
0.208
0.252
0.198
17
0.102
0.051
0.047
2
TabDPT
0.086
0.162
0.086
20
0.229
0.250
0.223
16
0.185
0.097
0.102
3
Appendix
Table 20: Three real datasets at the largest count: median error over twenty splits for Single , Expanded and Mean , and splits in which Expanded is worse than Single .
Scope
Near slope
Middle slope
MSE
Native
0.688
1.354
0.001345
Embedding-only
0.803
1.141
0.000203
ICL only
0.943
1.069
0.000196
Full
0.996
0.999
0.000065
Appendix
Table 21: Linear exact-copy interventions on the same table and model weights. Twelve sites, 64 copies, six site seeds, one ensemble member; arithmetic means over six cells.
Figure 9: Inside the table of Figure 1 . The twelve sites of its left design (here at the grid’s default noise SD 0.15 ), drawn at their mean reading and coloured by the pooled attention weight one query (star) places on each. Left: one reading per site. Right: 64 readings. One site’s weight rises from 0.28 to 0.51 and the prediction at the query moves from −0.16 to −0.42 against a truth of −0.17 .
Figure 10: Two-input prediction surfaces with independent readings. Additive target, 24 sites, noise SD 0.5 , r=64 ; Single uses the first reading. Height and colour share scales on a 61×61 grid; fresh errors use 512 separately drawn queries. Snear denotes Sr(A,h) on the near-site probes; W80 is the mean 80% -variation path fraction over eight neighboring-site paths. One design and one bank of readings, eight-member TabPFN-v3 (v3 checkpoint, tabpfn 9.0.0).
function
TabDPT
TabFM
quadratic
1.243
1.000
absolute value
0.712
0.999
linear
1.420
1.009
hyperbolic tangent
1.023
1.007
step
0.932
1.003
sine
0.484
0.999
Appendix
Table 22: Main-grid exact copies: median paired fresh-input MSE ratio Expanded / Single , twelve sites, 64 copies, twenty seed cells per function.
Figure 11: Effective number of attended sites and top pooled weight at a representative query as r grows, TabPFN-v3, Expanded table, m=12 , σ=0.15 ; mean and 95% interval over twenty seeds. Selective layers are 18,19,21,22 ; the all-layer curves pool all 24 ICL layers.
Figure 12: The embedding stage of TabPFN-v3 under repetition ( m=12 , r=64 , four functions, six seeds, medians). Spread of one site’s rows relative to the spread between sites along the forward pass; displacement of a site between the r=64 and r=1 contexts under count overrides; and the resulting error relative to Single .
Expanded
Mean
trial
shipped
trained
shipped
trained
wins
Four ICL layers + count + head
0.0648
0.0621
0.0079
0.0481
23/36
All ICL layers + count + head
0.0648
0.0730
0.0079
0.0307
17/36
Values-only adapter
0.0655
0.0727
0.0076
0.0076
17/36
Appendix
Table 23: Three fine-tuning trials on TabPFN-v3, each with one training seed. Median fresh-input MSE over 36 paired cells: six one-dimensional functions times six evaluation seeds ( 0 – 5 ), m=12 , r=64 , noise SD 0.05 . Each trial uses its own shipped baseline under the same inference settings; wins compare trained Expanded with shipped Expanded .
Transformer-based tabular foundation models (TFMs) achieve state-of-the-art performance on small to medium-sized tabular prediction tasks while processing diverse inputs through a shared stack of layers. Recent layerwise analyses identify strong predictive performance in early TFM representations and substantial overlap in computation across layers. However, early predictive accessibility alone does not establish whether later layers are dispensable for every input, and aggregate accuracy can obscure which inputs benefit from additional computation. We study all 24 in-context-learning layers of TabPFN-3 across 272 classification datasets, combining representation geometry, prediction trajectories, truncation, and structural interventions, while also examining related representation dynamics in four additional TFMs. We uncover a pronounced expansion-compression trajectory, with stronger late dimensional contraction associated with larger increases in class-prototype separation. A targeted analysis of early-stabilizing samples shows that substantial representational changes persist after predicted classes stabilize, while locally ambiguous queries obtain larger accuracy gains from shallow to late layers than well-separated queries. Repeating a single pretrained block or perturbing layer order reduces predictive performance on average, constraining simple post-hoc substitutions in the frozen model. These findings distinguish predictive accessibility, class stability, and the benefits of continuing computation, supporting an input-dependent assessment of depthwise redundancy in TFMs.
Wenwen Gong, Haotian Gu, Zhen Yang +6
China Agricultural University · Tsinghua University · Nankai University
Tabular foundation models (TFMs) such as TabPFN (Tabular Prior-Data Fitted Network) are designed to generalize across heterogeneous tabular datasets through in-context learning (ICL). They perform prediction in a single forward pass conditioned on labeled examples without dataset-specific parameter updates. This paradigm is particularly attractive in industrial domains (e.g., finance and healthcare) where tabular prediction is pervasive. Retraining a bespoke model for each new table can be costly or infeasible in these settings, while data quality issues such as irrelevant predictors, correlated feature groups, and label noise are common. In this paper, we provide strong empirical evidence that TabPFN is highly robust under these sub-optimal conditions. We study TabPFN and its attention mechanisms for binary classification problems with controlled synthetic perturbations that vary: (i) dataset width by injecting random uncorrelated features and by introducing nonlinearly correlated features, (ii) dataset size by increasing the number of training rows, and (iii) label quality by increasing the fraction of mislabeled targets. Beyond predictive performance, we analyze internal signals including attention concentration and attention-based feature ranking metrics. Across these parametric tests, TabPFN is remarkably resilient: ROC-AUC remains high, attention stays structured and sharp, and informative features are highly ranked by attention-based metrics. Qualitative visualizations with attention heatmaps, feature-token embeddings, and SHAP plots further support a consistent pattern across layers in which TabPFN increasingly concentrates on useful features while separating their signals from noise. Together, these findings suggest that TabPFN is a robust TFM capable of maintaining both predictive performance and coherent internal behavior under various scenarios of data imperfections.
James Hu, Mahdi Ghelichi
Model Development Innovation, Risk Management TD Bank, Toronto, Canada
Tabular foundation models (TFMs) have recently emerged as a promising paradigm for machine learning on tabular data, offering the ability to generalize across datasets without task-specific training. Since many machine learning datasets can be represented as tables, this raises the question: does TFM capability extend beyond tasks traditionally regarded as tabular? We address this question by using TabPFN v3 on three non-tabular classification problems: handwritten digit recognition on MNIST, language identification of French and German words, and image classification on Tiny ImageNet. In each case, the original data are represented as rows of a table and classification is formulated as prediction of a missing label. We evaluate performance as a function of the number of context samples provided to the pretrained model, with no additional training or fine-tuning. Despite having no explicit access to the spatial or sequential structure characterizing the data, TabPFN v3 in some cases achieves accuracies comparable with that of models or methods geared specifically toward the corresponding tasks.
Goran Nakerst, John Brennan, Wouter Beugeling +1
Institut für Theoretische Physik, Technische Universität Dresden, 01062 Dresden, Germany · Department of Physics, Maynooth University, Ireland · Physikalisches Institut (EP3) and Institute for Topological Insulators, Universität Würzburg, Am Hubland, 97074 Würzburg, Germany