When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models

Wenwen Gong; Haotian Gu; Zhen Yang; Jiaying Ma; Hanqing Zhang; Zhongshen Wang; Jingjun Wang; Shouyi Xu

cs.LGOct 14, 2026

When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models

Authors: Wenwen Gong, Haotian Gu, Zhen Yang, Jiaying Ma, Hanqing Zhang, Zhongshen Wang, Jingjun Wang, Shouyi Xu, +1 more

Organizations: China Agricultural University · Tsinghua University · Nankai University

Abstract

Transformer-based tabular foundation models (TFMs) achieve state-of-the-art performance on small to medium-sized tabular prediction tasks while processing diverse inputs through a shared stack of layers. Recent layerwise analyses identify strong predictive performance in early TFM representations and substantial overlap in computation across layers. However, early predictive accessibility alone does not establish whether later layers are dispensable for every input, and aggregate accuracy can obscure which inputs benefit from additional computation. We study all 24 in-context-learning layers of TabPFN-3 across 272 classification datasets, combining representation geometry, prediction trajectories, truncation, and structural interventions, while also examining related representation dynamics in four additional TFMs. We uncover a pronounced expansion-compression trajectory, with stronger late dimensional contraction associated with larger increases in class-prototype separation. A targeted analysis of early-stabilizing samples shows that substantial representational changes persist after predicted classes stabilize, while locally ambiguous queries obtain larger accuracy gains from shallow to late layers than well-separated queries. Repeating a single pretrained block or perturbing layer order reduces predictive performance on average, constraining simple post-hoc substitutions in the frozen model. These findings distinguish predictive accessibility, class stability, and the benefits of continuing computation, supporting an input-dependent assessment of depthwise redundancy in TFMs.

Figures & tables

Figure 1: Predictive accessibility, class stability, and the value of depth are distinct. (a) L9/L13/L23 accuracy vs. local class separation: ambiguous queries gain more from depth. (b) Effective dimensionality per layer for early-stabilizing queries: representations keep changing after the class stabilises, then contract sharply.

Figure 2: Representational expansion followed by late compression. Thin curves denote individual datasets and the thick curve denotes the dataset-level mean.

Panel of Figure 3 — Figure 3: Representation transformation dynamics across depth. (a) Mean pairwise layer similarity, with linear CKA above the diagonal and cosine similarity below. (b) Adjacent-layer displacement measured in a shared PCA basis. (c) Dataset-level displacement across the expansion, plateau, and compression regimes.

Figure 4: Candidate separation strengthens at late depth. (a) Prototype margins remain small through the early and middle layers and widen most strongly near the output. (b) Correct-class alignment emerges earlier on easier tasks, whereas harder tasks retain candidate ambiguity deeper into the network. (c) Stronger late effective-dimensionality contraction is associated with larger increases in prototype margin.

Figure 5: Sample-level predictive commitment across depth. (a) Cumulative fraction of samples stabilized to the final prediction across layers, shown per dataset and in aggregate. (b) Five recurring prediction-transition patterns and their pooled prevalence. (c) Pattern composition across tertiles of chance-normalized task difficulty; statistical analyses use difficulty as a continuous variable.

Figure 6: Local class separation across prediction-transition patterns. Separation decreases from ES to MS, indicating local ambiguity for more dynamic trajectories.

Figure 7: Depth has selective utility across samples and tasks. (a) Accuracy at L9, L13, and L23 across within-dataset local-separation percentiles; locally ambiguous samples show larger gains with depth. (b) Retention of L9-correct predictions and rescue of L9 errors across local-separation percentiles; both increase with separation. (c) Mean stabilization depth is positively associated with L13 truncation loss. (d) Accuracy increases with depth under layer-matched readouts.

Figure 8: Shared-weight repetition produces a distinct depthwise trajectory. (a) Shared-weight repetition weakens compression and class-discriminative growth. (b) The repeated-block trajectory remains distinct at matched computational depth. (c) The full model retains a predictive advantage under perturbation.

Figure 9: Layer order is functionally consequential. (a) Shuffling or reversing the learned sequence disrupts the depthwise representation dynamics. (b) Reordering reduces predictive performance on average.

Appendix figures & tables43 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Hyperparameter	Value
Stage 1: Feature-distribution embedding
	Base embedding dimension	128
	Features per circular-shift group	3
	Induced-attention blocks	3
	Attention heads	8
	Inducing points per column	128

Appendix

Table 1: Architecture of TabPFN-3 used in the primary analysis. Layer indices L0–L23 refer exclusively to the 24 Stage-3 ICL blocks.

Model	Parameters	ICL depth	ICL dim.	Pre-ICL stages	Normalization
TabPFN-v2	7.24M	12	192	Alternating feature/item attention	Post-LN
TabPFN-2.5	10.72M	24	192	Alternating feature/item attention	Post-LN
TabPFN-3	53M	24	512	Column embedding + row aggregation	RMSNorm
TabICL	27.05M	12	512	TFcol (3) + TFrow (3)	Pre-LN
TabICLv2	–	12	512	TFcol (3) + TFrow (3)	Pre-LN

Appendix

Table 2: Architectures used in the cross-model analyses. “ICL depth” denotes the sequence of transformer blocks over which layerwise representations are analyzed. Parameter counts correspond to the classification models where available.

Component	TabPFN-v2	TabPFN-2.5
Total parameters	$7{,}244{,}554$	$10{,}718{,}218$
Feature encoder	768	1,152
Label encoder	576	576
Positional embedding	9,408	9,408
Transformer depth	12	24
Transformer-block parameters	$7{,}077{,}888$	$10{,}616{,}832$

Appendix

Table 3: Component-level architecture of TabPFN-v2 and TabPFN-2.5.

Stage	Component	Parameters
Column embedder	Total	877,824
	Input linear	256
	Transformer blocks (3)	844,032
	Each block	281,344
	Attention 1	$66{,}048+16{,}512$
	MLP 1	$33{,}024+32{,}896$

Appendix

Table 4: Component-level architecture of TabICL.

Stage	Hyperparameter	Value
$\mathrm{TF}_{\rm col}$	Layers	3
	Inducing vectors	128
	Model dimension	128
	Attention heads	8
$\mathrm{TF}_{\rm row}$	Layers	3
	Model dimension	128

Appendix

Table 5: Architecture of TabICLv2 used in the cross-model analysis.

Layer	Role in subsequent analyses
L9	Early-depth anchor; end of the expansion summary interval
L13	Intermediate plateau anchor
L17	Late-depth anchor; start of the compression summary interval
L20	Late-depth anchor near the aggregate ED maximum
L23	Final-layer representation

Appendix

Table 6: Representative TabPFN-3 layers used in subsequent analyses. The Expansion–Compression trajectory is characterized from the complete layerwise measurements; these depths serve as recurring analytical anchors.

Figure 10: Cross-model representation dynamics. Results are shown for TabPFN-v2, TabPFN-2.5, TabICL, and TabICLv2 on matched datasets. (a) Effective dimensionality normalized by the first-layer value within each dataset–model pair. (b) Adjacent-layer PCA displacement expressed as a fraction of the total trajectory displacement. (c) Fisher discriminant ratio across depth. Thin curves denote individual datasets and thick curves denote the cross-dataset mean. Mean curves are interpolated only for visual smoothness; all statistics use the original layerwise measurements. Peak annotations in panel (a) refer to the cross-dataset mean trajectories and therefore need not equal the mean within-dataset expansion ratios reported in Table 7 .

Model	Depth	Expansion	Compression	Final ED	Late PCA
TabPFN-v2	12	$3.61\times$	$1.43\times$	$2.59$	$27.8\%$
TabPFN-2.5	24	$3.87\times$	$1.87\times$	$2.22$	$44.4\%$
TabICL	12	$1.82\times$	$3.88\times$	$0.62$	$64.9\%$
TabICLv2	12	$3.01\times$	$3.59\times$	$1.22$	$83.8\%$

Appendix

Table 7: Summary of cross-model representation dynamics. Expansion and compression ratios are first computed within each dataset and then averaged across matched datasets. Final ED denotes the final effective dimensionality normalized by the first-layer value and is likewise averaged from dataset-level values. Because these quantities are averaged independently, their cross-dataset means need not satisfy exact algebraic ratio identities. Late PCA reports the fraction of total PCA displacement occurring in the final quarter of layer transitions.

Figure 11: Pairwise representation similarity across architectures. For each model, the upper triangular matrix reports mean linear CKA between every pair of layers, while the lower triangular matrix reports mean cosine similarity. Darker entries indicate greater similarity.

Dataset	Pretrained	Randomized ICL blocks	Fully randomized
vehicle	$4.00\times/2.33\times$	$1.00\times/1.00\times$	$1.00\times/1.00\times$
segment	$4.67\times/2.33\times$	$1.00\times/1.00\times$	$1.00\times/1.00\times$
satimage	$7.50\times/3.33\times$	$1.00\times/1.00\times$	$1.00\times/1.00\times$
diabetes	$2.88\times/2.09\times$	$1.00\times/1.00\times$	$1.00\times/1.00\times$

Appendix

Table 8: Random-weight control for TabPFN-3. Each entry reports expansion ratio / compression ratio. The pretrained model exhibits substantial Expansion–Compression, whereas randomizing either the ICL stack or the complete model yields flat effective-dimensionality trajectories on the tested datasets.

Depth	ED trajectory	Expansion	Compression
3 layers	$1\rightarrow 2\rightarrow 2$	$2.00\times$	$1.00\times$
4 layers	$1\rightarrow 1\rightarrow 2\rightarrow 2$	$2.00\times$	$1.00\times$
5 layers	$1\rightarrow 2\rightarrow 2\rightarrow 2\rightarrow 1$	$2.00\times$	$2.00\times$

Appendix

Table 9: Controlled-depth experiment with nanoTabPFN. The three- and four-layer variants exhibit representation expansion without a subsequent compression phase, whereas the five-layer variant exhibits both expansion and compression.

Measurement	FT-Transformer
Mean expansion ratio	$1.11\times$
Mean compression ratio	$2.33\times$
Characteristic trajectory	Predominantly contraction

Appendix

Table 10: Representation geometry in the supervised FT-Transformer control. The supervised model exhibits substantial effective-dimensionality contraction without the pronounced preceding expansion characteristic of the Expansion–Compression trajectory.

Figure 12: Native-decoder and layer-matched readout performance across depth. Left: classification accuracy averaged over 72 OpenML-CC18 tasks. Right: negative log-likelihood. Error bars denote 95% bootstrap confidence intervals over datasets. Vertical dotted lines at L9 and L17 indicate representative analytical boundaries rather than estimated change points.

Figure 13: Decoder-independent predictive decodability across depth. (a) Linear and capacity-controlled MLP probe accuracy across the 24 ICL layers of TabPFN-3, averaged over 72 OpenML-CC18 tasks. Error bars denote 95% bootstrap confidence intervals over datasets; dotted vertical lines indicate the representative L9 and L17 analytical boundaries. (b) Change in MLP-probe accuracy from the first to the final third of the ICL layers for TabPFN-3, TabICL, and TabICLv2. Differences are computed within each dataset before aggregation, and error bars denote 95% bootstrap confidence intervals over the 72 tasks.

Model	$\Delta$ MLP Acc.	Positive	$\Delta$ ECE	$\Delta$ MLP NLL
TabPFN-3	+0.027	64%	-0.005	-0.056
TabICL	+0.029	66%	-0.017	-0.179
TabICLv2	+0.020	59%	-0.008	-0.104

Appendix

Table 11: Decoder-independent probing across the 72 OpenML-CC18 tasks.

\Delta

denotes the mean difference between the final and initial thirds of the layer sequence. “Positive” denotes the fraction of datasets with a positive MLP-accuracy difference.

Phase	$\Delta$ ED	$\Delta$ Linear Acc.	$\Delta$ MLP Acc.	$\Delta$ MLP NLL
Expansion	+10.95	+0.0274	+0.0278	-0.291
Plateau	+0.95	+0.0093	+0.0150	-0.011
Compression	-10.55	-0.0040	+0.0045	+0.023
Full depth	+1.40	+0.0327	+0.0473	-0.279

Appendix

Table 12: Phase-wise changes in effective dimensionality and decoder-independent probe performance for TabPFN-3. The phase boundaries are representative analytical boundaries derived from the aggregate ED trajectory.

Threshold	$\rho_{\mathrm{exp}}$	$\rho_{\mathrm{com}}$	Exp. $>1.5$	Comp. $>1.3$
80%	2.41 $\times$	2.80 $\times$	95%	100%
85%	2.40 $\times$	2.57 $\times$	95%	100%
90%	2.45 $\times$	2.24 $\times$	95%	95%
95%	2.54 $\times$	1.84 $\times$	100%	100%

Appendix

Table 13: Sensitivity to the effective-dimensionality threshold. Expansion and compression remain pronounced across variance thresholds from 80% to 95%. The last two columns report the fraction of datasets satisfying

\rho_{\mathrm{exp}}>1.5

and

\rho_{\mathrm{com}}>1.3

, respectively.

Representation	$\rho_{\mathrm{exp}}$	$\rho_{\mathrm{com}}$
Raw	4.00 $\times$	2.41 $\times$
Row normalization	3.70 $\times$	2.35 $\times$
Column standardization	3.16 $\times$	1.69 $\times$
Whitening	4.00 $\times$	2.41 $\times$

Appendix

Table 14: Effective-dimensionality robustness to representation normalization. The expansion–compression structure persists after removing sample-level norm, feature-wise scale, or covariance effects.

Metric	L0	L9	L13	L17	L20	L23	Consistent
Normalized candidate entropy	.997	.994	.995	.993	.992	.978	104/105
Active candidate fraction	.932	.599	.486	.529	.465	.465	103/105
Prototype margin	.055	.125	.104	.129	.147	.247	104/105
Mean correct-class rank	2.082	1.753	1.803	1.727	1.689	1.645	104/105
Correct-class top-rank rate	.634	.731	.710	.729	.739	.770	104/105

Appendix

Table 15: Candidate diagnostics across the 105 multiclass tasks. Entries are equal-weight dataset means. The final column counts tasks whose L23–L0 change matches the expected direction under candidate resolution.

Figure 14: Supplementary diagnostics of candidate structure across depth. (a) Normalized candidate entropy remains high through intermediate depth and decreases most visibly near the output. (b) The active candidate fraction contracts overall but shows a local rebound in the middle regime. (c) Correct-class top-rank rate improves overall, but its intermediate trajectory is non-monotonic. Dashed vertical lines mark the representative analytical boundaries at L9 and L17.

Figure 15: Class information becomes linearly readable before candidate resolution is complete. (a) Mean attn@V probe accuracy across depth, with 95% bootstrap confidence intervals over datasets; uniform@V serves as a matched aggregation baseline. The mean attn@V trajectory peaks at L2. (b) Dataset-level comparison of attn@V readability at L2 and L23. Most datasets lie below the diagonal (101/105), indicating higher readability at L2 than at L23. (c) Distribution of mean probe accuracy across datasets within the Expansion, Plateau, and Compression regimes; diamonds indicate equal-weight dataset means.

Statistic	Value
L0 attn@V accuracy	.460
L2 attn@V accuracy	.718
L23 attn@V accuracy	.600
Mean uniform@V accuracy	.334
Expansion mean	.667
Plateau mean	.664

Appendix

Table 16: Summary statistics for attn@V readability across the 105 multiclass tasks. Unless otherwise stated, values are equal-weight dataset means.

Figure 16: Robustness of the compression–resolution relationship. (a) Relative effective-dimensionality contraction from L20 to L23 versus the corresponding increase in prototype margin across matched datasets. (b) Bootstrap distribution of the Spearman correlation. The solid line marks the observed correlation and dashed lines denote the 95% bootstrap confidence interval.

Statistic	Value
Matched datasets	105
Analysis interval	L20–L23
Spearman $\rho(C_{d},R_{d})$	.511
$p$ -value	$5.85\times 10^{-7}$
Bootstrap 95% CI	$[.324,.664]$
Leave-one-dataset-out $\rho$ range	$[.495,.546]$

Appendix

Table 17: Compression–resolution coupling and robustness checks.

Pattern	Representative form	Assignment criterion
ES	$A\!\to\!A\!\to\!A\!\to\!A$	$p^{(9)}=p^{(23)},\ n_{\rm chg}\leq 1$
TS	$A\!\to\!B\!\to\!A\!\to\!A$	$p^{(9)}=p^{(23)},\ n_{\rm chg}>1$
IS	$A\!\to\!B\!\to\!B\!\to\!B$	$p^{(9)}\neq p^{(23)},\ p^{(13)}=p^{(23)}$
LS	$A\!\to\!A\!\to\!A\!\to\!B$	$p^{(9)}=p^{(13)}\neq p^{(23)}$
MS	$A\!\to\!B\!\to\!A\!\to\!C$	otherwise

Appendix

Table 18: Five prediction-transition patterns. The rules are applied to the complete 24-layer trajectory and do not use the ground-truth label.

Pattern	Pooled samples	Dataset macro-average
ES	72.2%	71.6% $\pm$ 22.3%
TS	15.8%	15.9% $\pm$ 10.9%
IS	5.3%	5.3% $\pm$ 5.2%
LS	5.8%	6.1% $\pm$ 8.0%
MS	1.0%	1.1% $\pm$ 3.5%

Appendix

Table 19: Pattern prevalence under pooled and dataset-macro aggregation. Macro values are mean

\pm

standard deviation across 272 datasets.

Difficulty	ES	TS	IS	LS	MS
Easy	86.3%	9.5%	2.1%	2.0%	0.2%
Medium	74.5%	15.0%	4.0%	5.4%	1.1%
Hard	54.2%	23.2%	9.8%	10.9%	1.9%

Appendix

Table 20: Pattern composition across chance-normalized difficulty tertiles. Tertiles are used only for visualization.

Representative depths	ES	TS	IS	LS	MS
Default	72.2%	15.8%	5.3%	5.8%	1.0%
$L8,L12$	72.1%	15.5%	4.9%	6.5%	1.1%
$L10,L14$	72.3%	16.7%	5.0%	5.0%	1.0%
$L9,L14$	72.2%	15.8%	6.1%	4.9%	1.1%
$L8,L13$	72.1%	15.5%	5.9%	5.5%	1.0%

Appendix

Table 21: Sensitivity of pooled pattern prevalence to representative layers. L23 is fixed as the final layer.

Difficulty definition	$n$	Spearman $\rho$	95% bootstrap CI	$p$
Raw error	272	0.775	[0.702, 0.831]	$3.8\times 10^{-52}$
Chance-normalized	272	0.694	[0.614, 0.761]	$8.2\times 10^{-38}$
Balanced error	272	0.505	[0.389, 0.608]	$7.5\times 10^{-18}$
Majority-normalized	272	0.361	[0.240, 0.469]	$3.1\times 10^{-9}$

Appendix

Table 22: Robustness of the difficulty–commitment association. Bootstrap intervals resample datasets (5,000 replicates). Positive

\rho

means harder datasets stabilize later on average.

Figure 17: Difficulty–commitment association under alternative task difficulty definitions. Points show Spearman correlations with mean stabilization depth; error bars are 95% dataset-bootstrap intervals.

Figure 18: Representation dynamics continue after predictive commitment. Results are computed using Early-Stable samples only; faded curves denote individual datasets and bold curves denote the dataset-level mean. (a) Effective dimensionality continues to increase after L9, rising from 22.8 at L9 to 28.0 at L20 before contracting to 13.3 at L23. (b) Representation movement remains substantial after L9 and is concentrated toward later depth.

Statistic	ES-only result
Analyzed datasets	60
ES queries	23,737
ES prevalence range	88.2–100%
Mean ED at L9	22.8
Mean ED at L20	28.0
Mean ED at L23	13.3

Appendix

Table 23: Summary of ES-only post-commitment representation dynamics. All statistics are computed after restricting the query representations to Early-Stable samples. Displacement fractions are first computed within each dataset and then macro-averaged across datasets.

Pattern	Pooled median [IQR]	Datasets	Dataset median [IQR]
ES	1.52 [1.11, 2.50]	270	1.90 [1.34, 6.26]
TS	1.03 [0.85, 1.34]	263	1.15 [1.05, 1.39]
IS	0.97 [0.76, 1.16]	251	1.03 [0.96, 1.17]
LS	0.96 [0.72, 1.16]	251	1.04 [0.97, 1.17]
MS	0.81 [0.61, 1.01]	93	0.87 [0.71, 1.02]

Appendix

Table 24: Local separation at sample and dataset levels. Sample entries report pooled median [IQR]. Dataset entries report the median [IQR] of within-dataset pattern means.

Figure 19: Dataset-aware local class separation. Each box summarizes within-dataset mean separation ratios for one trajectory pattern. The vertical axis is log-compressed and values above 10 are clipped for display; all reported statistics use the uncapped ratios. Tick labels remain on the original ratio scale. Numbers below the axis indicate datasets containing a finite estimate for that pattern.

Comparison	Shared datasets	Median $\Delta$	ES higher	Holm $p$
ES $>$ TS	263	0.57	94.7%	$<10^{-11}$
ES $>$ IS	251	0.78	96.5%	$<10^{-11}$
ES $>$ LS	251	0.72	94.8%	$<10^{-11}$
ES $>$ MS	93	0.86	94.5%	$<10^{-11}$

Appendix

Table 25: Dataset-aware paired comparisons of local separation.

\Delta

is the median within-dataset difference in mean separation ratio;

p

values use a one-sided paired Wilcoxon test with Holm correction.

Truncation metric	$\rho$	95% CI	$p$
Accuracy, L13	.551	[.443, .643]	$1.55\times 10^{-21}$
Balanced accuracy, L13	.468	[.365, .561]	$3.25\times 10^{-15}$
Accuracy, L17	.481	[.366, .583]	$3.95\times 10^{-16}$
Balanced accuracy, L17	.408	[.300, .504]	$1.32\times 10^{-11}$
Accuracy, L20	.344	[.220, .456]	$1.80\times 10^{-8}$
Balanced accuracy, L20	.308	[.191, .419]	$5.72\times 10^{-7}$

Appendix

Table 26: Robustness of the commitment–truncation association. Spearman correlations relate dataset-level mean stabilization depth to performance loss under truncation. Confidence intervals are obtained by dataset bootstrap.

Depth	Native Acc.	Native NLL	Matched Acc.	Matched NLL
L2	.804	.490	.822	.459
L5	.831	.434	.832	.464
L8	.845	.409	.831	.460
L9	.848	.402	.842	.414
L13	.848	.408	.847	.409
L17	.859	.396	.847	.421

Appendix

Table 27: Native and layer-specific matched readouts across depth. A separate matched readout is trained at each layer to evaluate predictive information available in the corresponding representation.

Figure 20: Retention and rescue across alternative shallow reference layers. (a) Retain,

P(\text{correct at L23}\mid\text{correct at the reference layer})

, and (b) Rescue,

P(\text{correct at L23}\mid\text{wrong at the reference layer})

, plotted against within-dataset local-separation percentile. Results are shown for L8, L9, and L10, with L9 used as the main-text reference. Curves denote cohort means over 272 datasets.

Shallow reference	Retain (%)	Rescue (%)	$\rho_{\text{retain}}$	$\rho_{\text{rescue}}$
L8	83.6 $\rightarrow$ 95.8	32.0 $\rightarrow$ 53.7	+0.203	+0.253
L9	83.2 $\rightarrow$ 95.9	30.4 $\rightarrow$ 50.2	+0.187	+0.244
L10	85.0 $\rightarrow$ 96.1	28.1 $\rightarrow$ 51.9	+0.194	+0.240

Appendix

Table 28: Retention and rescue under different shallow reference layers. Values are cohort means over 272 datasets. The first two columns summarize the endpoint changes across local-separation percentiles, and the final two columns report sample-level Spearman correlations with local separation (

n\!\approx\!9.6\times 10^{4}

; all

p<10^{-230}

Metric	Standard	Shared	$p$
Compression ratio	$2.85\pm 1.01$	$1.05\pm 0.08$	$9.5\times 10^{-7}$
FDR gain	$2.53\pm 1.94$	$0.84\pm 0.26$	$2.9\times 10^{-6}$

Appendix

Table 29: Standard depth-specific computation versus shared-weight iteration. Values are dataset means

\pm

standard deviations. Wilcoxon tests are one-sided for the hypothesis that the standard model yields the larger quantity.

Dataset	Standard 24-layer	Shared $\times 24$
vehicle	$4.00/2.33$	$1.57/1.22$
segment	$4.67/2.33$	$1.33/1.14$
satimage	$7.50/3.33$	$2.50/1.00$
diabetes	$2.88/2.09$	$1.62/1.62$

Appendix

Table 30: Matched-step shared-weight control. Each entry reports expansion/compression ratio. The shared block is iterated for 24 steps, matching the number of transformations in the standard network.

Dataset	Clean	20% label noise	Feature noise	Half samples
adult	+.005	+.030	.000	+.050
cmc	+.030	+.065	+.035	+.090
credit-g	+.030	+.015	+.040	+.085
diabetes	-.019	+.045	+.013	.000
balance-scale	+.232	+.288	+.056	+.256
vehicle	+.312	+.194	+.153	+.171

Appendix

Table 31: Predictive advantage of the standard model under data perturbations. Entries are

\Delta\mathrm{Acc}=\mathrm{Acc}_{\mathrm{standard}}-\mathrm{Acc}_{\mathrm{shared}}

. Positive values indicate higher accuracy for the standard model.

Metric	Learned	Perturbed	Learned better	$p$
Compression, shuffled	1.935	1.053	60/60	$8.65\times 10^{-7}$
Compression, reversed	1.935	1.012	60/60	$8.64\times 10^{-7}$
Accuracy, shuffled	.806	.650	58/60	$1.42\times 10^{-6}$
Accuracy, reversed	.806	.682	56/60	$5.53\times 10^{-6}$

Appendix

Table 32: Dataset-level effects of layer-order perturbation. “Learned better” reports the number of datasets on which the learned ordering yields the larger metric. Ties are omitted from this count.

Explore similar work

May 7, 2026

Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models

Transformer-based tabular foundation models (TFMs) dominate small to medium tabular predictive benchmark tasks, yet their inference mechanisms remain largely unexplored. We present the first large-scale mechanistic study of layerwise dynamics in 6 state-of-the-art tabular in-context learning models. We explore how predictions emerge across depth, identify distinct stages of inference and reveal latent-space dynamics that differ from those of language models. Our findings indicate substantial depthwise redundancy across multiple models, suggesting iterative refinement with overlapping computations during inference stages. Guided by these insights, we design a proof-of-concept, looped single-layer model that uses only 20% of the original model's parameters while achieving comparable performance. The code is available at https://github.com/amirbalef/is_one_layer_enough.

Amir Rezaei Balef, Mykhailo Koshil, Katharina Eggensperger

TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany · University of Tübingen, Tübingen, Germany

Tabular Foundation Models Transformer Interpretability Tabular ICL

Ask PDF

Oct 7, 2026

Thinking in Depth: Retrospective Inference for Tabular Foundation Models

Tabular foundation models (TFMs) are pretrained across diverse tabular tasks and make predictions on a new table at inference time using its labeled examples as context. Most recent TFMs perform such in-context prediction with stacked Transformer layers, repeatedly transforming how examples are represented and compared. By tracing individual queries through several strong TFMs, we find that predictive refinement is highly uneven across depth and is often concentrated in later layers. This uneven refinement motivates us to reconsider how intermediate representations are constructed and reused throughout the network. We introduce Retro, a tabular foundation model based on retrospective inference, where later stages can explicitly revisit and recombine intermediate information produced earlier in the network. Retro organizes this process around two complementary operations: which intermediate information to revisit, and how the resulting contextual update should be shaped for each query. Attention Residuals address the former by adaptively reweighting contributions from different depths, while query-conditioned Gated Attention addresses the latter by modulating the attention output element-wise across representation dimensions. Our analysis shows that Retro shifts predictive refinement earlier and more broadly across depth, with different stages revising different subsets of queries in a pattern suggestive of multi-view refinement. Across TabArena, TALENT, and RelArena, Retro ranks among the top three and lies on the Pareto frontier. These results indicate that directly reusing intermediate representations provides a practical way to better exploit depth in TFMs.

Hao-Run Cai, Si-Yang Liu, Zi-Jian Cheng +8

School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University

Tabular Foundation Models Tabular ICL Tabular Representation Learning

Ask PDF

Oct 9, 2026

Tabular Foundation Models Are Effectively Shallow

In-context-learning Tabular Foundation Models (TFMs) are uniquely suited to scenarios with limited resources or privacy constraints, as they solve new tasks without additional training. However, their architecture consists of multiple transformer blocks, resulting in high computational and memory footprints. Meanwhile, traditional alternatives like Gradient-Boosted Decision Trees (GBDTs) remain highly competitive and run efficiently on CPUs, suggesting that tabular tasks may not inherently require deep architectures. This contrast raises a fundamental question: can TFMs be simplified by removing architectural redundancies without sacrificing performance? To answer this, we introduce a closed-form, label-free affine translator that replaces a contiguous window of up to 96% of the blocks, leaving the rest unchanged. Evaluating eight frontier TFMs on 171 classification datasets, we reveal that tabular transformers are vastly overparameterized: replacing half of a model's blocks costs at most

0.002

median AUROC across all architectures, with the half-depth TabFM still outperforming every other full-depth TFMs and tuned GBDTs. Furthermore, different model families are overparameterized in different ways: TabPFN-3 relies on its encoders, maintaining performance regardless of which single block is retained. Other single-axis models spread computation thinly, needing only their first block, while dual-axis models concentrate it in one critical block. Ultimately, removing depth saves up to 1.5B parameters, speeds up GPU inference by up to

6\times

at a median cost of

0.03

AUROC, while a single-block TabPFN-3 serves cached queries

2.4\times

faster using

6\times

less memory at negligible cost.

Irene Cannistraci, Julia E. Vogt

Department of Computer Science ETH Zurich Switzerland

Tabular Foundation Models Tabular ICL Model Compression Efficient Transformer Inference Neural Network Pruning

Ask PDF