cs.LGOct 14, 2026

When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models

Authors: Wenwen Gong, Haotian Gu, Zhen Yang, Jiaying Ma, Hanqing Zhang, Zhongshen Wang, Jingjun Wang, Shouyi Xu, +1 more

Organizations: China Agricultural University · Tsinghua University · Nankai University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Transformer-based tabular foundation models (TFMs) achieve state-of-the-art performance on small to medium-sized tabular prediction tasks while processing diverse inputs through a shared stack of layers. Recent layerwise analyses identify strong predictive performance in early TFM representations and substantial overlap in computation across layers. However, early predictive accessibility alone does not establish whether later layers are dispensable for every input, and aggregate accuracy can obscure which inputs benefit from additional computation. We study all 24 in-context-learning layers of TabPFN-3 across 272 classification datasets, combining representation geometry, prediction trajectories, truncation, and structural interventions, while also examining related representation dynamics in four additional TFMs. We uncover a pronounced expansion-compression trajectory, with stronger late dimensional contraction associated with larger increases in class-prototype separation. A targeted analysis of early-stabilizing samples shows that substantial representational changes persist after predicted classes stabilize, while locally ambiguous queries obtain larger accuracy gains from shallow to late layers than well-separated queries. Repeating a single pretrained block or perturbing layer order reduces predictive performance on average, constraining simple post-hoc substitutions in the frozen model. These findings distinguish predictive accessibility, class stability, and the benefits of continuing computation, supporting an input-dependent assessment of depthwise redundancy in TFMs.

Figures & tables

Figure 1: Predictive accessibility, class stability, and the value of depth are distinct. (a) L9/L13/L23 accuracy vs. local class separation: ambiguous queries gain more from depth. (b) Effective dimensionality per layer for early-stabilizing queries: representations keep changing after the class stabilises, then contract sharply.
Figure 2: Representational expansion followed by late compression. Thin curves denote individual datasets and the thick curve denotes the dataset-level mean.
Figure 3: Representation transformation dynamics across depth. (a) Mean pairwise layer similarity, with linear CKA above the diagonal and cosine similarity below. (b) Adjacent-layer displacement measured in a shared PCA basis. (c) Dataset-level displacement across the expansion, plateau, and compression regimes.
Figure 4: Candidate separation strengthens at late depth. (a) Prototype margins remain small through the early and middle layers and widen most strongly near the output. (b) Correct-class alignment emerges earlier on easier tasks, whereas harder tasks retain candidate ambiguity deeper into the network. (c) Stronger late effective-dimensionality contraction is associated with larger increases in prototype margin.
Figure 5: Sample-level predictive commitment across depth. (a) Cumulative fraction of samples stabilized to the final prediction across layers, shown per dataset and in aggregate. (b) Five recurring prediction-transition patterns and their pooled prevalence. (c) Pattern composition across tertiles of chance-normalized task difficulty; statistical analyses use difficulty as a continuous variable.
Figure 6: Local class separation across prediction-transition patterns. Separation decreases from ES to MS, indicating local ambiguity for more dynamic trajectories.
Figure 7: Depth has selective utility across samples and tasks. (a) Accuracy at L9, L13, and L23 across within-dataset local-separation percentiles; locally ambiguous samples show larger gains with depth. (b) Retention of L9-correct predictions and rescue of L9 errors across local-separation percentiles; both increase with separation. (c) Mean stabilization depth is positively associated with L13 truncation loss. (d) Accuracy increases with depth under layer-matched readouts.
Figure 8: Shared-weight repetition produces a distinct depthwise trajectory. (a) Shared-weight repetition weakens compression and class-discriminative growth. (b) The repeated-block trajectory remains distinct at matched computational depth. (c) The full model retains a predictive advantage under perturbation.
Figure 9: Layer order is functionally consequential. (a) Shuffling or reversing the learned sequence disrupts the depthwise representation dynamics. (b) Reordering reduces predictive performance on average.
Appendix figures & tables43 assets

Supplementary material from the paper’s appendix.

Appendix

ComponentHyperparameterValue
Stage 1: Feature-distribution embedding
 Base embedding dimension128
 Features per circular-shift group3
 Induced-attention blocks3
 Attention heads8
 Inducing points per column128
Appendix
Table 1: Architecture of TabPFN-3 used in the primary analysis. Layer indices L0–L23 refer exclusively to the 24 Stage-3 ICL blocks.
ModelParametersICL depthICL dim.Pre-ICL stagesNormalization
TabPFN-v27.24M12192Alternating feature/item attentionPost-LN
TabPFN-2.510.72M24192Alternating feature/item attentionPost-LN
TabPFN-353M24512Column embedding + row aggregationRMSNorm
TabICL27.05M12512TFcol (3) + TFrow (3)Pre-LN
TabICLv2–12512TFcol (3) + TFrow (3)Pre-LN
Appendix
Table 2: Architectures used in the cross-model analyses. “ICL depth” denotes the sequence of transformer blocks over which layerwise representations are analyzed. Parameter counts correspond to the classification models where available.
ComponentTabPFN-v2TabPFN-2.5
Total parameters7,244,5547{,}244{,}55410,718,21810{,}718{,}218
Feature encoder7681,152
Label encoder576576
Positional embedding9,4089,408
Transformer depth1224
Transformer-block parameters7,077,8887{,}077{,}88810,616,83210{,}616{,}832
Appendix
Table 3: Component-level architecture of TabPFN-v2 and TabPFN-2.5.
StageComponentParameters
Column embedderTotal877,824
Input linear256
Transformer blocks (3)844,032
Each block281,344
Attention 166,048+16,51266{,}048+16{,}512
MLP 133,024+32,89633{,}024+32{,}896
Appendix
Table 4: Component-level architecture of TabICL.
StageHyperparameterValue
TFcol\mathrm{TF}_{\rm col}Layers3
Inducing vectors128
Model dimension128
Attention heads8
TFrow\mathrm{TF}_{\rm row}Layers3
Model dimension128
Appendix
Table 5: Architecture of TabICLv2 used in the cross-model analysis.
LayerRole in subsequent analyses
L9Early-depth anchor; end of the expansion summary interval
L13Intermediate plateau anchor
L17Late-depth anchor; start of the compression summary interval
L20Late-depth anchor near the aggregate ED maximum
L23Final-layer representation
Appendix
Table 6: Representative TabPFN-3 layers used in subsequent analyses. The Expansion–Compression trajectory is characterized from the complete layerwise measurements; these depths serve as recurring analytical anchors.
Figure 10: Cross-model representation dynamics. Results are shown for TabPFN-v2, TabPFN-2.5, TabICL, and TabICLv2 on matched datasets. (a) Effective dimensionality normalized by the first-layer value within each dataset–model pair. (b) Adjacent-layer PCA displacement expressed as a fraction of the total trajectory displacement. (c) Fisher discriminant ratio across depth. Thin curves denote individual datasets and thick curves denote the cross-dataset mean. Mean curves are interpolated only for visual smoothness; all statistics use the original layerwise measurements. Peak annotations in panel (a) refer to the cross-dataset mean trajectories and therefore need not equal the mean within-dataset expansion ratios reported in Table 7 .
ModelDepthExpansionCompressionFinal EDLate PCA
TabPFN-v2123.61×3.61\times1.43×1.43\times2.592.5927.8%27.8\%
TabPFN-2.5243.87×3.87\times1.87×1.87\times2.222.2244.4%44.4\%
TabICL121.82×1.82\times3.88×3.88\times0.620.6264.9%64.9\%
TabICLv2123.01×3.01\times3.59×3.59\times1.221.2283.8%83.8\%
Appendix
Table 7: Summary of cross-model representation dynamics. Expansion and compression ratios are first computed within each dataset and then averaged across matched datasets. Final ED denotes the final effective dimensionality normalized by the first-layer value and is likewise averaged from dataset-level values. Because these quantities are averaged independently, their cross-dataset means need not satisfy exact algebraic ratio identities. Late PCA reports the fraction of total PCA displacement occurring in the final quarter of layer transitions.
Figure 11: Pairwise representation similarity across architectures. For each model, the upper triangular matrix reports mean linear CKA between every pair of layers, while the lower triangular matrix reports mean cosine similarity. Darker entries indicate greater similarity.
DatasetPretrainedRandomized ICL blocksFully randomized
vehicle4.00×/2.33×4.00\times/2.33\times1.00×/1.00×1.00\times/1.00\times1.00×/1.00×1.00\times/1.00\times
segment4.67×/2.33×4.67\times/2.33\times1.00×/1.00×1.00\times/1.00\times1.00×/1.00×1.00\times/1.00\times
satimage7.50×/3.33×7.50\times/3.33\times1.00×/1.00×1.00\times/1.00\times1.00×/1.00×1.00\times/1.00\times
diabetes2.88×/2.09×2.88\times/2.09\times1.00×/1.00×1.00\times/1.00\times1.00×/1.00×1.00\times/1.00\times
Appendix
Table 8: Random-weight control for TabPFN-3. Each entry reports expansion ratio / compression ratio. The pretrained model exhibits substantial Expansion–Compression, whereas randomizing either the ICL stack or the complete model yields flat effective-dimensionality trajectories on the tested datasets.
DepthED trajectoryExpansionCompression
3 layers1→2→21\rightarrow 2\rightarrow 22.00×2.00\times1.00×1.00\times
4 layers1→1→2→21\rightarrow 1\rightarrow 2\rightarrow 22.00×2.00\times1.00×1.00\times
5 layers1→2→2→2→11\rightarrow 2\rightarrow 2\rightarrow 2\rightarrow 12.00×2.00\times2.00×2.00\times
Appendix
Table 9: Controlled-depth experiment with nanoTabPFN. The three- and four-layer variants exhibit representation expansion without a subsequent compression phase, whereas the five-layer variant exhibits both expansion and compression.
MeasurementFT-Transformer
Mean expansion ratio1.11×1.11\times
Mean compression ratio2.33×2.33\times
Characteristic trajectoryPredominantly contraction
Appendix
Table 10: Representation geometry in the supervised FT-Transformer control. The supervised model exhibits substantial effective-dimensionality contraction without the pronounced preceding expansion characteristic of the Expansion–Compression trajectory.
Figure 12: Native-decoder and layer-matched readout performance across depth. Left: classification accuracy averaged over 72 OpenML-CC18 tasks. Right: negative log-likelihood. Error bars denote 95% bootstrap confidence intervals over datasets. Vertical dotted lines at L9 and L17 indicate representative analytical boundaries rather than estimated change points.
Figure 13: Decoder-independent predictive decodability across depth. (a) Linear and capacity-controlled MLP probe accuracy across the 24 ICL layers of TabPFN-3, averaged over 72 OpenML-CC18 tasks. Error bars denote 95% bootstrap confidence intervals over datasets; dotted vertical lines indicate the representative L9 and L17 analytical boundaries. (b) Change in MLP-probe accuracy from the first to the final third of the ICL layers for TabPFN-3, TabICL, and TabICLv2. Differences are computed within each dataset before aggregation, and error bars denote 95% bootstrap confidence intervals over the 72 tasks.
ModelΔ\Delta MLP Acc.PositiveΔ\Delta ECEΔ\Delta MLP NLL
TabPFN-3+0.02764%-0.005-0.056
TabICL+0.02966%-0.017-0.179
TabICLv2+0.02059%-0.008-0.104
Appendix
Table 11: Decoder-independent probing across the 72 OpenML-CC18 tasks. Δ\Delta denotes the mean difference between the final and initial thirds of the layer sequence. “Positive” denotes the fraction of datasets with a positive MLP-accuracy difference.
PhaseΔ\Delta EDΔ\Delta Linear Acc.Δ\Delta MLP Acc.Δ\Delta MLP NLL
Expansion+10.95+0.0274+0.0278-0.291
Plateau+0.95+0.0093+0.0150-0.011
Compression-10.55-0.0040+0.0045+0.023
Full depth+1.40+0.0327+0.0473-0.279
Appendix
Table 12: Phase-wise changes in effective dimensionality and decoder-independent probe performance for TabPFN-3. The phase boundaries are representative analytical boundaries derived from the aggregate ED trajectory.
Thresholdρexp\rho_{\mathrm{exp}}ρcom\rho_{\mathrm{com}}Exp. >1.5>1.5Comp. >1.3>1.3
80%2.41 ×\times2.80 ×\times95%100%
85%2.40 ×\times2.57 ×\times95%100%
90%2.45 ×\times2.24 ×\times95%95%
95%2.54 ×\times1.84 ×\times100%100%
Appendix
Table 13: Sensitivity to the effective-dimensionality threshold. Expansion and compression remain pronounced across variance thresholds from 80% to 95%. The last two columns report the fraction of datasets satisfying ρexp>1.5\rho_{\mathrm{exp}}>1.5 and ρcom>1.3\rho_{\mathrm{com}}>1.3 , respectively.
Representationρexp\rho_{\mathrm{exp}}ρcom\rho_{\mathrm{com}}
Raw4.00 ×\times2.41 ×\times
Row normalization3.70 ×\times2.35 ×\times
Column standardization3.16 ×\times1.69 ×\times
Whitening4.00 ×\times2.41 ×\times
Appendix
Table 14: Effective-dimensionality robustness to representation normalization. The expansion–compression structure persists after removing sample-level norm, feature-wise scale, or covariance effects.
MetricL0L9L13L17L20L23Consistent
Normalized candidate entropy.997.994.995.993.992.978104/105
Active candidate fraction.932.599.486.529.465.465103/105
Prototype margin.055.125.104.129.147.247104/105
Mean correct-class rank2.0821.7531.8031.7271.6891.645104/105
Correct-class top-rank rate.634.731.710.729.739.770104/105
Appendix
Table 15: Candidate diagnostics across the 105 multiclass tasks. Entries are equal-weight dataset means. The final column counts tasks whose L23–L0 change matches the expected direction under candidate resolution.
Figure 14: Supplementary diagnostics of candidate structure across depth. (a) Normalized candidate entropy remains high through intermediate depth and decreases most visibly near the output. (b) The active candidate fraction contracts overall but shows a local rebound in the middle regime. (c) Correct-class top-rank rate improves overall, but its intermediate trajectory is non-monotonic. Dashed vertical lines mark the representative analytical boundaries at L9 and L17.
Figure 15: Class information becomes linearly readable before candidate resolution is complete. (a) Mean attn@V probe accuracy across depth, with 95% bootstrap confidence intervals over datasets; uniform@V serves as a matched aggregation baseline. The mean attn@V trajectory peaks at L2. (b) Dataset-level comparison of attn@V readability at L2 and L23. Most datasets lie below the diagonal (101/105), indicating higher readability at L2 than at L23. (c) Distribution of mean probe accuracy across datasets within the Expansion, Plateau, and Compression regimes; diamonds indicate equal-weight dataset means.
StatisticValue
L0 attn@V accuracy.460
L2 attn@V accuracy.718
L23 attn@V accuracy.600
Mean uniform@V accuracy.334
Expansion mean.667
Plateau mean.664
Appendix
Table 16: Summary statistics for attn@V readability across the 105 multiclass tasks. Unless otherwise stated, values are equal-weight dataset means.
Figure 16: Robustness of the compression–resolution relationship. (a) Relative effective-dimensionality contraction from L20 to L23 versus the corresponding increase in prototype margin across matched datasets. (b) Bootstrap distribution of the Spearman correlation. The solid line marks the observed correlation and dashed lines denote the 95% bootstrap confidence interval.
StatisticValue
Matched datasets105
Analysis intervalL20–L23
Spearman ρ(Cd,Rd)\rho(C_{d},R_{d}).511
pp -value5.85×10−75.85\times 10^{-7}
Bootstrap 95% CI[.324,.664][.324,.664]
Leave-one-dataset-out ρ\rho range[.495,.546][.495,.546]
Appendix
Table 17: Compression–resolution coupling and robustness checks.
PatternRepresentative formAssignment criterion
ESA ⁣→ ⁣A ⁣→ ⁣A ⁣→ ⁣AA\!\to\!A\!\to\!A\!\to\!Ap(9)=p(23), nchg≤1p^{(9)}=p^{(23)},\ n_{\rm chg}\leq 1
TSA ⁣→ ⁣B ⁣→ ⁣A ⁣→ ⁣AA\!\to\!B\!\to\!A\!\to\!Ap(9)=p(23), nchg>1p^{(9)}=p^{(23)},\ n_{\rm chg}>1
ISA ⁣→ ⁣B ⁣→ ⁣B ⁣→ ⁣BA\!\to\!B\!\to\!B\!\to\!Bp(9)≠p(23), p(13)=p(23)p^{(9)}\neq p^{(23)},\ p^{(13)}=p^{(23)}
LSA ⁣→ ⁣A ⁣→ ⁣A ⁣→ ⁣BA\!\to\!A\!\to\!A\!\to\!Bp(9)=p(13)≠p(23)p^{(9)}=p^{(13)}\neq p^{(23)}
MSA ⁣→ ⁣B ⁣→ ⁣A ⁣→ ⁣CA\!\to\!B\!\to\!A\!\to\!Cotherwise
Appendix
Table 18: Five prediction-transition patterns. The rules are applied to the complete 24-layer trajectory and do not use the ground-truth label.
PatternPooled samplesDataset macro-average
ES72.2%71.6% ±\pm 22.3%
TS15.8%15.9% ±\pm 10.9%
IS5.3%5.3% ±\pm 5.2%
LS5.8%6.1% ±\pm 8.0%
MS1.0%1.1% ±\pm 3.5%
Appendix
Table 19: Pattern prevalence under pooled and dataset-macro aggregation. Macro values are mean ±\pm standard deviation across 272 datasets.
DifficultyESTSISLSMS
Easy86.3%9.5%2.1%2.0%0.2%
Medium74.5%15.0%4.0%5.4%1.1%
Hard54.2%23.2%9.8%10.9%1.9%
Appendix
Table 20: Pattern composition across chance-normalized difficulty tertiles. Tertiles are used only for visualization.
Representative depthsESTSISLSMS
Default72.2%15.8%5.3%5.8%1.0%
L8,L12L8,L1272.1%15.5%4.9%6.5%1.1%
L10,L14L10,L1472.3%16.7%5.0%5.0%1.0%
L9,L14L9,L1472.2%15.8%6.1%4.9%1.1%
L8,L13L8,L1372.1%15.5%5.9%5.5%1.0%
Appendix
Table 21: Sensitivity of pooled pattern prevalence to representative layers. L23 is fixed as the final layer.
Difficulty definitionnnSpearman ρ\rho95% bootstrap CIpp
Raw error2720.775[0.702, 0.831]3.8×10−523.8\times 10^{-52}
Chance-normalized2720.694[0.614, 0.761]8.2×10−388.2\times 10^{-38}
Balanced error2720.505[0.389, 0.608]7.5×10−187.5\times 10^{-18}
Majority-normalized2720.361[0.240, 0.469]3.1×10−93.1\times 10^{-9}
Appendix
Table 22: Robustness of the difficulty–commitment association. Bootstrap intervals resample datasets (5,000 replicates). Positive ρ\rho means harder datasets stabilize later on average.
Figure 17: Difficulty–commitment association under alternative task difficulty definitions. Points show Spearman correlations with mean stabilization depth; error bars are 95% dataset-bootstrap intervals.
Figure 18: Representation dynamics continue after predictive commitment. Results are computed using Early-Stable samples only; faded curves denote individual datasets and bold curves denote the dataset-level mean. (a) Effective dimensionality continues to increase after L9, rising from 22.8 at L9 to 28.0 at L20 before contracting to 13.3 at L23. (b) Representation movement remains substantial after L9 and is concentrated toward later depth.
StatisticES-only result
Analyzed datasets60
ES queries23,737
ES prevalence range88.2–100%
Mean ED at L922.8
Mean ED at L2028.0
Mean ED at L2313.3
Appendix
Table 23: Summary of ES-only post-commitment representation dynamics. All statistics are computed after restricting the query representations to Early-Stable samples. Displacement fractions are first computed within each dataset and then macro-averaged across datasets.
PatternPooled median [IQR]DatasetsDataset median [IQR]
ES1.52 [1.11, 2.50]2701.90 [1.34, 6.26]
TS1.03 [0.85, 1.34]2631.15 [1.05, 1.39]
IS0.97 [0.76, 1.16]2511.03 [0.96, 1.17]
LS0.96 [0.72, 1.16]2511.04 [0.97, 1.17]
MS0.81 [0.61, 1.01]930.87 [0.71, 1.02]
Appendix
Table 24: Local separation at sample and dataset levels. Sample entries report pooled median [IQR]. Dataset entries report the median [IQR] of within-dataset pattern means.
Figure 19: Dataset-aware local class separation. Each box summarizes within-dataset mean separation ratios for one trajectory pattern. The vertical axis is log-compressed and values above 10 are clipped for display; all reported statistics use the uncapped ratios. Tick labels remain on the original ratio scale. Numbers below the axis indicate datasets containing a finite estimate for that pattern.
ComparisonShared datasetsMedian Δ\DeltaES higherHolm pp
ES >> TS2630.5794.7%<10−11<10^{-11}
ES >> IS2510.7896.5%<10−11<10^{-11}
ES >> LS2510.7294.8%<10−11<10^{-11}
ES >> MS930.8694.5%<10−11<10^{-11}
Appendix
Table 25: Dataset-aware paired comparisons of local separation. Δ\Delta is the median within-dataset difference in mean separation ratio; pp values use a one-sided paired Wilcoxon test with Holm correction.
Truncation metricρ\rho95% CIpp
Accuracy, L13.551[.443, .643]1.55×10−211.55\times 10^{-21}
Balanced accuracy, L13.468[.365, .561]3.25×10−153.25\times 10^{-15}
Accuracy, L17.481[.366, .583]3.95×10−163.95\times 10^{-16}
Balanced accuracy, L17.408[.300, .504]1.32×10−111.32\times 10^{-11}
Accuracy, L20.344[.220, .456]1.80×10−81.80\times 10^{-8}
Balanced accuracy, L20.308[.191, .419]5.72×10−75.72\times 10^{-7}
Appendix
Table 26: Robustness of the commitment–truncation association. Spearman correlations relate dataset-level mean stabilization depth to performance loss under truncation. Confidence intervals are obtained by dataset bootstrap.
DepthNative Acc.Native NLLMatched Acc.Matched NLL
L2.804.490.822.459
L5.831.434.832.464
L8.845.409.831.460
L9.848.402.842.414
L13.848.408.847.409
L17.859.396.847.421
Appendix
Table 27: Native and layer-specific matched readouts across depth. A separate matched readout is trained at each layer to evaluate predictive information available in the corresponding representation.
Figure 20: Retention and rescue across alternative shallow reference layers. (a) Retain, P(correct at L23∣correct at the reference layer)P(\text{correct at L23}\mid\text{correct at the reference layer}) , and (b) Rescue, P(correct at L23∣wrong at the reference layer)P(\text{correct at L23}\mid\text{wrong at the reference layer}) , plotted against within-dataset local-separation percentile. Results are shown for L8, L9, and L10, with L9 used as the main-text reference. Curves denote cohort means over 272 datasets.
Shallow referenceRetain (%)Rescue (%)ρretain\rho_{\text{retain}}ρrescue\rho_{\text{rescue}}
L883.6 →\rightarrow 95.832.0 →\rightarrow 53.7+0.203+0.253
L983.2 →\rightarrow 95.930.4 →\rightarrow 50.2+0.187+0.244
L1085.0 →\rightarrow 96.128.1 →\rightarrow 51.9+0.194+0.240
Appendix
Table 28: Retention and rescue under different shallow reference layers. Values are cohort means over 272 datasets. The first two columns summarize the endpoint changes across local-separation percentiles, and the final two columns report sample-level Spearman correlations with local separation ( n ⁣≈ ⁣9.6×104n\!\approx\!9.6\times 10^{4} ; all p<10−230p<10^{-230} ).
MetricStandardSharedpp
Compression ratio2.85±1.012.85\pm 1.011.05±0.081.05\pm 0.089.5×10−79.5\times 10^{-7}
FDR gain2.53±1.942.53\pm 1.940.84±0.260.84\pm 0.262.9×10−62.9\times 10^{-6}
Appendix
Table 29: Standard depth-specific computation versus shared-weight iteration. Values are dataset means ±\pm standard deviations. Wilcoxon tests are one-sided for the hypothesis that the standard model yields the larger quantity.
DatasetStandard 24-layerShared ×24\times 24
vehicle4.00/2.334.00/2.331.57/1.221.57/1.22
segment4.67/2.334.67/2.331.33/1.141.33/1.14
satimage7.50/3.337.50/3.332.50/1.002.50/1.00
diabetes2.88/2.092.88/2.091.62/1.621.62/1.62
Appendix
Table 30: Matched-step shared-weight control. Each entry reports expansion/compression ratio. The shared block is iterated for 24 steps, matching the number of transformations in the standard network.
DatasetClean20% label noiseFeature noiseHalf samples
adult+.005+.030.000+.050
cmc+.030+.065+.035+.090
credit-g+.030+.015+.040+.085
diabetes-.019+.045+.013.000
balance-scale+.232+.288+.056+.256
vehicle+.312+.194+.153+.171
Appendix
Table 31: Predictive advantage of the standard model under data perturbations. Entries are ΔAcc=Accstandard−Accshared\Delta\mathrm{Acc}=\mathrm{Acc}_{\mathrm{standard}}-\mathrm{Acc}_{\mathrm{shared}} . Positive values indicate higher accuracy for the standard model.
MetricLearnedPerturbedLearned betterpp
Compression, shuffled1.9351.05360/608.65×10−78.65\times 10^{-7}
Compression, reversed1.9351.01260/608.64×10−78.64\times 10^{-7}
Accuracy, shuffled.806.65058/601.42×10−61.42\times 10^{-6}
Accuracy, reversed.806.68256/605.53×10−65.53\times 10^{-6}
Appendix
Table 32: Dataset-level effects of layer-order perturbation. “Learned better” reports the number of datasets on which the learned ordering yields the larger metric. Ties are omitted from this count.

Explore similar work

CardsList

  1. Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models

    May 7, 2026Amir Rezaei Balef, Mykhailo Koshil, Katharina EggenspergerTabular Foundation ModelsTransformer Interpretability

    TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany · University of Tübingen, Tübingen, Germany

  2. Thinking in Depth: Retrospective Inference for Tabular Foundation Models

    Oct 7, 2026Hao-Run Cai, Si-Yang Liu, Zi-Jian Cheng +8Tabular Foundation ModelsTabular ICL

    School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University

  3. Tabular Foundation Models Are Effectively Shallow

    Oct 9, 2026Irene Cannistraci, Julia E. VogtTabular Foundation ModelsTabular ICL

    Department of Computer Science ETH Zurich Switzerland