Noise Immunity in In-Context Tabular Learning: An Empirical Robustness Analysis of TabPFN's Attention Mechanisms

James Hu; Mahdi Ghelichi

cs.LGApr 6, 2026

Noise Immunity in In-Context Tabular Learning: An Empirical Robustness Analysis of TabPFN's Attention Mechanisms

Authors: James Hu, Mahdi Ghelichi

Organizations: Model Development Innovation, Risk Management TD Bank, Toronto, Canada

Abstract

Tabular foundation models (TFMs) such as TabPFN (Tabular Prior-Data Fitted Network) are designed to generalize across heterogeneous tabular datasets through in-context learning (ICL). They perform prediction in a single forward pass conditioned on labeled examples without dataset-specific parameter updates. This paradigm is particularly attractive in industrial domains (e.g., finance and healthcare) where tabular prediction is pervasive. Retraining a bespoke model for each new table can be costly or infeasible in these settings, while data quality issues such as irrelevant predictors, correlated feature groups, and label noise are common. In this paper, we provide strong empirical evidence that TabPFN is highly robust under these sub-optimal conditions. We study TabPFN and its attention mechanisms for binary classification problems with controlled synthetic perturbations that vary: (i) dataset width by injecting random uncorrelated features and by introducing nonlinearly correlated features, (ii) dataset size by increasing the number of training rows, and (iii) label quality by increasing the fraction of mislabeled targets. Beyond predictive performance, we analyze internal signals including attention concentration and attention-based feature ranking metrics. Across these parametric tests, TabPFN is remarkably resilient: ROC-AUC remains high, attention stays structured and sharp, and informative features are highly ranked by attention-based metrics. Qualitative visualizations with attention heatmaps, feature-token embeddings, and SHAP plots further support a consistent pattern across layers in which TabPFN increasingly concentrates on useful features while separating their signals from noise. Together, these findings suggest that TabPFN is a robust TFM capable of maintaining both predictive performance and coherent internal behavior under various scenarios of data imperfections.

Figures & tables

Panel of Figure 1 — Figure 1: Feature-wise attention weights heatmaps in layer {3, 6, 9, 12} of TabPFN for the baseline case. In each plot, the first 2 indices represent informative features and last index is the target label.

Panel of Figure 2 — Figure 2: Feature-token embeddings in layer {3, 6, 9, 12} of TabPFN for the baseline case. In each plot, the green points represent informative features and gray points are random features.

Panel of Figure 3 — Figure 3: SHAP plots of TabPFN for the baseline case. First two indices represent informative features, and the rest are random features. Left: aggregate SHAP values across samples for all features; right: per-sample SHAP values distribution for all features.

Panel of Figure 4 — Figure 4: Performance and attention metrics of TabPFN with respect to increasing number of random features. Top left: ROC-AUC; top right: attention proportions and attention ratio; bottom left: KL-divergence with respect to different distributions; bottom right: average attention rank of informative features and their proportion in top-2 attended features.

Panel of Figure 5 — Figure 5: Feature-wise attention weights heatmaps in layer 12 of TabPFN. In each plot, the first 2 indices represent informative features and last index is the target label. Left: total number of features = 16; right: total number of features = 256, the first 16 features and the target label are shown.

Panel of Figure 6 — Figure 6: Performance and attention metrics of TabPFN with respect to increasing number of nonlinearly correlated features. Top left : ROC-AUC; top right: attention proportions and attention ratio; bottom left: KL-divergence with respect to different distributions; bottom right: average attention rank of informative features and their proportion in top-2 attended features.

Panel of Figure 7 — Figure 7: SHAP plots of TabPFN when the number of correlated features = 8. First two indices represent informative features, subsequent 8 indices are correlated features, and the rest are random features. Left: aggregate SHAP values across samples for all features; right: per-sample SHAP values distribution for top-20 features.

Panel of Figure 8 — Figure 8: Performance and attention metrics of TabPFN with respect to increasing the number of rows. Top left: ROC-AUC; top right: attention proportions and attention ratio; bottom left: KL-divergence with respect to different distributions; bottom right: average attention rank of informative features and their proportion in top-2 attended features.

Panel of Figure 9 — Figure 9: Feature-wise attention weights heatmaps and feature-token embeddings in layer {3, 12} of TabPFN when number of rows = 12000. In each attention heatmap plot, the first 2 indices represent informative features and last index is the target label. In each embeddings plot, the green points represent informative features and gray points are random features. Top left: attention heatmap in layer 3; top right: attention heatmap in layer 12; bottom left: feature embedding plot in layer 3; bottom right: feature embedding plot in layer 12.

Panel of Figure 10 — Figure 10: Performance and attention metrics of TabPFN with respect to increasing proportion of label noise. Top left: ROC-AUC; top right: attention proportions and attention ratio; bottom left: KL-divergence with respect to different distributions; bottom right: average attention rank of informative features and their proportion in top-2 attended features.

Panel of Figure 11 — Figure 11: Feature-wise attention weights heatmaps in layer {3, 12} of TabPFN when 35% of targets are mislabeled. In each plot, the first 2 indices represent informative features and last index is the target label. Left: in layer 3; right: in layer 12.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

ROC AUC
Data generation process	Total number of features
	8	16	32	64	128	256	512
Different seed	0.995	0.992	0.996	0.990	0.996	0.997	0.997
Larger number of clusters	0.986	0.985	0.981	0.987	0.978	0.985	0.984
Smaller class separation	0.888	0.890	0.882	0.886	0.886	0.874	0.875

Appendix

Table 1: ROC AUC using different data generation processes and various total numbers of features.

Attention Ratio of Informative over Random
Data generation process	Total number of features
	8	16	32	64	128	256	512
Different seed	47.997	41.788	159.154	178.965	264.194	227.691	167.835
Larger number of clusters	61.505	131.740	208.987	192.664	317.481	191.223	126.268
Smaller class separation	65.657	68.450	152.730	122.236	180.921	122.194	141.558

Appendix

Table 2: Attention ratio of informative features over random features in layer 12 using different data generation processes and various total numbers of features.

ROC AUC
Data generation process	Number of correlated features
	2	4	8	16	32	64	128
Different seed	0.997	0.997	0.997	0.997	0.997	0.997	0.997
Larger number of clusters	0.985	0.986	0.986	0.985	0.985	0.985	0.984
Smaller class separation	0.875	0.874	0.875	0.876	0.875	0.873	0.873

Appendix

Table 3: ROC AUC using different data generation processes and various numbers of correlated features.

Attention Ratio of Informative over Correlated+Random
Data generation process	Number of correlated features
	2	4	8	16	32	64	128
Different seed	81.124	67.135	63.581	50.756	19.162	15.262	6.182
Larger number of clusters	73.133	74.398	55.768	44.370	32.758	26.490	21.276
Smaller class separation	150.600	67.238	51.935	23.602	13.451	12.371	11.122

Appendix

Table 4: Attention ratio of informative features over correlated and random features in layer 12 using different data generation processes and various numbers of correlated features.

ROC AUC
Data generation process	Total number of training rows
	3000	4500	6000	7500	9000	10500	12000
Different seed	0.993	0.996	0.994	0.983	0.995	0.994	0.994
Larger number of clusters	0.984	0.983	0.972	0.984	0.983	0.991	0.982
Smaller class separation	0.978	0.995	0.928	0.991	0.926	0.873	0.968

Appendix

Table 5: ROC AUC using different data generation processes and various total number of training rows.

Attention Ratio of Informative over Random
Data generation process	Total number of training rows
	3000	4500	6000	7500	9000	10500	12000
Different seed	60.895	53.664	48.727	78.906	48.727	34.798	86.640
Larger number of clusters	60.354	93.889	90.220	79.762	88.258	92.712	119.712
Smaller class separation	91.052	88.664	118.795	123.444	134.075	126.989	108.147

Appendix

Table 6: Attention ratio of informative features over random features in layer 12 using different data generation processes and various total numbers of training rows.

ROC AUC
Data generation process	Fraction of label noise
	0.05	0.1	0.15	0.2	0.25	0.3	0.35
Different seed	0.999	0.999	0.999	0.999	0.999	0.999	0.999
Larger number of clusters	0.978	0.977	0.974	0.977	0.972	0.975	0.967
Smaller class separation	0.931	0.931	0.931	0.931	0.930	0.931	0.924

Appendix

Table 7: ROC AUC using different data generation processes at various fractions of label noise.

Attention Ratio of Informative over Random
Data generation process	Fraction of label noise
	0.05	0.1	0.15	0.2	0.25	0.3	0.35
Different seed	76.172	80.865	84.116	60.062	78.018	60.809	61.788
Larger number of clusters	61.319	45.238	48.377	29.345	32.015	60.531	45.640
Smaller class separation	109.531	111.949	110.681	83.177	76.726	56.233	57.198

Appendix

Table 8: Attention ratio of informative features over random features in layer 12 using different data generation processes at various fractions of label noise.

Tabular ML

Tabular machine learning develops models and tools for learning from structured data arranged in rows and columns.

See all 115 papers

Aug 23, 2026

Tabular foundation models for non-tabular tasks

Tabular foundation models (TFMs) have recently emerged as a promising paradigm for machine learning on tabular data, offering the ability to generalize across datasets without task-specific training. Since many machine learning datasets can be represented as tables, this raises the question: does TFM capability extend beyond tasks traditionally regarded as tabular? We address this question by using TabPFN v3 on three non-tabular classification problems: handwritten digit recognition on MNIST, language identification of French and German words, and image classification on Tiny ImageNet. In each case, the original data are represented as rows of a table and classification is formulated as prediction of a missing label. We evaluate performance as a function of the number of context samples provided to the pretrained model, with no additional training or fine-tuning. Despite having no explicit access to the spatial or sequential structure characterizing the data, TabPFN v3 in some cases achieves accuracies comparable with that of models or methods geared specifically toward the corresponding tasks.

Goran Nakerst, John Brennan, Wouter Beugeling +1

Institut für Theoretische Physik, Technische Universität Dresden, 01062 Dresden, Germany · Department of Physics, Maynooth University, Ireland · Physikalisches Institut (EP3) and Institute for Topological Insulators, Universität Würzburg, Am Hubland, 97074 Würzburg, Germany

Handwritten Digit Recognition Tabular Foundation Models Cross-Domain Generalization Tabular ICL

Ask PDF

May 6, 2026

Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustment

TabPFN has recently gained attention as a foundation model for tabular datasets, achieving strong performance by leveraging in-context learning on synthetic data. However, we find that TabPFN is vulnerable to label shift, often overfitting to the majority class in the training dataset. To address this limitation, we propose DistPFN, the first test-time posterior adjustment method designed for tabular foundation models. DistPFN rescales predicted class probabilities by downweighting the influence of the training prior (i.e., the class distribution of the context) and emphasizing the contribution of the model's predicted posterior, without architectural modification or additional training. We further introduce DistPFN-T, which incorporates temperature scaling to adaptively control the adjustment strength based on the discrepancy between prior and posterior. We evaluate our methods on over 250 OpenML datasets, demonstrating substantial improvements for various TabPFN-based models in classification tasks under label shift, while maintaining strong performance in standard settings without label shift. Code is available at this repository: https://github.com/seunghan96/DistPFN.

Seunghan Lee

LG AI Research, Seoul, South Korea.

Distribution Shift Robustness Tabular Foundation Models Label Shift Test-Time Adaptation Tabular ICL

Ask PDF

Oct 18, 2026

FTFM: Factorized Tabular Foundation Models

Tabular foundation models (TFMs) are becoming a powerful alternative to conventional tabular learning, adapting to new datasets in context without task-specific training. Current architectures, however, face a trade-off: models that preserve cell-level representations are expensive because they repeatedly attend across rows and columns, while faster models collapse each row into a fixed embedding and lose this fine-grained state. We introduce Factorized Tabular Foundation Models (FTFM), which preserve cell representations while factorizing their computation through one contextual latent per row and one per column. Cells no longer attend to one another directly; instead, each cell is updated through the interaction of its row and column latents, reducing repeated attention while retaining feature-specific information. FTFM achieves competitive predictive performance in tabular classification on major benchmarks such as TabArena and BeyondArena.

Josif Grabocka

University of Technology Nuremberg

Tabular Foundation Models Tabular ICL Tabular Classification Transformer Attention

Ask PDF

Abstract

Tabular ML

Explore similar work

Tabular foundation models for non-tabular tasks

Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustment

FTFM: Factorized Tabular Foundation Models