In-context learning (ICL) lets tabular foundation models make predictions without dataset-specific training or fine-tuning. These models achieve strong predictive accuracy, but interpreting how individual features contribute to the prediction remains difficult. In biology, for example, feature contributions can help identify biomarkers or risk genes. Post-hoc interpretability methods assign feature importance scores, but these scores do not indicate what the model actually uses and can lead to explanation errors. We introduce TICL3 (Tabular In-Context Local Linear Learning), an interpretable-by-design predictor that uses a tabular foundation model to guide a simple linear prediction rule. TICL3 first learns a shared linear model to capture dataset-level feature effects. Then, for each query, it retrieves similar training samples using representations from the foundation model and fits a second linear model to the residuals in that neighborhood. The shared model captures broad effects, while the local model corrects errors among similar samples. Their combined coefficients determine each feature contribution to the final query-specific linear score, so the same model makes and explains the prediction. Across 386 classification benchmark-dataset evaluations, TICL3 trails TabICLv2 by 1.1 percentage points and outperforms CatBoost in 266 evaluations. It improves average support F1 over MAPLE by 4.7 points and achieves the best performance on both real-data selected-feature utility metrics.
Figures & tables
Figure 1: Summary of results. (a) Average per-dataset rank in classification and regression over all methods in Tables 2 and 6 ; a subset is shown. (b) Deletion monotonicity and selected-feature utility; the latter averages Gen-SVM and Gen-kmeans, reported separately in Table 5 . (c) Per-dataset cost for 50 predictions and explanations, including method-specific explainer construction. All methods explain the same TICL 3 (global + residual) predictor, so its setup is excluded for every method. Online latency is similar for TICL 3 (global + residual) and the learned explainers (about 27 ms per query); the difference comes from explainer construction. LIME and KernelSHAP are extrapolated from the fixed 10-query-per-dataset subset.
Method
Interpretable by design
Explicit additive prediction
Query-specific deployed rule
ICL-guided locality
TabICLv2
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
SHAP/LIME on TabICLv2
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
ExplainerPFN
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
EBM/NAM
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
MAPLE
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
LLSPIN
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
Table 1: Conceptual positioning of TICL 3 . Interpretable by design : the explanation is part of the deployed predictor; explicit additive prediction : prediction sums inspectable feature terms mapped to input columns; query-specific deployed rule : an explicit feature-level rule is fitted or selected per test point (sample weights or post-hoc explanations alone do not qualify); ICL-guided locality : an ICL representation defines locality. KernelICL uses ICL-derived query weights but no query-specific feature-level rule. ShapPFN sums per-feature contributions computed by its transformer but deploys no explicit feature-level rule; ExplainerPFN estimates attributions without access to a predictor.
Figure 2: Overview of TICL 3 (global + residual). Offline we use a frozen TabICLv2 backbone with cross-fitting to build a reference bank of processed features, class probability targets and learned representations. We fit a shared ElasticNet model to the bank targets. For each query, we then use its representation to select similar reference rows and fit a second ElasticNet model to the shared model’s remaining errors on those rows. We add the shared model and the local correction to give a single linear model for the query. We clip and normalize the class scores from this to make the final prediction. We also use the model coefficients and the centered feature values to compute the contribution of each feature to the class scores.
Family
Method
CC-18 ↑
TabArena-Lite ↑
TALENT ↑
BeyondArena ↑
Tabular foundation models
TabICLv2
0.872±0.154
0.880±0.087
0.830±0.148
0.839±0.132
TabPFNv2
0.865±0.157
0.880±0.086
0.825±0.149
0.835±0.137
Interpretable by design
TICL 3 (global + residual)
0.856±0.161
0.877±0.089
0.817±0.160
0.833±0.134
TICL 3 (regional anchors)
0.855±0.162
0.874±0.087
0.814±0.164
0.834±0.136
MAPLE
0.830±0.164
0.862±0.091
0.790±0.166
0.820±0.137
EBM
0.840±0.159
0.865±0.093
0.779±0.186
0.825±0.136
Table 2: Predictive performance on CC-18, TabArena-Lite, TALENT, and BeyondArena under Section 4.1 . In that order, TabICLv2 uses 8/4/4/4 estimators and TabPFNv2 uses 8/8/4/8; both TICL 3 variants use one. The TICL 3 rows use a five-fold cross-fitted bank and the configurations in Section 3 and Appendix E.5 . TICL 3 (regional anchors) is grouped by its predictor form, but its predicted-class explanation is all zero for 73.3% of synthetic and 43.8% of real-data queries (Appendix E.5 ). The ± values describe variation across dataset means.
Method
Mean Acc. ↑
Fit (s) ↓
Predict (s) ↓
TabICLv2 (8 estimators)
0.8735
0.950
5.462
TabICLv2 (1 estimator)
0.8710
0.485
0.745
TabPFNv2
0.8641
3.139
9.660
TICL 3 (global + residual)
0.8557
5.515
6.849
TICL 3 -Fast (global + residual)
0.8609
5.842
1.495
TICL 3 (regional anchors)
0.8555
6.736
0.850
Table 3: CC-18 accuracy and mean fit/prediction time over 69 datasets and three seeds. The default TabICLv2 row uses eight estimators; the matched row uses one. Configuration and timing details are in Appendix E.5 .
Table 4: Generating-support F1 and accuracy on LSPIN E1–E4 (10 seeds; 100 queries per task–seed). Blocks distinguish own-predictor from masked- TabICLv2 accuracy; E4 uses group-conditional support. An all-zero coefficient vector selects no features and receives support F1 zero.
Method
Deletion Spearman ↑
Gen-SVM ↑
Gen-kmeans ↑
Neighbour stab. ↓
Online ms/query ↓
TICL 3 (global + residual)
0.470
0.886
0.697
0.032
26.974
KernelSHAP
0.384
0.708
0.555
0.037
53,686.935
LIME
0.449
0.813
0.601
0.032
14,607.496
FastSHAP
0.408
0.608
0.637
0.025
27.197
IG on surrogate
0.417
0.861
0.669
0.031
27.482
GradientSHAP on surrogate
0.446
0.856
0.666
0.031
27.560
Table 5: Explanation quality and online prediction-plus-explanation cost on 10 CC-18 datasets. We evaluate quality using 500 queries; all rows use the same 422 queries with valid values for deletion and 493 for the other metrics. We exclude undefined values rather than setting them to zero. We average runtime across datasets, giving each dataset equal weight and excluding one-off setup. For LIME and KernelSHAP attribution time, we use a fixed subset of 100 queries (10 per dataset, three repeats); for the common predictor latency and the remaining rows, we use 50 queries per dataset.
Family
Method
Mean nRMSE ↓
Mean R2↑
Tabular foundation models
TabICLv2
0.479±0.300
0.681
TabICLv2 (1 estimator)
0.482±0.303
0.676
TabPFNv2
0.506±0.310
0.648
Interpretable by design
TICL 3 (global + residual)
0.515±0.287
0.653
TICL 3 (regional anchors)
0.521±0.275
0.654
MAPLE
0.571±0.298
0.586
Table 6: TALENT regression on 119 splits. nRMSE uses the test-target standard deviation; ± reports across-dataset variation.
TabPFN; 8/8/4/8 estimators; native multiclass model for at most 10 classes and one-vs-rest above 10; packaged defaults otherwise.
No tuning or validation selection.
KernelICL
Repository implementation with four TabICL estimators; k NN uses k=20 and Gaussian uses scale 4.
Fixed a priori; no calibration or search.
MAPLE
scikit-learn random forest with 200 trees, max_features =0.5, minimum leaf size 10; per-query ridge penalty 0.001 .
Stratified 80/20 training/validation split; retained count chosen by validation RMSE from candidates up to min(D,100) , using at most 50 validation queries.
EBM
InterpretML; 128 bins, 3000 maximum rounds, learning rate 0.01, and no interactions.
Fixed; package-internal fitting only, with no external search.
NAM
One hidden layer of width 32 per feature; at most 300 epochs; batch size 256; AdamW with learning rate 10−3 and weight decay 10−4 .
Stratified 80/20 training/validation split; validation-loss early stopping with patience 30.
Appendix
Table 7: Configurations for the predictive baselines in Table 2 . Suite order for ensemble counts is CC-18/TabArena-Lite/TALENT/BeyondArena. “Defaults” means the named package’s released default configuration except for the explicitly listed settings.
Configuration
Acc. (%) ↑
Nonzero expl. (%) ↑
Deletion Spearman ↑
Gen-SVM (%) ↑
Gen-kmeans (%) ↑
Global and local components
Local ElasticNet only ( k=400 , matched)
90.18
68.0
0.5207
70.72
55.77
Global ElasticNet only
76.13
100.0
0.6097
95.40
67.80
TICL — primary ( k=400 , 300 iterations)
89.90
100.0
0.5297
88.80
71.40
Local reference configuration
Local ElasticNet reference ( k=100 )
90.29
41.8
0.2807
53.03
52.97
Appendix
Table 8: Component and configuration ablations. Bold identifies the primary configuration, not column-wise best results. Iteration caps apply to the local correction; shared global fits retain 300 iterations. A nonzero explanation has a finite contribution vector with maximum absolute contribution above 10−12 . Explanation metrics use configuration-specific valid populations and equally weight contributing datasets. Gen-SVM targets each predictor’s own classes; Gen-kmeans uses true labels. Evaluation counts appear in Table 9 .
Configuration
Valid deletion /500
Contributing datasets
Matched local K400
336
9 / 9 / 9
Global-only
500
10 / 10 / 10
TICL — primary
488
10 / 10 / 10
Local K100
209
7 / 6 / 6
TICL ( k=30 , 300 iterations)
493
10 / 10 / 10
TICL ( k=400 , 30 iterations)
488
10 / 10 / 10
Appendix
Table 9: Evaluation populations for the ablation. Every configuration has 500 attempted explanation queries and full-test accuracy on all ten datasets. Dataset counts refer to deletion Spearman, Gen-SVM and Gen-kmeans, respectively.
Large Language Models (LLMs) fine-tuned on serialized tabular data are emerging as powerful alternatives to traditional tree-based models, particularly for heterogeneous or context-rich datasets. However, their deployment in high-stakes domains is hindered by a lack of faithful interpretability; existing methods often rely on global linear proxies or scalar probability shifts that fail to capture the model's full probabilistic uncertainty. In this work, we introduce TabSHAP, a model-agnostic interpretability framework designed to directly attribute local query decision logic in LLM-based tabular classifiers. By adapting a Shapley-style sampled-coalition estimator with Jensen-Shannon divergence between full-input and masked-input class distributions, TabSHAP quantifies the distributional impact of each feature rather than simple prediction flips. To align with tabular semantics, we mask at the level of serialized key:value fields (atomic in the prompt string), not individual subword tokens. Experimental validation on the Adult Income and Heart Disease benchmarks demonstrates that TabSHAP isolates critical diagnostic features, achieving significantly higher faithfulness than random baselines and XGBoost proxies. We further run a distance-metric ablation on the same test instances and TabSHAP settings: attributions are recomputed with KL or L1 replacing JSD in the similarity step (results cached per metric), and we compare deletion faithfulness across all three.
Aryan Chaudhary, Prateek Agarwal, Tejasvi Alladi
Department of Computer Science Birla Institute of Technology and Science India
Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstrating the value of in-context learning for tabular data. We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars: (1) a novel synthetic data generation engine designed for high pretraining diversity; (2) various architectural innovations, including a new scalable softmax in attention improving generalization to larger datasets without prohibitive long-sequence pretraining; and (3) optimized pretraining protocols, notably replacing AdamW with the Muon optimizer. On the TabArena and TALENT benchmarks, TabICLv2 without any tuning surpasses the performance of the current state of the art, RealTabPFN-2.5 (hyperparameter-tuned, ensembled, and fine-tuned on real data). With only moderate pretraining compute, TabICLv2 generalizes effectively to million-scale datasets under 50 GB GPU memory while being markedly faster than RealTabPFN-2.5. We provide extensive ablation studies to quantify these contributions and foster open research by releasing code for inference, pretraining, and synthetic data generation at https://github.com/soda-inria/tabicl.
Jingang Qu, David Holzmüller, Gaël Varoquaux +1
SODA Team, INRIA Saclay, Palaiseau, France · Probabl, France
Interpretability is central for scientific machine learning, as understanding why models make predictions enables hypothesis generation and validation. While tabular foundation models show strong performance, existing explanation methods like SHAP are computationally expensive, limiting interactive exploration. We introduce ShapPFN, a foundation model that integrates Shapley value regression directly into its architecture, producing both predictions and explanations in a single forward pass. On standard benchmarks, ShapPFN achieves competitive performance while producing high-fidelity explanations (R2=0.96, cosine=0.99) over 1000\times faster than KernelSHAP (0.06s vs 610s). Our code is available at https://github.com/kunumi/ShapPFN
Luan Borges Teodoro Reis Sena, Francisco Galuppo Azevedo
Kunumi Institute, Belo Horizonte, Minas Gerais, Brazil · Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil