Closing the Context Gap: Activation Alignment for Tabular In-Context Learning

Yoel Zeldes

cs.LGOct 5, 2026

Closing the Context Gap: Activation Alignment for Tabular In-Context Learning

Authors: Yoel Zeldes

Organizations: Independent Researcher

Abstract

Tabular foundation models perform in-context learning (ICL) by conditioning predictions on labeled training examples provided as context. Unlike traditional models that separate training from inference, these models must process all training examples in every forward pass, making each prediction expensive. Restricting the number of training examples reduces this cost but substantially degrades performance. Instead of discarding context, we propose activation alignment, a method that leverages the full context to teach a model how to behave when seeing only a subset. This is achieved by training a lightweight linear transformation on synthetic unlabeled data to map the intermediate activations of a data-constrained "student" (using partial context) toward those of a full-context "teacher" (using all data). Training the aligner requires no GPU and converges in seconds to minutes on commodity hardware. We evaluate on 38 classification datasets from the TabArena benchmark using the leading two tabular foundation models, TabPFN-3 and TabFM. Across all context budgets, the aligned student yields broad, statistically significant improvements over the unaligned baseline for both models. In low-data regimes, alignment recovers nearly half of the teacher's predictive advantage. The method provides a practical, low-overhead approach to achieving the inference speed of compact contexts while closing a significant fraction of the performance gap to the full-context teacher.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Unlabeled synthetic queries $X_{\text{syn}}$ are appended to the full context (teacher) and to the reduced context (student), and both are passed through the same frozen model $\mathcal{M}$ . A linear aligner $f_{\theta}$ is fit by MSE so that the corrected student activation matches the teacher’s at layer $k$ . (b) At test time, the trained aligner is applied to the student’s layer- $k$ activations.

Figure 2: Pairwise win rates of the aligned student over the unaligned baseline student and full-training-budget XGBoost across context fractions $\alpha\in\{0.1,\dots,0.5\}$ on 38 TabArena classification datasets (5 repeats) for TabPFN and TabFM. Overall win rates across all context budgets are reported in the legend (82.2% vs. base and 81.1% vs. XGBoost for TabPFN; 79.8% vs. base and 80.3% vs. XGBoost for TabFM). Statistical significance annotations above each bar indicate two-sided paired Wilcoxon signed-rank test results against the respective comparator ( ${}^{***}p<0.001$ , ${}^{**}p<0.01$ , ${}^{*}p<0.05$ ; n.s.: not significant).

Figure 3: Median scaling and distillation efficiency across 38 TabArena classification datasets for TabPFN and TabFM, averaged over 5 repeats. Shaded bands denote the 25th–75th percentiles across datasets. (a) Sample Efficiency : Median effective baseline context fraction $E_{\alpha}$ achieved by the aligned student as a function of the actual context budget $\alpha$ . Both models achieve positive data gains across all context fractions. (b) Distillation Efficiency : Median relative gap closed between baseline student and full-context teacher across context fractions $\alpha$ , showing that alignment closes a significant fraction of the gap across all context budgets.

Figure 4: 3-way average rank (lower is better) across 38 TabArena classification datasets for (a) TabPFN and (b) TabFM, averaged over 5 repeats. At each context fraction $\alpha\in\{0.1,\dots,0.5\}$ , bars compare the aligned student, unaligned baseline student, and full-context XGBoost. Across all context budgets, the aligned student maintains the top average ranking.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Dataset	$0.1$		$0.2$		$0.3$		$0.4$		$0.5$		$0.6$		$0.7$		$0.8$		$0.9$		Mean Gain
	PFN	FM	PFN	FM	PFN	FM	PFN	FM	PFN	FM	PFN	FM	PFN	FM	PFN	FM	PFN	FM	PFN	FM
APSFailure	0.05	0.17	0.22	0.24	0.28	0.27	0.38	0.65	0.50	0.25	0.59	0.70	0.70	0.70	0.69	0.72	0.91	—	-0.02	+0.01
Amazon_employee_access	0.12	0.13	0.20	0.21	0.30	0.31	0.41	0.42	0.52	0.53	0.62	0.62	0.72	0.72	0.83	0.83	0.93	0.93	+0.02	+0.02
Bank_Customer_Churn	0.29	0.20	0.40	0.29	0.58	0.45	0.68	0.61	0.74	0.91	0.80	0.84	0.85	0.94	0.89	0.94	0.92	0.95	+0.18	+0.18
Bioresponse	0.12	0.13	0.19	0.24	0.26	0.35	0.36	0.44	0.46	0.50	0.55	0.58	0.64	0.69	0.76	0.81	0.91	0.91	-0.03	+0.02
Diabetes130US	0.16	0.17	0.25	0.34	0.34	0.47	0.42	0.70	0.49	0.70	0.58	0.74	0.72	0.78	0.82	0.82	0.89	0.93	+0.02	+0.13

Appendix

Table 1: Per-dataset effective baseline sample fraction

E_{\alpha}

and Mean Gain (

E_{\alpha}-\alpha

) across 38 TabArena classification datasets for TabPFN (PFN) and TabFM (FM). Shaded green cells indicate effective samples (

E_{\alpha}>\alpha

or Mean Gain

>0

). Dashes (—) denote slices excluded by the teacher superiority filter (Section 4 ).

Explore similar work

Aug 2, 2026cs.LG

TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction

Tabular foundation models, driven by in-context learning, have rapidly grown in quality and popularity. However, recent approaches with either cell-based architectures or retrieval have sacrificed efficiency for raw performance, restricting their utility in situations where compute is limited or inference speed is crucial. We adopt an alternate approach, sticking with row-based attention while incorporating long context pre-training to eliminate the need for retrieval. By combining this with architectural improvements and SSL pre-training on a newly-sourced, larger corpus of real data results, we present TabDPT-Turbo, a model that provides comparable default performance to TabDPT v1.1 on TabArena-Lite, CC18, and CTR23, at orders of magnitude faster. In our experiments, TabDPT-Turbo is the fastest model overall among leading foundation models. We have released the new model as TabDPT v1.2 at https://github.com/layer6ai-labs/TabDPT-inference.

Rasa Hosseinzadeh, Alex Labach, Zexin Xue +3

Layer 6 AI, Toronto, Canada · Cohere, Toronto, Canada

Tabular Foundation Models In-Context Learning

Ask PDF

Feb 11, 2026cs.LG

TabICLv2: A better, faster, scalable, and open tabular foundation model

Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstrating the value of in-context learning for tabular data. We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars: (1) a novel synthetic data generation engine designed for high pretraining diversity; (2) various architectural innovations, including a new scalable softmax in attention improving generalization to larger datasets without prohibitive long-sequence pretraining; and (3) optimized pretraining protocols, notably replacing AdamW with the Muon optimizer. On the TabArena and TALENT benchmarks, TabICLv2 without any tuning surpasses the performance of the current state of the art, RealTabPFN-2.5 (hyperparameter-tuned, ensembled, and fine-tuned on real data). With only moderate pretraining compute, TabICLv2 generalizes effectively to million-scale datasets under 50 GB GPU memory while being markedly faster than RealTabPFN-2.5. We provide extensive ablation studies to quantify these contributions and foster open research by releasing code for inference, pretraining, and synthetic data generation at https://github.com/soda-inria/tabicl.

Jingang Qu, David Holzmüller, Gaël Varoquaux +1

SODA Team, INRIA Saclay, Palaiseau, France · Probabl, France

Tabular Foundation Models Pretraining Gumbel-Softmax Relaxation

Ask PDF

May 7, 2026cs.LG

Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models

Transformer-based tabular foundation models (TFMs) dominate small to medium tabular predictive benchmark tasks, yet their inference mechanisms remain largely unexplored. We present the first large-scale mechanistic study of layerwise dynamics in 6 state-of-the-art tabular in-context learning models. We explore how predictions emerge across depth, identify distinct stages of inference and reveal latent-space dynamics that differ from those of language models. Our findings indicate substantial depthwise redundancy across multiple models, suggesting iterative refinement with overlapping computations during inference stages. Guided by these insights, we design a proof-of-concept, looped single-layer model that uses only 20% of the original model's parameters while achieving comparable performance. The code is available at https://github.com/amirbalef/is_one_layer_enough.

Amir Rezaei Balef, Mykhailo Koshil, Katharina Eggensperger

TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany · University of Tübingen, Tübingen, Germany

Tabular Foundation Models Transformer Architectures Layer-Wise Foundation Model Inference Dynamics Latent Dynamics

Ask PDF