Closing the Context Gap: Activation Alignment for Tabular In-Context Learning

Yoel Zeldes

cs.LGOct 5, 2026

Closing the Context Gap: Activation Alignment for Tabular In-Context Learning

Authors: Yoel Zeldes

Organizations: Independent Researcher

Abstract

Tabular foundation models perform in-context learning (ICL) by conditioning predictions on labeled training examples provided as context. Unlike traditional models that separate training from inference, these models must process all training examples in every forward pass, making each prediction expensive. Restricting the number of training examples reduces this cost but substantially degrades performance. Instead of discarding context, we propose activation alignment, a method that leverages the full context to teach a model how to behave when seeing only a subset. This is achieved by training a lightweight linear transformation on synthetic unlabeled data to map the intermediate activations of a data-constrained "student" (using partial context) toward those of a full-context "teacher" (using all data). Training the aligner requires no GPU and converges in seconds to minutes on commodity hardware. We evaluate on 38 classification datasets from the TabArena benchmark using the leading two tabular foundation models, TabPFN-3 and TabFM. Across all context budgets, the aligned student yields broad, statistically significant improvements over the unaligned baseline for both models. In low-data regimes, alignment recovers nearly half of the teacher's predictive advantage. The method provides a practical, low-overhead approach to achieving the inference speed of compact contexts while closing a significant fraction of the performance gap to the full-context teacher.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Unlabeled synthetic queries $X_{\text{syn}}$ are appended to the full context (teacher) and to the reduced context (student), and both are passed through the same frozen model $\mathcal{M}$ . A linear aligner $f_{\theta}$ is fit by MSE so that the corrected student activation matches the teacher’s at layer $k$ . (b) At test time, the trained aligner is applied to the student’s layer- $k$ activations.

Figure 2: Pairwise win rates of the aligned student over the unaligned baseline student and full-training-budget XGBoost across context fractions $\alpha\in\{0.1,\dots,0.5\}$ on 38 TabArena classification datasets (5 repeats) for TabPFN and TabFM. Overall win rates across all context budgets are reported in the legend (82.2% vs. base and 81.1% vs. XGBoost for TabPFN; 79.8% vs. base and 80.3% vs. XGBoost for TabFM). Statistical significance annotations above each bar indicate two-sided paired Wilcoxon signed-rank test results against the respective comparator ( ${}^{***}p<0.001$ , ${}^{**}p<0.01$ , ${}^{*}p<0.05$ ; n.s.: not significant).

Figure 3: Median scaling and distillation efficiency across 38 TabArena classification datasets for TabPFN and TabFM, averaged over 5 repeats. Shaded bands denote the 25th–75th percentiles across datasets. (a) Sample Efficiency : Median effective baseline context fraction $E_{\alpha}$ achieved by the aligned student as a function of the actual context budget $\alpha$ . Both models achieve positive data gains across all context fractions. (b) Distillation Efficiency : Median relative gap closed between baseline student and full-context teacher across context fractions $\alpha$ , showing that alignment closes a significant fraction of the gap across all context budgets.

Figure 4: 3-way average rank (lower is better) across 38 TabArena classification datasets for (a) TabPFN and (b) TabFM, averaged over 5 repeats. At each context fraction $\alpha\in\{0.1,\dots,0.5\}$ , bars compare the aligned student, unaligned baseline student, and full-context XGBoost. Across all context budgets, the aligned student maintains the top average ranking.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Dataset	$0.1$		$0.2$		$0.3$		$0.4$		$0.5$		$0.6$		$0.7$		$0.8$		$0.9$		Mean Gain
	PFN	FM	PFN	FM	PFN	FM	PFN	FM	PFN	FM	PFN	FM	PFN	FM	PFN	FM	PFN	FM	PFN	FM
APSFailure	0.05	0.17	0.22	0.24	0.28	0.27	0.38	0.65	0.50	0.25	0.59	0.70	0.70	0.70	0.69	0.72	0.91	—	-0.02	+0.01
Amazon_employee_access	0.12	0.13	0.20	0.21	0.30	0.31	0.41	0.42	0.52	0.53	0.62	0.62	0.72	0.72	0.83	0.83	0.93	0.93	+0.02	+0.02
Bank_Customer_Churn	0.29	0.20	0.40	0.29	0.58	0.45	0.68	0.61	0.74	0.91	0.80	0.84	0.85	0.94	0.89	0.94	0.92	0.95	+0.18	+0.18
Bioresponse	0.12	0.13	0.19	0.24	0.26	0.35	0.36	0.44	0.46	0.50	0.55	0.58	0.64	0.69	0.76	0.81	0.91	0.91	-0.03	+0.02
Diabetes130US	0.16	0.17	0.25	0.34	0.34	0.47	0.42	0.70	0.49	0.70	0.58	0.74	0.72	0.78	0.82	0.82	0.89	0.93	+0.02	+0.13

Appendix

Table 1: Per-dataset effective baseline sample fraction

E_{\alpha}

and Mean Gain (

E_{\alpha}-\alpha

) across 38 TabArena classification datasets for TabPFN (PFN) and TabFM (FM). Shaded green cells indicate effective samples (

E_{\alpha}>\alpha

or Mean Gain

>0

). Dashes (—) denote slices excluded by the teacher superiority filter (Section 4 ).

Explore similar work

TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction
Aug 2, 2026Rasa Hosseinzadeh, Alex Labach, Zexin Xue +3Tabular Foundation Models In-Context Learning
Layer 6 AI, Toronto, Canada · Cohere, Toronto, Canada
TabICLv2: A better, faster, scalable, and open tabular foundation model
Feb 11, 2026Jingang Qu, David Holzmüller, Gaël Varoquaux +1Tabular Foundation Models Pretraining
SODA Team, INRIA Saclay, Palaiseau, France · Probabl, France
Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models
May 7, 2026Amir Rezaei Balef, Mykhailo Koshil, Katharina EggenspergerTabular Foundation Models Transformer Architectures
TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany · University of Tübingen, Tübingen, Germany

Aug 2, 2026, cs.LGJ/K move · Enter open · S save

TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction

Rasa Hosseinzadeh, Alex Labach, Zexin Xue, Shuyi Han+2

Layer 6 AI, Toronto, Canada · Cohere, Toronto, Canada

Tabular Foundation Models In-Context Learning

Open paper AskPDF arXiv

Tabular foundation models, driven by in-context learning, have rapidly grown in quality and popularity. However, recent approaches with either cell-based architectures or retrieval have sacrificed efficiency for raw performance, restricting their utility in situations where compute is limited or inference speed is crucial. We adopt an alternate approach, sticking with row-based attention while incorporating long context pre-training to eliminate the need for retrieval. By combining this with architectural improvements and SSL pre-training on a newly-sourced, larger corpus of real data results, we present TabDPT-Turbo, a model that provides comparable default performance to TabDPT v1.1 on TabArena-Lite, CC18, and CTR23, at orders of magnitude faster. In our experiments, TabDPT-Turbo is the fastest model overall among leading foundation models. We have released the new model as TabDPT v1.2 at https://github.com/layer6ai-labs/TabDPT-inference.

Abstract

Explore similar work

TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction

TabICLv2: A better, faster, scalable, and open tabular foundation model

Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models