Active In-Context Learning for Tabular Foundation Models

Wilailuck Treerath; Fabrizio Pittorino

cs.LGMar 28, 2026

Active In-Context Learning for Tabular Foundation Models

Authors: Wilailuck Treerath, Fabrizio Pittorino

Organizations: Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milano, Italy

Abstract

Active learning (AL) reduces labeling cost by querying informative samples, but in tabular settings its cold-start gains are often limited because uncertainty estimates are unreliable when models are trained on very few labels. Tabular foundation models such as TabPFN provide calibrated probabilistic predictions via in-context learning (ICL), i.e., without task-specific weight updates, enabling an AL regime in which the labeled context - rather than parameters - is iteratively optimized. We formalize Tabular Active In-Context Learning (Tab-AICL) and instantiate it with four acquisition rules: uncertainty (TabPFN-Margin), diversity (TabPFN-Coreset), an uncertainty-diversity hybrid (TabPFN-Hybrid), and a scalable two-stage method (TabPFN-Proxy-Hybrid) that shortlists candidates using a lightweight linear proxy before TabPFN-based selection. Across 20 classification benchmarks, Tab-AICL improves cold-start sample efficiency over retrained gradient-boosting baselines (CatBoost-Margin and XGBoost-Margin), measured by normalized AULC up to 100 labeled samples.

Figures & tables

Dataset	ID	Instances	Features	Classes	Domain
Iris	61	150	4	3	Botany
Glass	41	214	9	6	Forensic
Ionosphere	59	351	34	2	Physics
Balance-scale	997	625	4	2	Psychology
Vehicle	994	846	18	2	Image Rec.
Page-blocks	1021	5,473	10	2	Document

TABLE I: Benchmark datasets (post-preprocessing).

Dataset	Tab-AICL Strategies					Baselines
Dataset	TabPFN-Coreset	TabPFN-Hybrid	TabPFN-Proxy-Hybrid	TabPFN-Margin	TabPFN-Random	CatBoost-Margin	XGBoost-Margin	LabelSpreading-Random
Iris	0.934 $\pm$ 0.023	0.929 $\pm$ 0.029	0.931 $\pm$ 0.019	0.917 $\pm$ 0.027	0.902 $\pm$ 0.031	0.880 $\pm$ 0.060	0.743 $\pm$ 0.074	0.831 $\pm$ 0.058
Glass	0.397 $\pm$ 0.055	0.472 $\pm$ 0.041	0.441 $\pm$ 0.061	0.481 $\pm$ 0.056	0.489 $\pm$ 0.077	0.456 $\pm$ 0.055	0.400 $\pm$ 0.033	0.382 $\pm$ 0.061
Ionosphere	0.617 $\pm$ 0.132	0.810 $\pm$ 0.042	0.800 $\pm$ 0.032	0.788 $\pm$ 0.038	0.751 $\pm$ 0.052	0.641 $\pm$ 0.051	0.640 $\pm$ 0.049	0.440 $\pm$ 0.154
Balance-scale	0.758 $\pm$ 0.032	0.808 $\pm$ 0.024	0.811 $\pm$ 0.040	0.808 $\pm$ 0.044	0.721 $\pm$ 0.055	0.609 $\pm$ 0.042	0.578 $\pm$ 0.058	0.665 $\pm$ 0.049
Vehicle	0.742 $\pm$ 0.067	0.833 $\pm$ 0.022	0.811 $\pm$ 0.054	0.794 $\pm$ 0.040	0.747 $\pm$ 0.052	0.665 $\pm$ 0.057	0.640 $\pm$ 0.062	0.685 $\pm$ 0.056

TABLE II: AULC Performance Summary (Mean

\pm

std of AULC up to

N=100

, Batch Size 10)

Dataset	$\Delta\text{AULC}$	$p$ -Value (Raw)	$p$ -Value (Adj)	Result
Iris	+0.050	0.037	0.080	No significant difference
Glass	+0.016	0.375	0.526	No significant difference
Ionosphere	+0.169	0.002	0.007	Significantly higher
Balance-scale	+0.199	0.002	0.007	Significantly higher
Vehicle	+0.168	0.002	0.007	Significantly higher
Page-blocks	+0.095	0.002	0.007	Significantly higher

TABLE III: Statistical Significance: TabPFN-Hybrid vs. CatBoost-Margin (Wilcoxon Signed-Rank Test on AULC)

Fig. 1: Comprehensive AULC Performance: Area Under Learning Curve scores across 20 datasets, grouped by acquisition strategy.

Fig. 2: Learning Curve for Ionosphere: Tab-AICL strategies achieve rapid convergence, with TabPFN-Proxy-Hybrid offering an effective balance of speed and stability. The shaded regions represent 95% confidence intervals across 10 random seeds.

Fig. 3: Learning Curve for Adult: The TabPFN-Proxy-Hybrid strategy (pink) demonstrates stable performance and consistent gains. Note the narrower confidence intervals compared to other strategies.

Fig. 4: Learning Curve for Vehicle: The TabPFN-Hybrid strategy (purple) achieves higher performance than pure uncertainty sampling and baselines.

Fig. 5: Learning Curve for Page Blocks: The TabPFN-Margin strategy (blue) achieves rapid convergence, suggesting that for certain data manifolds, pure uncertainty sampling remains an effective approach.

Dataset	B=5	B=10	B=15	B=20
Iris	0.920 $\pm$ 0.036	0.931 $\pm$ 0.019	0.910 $\pm$ 0.016	0.907 $\pm$ 0.022
Glass	0.446 $\pm$ 0.059	0.441 $\pm$ 0.061	0.467 $\pm$ 0.040	0.442 $\pm$ 0.027
Ionosphere	0.809 $\pm$ 0.044	0.800 $\pm$ 0.032	0.784 $\pm$ 0.049	0.766 $\pm$ 0.092
Balance-scale	0.836 $\pm$ 0.038	0.811 $\pm$ 0.040	0.791 $\pm$ 0.037	0.772 $\pm$ 0.044
Vehicle	0.837 $\pm$ 0.040	0.811 $\pm$ 0.054	0.817 $\pm$ 0.040	0.776 $\pm$ 0.054
Page-blocks	0.651 $\pm$ 0.075	0.657 $\pm$ 0.043	0.623 $\pm$ 0.069	0.632 $\pm$ 0.059

TABLE IV: Batch Size Ablation (AULC

\pm

std, TabPFN-Proxy-Hybrid)

Dataset	Tab-AICL Strategies					Baselines
Dataset	TabPFN-Coreset	TabPFN-Hybrid	TabPFN-Proxy-Hybrid	TabPFN-Margin	TabPFN-Random	CatBoost-Margin	XGBoost-Margin	LabelSpreading-Random
Iris	0.950 $\pm$ 0.043	0.933 $\pm$ 0.049	0.950 $\pm$ 0.037	0.943 $\pm$ 0.047	0.947 $\pm$ 0.034	0.927 $\pm$ 0.047	0.880 $\pm$ 0.092	0.877 $\pm$ 0.060
Glass	0.564 $\pm$ 0.053	0.580 $\pm$ 0.060	0.585 $\pm$ 0.061	0.637 $\pm$ 0.055	0.592 $\pm$ 0.094	0.588 $\pm$ 0.091	0.496 $\pm$ 0.024	0.441 $\pm$ 0.066
Ionosphere	0.805 $\pm$ 0.078	0.875 $\pm$ 0.039	0.888 $\pm$ 0.044	0.878 $\pm$ 0.034	0.860 $\pm$ 0.037	0.821 $\pm$ 0.047	0.800 $\pm$ 0.065	0.574 $\pm$ 0.069
Balance Scale	0.871 $\pm$ 0.033	0.918 $\pm$ 0.029	0.940 $\pm$ 0.024	0.949 $\pm$ 0.018	0.885 $\pm$ 0.034	0.742 $\pm$ 0.050	0.756 $\pm$ 0.065	0.762 $\pm$ 0.035
Vehicle	0.916 $\pm$ 0.070	0.966 $\pm$ 0.022	0.980 $\pm$ 0.014	0.989 $\pm$ 0.009	0.921 $\pm$ 0.065	0.929 $\pm$ 0.018	0.897 $\pm$ 0.024	0.810 $\pm$ 0.053

TABLE V: Final Kappa Performance Summary (Mean

\pm

std at

N=100

, Batch Size 10)

Dataset	Tab-AICL Strategies					Baselines
Dataset	TabPFN-Coreset	TabPFN-Hybrid	TabPFN-Proxy-Hybrid	TabPFN-Margin	TabPFN-Random	CatBoost-Margin	XGBoost-Margin	LabelSpreading-Random
Iris	0.997 $\pm$ 0.003	0.995 $\pm$ 0.005	0.996 $\pm$ 0.003	0.996 $\pm$ 0.004	0.996 $\pm$ 0.004	0.994 $\pm$ 0.006	0.956 $\pm$ 0.044	0.986 $\pm$ 0.009
Glass	0.921 $\pm$ 0.026	0.929 $\pm$ 0.018	0.926 $\pm$ 0.016	0.936 $\pm$ 0.013	0.929 $\pm$ 0.033	0.907 $\pm$ 0.022	0.849 $\pm$ 0.017	0.854 $\pm$ 0.041
Ionosphere	0.973 $\pm$ 0.009	0.984 $\pm$ 0.007	0.985 $\pm$ 0.006	0.987 $\pm$ 0.006	0.981 $\pm$ 0.008	0.964 $\pm$ 0.016	0.947 $\pm$ 0.018	0.899 $\pm$ 0.044
Balance Scale	0.989 $\pm$ 0.003	0.994 $\pm$ 0.003	0.996 $\pm$ 0.002	0.998 $\pm$ 0.002	0.990 $\pm$ 0.006	0.956 $\pm$ 0.012	0.955 $\pm$ 0.030	0.957 $\pm$ 0.011
Vehicle	0.996 $\pm$ 0.003	0.999 $\pm$ 0.001	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	0.998 $\pm$ 0.001	0.995 $\pm$ 0.003	0.986 $\pm$ 0.008	0.975 $\pm$ 0.011

TABLE VI: Final ROC AUC Performance Summary (Mean

\pm

std at

N=100

, Batch Size 10)

Active Learning

Active learning selects the examples, labels, features, or experiments most useful for improving a model under limited budgets.

See all 280 papers

Closing the Context Gap: Activation Alignment for Tabular In-Context Learning
Oct 5, 2026Yoel ZeldesTabular Foundation Models Tabular ICL
Independent Researcher
TabICLv2: A better, faster, scalable, and open tabular foundation model
Feb 11, 2026Jingang Qu, David Holzmüller, Gaël Varoquaux +1Synthetic Data Pretraining Tabular Foundation Models
SODA Team, INRIA Saclay, Palaiseau, France · Probabl, France
LUCoS: Latent Unsupervised Context Selection for Tabular Foundation Models
May 26, 2026Oroel Ipas, Guillermo Gomez-Trenado, Rocío Romero-Zaliz +1Tabular Foundation Models Tabular ICL
1Andalusian Research Institute in Data Science and Computational Intelligence (DaSCI) · Department of Computer Science and Artificial Intelligence (DECSAI) · 3Research Center in Information and Communication Technologies (CITIC) +1

Oct 5, 2026J/K move · Enter open · S save

Closing the Context Gap: Activation Alignment for Tabular In-Context Learning

Yoel Zeldes

Independent Researcher

Tabular Foundation Models Tabular ICL Feature-Based Knowledge Distillation Representation Alignment

Open paper AskPDF arXiv

Tabular foundation models perform in-context learning (ICL) by conditioning predictions on labeled training examples provided as context. Unlike traditional models that separate training from inference, these models must process all training examples in every forward pass, making each prediction expensive. Restricting the number of training examples reduces this cost but substantially degrades performance. Instead of discarding context, we propose activation alignment, a method that leverages the full context to teach a model how to behave when seeing only a subset. This is achieved by training a lightweight linear transformation on synthetic unlabeled data to map the intermediate activations of a data-constrained "student" (using partial context) toward those of a full-context "teacher" (using all data). Training the aligner requires no GPU and converges in seconds to minutes on commodity hardware. We evaluate on 38 classification datasets from the TabArena benchmark using the leading two tabular foundation models, TabPFN-3 and TabFM. Across all context budgets, the aligned student yields broad, statistically significant improvements over the unaligned baseline for both models. In low-data regimes, alignment recovers nearly half of the teacher's predictive advantage. The method provides a practical, low-overhead approach to achieving the inference speed of compact contexts while closing a significant fraction of the performance gap to the full-context teacher.

Abstract

Active Learning

Explore similar work

Closing the Context Gap: Activation Alignment for Tabular In-Context Learning

TabICLv2: A better, faster, scalable, and open tabular foundation model

LUCoS: Latent Unsupervised Context Selection for Tabular Foundation Models