Foundation-Model Surrogates Enable Data-Efficient Active Learning for Materials Discovery

Jeffrey Hu; Rongzhi Dong; Ying Feng; Ming Hu; Jianjun Hu

cond-mat.mtrl-sciMar 13, 2026

Foundation-Model Surrogates Enable Data-Efficient Active Learning for Materials Discovery

Authors: Jeffrey Hu, Rongzhi Dong, Ying Feng, Ming Hu, Jianjun Hu

Organizations: Department of Materials Science and Engineering, University of Illinois Urbana Champaign, Champaign,IL, USA · Department of Computer Science & Engineering, University of South Carolina, Columbia, SC, USA · Zuoyue Honors College, Hangzhou Dianzhi University, Hangzhou, China · Department of Mechanical Engineering, University of South Carolina, Columbia, SC, USA

Abstract

Active learning (AL) has emerged as a powerful paradigm for accelerating materials discovery by iteratively steering experiments toward promising candidates, reducing the number of costly synthesis-and-characterization cycles needed to identify optimal materials. However, current AL relies predominantly on Gaussian Process (GP) and Random Forest (RF) surrogates, which suffer from complementary limitations: GP underfits complex composition-property landscapes due to rigid kernel assumptions, while RF produces unreliable heuristic uncertainty estimates in small-data regimes. This small-data challenge is pervasive in materials science, making reliable surrogate modeling extremely difficult with models trained from scratch on each new dataset. Here we propose In-Context Active Learning (ICAL), which addresses this bottleneck by replacing conventional surrogates with TabPFN, a transformer-based foundation model (FM) pre-trained on millions of synthetic regression tasks to meta-learn a universal prior over tabular data, upon which TabPFN performs principled Bayesian inference in a single forward pass without dataset-specific retraining, delivering strong small-data regression performance and well-calibrated predictive uncertainty (required for effective AL). We benchmark ICAL against GP and RF across 10 materials datasets and TabPFN wins on 8 out of 10 datasets, achieving a mean saving of 52% in extra evaluations relative to GP and 29.77% relative to RF. Cross-validation analysis confirms that TabPFN's advantage stems from superior uncertainty calibration, achieving the lowest Negative Log-Likelihood and Area Under the Sparsification Error curve among all surrogates. These results demonstrate that pre-trained FMs can serve as effective surrogates for active learning, enabling data-efficient discovery across diverse materials systems and small-data experimental sciences.

Figures & tables

Figure 1: Foundation models resolve the surrogate modeling dilemma in active learning. (a) Conventional surrogate models used in active learning face a trade-off between predictive capacity and reliable uncertainty estimation. Gaussian processes provide principled uncertainty but limited modeling flexibility, while high-capacity models such as random forests and neural networks can capture complex relationships but often produce poorly calibrated uncertainty in small-data regimes. (b) Foundation models trained to approximate Bayesian inference over tabular data combine expressive prediction with calibrated uncertainty through meta-learned priors. (c) Active learning with foundation-model surrogates identifies achieve good balance of modeling capacity and uncertainty estimation capability. (d) FM-based AL can find optimal materials with fewer experimental evaluations.

Figure 2: Pool-based active learning pipeline for discovering new materials. F: EI can be replaced with any other acquisition functions. G: k=1 is used for maximum sample efficiency. H: Top-1 task can be replaced with discovering Any of All of Top-K tasks.

Acquisition Strategy	Mathematical Formulation	Description & Scaling Notes
Standard UCB	$\mu(x)+\beta\sigma(x)$	Balances mean and uncertainty using raw, unnormalized values (prone to scale issues).
Normalized UCB	$\mu_{norm}(x)+\beta\sigma_{norm}(x)$	Scale-invariant. Both $\mu$ and $\sigma$ are normalized to $[0,1]$ before scaling with $\beta$ .
Hybrid UCB	$\mu_{norm}(x)+\beta\sigma_{GP\_norm}(x)$	Uses normalized mean from TabPFN and normalized uncertainty from a Gaussian Process.
Standard EI	$(\mu(x)-f_{best})\Phi(Z)+\sigma(x)\phi(Z)$	Calculates Expected Improvement using raw units without a trade-off parameter.
Normalized EI	$(\hat{\mu}(x)-\hat{f}_{best}-\xi)\Phi(Z)+\hat{\sigma}(x)\phi(Z)$	Calculates Expected Improvement using min-max normalized predictions and a trade-off parameter $\xi$ , where $Z=\frac{\hat{\mu}(x)-\hat{f}_{best}-\xi}{\hat{\sigma}(x)}$ .
Hybrid EI	Same as Normalized EI	Computes Normalized EI $\xi$ utilizing the mean from TabPFN and the uncertainty from the GP.

Table 1: Summary of Active Learning Acquisition Functions

Surrogate Model	Predictive Mean ( $\mu(x)$ )	Predictive Uncertainty ( $\sigma(x)$ )
TabPFN	The median (50th percentile) of the output distribution predicted directly by the pre-trained transformer network.	Derived from the predicted 95% confidence interval. Calculated as the difference between the 97.5th and 2.5th predicted percentiles divided by $3.92$ .
Random Forest (RF)	The average prediction across all $T$ individual decision trees in the ensemble: $\mu(x)=\frac{1}{T}\sum_{t=1}^{T}\hat{y}_{t}(x)$	The standard deviation of predictions across the ensemble of trees, serving as a measure of epistemic uncertainty (model disagreement): $\sigma(x)=\sqrt{\frac{1}{T}\sum_{t=1}^{T}(\hat{y}_{t}(x)-\mu(x))^{2}}$
Gaussian Process (GP)	The analytical posterior predictive mean, exacted via exact probabilistic conditioning on the training data.	The analytical posterior predictive standard deviation, representing geometric uncertainty governed heavily by distance in the kernel space (e.g., RBF).

Table 2: Summary of Predictive Mean and Uncertainty Estimation by Surrogate Models

Dataset Code	Dataset	Samples	Features	Opt. Objective
LTC_C	LTC Concentration	3,148	62	LTC
LTC_M	LTC Magpie	3,148	126	LTC
LTC_S	LTC Structure	3,148	146	LTC
Cu_HD_C	Cu Alloy Hardness Concentration	1,614	34	Hardness	[ 54 ]
Cu_HD_M	Cu Alloy Hardness Magpie	1,614	132	Hardness	[ 54 ]
Cu_EC_C	Cu Alloy Elect. Conduct. Concentration	1,826	34	Electrical Conductivity	[ 54 ]

Table 3: Summary of 25 Benchmark Datasets

Figure 3: Performance of ICAL on Cu_hardness dataset compared to GP and RF based AL. (a-b) Tabpfn against GP; (c-d): Tabpfn against RF.

Panel of Figure 4 — Figure 4: Performance of ICAL on (a) (c) Cu_electric dataset and (b) (d) Glass_DS3 dataset compared to GP and RF based AL.

Figure 5: Performance of ICAL on LTC_conc dataset with element concentration features compared to GP and RF based AL.

Dataset	Gauss GP	RF	TabPFN	Winner	Best Acq (TabPFN)	% Saving. (vs GP)	% Saving. (vs RF)
cu_electric	58.5	37.5	[ 30.1 ]	TabPFN	ei	48.5%	19.7%
cu_electric_magpie	82.5	90.3	83.9	Gauss	ucb_norm	$-$ 1.7%	7.1%
cu_hardness	31.3	18.8	[ 8.8 ]	TabPFN	ucb	71.9%	53.2%
cu_hardness_magpie	21.9	23.5	24.1	Gauss	ei_hybrid	$-$ 10.0%	$-$ 2.6%
glass_ds1_conc	72.4	66.0	48.8	TabPFN	ucb_norm	32.6%	26.1%
glass_ds2_physical	37.6	43.7	32.6	TabPFN	ucb_norm	13.3%	25.4%

Table 4: Mean number of extra evaluations per surrogate model averaged across all 19 init_ratio levels (5%–95%), using each model’s best acquisition function for that dataset. Percentage of extra evaluation saving achieved by TabPFN relative to GP and RF are shown in the final two columns; negative values indicate TabPFN underperforms the baseline (only for the inappropriate magppie feature sets).

Model	RMSE $\downarrow$	$R^{2}$ $\uparrow$	Spearman $\uparrow$	NLL $\downarrow$	PICP (0.95)	MPIW $\downarrow$	AUSE $\downarrow$
Gaussian Process	$8.38\pm 0.51$	$0.73\pm 0.04$	$0.81\pm 0.02$	$3.46\pm 0.03$	$\mathbf{0.93\pm 0.01}$	$32.50\pm 1.02$	$5.58\pm 0.16$
Random Forest	$5.86\pm 0.28$	$0.87\pm 0.02$	$0.89\pm 0.02$	$3.11\pm 0.25$	$0.87\pm 0.02$	$15.93\pm 0.35$	$1.77\pm 0.42$
TabPFN	$\mathbf{5.56\pm 0.42}$	$\mathbf{0.88\pm 0.03}$	$\mathbf{0.92\pm 0.01}$	$\mathbf{2.34\pm 0.15}$	$0.88\pm 0.02$	$\mathbf{11.16\pm 0.34}$	$\mathbf{0.83\pm 0.12}$

Table 5: Uncertainty Quantification & Regression Performance Comparison. Values are reported as Mean

\pm

Standard Deviation. Arrows indicate whether lower (

\downarrow

) or higher (

\uparrow

) values are better. All are 5-Fold Cross-Validation Results for Cu-alloy Electrical Conductivity dataset (%IACS). Best results are highlighted in bold .

Model	RMSE $\downarrow$	R 2 $\uparrow$	Spearman $\uparrow$	NLL $\downarrow$	PICP (95%)	MPIW $\downarrow$	AUSE $\downarrow$
Gaussian Process	53.26 $\pm$ 33.75	$-$ 0.01 $\pm$ 0.01	0.61 $\pm$ 0.04	5.81 $\pm$ 1.08	0.99 $\pm$ 0.00	237.23 $\pm$ 39.87	10.80 $\pm$ 3.56
Random Forest	48.62 $\pm$ 32.22	0.18 $\pm$ 0.06	0.83 $\pm$ 0.01	7.53 $\pm$ 3.42	0.86 $\pm$ 0.02	24.37 $\pm$ 2.74	4.50 $\pm$ 1.84
TabPFN	48.86 $\pm$ 32.78	0.18 $\pm$ 0.06	0.86 $\pm$ 0.01	3.22 $\pm$ 0.32	0.94 $\pm$ 0.01	33.16 $\pm$ 3.47	2.16 $\pm$ 0.43

Table 6: Five-fold cross-validation results on the LTC dataset with concentration features. Values are reported as mean

\pm

standard deviation across folds. Arrows indicate whether lower (

\downarrow

) or higher (

\uparrow

) values are better. PICP: prediction interval coverage probability; MPIW: mean prediction interval width; AUSE: area under the sparsification error curve. Best results are highlighted in bold.

Active Learning

Active learning selects the examples, labels, features, or experiments most useful for improving a model under limited budgets.

See all 280 papers

Active In-Context Learning for Tabular Foundation Models
Mar 28, 2026Wilailuck Treerath, Fabrizio PittorinoActive Learning Tabular ICL
Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milano, Italy
Surrogate-Gated Generation and Foundation-Model Embeddings for Bayesian Materials Design
Jun 26, 2026Sk Md Ahnaf Akif Alvi, Jan Janssen, Danny Perez +2Surrogate-Assisted Optimization Bayesian Optimization
Department of Materials Science and Engineering, Texas A&M University, College Station, TX 77843, USA · Theoretical Division T-1, Los Alamos National Laboratory, Los Alamos, NM 87544, USA · Max-Planck-Institute for Sustainable Materials, Düsseldorf, Germany 40237 +3
ALF: An Active Learning Framework for Scientific Discovery
Sep 25, 2026Shikha Surana, Alex Hawkins-Hooker, Olivia Gallup +3Active Learning Scientific ML
InstaDeep · University of Oxford

Mar 28, 2026J/K move · Enter open · S save

Active In-Context Learning for Tabular Foundation Models

Wilailuck Treerath, Fabrizio Pittorino

Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milano, Italy

Active Learning Tabular ICL In-Context Learning Tabular Foundation Models

Open paper AskPDF arXiv

Active learning (AL) reduces labeling cost by querying informative samples, but in tabular settings its cold-start gains are often limited because uncertainty estimates are unreliable when models are trained on very few labels. Tabular foundation models such as TabPFN provide calibrated probabilistic predictions via in-context learning (ICL), i.e., without task-specific weight updates, enabling an AL regime in which the labeled context - rather than parameters - is iteratively optimized. We formalize Tabular Active In-Context Learning (Tab-AICL) and instantiate it with four acquisition rules: uncertainty (TabPFN-Margin), diversity (TabPFN-Coreset), an uncertainty-diversity hybrid (TabPFN-Hybrid), and a scalable two-stage method (TabPFN-Proxy-Hybrid) that shortlists candidates using a lightweight linear proxy before TabPFN-based selection. Across 20 classification benchmarks, Tab-AICL improves cold-start sample efficiency over retrained gradient-boosting baselines (CatBoost-Margin and XGBoost-Margin), measured by normalized AULC up to 100 labeled samples.

Abstract

Active Learning

Explore similar work

Active In-Context Learning for Tabular Foundation Models

Surrogate-Gated Generation and Foundation-Model Embeddings for Bayesian Materials Design

ALF: An Active Learning Framework for Scientific Discovery