Foundation-Model Surrogates Enable Data-Efficient Active Learning for Materials Discovery
Authors: Jeffrey Hu, Rongzhi Dong, Ying Feng, Ming Hu, Jianjun Hu
Organizations: Department of Materials Science and Engineering, University of Illinois Urbana Champaign, Champaign,IL, USA · Department of Computer Science & Engineering, University of South Carolina, Columbia, SC, USA · Zuoyue Honors College, Hangzhou Dianzhi University, Hangzhou, China · Department of Mechanical Engineering, University of South Carolina, Columbia, SC, USA
Active learning (AL) has emerged as a powerful paradigm for accelerating materials discovery by iteratively steering experiments toward promising candidates, reducing the number of costly synthesis-and-characterization cycles needed to identify optimal materials. However, current AL relies predominantly on Gaussian Process (GP) and Random Forest (RF) surrogates, which suffer from complementary limitations: GP underfits complex composition-property landscapes due to rigid kernel assumptions, while RF produces unreliable heuristic uncertainty estimates in small-data regimes. This small-data challenge is pervasive in materials science, making reliable surrogate modeling extremely difficult with models trained from scratch on each new dataset. Here we propose In-Context Active Learning (ICAL), which addresses this bottleneck by replacing conventional surrogates with TabPFN, a transformer-based foundation model (FM) pre-trained on millions of synthetic regression tasks to meta-learn a universal prior over tabular data, upon which TabPFN performs principled Bayesian inference in a single forward pass without dataset-specific retraining, delivering strong small-data regression performance and well-calibrated predictive uncertainty (required for effective AL). We benchmark ICAL against GP and RF across 10 materials datasets and TabPFN wins on 8 out of 10 datasets, achieving a mean saving of 52% in extra evaluations relative to GP and 29.77% relative to RF. Cross-validation analysis confirms that TabPFN's advantage stems from superior uncertainty calibration, achieving the lowest Negative Log-Likelihood and Area Under the Sparsification Error curve among all surrogates. These results demonstrate that pre-trained FMs can serve as effective surrogates for active learning, enabling data-efficient discovery across diverse materials systems and small-data experimental sciences.
Figures & tables
Figure 1: Foundation models resolve the surrogate modeling dilemma in active learning. (a) Conventional surrogate models used in active learning face a trade-off between predictive capacity and reliable uncertainty estimation. Gaussian processes provide principled uncertainty but limited modeling flexibility, while high-capacity models such as random forests and neural networks can capture complex relationships but often produce poorly calibrated uncertainty in small-data regimes. (b) Foundation models trained to approximate Bayesian inference over tabular data combine expressive prediction with calibrated uncertainty through meta-learned priors. (c) Active learning with foundation-model surrogates identifies achieve good balance of modeling capacity and uncertainty estimation capability. (d) FM-based AL can find optimal materials with fewer experimental evaluations.
Figure 2: Pool-based active learning pipeline for discovering new materials. F: EI can be replaced with any other acquisition functions. G: k=1 is used for maximum sample efficiency. H: Top-1 task can be replaced with discovering Any of All of Top-K tasks.
Acquisition Strategy
Mathematical Formulation
Description & Scaling Notes
Standard UCB
μ(x)+βσ(x)
Balances mean and uncertainty using raw, unnormalized values (prone to scale issues).
Normalized UCB
μnorm(x)+βσnorm(x)
Scale-invariant. Both μ and σ are normalized to [0,1] before scaling with β .
Hybrid UCB
μnorm(x)+βσGP_norm(x)
Uses normalized mean from TabPFN and normalized uncertainty from a Gaussian Process.
Standard EI
(μ(x)−fbest)Φ(Z)+σ(x)ϕ(Z)
Calculates Expected Improvement using raw units without a trade-off parameter.
Normalized EI
(μ^(x)−f^best−ξ)Φ(Z)+σ^(x)ϕ(Z)
Calculates Expected Improvement using min-max normalized predictions and a trade-off parameter ξ , where Z=σ^(x)μ^(x)−f^best−ξ .
Hybrid EI
Same as Normalized EI
Computes Normalized EI ξ utilizing the mean from TabPFN and the uncertainty from the GP.
Table 1: Summary of Active Learning Acquisition Functions
Surrogate Model
Predictive Mean ( μ(x) )
Predictive Uncertainty ( σ(x) )
TabPFN
The median (50th percentile) of the output distribution predicted directly by the pre-trained transformer network.
Derived from the predicted 95% confidence interval. Calculated as the difference between the 97.5th and 2.5th predicted percentiles divided by 3.92 .
Random Forest (RF)
The average prediction across all T individual decision trees in the ensemble: μ(x)=T1∑t=1Ty^t(x)
The standard deviation of predictions across the ensemble of trees, serving as a measure of epistemic uncertainty (model disagreement): σ(x)=T1∑t=1T(y^t(x)−μ(x))2
Gaussian Process (GP)
The analytical posterior predictive mean, exacted via exact probabilistic conditioning on the training data.
The analytical posterior predictive standard deviation, representing geometric uncertainty governed heavily by distance in the kernel space (e.g., RBF).
Table 2: Summary of Predictive Mean and Uncertainty Estimation by Surrogate Models
Dataset Code
Dataset
Samples
Features
Opt. Objective
LTC_C
LTC Concentration
3,148
62
LTC
LTC_M
LTC Magpie
3,148
126
LTC
LTC_S
LTC Structure
3,148
146
LTC
Cu_HD_C
Cu Alloy Hardness Concentration
1,614
34
Hardness
[ 54 ]
Cu_HD_M
Cu Alloy Hardness Magpie
1,614
132
Hardness
[ 54 ]
Cu_EC_C
Cu Alloy Elect. Conduct. Concentration
1,826
34
Electrical Conductivity
[ 54 ]
Table 3: Summary of 25 Benchmark Datasets
Figure 3: Performance of ICAL on Cu_hardness dataset compared to GP and RF based AL. (a-b) Tabpfn against GP; (c-d): Tabpfn against RF.
Figure 4: Performance of ICAL on (a) (c) Cu_electric dataset and (b) (d) Glass_DS3 dataset compared to GP and RF based AL.
Figure 5: Performance of ICAL on LTC_conc dataset with element concentration features compared to GP and RF based AL.
Dataset
Gauss GP
RF
TabPFN
Winner
Best Acq (TabPFN)
% Saving. (vs GP)
% Saving. (vs RF)
cu_electric
58.5
37.5
[ 30.1 ]
TabPFN
ei
48.5%
19.7%
cu_electric_magpie
82.5
90.3
83.9
Gauss
ucb_norm
− 1.7%
7.1%
cu_hardness
31.3
18.8
[ 8.8 ]
TabPFN
ucb
71.9%
53.2%
cu_hardness_magpie
21.9
23.5
24.1
Gauss
ei_hybrid
− 10.0%
− 2.6%
glass_ds1_conc
72.4
66.0
48.8
TabPFN
ucb_norm
32.6%
26.1%
glass_ds2_physical
37.6
43.7
32.6
TabPFN
ucb_norm
13.3%
25.4%
Table 4: Mean number of extra evaluations per surrogate model averaged across all 19 init_ratio levels (5%–95%), using each model’s best acquisition function for that dataset. Percentage of extra evaluation saving achieved by TabPFN relative to GP and RF are shown in the final two columns; negative values indicate TabPFN underperforms the baseline (only for the inappropriate magppie feature sets).
Model
RMSE ↓
R2↑
Spearman ↑
NLL ↓
PICP (0.95)
MPIW ↓
AUSE ↓
Gaussian Process
8.38±0.51
0.73±0.04
0.81±0.02
3.46±0.03
0.93±0.01
32.50±1.02
5.58±0.16
Random Forest
5.86±0.28
0.87±0.02
0.89±0.02
3.11±0.25
0.87±0.02
15.93±0.35
1.77±0.42
TabPFN
5.56±0.42
0.88±0.03
0.92±0.01
2.34±0.15
0.88±0.02
11.16±0.34
0.83±0.12
Table 5: Uncertainty Quantification & Regression Performance Comparison. Values are reported as Mean ± Standard Deviation. Arrows indicate whether lower ( ↓ ) or higher ( ↑ ) values are better. All are 5-Fold Cross-Validation Results for Cu-alloy Electrical Conductivity dataset (%IACS). Best results are highlighted in bold .
Model
RMSE ↓
R 2 ↑
Spearman ↑
NLL ↓
PICP (95%)
MPIW ↓
AUSE ↓
Gaussian Process
53.26 ± 33.75
− 0.01 ± 0.01
0.61 ± 0.04
5.81 ± 1.08
0.99 ± 0.00
237.23 ± 39.87
10.80 ± 3.56
Random Forest
48.62 ± 32.22
0.18 ± 0.06
0.83 ± 0.01
7.53 ± 3.42
0.86 ± 0.02
24.37 ± 2.74
4.50 ± 1.84
TabPFN
48.86 ± 32.78
0.18 ± 0.06
0.86 ± 0.01
3.22 ± 0.32
0.94 ± 0.01
33.16 ± 3.47
2.16 ± 0.43
Table 6: Five-fold cross-validation results on the LTC dataset with concentration features. Values are reported as mean ± standard deviation across folds. Arrows indicate whether lower ( ↓ ) or higher ( ↑ ) values are better. PICP: prediction interval coverage probability; MPIW: mean prediction interval width; AUSE: area under the sparsification error curve. Best results are highlighted in bold.
Active learning (AL) reduces labeling cost by querying informative samples, but in tabular settings its cold-start gains are often limited because uncertainty estimates are unreliable when models are trained on very few labels. Tabular foundation models such as TabPFN provide calibrated probabilistic predictions via in-context learning (ICL), i.e., without task-specific weight updates, enabling an AL regime in which the labeled context - rather than parameters - is iteratively optimized. We formalize Tabular Active In-Context Learning (Tab-AICL) and instantiate it with four acquisition rules: uncertainty (TabPFN-Margin), diversity (TabPFN-Coreset), an uncertainty-diversity hybrid (TabPFN-Hybrid), and a scalable two-stage method (TabPFN-Proxy-Hybrid) that shortlists candidates using a lightweight linear proxy before TabPFN-based selection. Across 20 classification benchmarks, Tab-AICL improves cold-start sample efficiency over retrained gradient-boosting baselines (CatBoost-Margin and XGBoost-Margin), measured by normalized AULC up to 100 labeled samples.
Wilailuck Treerath, Fabrizio Pittorino
Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milano, Italy
Closed-loop materials discovery iterates between proposing candidate structures and evaluating their properties, and property evaluation dominates the cost. In the generative variant, a learned prior proposes candidate crystals and a property oracle scores them; we ask whether a cheap probabilistic surrogate can triage the generator's output, and what such a surrogate must do well. Across three architecturally distinct pretrained diffusion priors (MatterGen, CrystalFlow, ADiT) and two targets (room-temperature heat capacity and bulk modulus), we insert a Gaussian process acquisition gate between structure generation and the oracle in an RL-steered generative workflow. The gate matches or exceeds ungated fine-tuning of the generative model while capping oracle calls at a fixed per-cycle budget. Budget-matched ablations isolate the mechanism. At an identical four-call budget, ranking-based selection outperforms arbitrary selection, confirming that the gain comes from the surrogate's choice; the gate comes within ∼9% of exhaustive oracle spending at roughly one-fifth of the calls. A density-functional-theory check of the bulk-modulus discoveries confirms the learned oracle to within 2.5% on average and the surrogate's ranking of the generated structures at Spearman ρ=0.94. A cross-factorial benchmark of surrogate performance spanning mechanical, electronic, and vibrational properties identifies pretrained ORB embeddings with a Gaussian process as the most reliable combination, which we adopt as the building blocks of the proposed workflow. The complete pipeline is released as open-source software.
Sk Md Ahnaf Akif Alvi, Jan Janssen, Danny Perez +2
Department of Materials Science and Engineering, Texas A&M University, College Station, TX 77843, USA · Theoretical Division T-1, Los Alamos National Laboratory, Los Alamos, NM 87544, USA · Max-Planck-Institute for Sustainable Materials, Düsseldorf, Germany 40237 +3
Machine learning for scientific discovery is almost systematically data bound. Producing relevant high quality data, under budget constraints, is amongst the most promising ways to advance the field. Active learning (AL) offers promise wherever labelling requires expensive experiment, measurement, or simulation. Most existing tools cover only part of the data acquisition loop, and typically focus on either offline benchmarking or online deployment, but not both. We present ALF, a modular AL Framework that runs the full data acquisition loop via five modular components. One clear API for both settings: offline, against an existing dataset for controlled and reproducible experimentation; and online, against an oracle for acquiring new candidates in real-world deployments. ALF is open-source and available at https://github.com/instadeepai/alf.
Shikha Surana, Alex Hawkins-Hooker, Olivia Gallup +3