TabPFN Extensions for Interpretable Geotechnical Modelling

Taiga Saito; Yu Otake; Daijiro Mizutani; Stephen Wu

cs.CEMar 22, 2026

TabPFN Extensions for Interpretable Geotechnical Modelling

Authors: Taiga Saito, Yu Otake, Daijiro Mizutani, Stephen Wu

Organizations: Department of Civil and Environmental Engineering, Tohoku University, 6-6-06 Aramaki Aoba, Aoba-ku, Sendai, Miyagi 980-8579, Japan · Research Organization of Information and Systems, The Institute of Statistical Mathematics 10-3 Midori-cho, Tachikawa, Tokyo 190-8562, Japan · Department of Statistical Science, The Graduate University for Advanced Studies 10-3 Midori-cho, Tachikawa, Tokyo 190-8562, Japan

Abstract

Geotechnical site characterisation relies on sparse, heterogeneous borehole data, where uncertainty quantification and interpretability matter as much as predictive accuracy. We evaluate TabPFN~\citep{Hollmann2025}, a tabular foundation model, and its tabpfn-extensions library on two geotechnical tasks: (1) soil-type classification from N-value and shear-wave velocity data as a controlled illustrative case, and (2) iterative imputation of five mechanical parameters ( $s_\mathrm{u}$ , $E_{\mathrm{u}}$ , ${σ'}_\mathrm{p}$ , $C_\mathrm{c}$ , $C_\mathrm{v}$ ) in BM/AirportSoilProperties/2/2025. Without retraining, we apply cosine-similarity analysis to TabPFN embeddings, visualise predictive distributions, and compute SHAP attributions. On the regression benchmark we compare TabPFN with mean imputation, linear regression, random forests, XGBoost, and HBM; introduce a proxy decomposition of predictive uncertainty across context-perturbation classes; and propagate marginal $C_\mathrm{c}$ and ${σ'}_\mathrm{p}$ distributions through a one-dimensional consolidation model to obtain the reliability index $β$ and serviceability exceedance probability $P_\mathrm{f}$ . Embeddings exhibit label-consistent Clay/Sand grouping; iterative imputation reduces RMSE for all five targets, with TabPFN lowest on four; SHAP attributions are consistent with the Skempton compression-index correlation and the inverse preconsolidation-pressure-water-content dependence; the within-posterior component is largest in the proxy decomposition. We position the contribution as a worked evaluation workflow that may complement established methods for data-scarce geotechnics, not as algorithmic innovation.

Figures & tables

Training set ( $n=16$ )				Test set ( $n=16$ )
No.	$N$	$V_{\mathrm{s}}$ (m/s)	Soil	No.	$N$	$V_{\mathrm{s}}$ (m/s)	Soil
1	1	100.000	Clay	1	4	158.740	Clay
2	2	125.992	Clay	2	5	170.998	Clay
3	7	191.293	Clay	3	9	208.008	Clay
4	12	228.943	Clay	4	10	215.443	Clay
5	15	246.621	Clay	5	11	222.398	Clay

Table 1: All samples in the synthetic geotechnical dataset. No. corresponds to the axis labels in Figure 3 : training samples are ordered along the row axis and test samples along the column axis (Clay first, sorted by ascending

N

-value; Sand second, sorted by ascending

N

-value).

Figure 1: Scatter plots of training (circles) and test (stars) data in the $N$ – $V_{\mathrm{s}}$ space, coloured by soil class (Clay: blue, Sand: orange). Training and test sets each contain 16 samples. Reference curves $V_{\mathrm{s}}=100N^{1/3}$ and $V_{\mathrm{s}}=80N^{1/3}$ from the Japanese railway seismic design standard ( Railway Technical Research Institute 2012 ) are overlaid; Clay samples follow the upper curve and Sand samples the lower curve.

Figure 2: Predicted probability of Sand class $P(\text{Sand})$ over the $N$ – $V_{\mathrm{s}}$ domain, with the training samples overlaid (circles). The bold contour marks the decision boundary at $P(\text{Sand})=0.5$ ; grey contours indicate the 0.1, 0.25, 0.75, and 0.9 levels.

Figure 3: Cosine similarity heatmap of TabPFN embeddings between test and training samples (axis labels correspond to Table 1 No.). The block-diagonal structure indicates that, in this controlled case, test embeddings are more similar to training embeddings from the same constructed class. Test sample No. 8 (first Sand test sample; $N=14$ , $V_{\mathrm{s}}=193$ m/s) shows visibly lower similarity with the training Sand block, illustrating how the heatmap can flag a boundary sample.

Figure 4: Normalised RMSE (RMSE / RMSE at iteration 1) per iteration for each mechanical parameter. Values below 1.0 indicate improvement over the initial estimate.

Parameter	RMSE (iter. 1)	RMSE (iter. 10)
$s_{\mathrm{u}}$ (kN/m 2 )	13.08	9.30
$E_{\mathrm{u}}$ (kN/m 2 )	3277	2692
${\sigma^{\prime}}_{\mathrm{p}}$ (kN/m 2 )	30.47	25.89
$C_{\mathrm{c}}$ (—)	0.118	0.106
$C_{\mathrm{v}}$ (m 2 /s)	35.72	34.58

Table 2: RMSE at iteration 1 and iteration 10 for each mechanical parameter.

Figure 5: Posterior distributions at iteration 10 for all five mechanical parameters across test samples. Violin width represents the probability density; black bar = median; yellow circle = true value; black filled circle = observed value (parameter was not missing for that sample and therefore not predicted).

Figure 6: Mean absolute SHAP values for each input feature across all test samples, shown separately for each mechanical parameter. Blue bars: index properties; orange bars: non-target mechanical parameters used as features in the iterative imputation scheme.

Figure 7: SHAP value versus feature value for the dominant predictors of each mechanical target. Columns (a)–(e) correspond to

s_{\mathrm{u}}

E_{\mathrm{u}}

{\sigma^{\prime}}_{\mathrm{p}}

C_{\mathrm{c}}

, and

C_{\mathrm{v}}

, respectively. The upper row shows the best non-target mechanical predictor per target by mean

|\mathrm{SHAP}|

(

s_{\mathrm{u}}\!\leftarrow\!E_{\mathrm{u}}

E_{\mathrm{u}}\!\leftarrow\!s_{\mathrm{u}}

{\sigma^{\prime}}_{\mathrm{p}}\!\leftarrow\!s_{\mathrm{u}}

C_{\mathrm{c}}\!\leftarrow\!C_{\mathrm{v}}

C_{\mathrm{v}}\!\leftarrow\!C_{\mathrm{c}}

); the lower row shows the best index-property predictor per target (

s_{\mathrm{u}},E_{\mathrm{u}}\!\leftarrow\!PL

;

{\sigma^{\prime}}_{\mathrm{p}}\!\leftarrow\!w

;

C_{\mathrm{c}},C_{\mathrm{v}}\!\leftarrow\!LL

). Positive SHAP values indicate that the feature increases the predicted value; negative values indicate a decrease.

Figure 8: Embedding-based borehole-structure diagnostic for the five BM/AirportSoilProperties mechanical targets. (a)

k\!=\!3,\,5,\,10,\,20

nearest-neighbour borehole purity per target at the default

n_{\mathrm{top}}\!=\!8

; the dashed line is the random-neighbour baseline. (b) PCA-reduced cosine

\Delta_{\mathrm{cos}}

n_{\mathrm{top}}\!=\!8

as a function of dimensionality

d

. (c)

k\!=\!3

borehole-purity ratio (mean purity divided by the random-neighbour baseline) against

n_{\mathrm{top}}\!\in\!\{4,8,12,16\}

; the dotted line marks ratio

=\!1

(chance level). (d) PCA-reduced cosine

\Delta_{\mathrm{cos}}

d\!=\!20

against

n_{\mathrm{top}}

	Mean imp.	Linear	HBM	RF	XGBoost	TabPFN
$C_{\mathrm{c}}$	0.332	0.127	0.128	0.108	0.123	0.106
$E_{\mathrm{u}}$ (kN/m 2 )	3834	2952	2588	3212	2376	2692
$s_{\mathrm{u}}$ (kN/m 2 )	24.50	16.19	9.74	14.64	16.69	9.30
${\sigma^{\prime}}_{\mathrm{p}}$ (kN/m 2 )	49.93	35.44	28.24	32.92	38.79	25.89
$C_{\mathrm{v}}$ (m 2 /s)	168.71	119.16	59.32	46.91	60.90	34.58

Table 3: Final-iteration RMSE on the BM/AirportSoilProperties 20-sample missing pattern. Units follow each target. Lowest value per target is in bold.

Figure 9: Matched-

\kappa

proxy decomposition of TabPFN’s predictive variance on the BM/AirportSoilProperties 20-sample missing pattern. Coloured curves are the across-scenario components for three context perturbations against

\kappa

: bootstrap with replacement (WR; dashed), without replacement (WOR; solid), and borehole-level holdout (holdout; solid), with

95\%

percentile-bootstrap CI bands from

20

scenarios per cell. The dashed blue baseline is the multi-seed run-to-run component estimated at

\kappa=1.0

with the context fixed. The violet line is the within-posterior component averaged across the three context classes; no CI band is shown for this cross-class average.

Figure 10: Per-sample settlement comparison between TabPFN (blue circles), HBM (orange squares), and the deterministic ground truth (gold circles) for the 15 jointly imputed samples. Markers show

S

evaluated at the saved posterior mean of

(C_{\mathrm{c}},{\sigma^{\prime}}_{\mathrm{p}})

; error bars show 5%/95% Monte Carlo intervals propagated through Eq. ( 3 ). HBM marginals are reconstructed by log-normal quantile matching.

Capability	HBM	Conv. ML †	TabPFN + Ext.
Posterior predictive distribution	Yes (parametric)	No ∗	Yes (native)
Embedding similarity	Not native	No	Yes
SHAP feature importance	Indirect ‡	Yes	Yes
No retraining required	No	No	Yes
Correlation structure	Parametric	—	Implicit / prior-learned
User-specified priors	Yes	No	No

Table 4: TabPFN-centric capability summary; complementary strengths of HBM and conventional ML are discussed in the surrounding paragraph.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 649 papers

May 31, 2026

On the Uncertainty Quantification Ability of Tabular Foundation Models

Foundation models (FMs) have achieved substantial success in generalizing across tasks without problemspecific training or fine-tuning. However, many critical applications in mechanics and computational science require not only accurate predictions but also reliable uncertainty quantification (UQ). Herein we investigate the UQ capabilities of tabular FMs in regression tasks through a comprehensive empirical study comparing Tabular Prior-Data Fitted Networks (TabPFN) against Gaussian processes (GPs). We systematically evaluate these two methods across a host of regression problems with varying complexity, dataset sizes, and input dimensionalities. We use a default setting to build all the GPs and for a fair comparison against TabPFN v2.5. Our findings highlight an important trade-off between explicit and learned priors: while TabPFN achieves highly competitive performance for complex, high-dimensional problems with sufficient data, GPs often provide superior predictive accuracy and UQ in data-scarce settings. Moreover, when the chosen kernel constitutes a good prior for the underlying function, GP performance can substantially exceed that of TabPFN. Our results can be reproduced from https://github.com/kianswarehouse/GPvsPFN.

Tyler R. Johnson, Kian Ben-Jacob, Nima Negarandeh +2

Department of Mechanical and Aerospace Engineering, University of California, Irvine · Department of Civil and Environmental Engineering, University of California, Irvine

Tabular Foundation Models Uncertainty Quantification Prior-Data Fitted Networks Tabular Regression Gaussian Process Regression

Ask PDF

Sep 15, 2026

TabPFN-3.5: Technical Report

We introduce TabPFN-3.5, our new flagship Tabular Foundation Model. It significantly outperforms its predecessor, TabPFN-3, and all existing baselines across a broad range of tabular problems. TabPFN-3.5 sets a new state of the art on standard tabular prediction in TabArena, and extends it to the data practitioners encounter in practice: non-i.i.d. data with temporal or grouped splits, tables with strings, text and images, high-cardinality categorical features, and wide tables with many features. These gains carry over to our task-specific harnesses: state of the art on relational data and stronger time-series forecasting. For faster inference, our variant TabPFN-3.5-Fast runs up to 3x faster than TabPFN-3 while keeping most of the accuracy gains. In addition, we upgrade TabPFN-3.5-Plus, expanding our multimodal capabilities with advanced text and date handling alongside proprietary inference optimizations. Finally, we release a new version of our Thinking mode, TabPFN-3.5-Thinking, which scales inference-time computation to push the state of the art further. It benefits from our stronger base model and from inference-time improvements that make it up to 12x faster than TabPFN-3-Thinking.

Benjamin Jäger, Nick Erickson, Léo Grinsztajn +44

Tabular Foundation Models Inference-Time Optimization Tabular ICL Tabular Prediction

Ask PDF

Aug 14, 2026

Probabilistic indirect models for undrained shear strength: addressing significant data missing and variability with advanced imputation and machine learning techniques

Accurate prediction of undrained shear strength (su) is crucial for geotechnical design, but is often hampered by substantial uncertainty in traditional empirical methods. This study uses the CLAY/10/7490 global database to develop probabilistic indirect models to predict su based on Atterberg limits and piezocone cone penetration (CPTU) measurements. Firstly, the dataset has a high missing data rate and variability. We test three imputation methods - multivariate normal (MN), multiple imputation by chained equations (MICE), and miss forest (MF) - to fill the missing values. To validate their effectiveness, a Probabilistic Extreme Gradient Boosting (PXGB) model is developed, and the imputation methods are evaluated by comparing the PXGB's performance when trained on the imputed datasets against that on the original incomplete data. Secondly, the indirect model is built by integrating a multi-head attention (MHA) mechanism into an artificial neural network (ANN) to enhance information extraction from limited data, which leads to the MHA-based probabilistic neural networks (MHA-PNN) model. The models' performance, alongside a conventional MN-based prediction model, was evaluated using root mean square error (RMSE), coefficient of determination (R2), mean absolute percentage error (MAPE), conditional interval width (wCI), and coverage rate (CR). Results demonstrate that the proposed MN-enhanced MHA-PNN model substantially outperforms other models in both prediction accuracy and uncertainty quantification. These findings highlight the potential of this integrated strategy for building robust probabilistic indirect models in geotechnical applications, particularly when confronted with sparse and incomplete datasets.

Haibin Xiong, Shaoheng Dai, Peng Lan +4

School of Civil and Environmental Engineering, University of Technology Sydney, Ultimo, NSW 2007, Australia. · School of Infrastructure Engineering, Nanchang University, Xuefu Road, 999, Nanchang 330031, China. · School of Civil Engineering, Central South University, Changsha, 410075, China.

Materials Property Prediction Uncertainty Quantification Incomplete Data Imputation

Ask PDF

Abstract

Uncertainty Quantification

Explore similar work

On the Uncertainty Quantification Ability of Tabular Foundation Models

TabPFN-3.5: Technical Report

Probabilistic indirect models for undrained shear strength: addressing significant data missing and variability with advanced imputation and machine learning techniques