Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression

Rafael Izbicki; Pedro L. C. Rodrigues

cs.LGMar 27, 2026

Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression

Authors: Rafael Izbicki, Pedro L. C. Rodrigues

Organizations: Department of Statistics, Federal University of São Carlos (UFSCar), Brazil. · Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, France.

Abstract

Conditional density estimation (CDE) - recovering the full conditional distribution of a response given tabular covariates - is essential in settings with heteroscedasticity, multimodality, or asymmetric uncertainty. Recent tabular foundation models, such as TabPFN and TabICL, naturally produce predictive distributions, but their effectiveness as general-purpose CDE methods has not been systematically evaluated, unlike their performance for point prediction, which is well studied. We benchmark three tabular foundation model variants against a diverse set of parametric, tree-based, and neural CDE baselines on 39 real-world datasets, across training sizes from 50 to 20,000, using six metrics covering density accuracy, calibration, and computation time. Across all sample sizes, foundation models achieve the best CDE loss, log-likelihood, and CRPS on the large majority of datasets tested. Calibration is competitive at small sample sizes but, for some metrics and datasets, lags behind task-specific neural baselines at larger sample sizes, suggesting that post-hoc recalibration may be a valuable complement. In a photometric redshift case study using SDSS DR18, TabPFN exposed to 50,000 training galaxies outperforms all baselines trained on the full 500,000-galaxy dataset. Taken together, these results establish tabular foundation models as strong off-the-shelf conditional density estimators.

Figures & tables

Figure 1: Illustration of CDE on a bimodal synthetic DGP. Each row corresponds to a different test instance whose true conditional density (dashed black) is a two-component Gaussian mixture with covariate-dependent means, variances, and weights. Columns show three training sizes ( $n\in\{50,200,2{,}000\}$ ). TabPFN-2.5 (orange) already captures the bimodal structure at $n=200$ , while Flow-Spline (blue) and FlexCode-RF (red) require considerably more data and still show spurious peaks or roughness. Note that this is a controlled synthetic example designed to illustrate differences among methods; see Section 5 for real data analyses.

Figure 2: CDE loss across real-world datasets at $n=50$ . Per-dataset raw CDE loss values (lower/greener is better). Datasets are sorted by covariate dimension $d$ . A $*$ marks foundation models that significantly outperform all parametric and nonparametric competitors on that dataset. The two TabPFN variants (orange) achieve the top two average ranks (bottom row), while TabICL-Quantiles (7.0) is narrowly outranked by Student-t-Ridge (6.6). Even at this extremely small sample size, a foundation model achieves the best CDE loss on 82% of datasets.

Figure 3: CDE loss across real-world datasets at $n=1{,}000$ . Per-dataset raw CDE loss values (lower/greener is better). Datasets are sorted by $d$ . A $*$ marks foundation models that significantly outperform all competitors on that dataset. All foundation models occupy the top three average ranks. Foundation models achieve the best CDE loss on 92% of datasets.

Figure 4: CDE loss across real-world datasets at $n=20{,}000$ . Per-dataset raw CDE loss values (lower/greener is better). Datasets are sorted by $d$ . A $*$ marks foundation models that significantly outperform all competitors on that dataset; $\times$ marks foundation models that encountered out-of-memory errors. All three foundation models occupy the top three average ranks. On CTSlices, both TabPFN variants ran out of memory at this sample size.

Figure 5: CDE loss vs. sample size for selected real-world datasets. Each panel shows one dataset, arranged by increasing $d$ . Foundation models (orange, bold) are contrasted against parametric (green, faded) and nonparametric (blue, faded) baselines; the $y$ -axis is oriented so that higher values are better. Foundation models are already competitive at $n=50$ and often exhibit the steepest improvement with growing $n$ . Parametric baselines plateau early and rarely reach the top regardless of sample size. Nonparametric methods show greater variability, occasionally matching foundation models in higher-dimensional settings but without consistency.

Figure 6: Diagnostic examples of foundation-model failures on real data. Each row shows a task where the best foundation model (orange) is beaten in CDE loss by the best non-foundation method (blue); five test cases are shown per row, with dashed lines marking observed outcomes. On Digits, CatMLP captures the discrete support more precisely than TabPFN-2.5 . On VideoTranscoding, LogNormal-Homo-Ridge produces sharper tail densities. On BlackFriday, CatMLP better captures the quasi-discrete response structure—a gap that persists even at the largest available sample size.

Figure 7: CDE loss (panel a), CRPS (panel b), and log-likelihood (panel c) vs. training sample size on the SDSS photometric redshift dataset. All $y$ -axes are oriented so that top = better (CDE loss and CRPS are inverted; log-likelihood is shown in natural order). Parametric methods (green): the shaded band spans the full range across all parametric models; the best-performing parametric model ( LinearGauss-Hetero ) is highlighted as a solid line. Ridge-regularized variants are omitted from the band as they are uniformly worse and would distort the $y$ -axis scale. Nonparametric methods (blue): each method is shown as a separate line; shaded $\pm 1$ SE bands are displayed only for the three best-performing nonparametric methods to avoid visual clutter. A terminal $\times$ marker indicates the last sample size at which a method was evaluated. Key finding: across all three metrics, TabPFN trained on 50k galaxies outperforms all methods trained on 500k galaxies, demonstrating a ten-fold data efficiency advantage.

Figure 8: Fit time (training + prediction) for each method across sample sizes $n$ on the SDSS photometric redshift dataset. Each cell reports the mean fit time (top) and standard error in parentheses. Cells are colour-coded within each method column from green (fastest) to red (slowest); grey cells indicate that the method was not run at that sample size due to computational limits. Fit time includes both training and inference on the test set. Key finding: Foundation models are substantially faster than neural baselines ( Flow-Spline , MDN ) at the same sample sizes, while parametric baselines are the cheapest overall but sacrifice density accuracy.

Appendix figures & tables43 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Search space
MDN	number of components $\in\{2,3,5\}$ ; hidden units $\in\{16,32,64\}$ ; learning rate $\in\{0.005,0.01,0.02\}$ ; epochs $\in\{300,500,800\}$
Flow-Spline	spline bins $\in\{4,8,12\}$ ; layers $\in\{2,3,4\}$ ; hidden units $\in\{32,64,128\}$ ; learning rate $\in\{10^{-3},2\times 10^{-3},5\times 10^{-3}\}$ ; weight decay $\in\{10^{-6},10^{-5},10^{-4}\}$
Quantile-Tree	number of boosting rounds $\in\{50,100,200\}$ ; max depth $\in\{3,4,6\}$ ; learning rate $\in\{0.05,0.1,0.2\}$
BART-Homo , BART-Hetero	number of trees $\in\{20,30,50\}$ ; number of sweeps $\in\{40,60,80\}$
CatMLP	number of bins $\in\{30,50,100\}$ ; hidden units $\in\{32,64,128\}$ ; learning rate $\in\{0.005,0.01,0.02\}$ ; epochs $\in\{300,500,800\}$

Appendix

Table 1: Random-search spaces for tuned baselines.

Dataset	Source	Max $n$	Domain
SpaceGA	OpenML	3,107	Spatial analysis
Elevators	OpenML	16,599	Control systems
Kin8nm	OpenML	8,192	Robotics
Puma8NH	OpenML	8,192	Robotics
Bank8FM	OpenML	22,784	Finance
CpuSmall	OpenML	8,192	Computer hardware

Appendix

Table 2: Real-world datasets.

Figure 9: Raw CDE Loss – $n=50$ , real data.

Figure 10: Raw CDE Loss – $n=500$ , real data.

Figure 11: Raw CDE Loss – $n=1000$ , real data.

Figure 12: Raw CDE Loss – $n=5000$ , real data.

Figure 13: Raw CDE Loss – $n=10000$ , real data.

Figure 14: Raw CDE Loss – $n=20000$ , real data.

Figure 15: Raw Log-Likelihood – $n=50$ , real data.

Figure 16: Raw Log-Likelihood – $n=500$ , real data.

Figure 17: Raw Log-Likelihood – $n=1000$ , real data.

Figure 18: Raw Log-Likelihood – $n=5000$ , real data.

Figure 19: Raw Log-Likelihood – $n=10000$ , real data.

Figure 20: Raw Log-Likelihood – $n=20000$ , real data.

Figure 21: Raw CRPS – $n=50$ , real data.

Figure 22: Raw CRPS – $n=500$ , real data.

Figure 23: Raw CRPS – $n=1000$ , real data.

Figure 24: Raw CRPS – $n=5000$ , real data.

Figure 25: Raw CRPS – $n=10000$ , real data.

Figure 26: Raw CRPS – $n=20000$ , real data.

Figure 27: Raw PIT KS – $n=50$ , real data.

Figure 28: Raw PIT KS – $n=500$ , real data.

Figure 29: Raw PIT KS – $n=1000$ , real data.

Figure 30: Raw PIT KS – $n=5000$ , real data.

Figure 31: Raw PIT KS – $n=10000$ , real data.

Figure 32: Raw PIT KS – $n=20000$ , real data.

Figure 33: Raw 90% Coverage – $n=50$ , real data.

Figure 34: Raw 90% Coverage – $n=500$ , real data.

Figure 35: Raw 90% Coverage – $n=1000$ , real data.

Figure 36: Raw 90% Coverage – $n=5000$ , real data.

Figure 37: Raw 90% Coverage – $n=10000$ , real data.

Figure 38: Raw 90% Coverage – $n=20000$ , real data.

Figure 39: Raw Fit Time – $n=50$ , real data.

Figure 40: Raw Fit Time – $n=500$ , real data.

Figure 41: Raw Fit Time – $n=1000$ , real data.

Figure 42: Raw Fit Time – $n=5000$ , real data.

Figure 43: Raw Fit Time – $n=10000$ , real data.

Figure 44: Raw Fit Time – $n=20000$ , real data.

Figure 45: Raw CDE Loss – SDSS, real data.

Figure 46: Raw Log-Likelihood – SDSS, real data.

Figure 48: Raw PIT KS – SDSS, real data.

Figure 49: Raw 90% Coverage – SDSS, real data.

Conditional Distribution Estimation

Conditional distribution estimation models the full distribution of an outcome given covariates, enabling uncertainty-aware prediction and decisions.

See all 74 papers

Sep 28, 2026

Conformal Prediction and Conditional Coverage for Tabular Foundation Models

Tabular foundation models (TFMs) provide predictive distributions for regression, but their prediction regions can exhibit undercoverage or overcoverage even when point predictions are accurate. We introduce C-USIM (Conditionally-Uniformized Score Integration Method), a lightweight application of highest predictive density split conformal prediction that accommodates multimodal predictions. Given calibration and test outputs, it requires no additional training or model inference. It provides finite-sample marginal validity under our assumptions. We bound conditional-marginal coverage gaps using distribution-estimation error and score discreteness, and examine coverage heterogeneity through percentile rank-score plots. Experiments with TabPFN and TabICL show improved marginal coverage accuracy and lower average conditional and group coverage errors. Under a fixed data budget, allocating more observations to calibration can reduce marginal coverage error despite less accurate point predictions.

Sungwoo Park, Sunghee Park, Won Chang

Department of Statistics Seoul National University Seoul, Republic of Korea

Tabular Foundation Models Group-Conditional Conformal Prediction Conformal Prediction Regression Prediction Intervals

Ask PDF

Mar 9, 2026

Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules

We study how the objective used to train or fine-tune a tabular foundation model affects its distributional predictions. Models such as TabPFN and TabICL provide forecasts of the full conditional distribution, but regression benchmarks including TabArena and TALENT assess them mainly through RMSE and

R^2

. Such assessments reward accurate conditional means while leaving the remaining distributional information unassessed. We therefore examine proper scoring rules as objectives for training and fine-tuning and as criteria for evaluating and ranking these models. Every strictly proper rule favors the true distribution at the population level, however, this shared optimum does not determine how learning proceeds with a finite sample. Our examples illustrate score-dependent preferences between forecasts, and our fine-tuning experiments show differences in predictive performance under different objectives. Experiments that fine-tune TabPFN and TabICL on several datasets reveal that changing the training objective can change performance across evaluation scores, with non-trivial interactions between the two. By connecting established methods from distributional regression, scoring-rule theory, and foundation-model adaptation, we show that selecting the score is part of specifying the model and can measurably alter the distribution it learns.

Jonas Landsgesell, Pascal Knoll, Tizian Wenzel

University of Stuttgart (Stuttgart, Germany) · Ludwig Maximilian University of Munich (Munich, Germany) · Munich Center for Machine Learning (Munich, Germany)

Tabular Foundation Models Proper Scoring Rules Distributional Regression Tabular Regression

Ask PDF

Oct 9, 2026

TabReliBench: Evaluating Tabular Foundation Models Beyond Predictive Accuracy

Tabular foundation models (TFMs) are increasingly popular for their strong predictive performance. Many also provide posterior predictive distributions (PPDs), whose reliability remains less studied. We assess reliability through point prediction, distributional quality, marginal and subgroup calibration, and interval efficiency. We then introduce TabReliBench, a large classification and regression benchmark for these comparisons and dataset-level analyses. We find that TFMs' rankings change between TabArena and our benchmark and across reliability metrics. Regression TFMs usually achieve near-nominal marginal coverage and retain narrow intervals, but this favorable marginal behavior does not imply valid conditional coverage. Dataset-level analyses show that classification difficulty often accompanies poor calibration, whereas regression predictive performance is not a reliable indicator of calibration. Ensembling improves predictive performance but does not consistently improve calibration of PPDs; TFMs also benefit little from conformal post-processing, which does not reliably resolve subgroup gaps. Reliable TFM selection therefore requires multidimensional evaluation beyond aggregate rankings. Code is available at https://github.com/Tianqi-Zhao/TabReliBench.

Tianqi Zhao, Jia Guo, Yan Shuo Tan +1

Renmin University of China · Cerebras Systems, Inc. · National University of Singapore +1

Tabular Foundation Models Probability Calibration Regression Prediction Intervals Multicalibration

Ask PDF

Abstract

Conditional Distribution Estimation

Explore similar work

Conformal Prediction and Conditional Coverage for Tabular Foundation Models

Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules

TabReliBench: Evaluating Tabular Foundation Models Beyond Predictive Accuracy