ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules
Authors: Jonas Landsgesell, Pascal Knoll, Tizian Wenzel
Organizations: University of Stuttgart (Stuttgart, Germany) · Ludwig Maximilian University of Munich (Munich, Germany) · Munich Center for Machine Learning (Munich, Germany)
Tabular foundation models such as TabPFN, TabICL, ExaOneTabular, and Nori already produce full predictive distributions, yet prevailing regression benchmarks evaluate them almost exclusively via point-estimate metrics such as RMSE. This overlooks the distributional information these models are designed to provide. We introduce ScoringBench, an open and extensible benchmark that evaluates tabular regression models under a comprehensive suite of proper scoring rules - including (weighted) CRPS, energy scores and interval scores - alongside standard point metrics. ScoringBench covers 101 regression datasets from diverse domains, supports transparent community contributions via a git-based leaderboard, and provides two complementary ranking protocols: an ordinal Demsar/autorank approach and a magnitude-preserving z-score ranking approach. Our evaluation of various model types, including (fine-tuned) tabular foundation models, gradient-boosted trees, and MLPs, shows that model rankings shift depending on the chosen scoring rule. Models that perform well on point-estimate metrics can rank lower on probabilistic ones. These findings demonstrate that evaluation metric selection is a core modeling decision and a domain-specific requirement for applications where certain types of errors carry higher consequences.
Figure 2: Ranking heatmaps summarizing the performance of different models on different scoring rules.
Figure 3: Critical Difference (CD) diagram for CRPS: Models are positioned by autorank on a horizontal axis according to their mean rank rˉm,p (see Eq. ( 1 )), and those whose mean-rank difference does not exceed CD are connected by a horizontal bar, indicating no statistically significant difference in performance at the α=0.05 level. Critical difference (CD) is defined in Eq. ( 2 ) ( Demšar, 2006 ) .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Dataset
1000-Cameras-Dataset
1027_ESL
1028_SWD
1029_LEV
1030_ERA
1193_BNG_lowbwt
1199_BNG_echoMonths
1201_BNG_breastTumor
1203_BNG_pwLinear
215_2dplanes
218_house_8L
225_puma8NH
Appendix
Table 1: The 101 datasets included in the reported ScoringBench results.
Figure 4: Winner stability between ranking methods. Evaluation of Top-1 and Top-3 agreement as described in the text.
Metric
Rank Correlation ( r )
cde_loss
0.9743
crps
0.9910
dispersion
0.9680
dpd_beta_0.01
0.9880
dpd_beta_0.2
0.9720
dpd_beta_0.5
0.9716
Appendix
Table 2: Pearson correlation of ranks between Autorank and Zscore ranking across metrics (computed over 51 models).
MR
MED
MAD
CI
γ
Magnitude
exaonetabular
8.723
0.378
0.362
[0.073, 1.597]
0.000
negligible
tabpfn_v3
8.733
0.373
0.356
[0.092, 1.608]
0.010
negligible
finetune_tabpfn_realv2_5_crls
11.084
0.377
0.363
[0.091, 1.652]
0.001
negligible
finetune_tabiclv2
11.871
0.392
0.377
[0.092, 1.600]
-0.025
negligible
finetune_tabpfn_realv2_5_beta_0.7
12.386
0.379
0.364
[0.092, 1.651]
-0.002
negligible
finetune_tabpfn_realv2_5_beta_0.5
12.569
0.379
0.365
[0.092, 1.651]
-0.003
negligible
Appendix
Table 3: Autorank leaderboard for CRPS. Columns: MR = mean rank across datasets (lower is better); MED = median score; MAD = median absolute deviation; CI = confidence interval for the median; γ = Akinshin’s gamma effect size relative to the top-ranked model. Note: MED, MAD, and γ are computed by autorank without further standardization, thus reflecting the original score scale. While this may hamper direct comparison across datasets with different units, we report it for transparency; ranking is unaffected as autorank sorts by mean rank (scale-independent). The z -score ranking approach properly standardizes for dataset difficulty and produces similar ordering as the autorank approach due to high correlation between both methods (Appendix D.3 ) while having interpretable z-scores; Live rankings available at https://scoringbench.com/ .
MR
MED
MAD
CI
γ
Magnitude
exaonetabular
8.604
2.963
2.816
[0.663, 12.150]
0.000
negligible
tabpfn_v3
10.327
3.007
2.811
[0.690, 12.049]
-0.011
negligible
finetune_tabpfn_realv2_5_crls
10.678
2.968
2.833
[0.688, 12.285]
-0.001
negligible
finetune_tabiclv2
11.020
3.006
2.879
[0.689, 12.238]
-0.010
negligible
tabiclv2
12.584
2.997
2.852
[0.685, 12.168]
-0.008
negligible
finetune_tabpfn_realv2_5_beta_0.7
13.485
3.001
2.824
[0.692, 12.252]
-0.009
negligible
Appendix
Table 4: Autorank leaderboard for the 90% interval score (lower is better). Column definitions as in Table 3 .
MR
MED
MAD
CI
γ
Magnitude
exaonetabular
8.564
0.758
0.726
[0.181, 3.681]
0.000
negligible
tabpfn_v3
9.851
0.763
0.732
[0.183, 3.809]
-0.005
negligible
finetune_tabpfn_realv2_5_crls
12.649
0.774
0.742
[0.180, 3.832]
-0.015
negligible
finetune_tabiclv2
13.287
0.837
0.805
[0.182, 3.768]
-0.070
negligible
finetune_tabpfn_realv2_5_mse
14.104
0.770
0.738
[0.181, 3.791]
-0.012
negligible
finetune_tabpfn_realv2_5_beta_0.5
14.282
0.770
0.738
[0.181, 3.829]
-0.012
negligible
Appendix
Table 5: Autorank leaderboard for RMSE (lower is better). Column definitions as in Table 3 .
Figure 5: Ablation study on dataset size. We compare model rankings across all six pairs of sample-size caps (1000, 2000, 3000 and 8000 training plus test observations) using Spearman correlation over the 51 models common to all four runs. The heatmap displays 47 metrics, excluding the PIT KS test p -value, and shows high correlations across sample sizes.
University of Stuttgart (Stuttgart, Germany) · Ludwig Maximilian University of Munich (Munich, Germany) · Munich Center for Machine Learning (Munich, Germany)