TabReliBench: Evaluating Tabular Foundation Models Beyond Predictive Accuracy

Tianqi Zhao; Jia Guo; Yan Shuo Tan; Qiong Zhang

Task

Dataset identifiers

Classification

3, 11, 20, 23, 24, 35, 38, 39, 43, 50, 56, 57, 60, 156, 333, 334, 335, 377, 446, 451, 455, 460, 464, 1049, 1050, 1056, 1063, 1065, 1067, 1068, 1069, 1071, 1100, 1112, 1114, 1120, 1460, 1462, 1475, 1479, 1480, 1495, 1496, 1498, 1499, 1507, 1510, 1511, 1512, 1523, 1547, 1548, 1551, 1553, 1554, 1555, 1565, 1567, 1590, 1600, 4329, 23381, 40497, 40646, 40647, 40648, 40649, 40650, 40668, 40677, 40682, 40693, 40706, 40707, 40708, 40711, 40713, 40945, 40994, 41027, 41150, 41162, 41919, 42395, 42585, 42742, 43786, 43896, 43897, 43920, 43921, 43938, 44095, 44155, 44227, 45547, 45548, 45550, 45563, 45568, 45570, 45660, 46298, 46331, 46334, 46483, 46752, 46758, 46801, 46831, 46839, 46846, 46856, 46857, 46859, 46866, 46868, 46869, 46905, 46906, 46908, 46910, 46911, 46912, 46913, 46915, 46916, 46918, 46919, 46920, 46921, 46922, 46924, 46927, 46929, 46930, 46932, 46933, 46935, 46937, 46938, 46939, 46940, 46941, 46944, 46947, 46950, 46951, 46952, 46955, 46956, 46958, 46959, 46960, 46962, 46963, 46969, 46979, 46980, 46999

Regression

8, 196, 202, 204, 215, 225, 229, 230, 308, 344, 509, 511, 525, 529, 534, 538, 558, 560, 561, 564, 572, 583, 586, 588, 589, 590, 592, 593, 595, 598, 599, 606, 607, 608, 609, 612, 618, 620, 622, 623, 628, 699, 1097, 41265, 42224, 42352, 42563, 42571, 42723, 42724, 43141, 43714, 44223, 44960, 44967, 44980, 44981, 44984, 44993, 45042, 45702, 46295, 46299, 46612, 46613, 46904, 46907, 46914, 46917, 46923, 46928, 46931, 46934, 46942, 46946, 46949, 46953, 46954, 46961, 46964, 46976

Appendix

Table 2: OpenML dataset identifiers defining the benchmark.

Model	Checkpoint / implementation	Source	C	R	D	Reference
Tabular foundation models
TabPFNv2	v2 default; tabpfn 8.0.2	code	$\checkmark$	$\checkmark$	$\checkmark$	( Hollmann et al.,, 2025 )
TabPFNv2.5	v2.5 default; tabpfn 8.0.2	code	$\checkmark$	$\checkmark$	$\checkmark$	( Grinsztajn et al., 2026a, )
TabPFNv3	v3 default; tabpfn 8.0.2	code	$\checkmark$	$\checkmark$	$\checkmark$	( Grinsztajn et al., 2026b, )
TabPFNv3.5	20260909 checkpoint; tabpfn 9.1.0	code	$\checkmark$	$\checkmark$	$\checkmark$	( Jäger et al.,, 2026 )
TabICLv1	classifier-v1.1-20250506 ; tabicl 2.1.0	code	$\checkmark$	–	–	( Qu et al.,, 2025 )

Appendix

Table 3: Evaluated model versions, public implementations, and prediction types. A check mark indicates that the model enters the corresponding classification-probability (C), regression point-prediction (R), or regression distributional (D) comparison. TabPFN v2–v3 use the task-specific default checkpoint of the stated version.

Index	Numeric transform	Categorical encoding	Permutation	Additional features
0	Quantile-normal, all data	Strict shuffled ordinal	Rotation	Fingerprint
1	Quantile-uniform, 10	Numeric	Shuffle	Original, SVD (64)
2	Quantile-normal, 5; discrete	Shuffled ordinal	Shuffle	Fingerprint
3	KDI-uniform	One-hot	Rotation	10 interactions, fingerprint
4	Quantile-normal, 5	Shuffled ordinal	Shuffle	—
5	Robust	Strict shuffled ordinal	Rotation	SVD (32), fingerprint

Appendix

Table 4: LimiX-2 classification pipelines. Transform names follow the released recipe. “Original” retains original numeric features alongside the transformed representation.

Parameter	Distribution
Learning rate	$\mathrm{LogU}(0.02,0.1)$
Maximum depth	$\mathrm{LogU}_{\mathbb{Z}}(4,10)$
Minimum child weight	$\mathrm{LogU}(10^{-3},5)$
Row subsampling	$\mathrm{U}(0.6,1)$
Column sampling by level / node	Independently $\mathrm{U}(0.6,1)$
$L_{1}$ / $L_{2}$ penalties	Independently $\mathrm{U}(10^{-4},5)$

Appendix

Table 5: Prespecified TabArena-based search space for XGBoost-HPO.

\mathrm{U}

is continuous uniform sampling;

\mathrm{LogU}_{\mathbb{Z}}

is rounded log-uniform sampling. Column sampling by tree remains one.

Model	Accuracy	Brier	Conf. ECE	Classwise ECE
LimiX-2	4.66 [4.07, 5.28]	4.13 [3.59, 4.69]	6.24 [5.61, 6.88]	6.33 [5.71, 6.99]
TabPFNv3.5	4.91 [4.47, 5.38]	3.77 [3.38, 4.19]	5.31 [4.71, 5.91]	5.21 [4.64, 5.80]
Causilo	5.46 [5.04, 5.88]	4.83 [4.44, 5.22]	5.11 [4.62, 5.61]	5.24 [4.76, 5.72]
TabFM	4.77 [4.23, 5.32]	3.82 [3.38, 4.31]	5.92 [5.35, 6.51]	5.42 [4.88, 5.98]
TabPFNv3	6.74 [6.34, 7.17]	6.77 [6.42, 7.11]	7.34 [6.78, 7.91]	7.19 [6.64, 7.76]
TabICLv2	6.66 [6.20, 7.13]	6.34 [5.92, 6.77]	6.29 [5.78, 6.81]	6.03 [5.54, 6.53]

Appendix

Table 6: Classification mean ranks and 95% intervals over 160 datasets and 16 configurations.

Model	$R^{2}$	CRPS	PIT–KS
LimiX-2	3.30 [2.58, 4.05]	–	–
TabPFNv3.5	4.05 [3.52, 4.62]	2.33 [1.98, 2.72]	4.37 [3.86, 4.88]
Causilo	4.54 [4.09, 5.00]	3.21 [2.85, 3.58]	3.32 [2.94, 3.72]
TabFM	4.42 [3.91, 4.94]	–	–
TabPFNv3	5.02 [4.54, 5.51]	3.46 [3.11, 3.81]	4.54 [4.14, 4.95]
TabICLv2	7.25 [6.69, 7.79]	5.65 [5.26, 6.04]	3.49 [3.02, 3.96]

Appendix

Table 7: Regression mean ranks and 95% intervals over 81 datasets.

R^{2}

uses 14 configurations; CRPS and PIT–KS use the 12 configurations with distributional outputs.

Model	MC error	WSC error	Total error
LimiX-2	–	–	–
TabPFNv3.5	4.67 [4.14, 5.21]	4.27 [3.79, 4.76]	4.40 [3.98, 4.83]
Causilo	3.90 [3.44, 4.36]	3.85 [3.33, 4.40]	3.67 [3.19, 4.19]
TabFM	–	–	–
TabPFNv3	5.35 [4.70, 6.00]	5.46 [4.89, 6.07]	5.20 [4.64, 5.78]
TabICLv2	4.31 [3.77, 4.88]	4.60 [4.09, 5.14]	4.06 [3.58, 4.57]

Appendix

Table 8: Regression coverage-error mean ranks and 95% intervals over 81 datasets and 12 distributional configurations at 95% nominal coverage (

\alpha=.05

). MC and WSC denote marginal and worst-slab coverage; total error is the sum of their absolute deviations from nominal coverage.

Model	MC error	WSC error	Total error
LimiX-2	–	–	–
TabPFNv3.5	4.31 [3.78, 4.89]	4.34 [3.76, 4.94]	4.10 [3.58, 4.65]
Causilo	3.86 [3.36, 4.39]	4.02 [3.54, 4.52]	3.80 [3.32, 4.31]
TabFM	–	–	–
TabPFNv3	5.41 [4.84, 5.99]	5.71 [5.14, 6.27]	5.44 [4.87, 5.99]
TabICLv2	4.31 [3.81, 4.80]	4.28 [3.74, 4.85]	4.01 [3.52, 4.53]

Appendix

Table 9: Regression coverage-error mean ranks and 95% intervals at 90% nominal coverage (

\alpha=.10

), using the same 81 datasets, 12 distributional configurations, and metric definitions as the 95% table.

Model	MC error	WSC error	Total error
LimiX-2	–	–	–
TabPFNv3.5	4.57 [4.07, 5.10]	4.64 [4.08, 5.22]	4.45 [3.90, 5.01]
Causilo	3.96 [3.46, 4.49]	4.33 [3.77, 4.91]	3.85 [3.31, 4.42]
TabFM	–	–	–
TabPFNv3	5.62 [5.05, 6.20]	5.64 [5.02, 6.24]	5.49 [4.94, 6.04]
TabICLv2	4.10 [3.58, 4.64]	4.07 [3.56, 4.60]	3.75 [3.33, 4.20]

Appendix

Table 10: Regression coverage-error mean ranks and 95% intervals at 85% nominal coverage (

\alpha=.15

), using the same 81 datasets, 12 distributional configurations, and metric definitions as the 95% table.

Model	MC error	WSC error	Total error
LimiX-2	–	–	–
TabPFNv3.5	4.83 [4.29, 5.39]	4.60 [4.01, 5.20]	4.52 [3.98, 5.09]
Causilo	4.15 [3.61, 4.70]	3.83 [3.33, 4.35]	3.43 [2.96, 3.94]
TabFM	–	–	–
TabPFNv3	5.61 [5.07, 6.16]	5.62 [5.05, 6.19]	5.61 [5.06, 6.15]
TabICLv2	3.63 [3.19, 4.08]	4.51 [3.94, 5.07]	3.99 [3.49, 4.49]

Appendix

Table 11: Regression coverage-error mean ranks and 95% intervals at 80% nominal coverage (

\alpha=.20

), using the same 81 datasets, 12 distributional configurations, and metric definitions as the 95% table.

Figure 10: Absolute prediction and calibration profiles for classification and regression (reading order). Coordinates are medians across datasets of five-split means; lower is better except for accuracy and

R^{2}

. Regression CRPS is normalized by the training-response standard deviation, and point-only regression models are omitted. The inset enlarges the outlined TFM region; dashed lines mark aggregate Pareto fronts when defined.

Model	Accuracy $\uparrow$	Brier $\downarrow$	Conf. ECE $\downarrow$	Classwise ECE $\downarrow$
TabPFNv3.5	0.902 [0.785, 0.970]	0.153 [0.045, 0.301]	0.0184 [0.0077, 0.0502]	0.0170 [0.0073, 0.0455]
TabPFNv3	0.897 [0.781, 0.970]	0.158 [0.045, 0.310]	0.0204 [0.0092, 0.0537]	0.0191 [0.0086, 0.0455]
TabPFNv2.5	0.896 [0.784, 0.966]	0.162 [0.048, 0.303]	0.0206 [0.0112, 0.0502]	0.0198 [0.0095, 0.0478]
TabPFNv2	0.899 [0.779, 0.966]	0.161 [0.056, 0.309]	0.0241 [0.0130, 0.0548]	0.0225 [0.0110, 0.0478]
TabICLv2	0.899 [0.781, 0.970]	0.158 [0.046, 0.306]	0.0210 [0.0099, 0.0474]	0.0189 [0.0087, 0.0451]
TabICLv1	0.890 [0.776, 0.964]	0.170 [0.054, 0.312]	0.0212 [0.0117, 0.0495]	0.0203 [0.0101, 0.0471]

Appendix

Table 12: Classification absolute profiles over 160 datasets. Entries are median

[Q_{1},Q_{3}]

of dataset-level five-split means. Arrows indicate the preferred direction; bold marks the best displayed median.

Model	$R^{2}$ $\uparrow$	Norm. CRPS $\downarrow$	PIT–KS $\downarrow$	Total error $\downarrow$
TabPFNv3.5	0.881 [0.563, 0.972]	0.134 [0.088, 0.299]	0.0473 [0.0352, 0.0867]	0.0559 [0.0356, 0.0761]
TabPFNv3	0.869 [0.556, 0.972]	0.135 [0.087, 0.304]	0.0463 [0.0349, 0.0796]	0.0610 [0.0437, 0.0808]
TabPFNv2.5	0.864 [0.465, 0.973]	0.144 [0.088, 0.306]	0.0444 [0.0317, 0.0795]	0.0607 [0.0405, 0.0813]
TabPFNv2	0.866 [0.544, 0.972]	0.152 [0.088, 0.309]	0.0440 [0.0363, 0.0855]	0.0553 [0.0361, 0.0831]
TabICLv2	0.871 [0.547, 0.969]	0.145 [0.093, 0.304]	0.0471 [0.0311, 0.0653]	0.0541 [0.0315, 0.0799]
TabDPT1.3	0.888 [0.544, 0.969]	0.148 [0.092, 0.310]	0.0541 [0.0407, 0.0822]	0.0583 [0.0365, 0.0924]

Appendix

Table 13: Regression absolute profiles over 81 datasets. Normalized CRPS uses the population standard deviation of the corresponding training responses; total coverage error uses 90% nominal coverage. Entries are median

[Q_{1},Q_{3}]

of dataset-level five-split means. Point-only models have no distributional entries. Arrows indicate the preferred direction; bold marks the best displayed median.

Figure 11: Coverage–width trade-offs across 81 regression datasets at 95%, 85%, and 80% nominal coverage (reading order). Coordinates are separate medians across datasets of five-split means; circles show marginal coverage and triangles WSC. Dashed lines mark nominal coverage, and each marker pair shares one normalized interval width. Connecting segments are visual guides rather than uncertainty intervals; insets enlarge the outlined TFM regions.

Figure 12: Score-variation decomposition after adding the five conventional configurations. Definitions and visual encodings match Figure 3 . Regression CRPS is normalized by the training-response standard deviation, and coverage errors use 90% nominal coverage.

Task	Metric	Original (%)	Corrected (%)	$\hat{\rho}_{\rm single}$	$\hat{\rho}_{\rm mean}$
Classification	Accuracy	2.28	2.13	0.733	0.932
Classification	Brier	1.63	1.60	0.906	0.980
Classification	Confidence ECE	11.29	8.78	0.412	0.778
Classification	Classwise ECE	10.94	8.52	0.413	0.779
Regression	$R^{2}$	4.50	3.81	0.528	0.848
Regression	PIT-KS	13.20	12.26	0.722	0.929

Appendix

Table 14: TFM interaction size and persistence under the nested model. Original and corrected contributions use the same split-averaged total sum of squares. Persistence estimates refer to a single split and the mean of five splits, respectively.

Figure 13: Exact decomposition of split-level TFM scores. The five contributions sum to 100% of total split-level squared variation. The dataset–model term still includes residual variation in the split mean; the five-term identity alone does not correct it. , , , , and denote dataset effects, model effects, dataset–model interactions, splits within datasets, and model–split residuals, respectively.

Figure 14: Estimated persistent interaction contribution ( ) and split-noise contribution ( ), expressed relative to the original split-averaged total squared variation. Labels give corrected/original percentages. The correction uses the nested working model; the denominator is held fixed.

Figure 15: Estimated interaction persistence for one split ( ) and the five-split mean ( ), with pointwise 95% dataset-bootstrap intervals. crosses show the original median correlation across the ten split pairs. All estimates are conditional on the observed samples and evaluated models.

Group added	Information supplied
$X$	Sample size, dimensionality, missingness, numeric feature moments, inter-feature correlations, and categorical cardinality and frequency summaries.
$Y$	Class balance and entropy for classification; response shape, entropy, outlier frequency, and response-bin dispersion for regression.
$XY$	Direct feature–response association, mutual information, neighborhood label mixing or response variation, and class separation.
Aux	Scores of auxiliary linear and nonlinear predictors, supervised tree structure, and hierarchical-shrinkage tree summaries.

Appendix

Table 15: Cumulative meta-feature sets for component prediction. Every meta-feature is computed from benchmark training data.

Figure 16: Sensitivity of component prediction to the evaluated models and meta-predictor. Left: classification; right: regression. denotes the average shift across models, and denotes model-specific deviations.

\circ

TFM Ridge;

\diamond

TFM Extra-Trees;

\square

expanded-set Ridge;

\triangle

expanded-set Extra-Trees.

Figure 17: Out-of-fold

R^{2}

for predicting each model’s score from dataset meta-features. Dark points and bars show means and one standard deviation across repetitions, while points show individual repetitions. Panels show accuracy, Brier score, and confidence ECE for classification (top) and

R^{2}

, normalized CRPS, and PIT–KS for regression (bottom).

Task	$X$	$+Y$	$+XY$	$+\mathrm{Aux}$
Classification	27	$33\,(+6)$	$49\,(+16)$	$62\,(+13)$
Regression	27	$36\,(+9)$	$45\,(+9)$	$59\,(+14)$

Appendix

Table 16: Incremental and cumulative meta-feature counts. Parentheses give the number added by each information group.

Meta-feature family	Count	Definition
Size and dimensionality	4	$n$ , $p$ , $n/p$ , and the number of principal components needed for 95% explained variance divided by $p_{n}$ .
Feature types / missingness	3	Fractions of nonnumeric columns, columns with exactly two observed values, and missing cells.
Near-constant features	1	Fraction of numeric columns with standard deviation divided by range below 0.01; columns with at most one observation also count. The range denominator is floored at $10^{-10}$ .
Inter-feature dependence	2	Mean/max absolute Pearson correlation over distinct pairs of nonconstant numeric columns.
Neighborhood geometry	1	Mean distance to five non-self nearest neighbors, divided by its mean under five independent column-wise permutations.
Feature skewness	2	Mean/max absolute skewness across qualifying numeric columns.

Appendix

Table 17: The 27 covariate-only meta-features are shared by both tasks. “Mean/max” denotes two distinct meta-features. Let

n

be training size,

p

total feature count, and

p_{n}

retained numeric feature count.

Task / family	Count	Definition
Classification	6	Number of classes $K$ ; entropy $-\sum_{k}p_{k}\log p_{k}/\log K$ ; largest and smallest class frequencies; their ratio; normalized class Gini $(1-\sum_{k}p_{k}^{2})/(1-1/K)$ .
Regression shape	4	Signed skewness, excess kurtosis, standard deviation divided by absolute mean, and $(\mathrm{skew}^{2}+1)/(\mathrm{kurtosis}+3)$ .
Regression normality / tails	3	KS distance between standardized responses and $\mathcal{N}(0,1)$ , and fractions with absolute standardized response above two and three.
Regression entropy	1	Entropy of the response histogram using ten equal-width bins from its minimum to maximum.
Response-bin dispersion	1	Partition responses at their 20th, 40th, 60th, and 80th percentiles. Among bins with more than five observations, divide the largest standard deviation by the smallest, flooring the denominator at $10^{-10}$ .

Appendix

Table 18: Response-only meta-features. Class frequencies here use the full training response vector.

Task / family	Count	Definition
Both: MI summaries	5	Mean and maximum MI (entropy-normalized for classification), MI entropy, MI Gini concentration, and weak-signal fraction.
Classification: ANOVA	2	Mean/max $\eta_{j}^{2}=F_{j}(K-1)/[F_{j}(K-1)+n_{c}-K]$ , where $F_{j}$ is the finite one-way ANOVA statistic and $n_{c}$ is complete-case sample size.
Classification: neighbors	2	Mean fraction of the five nearest neighbors having another label; fraction of observations with at least one such neighbor.
Classification: N2	1	Sum of nearest same-class distances divided by sum of nearest other-class distances, over observations with both distances defined.
Classification: range overlap	2	For each class pair, form each feature’s 5–95% ranges. F2 averages intersection length divided by total span over features and pairs. F3 averages over pairs the maximum fraction of their observations outside a feature’s intersection.
Classification: clustering	4	Silhouette, Davies–Bouldin, Calinski–Harabasz, and mean within-class radius divided by mean inter-centroid distance, each divided by its mean under five label permutations. Silhouette uses $(s+1)/(s_{\mathrm{null}}+1)$ .

Appendix

Table 19: Direct feature–response meta-features: 16 for classification and nine for regression. Distances use standardized numeric features.

Classification		Regression
Abbreviation	Meta-feature	Abbreviation	Meta-feature
NLD	Neighbor label disagreement	$\mathrm{Out}_{\mathrm{mean}}$	Mean feature-outlier fraction
NBF	Neighborhood boundary fraction	$\mathrm{MI}_{\max}$	Maximum feature–target MI
N2	Within/between-class distance	$\mathrm{MI}_{\mathrm{mean}}$	Mean feature–target MI
$\mathrm{NMI}_{\max}$	Maximum normalized MI	$\mathrm{Skew}_{\mathrm{mean}}$	Mean feature skewness
$\mathrm{Sil}_{\mathrm{perm}}$	Permutation-normalized silhouette	$\mathrm{Kurt}_{\mathrm{mean}}$	Mean feature kurtosis

Appendix

Table 20: Meta-feature abbreviations in Figures 5 and 6 .

Figure 18: Classification: individual-model associations with Accuracy. Rows are the union of each model’s top five non-auxiliary descriptors; parentheses give complete-pair dataset counts. Stars indicate BH

q<0.05

over all eligible descriptor–model tests for this task and metric; unstarred cells retain their signed estimates and colors but are not significant after correction. Gray cells with em dashes denote unavailable or ineligible coefficients, not zero associations. Positive correlations increase the raw metric. Model abbreviations and selection rules are defined in Appendix F .

Figure 19: Classification: individual-model associations with Brier. Rows are the union of each model’s top five non-auxiliary descriptors; parentheses give complete-pair dataset counts. Stars indicate BH

q<0.05

over all eligible descriptor–model tests for this task and metric; unstarred cells retain their signed estimates and colors but are not significant after correction. Gray cells with em dashes denote unavailable or ineligible coefficients, not zero associations. Positive correlations increase the raw metric. Model abbreviations and selection rules are defined in Appendix F .

Figure 20: Classification: individual-model associations with Confidence ECE. Rows are the union of each model’s top five non-auxiliary descriptors; parentheses give complete-pair dataset counts. Stars indicate BH

q<0.05

over all eligible descriptor–model tests for this task and metric; unstarred cells retain their signed estimates and colors but are not significant after correction. Gray cells with em dashes denote unavailable or ineligible coefficients, not zero associations. Positive correlations increase the raw metric. Model abbreviations and selection rules are defined in Appendix F .

Figure 21: Classification: individual-model associations with Classwise ECE. Rows are the union of each model’s top five non-auxiliary descriptors; parentheses give complete-pair dataset counts. Stars indicate BH

q<0.05

over all eligible descriptor–model tests for this task and metric; unstarred cells retain their signed estimates and colors but are not significant after correction. Gray cells with em dashes denote unavailable or ineligible coefficients, not zero associations. Positive correlations increase the raw metric. Model abbreviations and selection rules are defined in Appendix F .

Figure 22: Regression: individual-model associations with

R^{2}

. Rows are the union of each model’s top five non-auxiliary descriptors; parentheses give complete-pair dataset counts. Stars indicate BH

q<0.05

over all eligible descriptor–model tests for this task and metric; unstarred cells retain their signed estimates and colors but are not significant after correction. Gray cells with em dashes denote unavailable or ineligible coefficients, not zero associations. Positive correlations increase the raw metric. Model abbreviations and selection rules are defined in Appendix F .

Figure 23: Regression: individual-model associations with Normalized CRPS. Rows are the union of each model’s top five non-auxiliary descriptors; parentheses give complete-pair dataset counts. Stars indicate BH

q<0.05

over all eligible descriptor–model tests for this task and metric; unstarred cells retain their signed estimates and colors but are not significant after correction. Gray cells with em dashes denote unavailable or ineligible coefficients, not zero associations. Positive correlations increase the raw metric. Model abbreviations and selection rules are defined in Appendix F .

Figure 24: Regression: individual-model associations with PIT–KS. Rows are the union of each model’s top five non-auxiliary descriptors; parentheses give complete-pair dataset counts. Stars indicate BH

q<0.05

over all eligible descriptor–model tests for this task and metric; unstarred cells retain their signed estimates and colors but are not significant after correction. Gray cells with em dashes denote unavailable or ineligible coefficients, not zero associations. Positive correlations increase the raw metric. Model abbreviations and selection rules are defined in Appendix F .

Figure 25: Regression: individual-model associations with Coverage error. Rows are the union of each model’s top five non-auxiliary descriptors; parentheses give complete-pair dataset counts. Stars indicate BH

q<0.05

over all eligible descriptor–model tests for this task and metric; unstarred cells retain their signed estimates and colors but are not significant after correction. Gray cells with em dashes denote unavailable or ineligible coefficients, not zero associations. Positive correlations increase the raw metric. Model abbreviations and selection rules are defined in Appendix F .

Figure 26: Regression: individual-model associations with WSC error. Rows are the union of each model’s top five non-auxiliary descriptors; parentheses give complete-pair dataset counts. Stars indicate BH

q<0.05

over all eligible descriptor–model tests for this task and metric; unstarred cells retain their signed estimates and colors but are not significant after correction. Gray cells with em dashes denote unavailable or ineligible coefficients, not zero associations. Positive correlations increase the raw metric. Model abbreviations and selection rules are defined in Appendix F .

Figure 27: Regression: individual-model associations with Total error. Rows are the union of each model’s top five non-auxiliary descriptors; parentheses give complete-pair dataset counts. Stars indicate BH

q<0.05

over all eligible descriptor–model tests for this task and metric; unstarred cells retain their signed estimates and colors but are not significant after correction. Gray cells with em dashes denote unavailable or ineligible coefficients, not zero associations. Positive correlations increase the raw metric. Model abbreviations and selection rules are defined in Appendix F .

Figure 28: Mean marginal coverage minus nominal coverage, in percentage points. Zero indicates the target; stars mark vanilla. Plots in reading order: TabPFNv3.5, TabPFNv3, TabPFNv2.5, TabPFNv2, TabICLv2, Causilo, TabDPT1.3, and BART.

Figure 29: Mean held-out worst-slab coverage minus nominal coverage. Each method selects its subgroup on the search half and evaluates it on the other half. Plots in reading order: TabPFNv3.5, TabPFNv3, TabPFNv2.5, TabPFNv2, TabICLv2, Causilo, TabDPT1.3, and BART.

Figure 30: Changes in absolute marginal (left) and worst-slab (right) coverage error after SCP-CQR calibration, relative to the vanilla intervals, in percentage points. Each curve is one base model, averaged with equal dataset weights; negative values favor calibration. Error bars are paired 95% dataset-bootstrap intervals.

Figure 31: All post-processors and base models at 90% nominal coverage. Left and center: paired changes in marginal and worst-slab coverage error relative to Vanilla, in percentage points; blue cells reduce error and red cells increase it. Bold values have paired 95% dataset-bootstrap intervals excluding zero. Right: median across datasets of the percentage change in normalized width relative to Vanilla; gray

\infty

cells have infinite median width. Rows are grouped into global split-conformal, localized, and learned-adjustment methods; colors share a common scale across nominal levels.

Figure 32: All post-processors and base models at 95% nominal coverage. Left and center: paired changes in marginal and worst-slab coverage error relative to Vanilla, in percentage points; blue cells reduce error and red cells increase it. Bold values have paired 95% dataset-bootstrap intervals excluding zero. Right: median across datasets of the percentage change in normalized width relative to Vanilla; gray

\infty

cells have infinite median width. Rows are grouped into global split-conformal, localized, and learned-adjustment methods; colors share a common scale across nominal levels.

Figure 33: All post-processors and base models at 85% nominal coverage. Left and center: paired changes in marginal and worst-slab coverage error relative to Vanilla, in percentage points; blue cells reduce error and red cells increase it. Bold values have paired 95% dataset-bootstrap intervals excluding zero. Right: median across datasets of the percentage change in normalized width relative to Vanilla; gray

\infty

cells have infinite median width. Rows are grouped into global split-conformal, localized, and learned-adjustment methods; colors share a common scale across nominal levels.

Figure 34: All post-processors and base models at 80% nominal coverage. Left and center: paired changes in marginal and worst-slab coverage error relative to Vanilla, in percentage points; blue cells reduce error and red cells increase it. Bold values have paired 95% dataset-bootstrap intervals excluding zero. Right: median across datasets of the percentage change in normalized width relative to Vanilla; gray

\infty

cells have infinite median width. Rows are grouped into global split-conformal, localized, and learned-adjustment methods; colors share a common scale across nominal levels.

Figure 35: Coverage and normalized width at 80% for all eight base models. Stars mark Vanilla and arrows show post-processing changes. The strip denotes infinite mean width. The upper two rows show marginal coverage; the lower two rows show held-out worst-slab coverage (WSC). Among methods reaching the target (on or to the right of the dashed line), lower normalized width is preferable.

Figure 36: Coverage and normalized width at 85% for all eight base models. Stars mark Vanilla and arrows show post-processing changes. The strip denotes infinite mean width. The upper two rows show marginal coverage; the lower two rows show held-out worst-slab coverage (WSC). Among methods reaching the target (on or to the right of the dashed line), lower normalized width is preferable.

Figure 37: Coverage and normalized width at 90% for all eight base models. Stars mark Vanilla and arrows show post-processing changes. The strip denotes infinite mean width. The upper two rows show marginal coverage; the lower two rows show held-out worst-slab coverage (WSC). Among methods reaching the target (on or to the right of the dashed line), lower normalized width is preferable.

Figure 38: Coverage and normalized width at 95% for all eight base models. Stars mark Vanilla and arrows show post-processing changes. The strip denotes infinite mean width. The upper two rows show marginal coverage; the lower two rows show held-out worst-slab coverage (WSC). Among methods reaching the target (on or to the right of the dashed line), lower normalized width is preferable.

Figure 39: Infinite-interval rates across target coverage levels. Lines average the eight base regressors, bands span their model-specific means, and the vertical axis is logarithmic. Methods with zero infinite-interval rates are omitted.

Figure 40: Infinite-interval rates across nominal levels, averaged within datasets and then across datasets. Methods with zero rates throughout are omitted. Plots in reading order: TabPFNv3.5, TabPFNv3, TabPFNv2.5, TabPFNv2, TabICLv2, Causilo, TabDPT1.3, and BART.

Figure 41: Empty-set rates for signed-score methods. Empty sets are retained as uncovered observations with zero width. Plots in reading order: TabPFNv3.5, TabPFNv3, TabPFNv2.5, TabPFNv2, TabICLv2, Causilo, TabDPT1.3, and BART.

Figure 42: Median metrics for the complete classification ensemble-size sweep. Plots in reading order show accuracy, Brier score, confidence ECE, and classwise ECE. Each curve is one model, and every point uses the same 160 datasets and five seeds. Higher accuracy is better; lower Brier score and ECE are better.

Figure 43: Median metrics for the complete regression ensemble-size sweep. Plots in reading order show

R^{2}

, normalized CRPS, PIT–KS, marginal coverage error, worst-slab coverage error, and their sum. Each curve is one model, and every point uses the same 81 datasets and five seeds.

R^{2}

includes all nine regression TFMs, whereas the distributional plots use the seven models that provide predictive distributions. Coverage quantities use 90% nominal intervals. Higher

R^{2}

is better; lower values are better for all other panels.

Figure 44: Absolute coverage–width trajectories as ensemble size changes at 90% nominal coverage. The left plot uses marginal coverage and the right plot uses worst-slab coverage; the vertical axis is normalized interval width. Squares mark

E=1

, stars mark

E=32

, and circles mark intermediate ensemble sizes. Each coordinate is the median across the same 81 datasets after averaging five seeds.

Property	Classification	Regression
Point prediction	Accuracy $\uparrow$	Predictive $R^{2}$ $\uparrow$
Proper score	Brier score $\downarrow$	CRPS $\downarrow$
Marginal calibration	Confidence ECE $\downarrow$	PIT–KS $\downarrow$ , marginal coverage $\to$ nominal
Classwise / subgroup calibration	Classwise ECE $\downarrow$	Worst-slab coverage $\to$ nominal
Sharpness	–	Interval width $\downarrow$

TabReliBench: Evaluating Tabular Foundation Models Beyond Predictive Accuracy

Abstract

Figures & tables

Appendix

Tabular Foundation Models

High Performance, Low Reliability: Uncertainty Benchmarking for Tabular Foundation Models

Do Tabular Foundation Models Agree with Themselves?

Conformal Prediction and Conditional Coverage for Tabular Foundation Models

Abstract

Tabular Foundation Models

Explore similar work

High Performance, Low Reliability: Uncertainty Benchmarking for Tabular Foundation Models

Do Tabular Foundation Models Agree with Themselves?

Conformal Prediction and Conditional Coverage for Tabular Foundation Models