Tabular foundation models are pretrained networks that predict risk from a table without fitting a model. Whether their risks are correct (calibrated) has been judged on real data, where true risk is unknown. In a pre-registered simulation with known true risks, we compared TabPFN-3.5, TabICLv2 and Mitra-v2 with Firth logistic regression with intercept correction (FLIC) in 58 scenarios and 1,000 random designs, then on four clinical data sets. Foundation models were not a shortcut to calibration. Their calibration slope matched FLIC's within the pre-registered margin in 36 or 37 of 48 core scenarios and was better only at n = 200 with a 3% event rate. Even 3,000 patients were often not enough: with FLIC, TabPFN-3.5 or Mitra-v2, 63%-83% of samples gave risks close to the truth in average, intercept and slope at event rates of 10% and 30%, and 35%-50% at 3%. TabICLv2 did worse (8%-39% and 1%-3%). Across the random designs, foundation models had a lower true calibration error than FLIC beyond the margin in 25%-38% and a higher one in none, though small losses were predicted at n = 3000. On real data, no model resisted a change in event rate between populations; recalibration on 500 local patients largely repaired it.
Figures & tables
Question (cells)
TabPFN-3.5
TabICLv2
Mitra-v2
True ICI against FLIC (48 core scenarios; primary)
44 eq., 4 better
40 eq., 4 better, 4 worse
44 eq., 2 better, 2 inc.
Q1 closeness of b∗ to 1 (8 main cells)
5 eq., 3 inc.
4 eq., 4 inc.
4 eq., 4 inc.
Q1 closeness of b∗ to 1 (48 core scenarios)
37 eq., 4 better, 7 inc.
36 eq., 4 better, 8 inc.
36 eq., 4 better, 8 inc.
Q1 direction of b∗ (8 main cells)
2 close to 1, 1 under-confident, 5 unclear
4 over-confident, 4 unclear
4 close to 1, 4 unclear
Q2 minimum n for weak calibration (12 shape–event-rate cells)
9 the same n , 1 smaller, 2 larger
10 the same n , 2 larger
10 the same n , 2 larger
Q2 clarity of the ordering
9 clear, 3 sensitive to Monte Carlo error
10 clear, 2 sensitive
10 clear, 2 sensitive
Table 1: Summary of findings. Counts of cells in each pre-registered label, for each TFM paired with FLIC on the replicates common to both. Labels come from 95% intervals and margins fixed before the run: true ICI ±0.01 ( ±0.003 at π=3% ), ∣logb∗∣±log1.1 , ∣a∗∣±0.05 . Eq., equivalent; inc., inconclusive. Q2 compares the smallest n at which 80% of replicates are weakly calibrated; an ordering is “sensitive to Monte Carlo error” when it changes if the lower 95% limit of each share replaces the share. Q4a, size of the goodness-of-fit tests; Q4b, detection; Q5a, true ICI after logistic recalibration on 500 rows, against FLIC after the same recalibration and against the best arm (the lowest mean true ICI among FLIC and the TFMs after this recalibration and the other established methods as fitted); Q5b, positive predictive value of the test-guided rule. Shipped softmax temperature: TabPFN-3.5 1.0, TabICLv2 0.9; Mitra-v2 has none.
Figure 1: Distance of each model’s predictions from the true risk. Mean true integrated calibration index (ICI) against development size n , for event rates of 3%, 10% and 30% (rows) and the four outcome shapes (columns: S1 linear, S2 smooth non-linear, S3 interactions, S4 asymmetric link). Points are means over replicates with ±1.96 Monte Carlo standard errors (often smaller than the symbols). The dashed oracle line is the floor of the estimator. GAM, generalized additive model; LightGBM + Platt, gradient-boosted trees recalibrated by Platt scaling. Maximum-likelihood logistic regression and ridge regression are shown in Supplementary Information.
Figure 2: Development size needed for calibration. a , b , The smallest n in the design at which each model reaches weak ( a ) or moderate ( b ) calibration in at least 80% of replicates, by event rate and outcome shape (“ >3000 ” when no n qualifies); the column “all” takes the largest value over the four shapes, and Riley’s minimum sample size is given below each column. c – h , Share of development sets on which each model is weakly ( c – e ) or moderately ( f – h ) calibrated, against n ; lines are the mean over the four outcome shapes and bands their range, and the dashed line marks the 80% rule of panels a and b . The MCSE of each share is at most 0.022 for the established methods and 0.041 for the TFMs. LR-MLE, maximum-likelihood logistic regression; Ridge-CV, ridge logistic regression with the penalty chosen by cross-validation; GAM, generalized additive model; LightGBM + Platt and LightGBM (raw), gradient-boosted trees with and without Platt scaling. Versions at half and double the thresholds, with the lower 95% limit of the share, and with the log-scale ICI are in Supplementary Information.
Figure 3: Calibration-in-the-large with rare events and the effect of the softmax temperature. a , Paired difference in ∣a∗∣ (TFM minus FLIC; Hodges-Lehmann estimate with its 95% signed-rank interval) in the 16 cells at an event rate of 3%, the cells with n≥1000 first; dashed lines mark the margin ±0.05 , and a negative difference favours the TFM. T , softmax temperature; the shipped T of TabPFN-3.5 and TabICLv2 is in the legend; Mitra-v2 has no temperature setting. b – d , Paired effect of the temperature (temperature arm minus main arm) at an event rate of 10% on a∗ ( b ), logb∗ ( c ) and the true ICI ( d ; mean ±1.96 MCSE). TabPFN-3.5 was moved from 1.0 to 0.9 and TabICLv2 from 0.9 to 1.0.
Figure 4: Detection and size of the goodness-of-fit tests. Each panel shows the probability that a test rejects at the 0.05 level on the observed outcomes, against the true ICI, from a logistic generalized additive model pooled over models, scenarios (with evaluation sets of 1,000 and 3,334 rows) and replicates, with a 95% band from a bootstrap that resamples whole replicates. Vertical lines mark true ICI values of 0.01, 0.02, 0.03 and 0.05. Insets show the pooled size of the test on outcomes drawn from each model’s own predicted risks ( ±1.96 MCSE) against the band [0.04,0.06] ; for the oracle, whose observed outcomes are themselves drawn from the true risk, both its observed and its redrawn outcomes are null outcomes. G , number of risk groups of EDGE (10, or chosen automatically); CITL, calibration-in-the-large test. DeepGOF-1 is a pretrained neural goodness-of-fit test; its size comes from a separate size study. Panel l is the localization procedure, which names the part of the misfit (overall level, slope, curve shape or covariate subgroup) with the familywise error rate (FWER) controlled at 0.05; it shows the rate of naming any part.
Figure 5: Recalibration with a local sample. Mean true ICI at n=500 after each level of the ladder: R0, none; R1, intercept only; R2, logistic recalibration; R3, isotonic regression; fitted on a local sample of 500 (solid) or 100 (dashed) rows, for FLIC and the three TFMs. Rows are event rates and columns outcome shapes. Points are means ±1.96 MCSE; with 100 rows, only replicates in which a fit was possible (both outcome classes present) are included. The same figure at n=200 , 1,000 and 3,000 is in Supplementary Information.
Figure 6: Where each foundation model is better or worse calibrated than FLIC across random designs. Meta-model prediction of the paired difference in true ICI (TFM minus FLIC) over development size n and event rate π , with every other design factor at its median. Dark grey: the 95% simultaneous band lies below −0.01 (TFM better). Hatched light grey: the band lies above +0.01 (TFM worse; no such region occurred). White: within 0.01. Dashed and solid lines are contours of the predicted difference. Cross-hatched: Mitra-v2 not run because it does not fit in 6 GB of graphics memory. Based on 1,000 random designs with one replicate each.
Figure 7: Calibration of each model on the external real-data sets. Empirical calibration on the external evaluation set against development size n , for FLIC, LightGBM with Platt scaling and the three TFMs, in the International Stroke Trial (IST; two outcomes; developed in the United Kingdom, evaluated in the other countries), CRASH-2 (74 held-out hospitals), the sepsis data (developed in admissions not meeting the Sepsis-3 definition, evaluated in Sepsis-3 admissions) and the PhysioNet 2012 data (sets A and B against set C) (columns). In IST death or dependency ( b , g , l ) and sepsis ( d , i , n ), the event rate differs between the two populations. a – e , Mean empirical ICI over 100 development subsamples at each n , with 95% intervals ( ±1.96 MCSE). f – j , Median empirical calibration slope with its interquartile range; the dashed line marks 1. k – o , Median calibration-in-the-large (log-odds scale, slope fixed at 1; positive means predicted risks too low) with its interquartile range; the dashed line marks 0. Mitra-v2 was not run at n=3000 in IST and PhysioNet 2012, where it does not fit in graphics memory. CRASH-2 results are aggregates only. The real-data results are exploratory.
Tabular foundation models (TFMs) are increasingly popular for their strong predictive performance. Many also provide posterior predictive distributions (PPDs), whose reliability remains less studied. We assess reliability through point prediction, distributional quality, marginal and subgroup calibration, and interval efficiency. We then introduce TabReliBench, a large classification and regression benchmark for these comparisons and dataset-level analyses. We find that TFMs' rankings change between TabArena and our benchmark and across reliability metrics. Regression TFMs usually achieve near-nominal marginal coverage and retain narrow intervals, but this favorable marginal behavior does not imply valid conditional coverage. Dataset-level analyses show that classification difficulty often accompanies poor calibration, whereas regression predictive performance is not a reliable indicator of calibration. Ensembling improves predictive performance but does not consistently improve calibration of PPDs; TFMs also benefit little from conformal post-processing, which does not reliably resolve subgroup gaps. Reliable TFM selection therefore requires multidimensional evaluation beyond aggregate rankings. Code is available at https://github.com/Tianqi-Zhao/TabReliBench.
Tianqi Zhao, Jia Guo, Yan Shuo Tan +1
Renmin University of China · Cerebras Systems, Inc. · National University of Singapore +1
Tabular data is the most common format in clinical practice, encompassing laboratory results, medication records, diagnostic codes, and patient demographics. As foundation models for tabular data have grown in number and variety, a practical question has become harder to answer: which model should a clinician or data scientist actually choose for a given task, and why? Existing surveys catalogue what these models can do, but they stop short of providing a structured way to compare them against the specific demands of a real application. We introduce \system{}, a comparative evaluation framework that scores and ranks tabular foundation models (TFMs) across six clinically meaningful dimensions: how well a model generalizes to new datasets, how effectively it protects patient privacy, how much data it needs to perform well, how it scales with growing datasets and feature spaces, how interpretable its predictions are to clinicians, and how fairly it performs across patient subgroups. Each dimension is broken down into measurable sub-components, and groups of sub-components can optionally be combined into supplementary compound scores, called super-metrics, that provide a diagnostic view of how a model performs across several dimensions simultaneously. To show how the framework works in practice, we apply it to two healthcare use cases, screening for iron deficiency and predicting heart failure, demonstrating how the same set of metrics leads to different model rankings depending on what matters most in each clinical context. We also provide a taxonomy of 45 TFMs organized by their underlying architecture, which serves as a reference for researchers and practitioners looking to navigate this rapidly expanding field.
Majid Lotfian Delouee, Sjors G. J. G. In 't Veld, Martijn C. Schut
Tabular foundation models (TFMs) provide predictive distributions for regression, but their prediction regions can exhibit undercoverage or overcoverage even when point predictions are accurate. We introduce C-USIM (Conditionally-Uniformized Score Integration Method), a lightweight application of highest predictive density split conformal prediction that accommodates multimodal predictions. Given calibration and test outputs, it requires no additional training or model inference. It provides finite-sample marginal validity under our assumptions. We bound conditional-marginal coverage gaps using distribution-estimation error and score discreteness, and examine coverage heterogeneity through percentile rank-score plots. Experiments with TabPFN and TabICL show improved marginal coverage accuracy and lower average conditional and group coverage errors. Under a fixed data budget, allocating more observations to calibration can reduce marginal coverage error despite less accurate point predictions.
Sungwoo Park, Sunghee Park, Won Chang
Department of Statistics Seoul National University Seoul, Republic of Korea