stat.MEOct 28, 2026
SaveTabular foundation models are not a shortcut to calibrated clinical risk prediction: a pre-registered simulation against the true risk
Organizations: Department of Applied Statistics, Faculty of Business, Alexandria University, Alexandria, Egypt.
Abstract
Tabular foundation models are pretrained networks that predict risk from a table without fitting a model. Whether their risks are correct (calibrated) has been judged on real data, where true risk is unknown. In a pre-registered simulation with known true risks, we compared TabPFN-3.5, TabICLv2 and Mitra-v2 with Firth logistic regression with intercept correction (FLIC) in 58 scenarios and 1,000 random designs, then on four clinical data sets. Foundation models were not a shortcut to calibration. Their calibration slope matched FLIC's within the pre-registered margin in 36 or 37 of 48 core scenarios and was better only at n = 200 with a 3% event rate. Even 3,000 patients were often not enough: with FLIC, TabPFN-3.5 or Mitra-v2, 63%-83% of samples gave risks close to the truth in average, intercept and slope at event rates of 10% and 30%, and 35%-50% at 3%. TabICLv2 did worse (8%-39% and 1%-3%). Across the random designs, foundation models had a lower true calibration error than FLIC beyond the margin in 25%-38% and a higher one in none, though small losses were predicted at n = 3000. On real data, no model resisted a change in event rate between populations; recalibration on 500 local patients largely repaired it.
Figures & tables
| Question (cells) | TabPFN-3.5 | TabICLv2 | Mitra-v2 |
|---|---|---|---|
| True ICI against FLIC (48 core scenarios; primary) | 44 eq., 4 better | 40 eq., 4 better, 4 worse | 44 eq., 2 better, 2 inc. |
| Q1 closeness of to 1 (8 main cells) | 5 eq., 3 inc. | 4 eq., 4 inc. | 4 eq., 4 inc. |
| Q1 closeness of to 1 (48 core scenarios) | 37 eq., 4 better, 7 inc. | 36 eq., 4 better, 8 inc. | 36 eq., 4 better, 8 inc. |
| Q1 direction of (8 main cells) | 2 close to 1, 1 under-confident, 5 unclear | 4 over-confident, 4 unclear | 4 close to 1, 4 unclear |
| Q2 minimum for weak calibration (12 shape–event-rate cells) | 9 the same , 1 smaller, 2 larger | 10 the same , 2 larger | 10 the same , 2 larger |
| Q2 clarity of the ordering | 9 clear, 3 sensitive to Monte Carlo error | 10 clear, 2 sensitive | 10 clear, 2 sensitive |
Table 1: Summary of findings. Counts of cells in each pre-registered label, for each TFM paired with FLIC on the replicates common to both. Labels come from 95% intervals and margins fixed before the run: true ICI ( at ), , . Eq., equivalent; inc., inconclusive. Q2 compares the smallest at which 80% of replicates are weakly calibrated; an ordering is “sensitive to Monte Carlo error” when it changes if the lower 95% limit of each share replaces the share. Q4a, size of the goodness-of-fit tests; Q4b, detection; Q5a, true ICI after logistic recalibration on 500 rows, against FLIC after the same recalibration and against the best arm (the lowest mean true ICI among FLIC and the TFMs after this recalibration and the other established methods as fitted); Q5b, positive predictive value of the test-guided rule. Shipped softmax temperature: TabPFN-3.5 1.0, TabICLv2 0.9; Mitra-v2 has none.
Figure 1: Distance of each model’s predictions from the true risk. Mean true integrated calibration index (ICI) against development size , for event rates of 3%, 10% and 30% (rows) and the four outcome shapes (columns: S1 linear, S2 smooth non-linear, S3 interactions, S4 asymmetric link). Points are means over replicates with Monte Carlo standard errors (often smaller than the symbols). The dashed oracle line is the floor of the estimator. GAM, generalized additive model; LightGBM + Platt, gradient-boosted trees recalibrated by Platt scaling. Maximum-likelihood logistic regression and ridge regression are shown in Supplementary Information.
Figure 2: Development size needed for calibration. a , b , The smallest in the design at which each model reaches weak ( a ) or moderate ( b ) calibration in at least 80% of replicates, by event rate and outcome shape (“ ” when no qualifies); the column “all” takes the largest value over the four shapes, and Riley’s minimum sample size is given below each column. c – h , Share of development sets on which each model is weakly ( c – e ) or moderately ( f – h ) calibrated, against ; lines are the mean over the four outcome shapes and bands their range, and the dashed line marks the 80% rule of panels a and b . The MCSE of each share is at most 0.022 for the established methods and 0.041 for the TFMs. LR-MLE, maximum-likelihood logistic regression; Ridge-CV, ridge logistic regression with the penalty chosen by cross-validation; GAM, generalized additive model; LightGBM + Platt and LightGBM (raw), gradient-boosted trees with and without Platt scaling. Versions at half and double the thresholds, with the lower 95% limit of the share, and with the log-scale ICI are in Supplementary Information.
Figure 3: Calibration-in-the-large with rare events and the effect of the softmax temperature. a , Paired difference in (TFM minus FLIC; Hodges-Lehmann estimate with its 95% signed-rank interval) in the 16 cells at an event rate of 3%, the cells with first; dashed lines mark the margin , and a negative difference favours the TFM. , softmax temperature; the shipped of TabPFN-3.5 and TabICLv2 is in the legend; Mitra-v2 has no temperature setting. b – d , Paired effect of the temperature (temperature arm minus main arm) at an event rate of 10% on ( b ), ( c ) and the true ICI ( d ; mean MCSE). TabPFN-3.5 was moved from 1.0 to 0.9 and TabICLv2 from 0.9 to 1.0.
Figure 4: Detection and size of the goodness-of-fit tests. Each panel shows the probability that a test rejects at the 0.05 level on the observed outcomes, against the true ICI, from a logistic generalized additive model pooled over models, scenarios (with evaluation sets of 1,000 and 3,334 rows) and replicates, with a 95% band from a bootstrap that resamples whole replicates. Vertical lines mark true ICI values of 0.01, 0.02, 0.03 and 0.05. Insets show the pooled size of the test on outcomes drawn from each model’s own predicted risks ( MCSE) against the band ; for the oracle, whose observed outcomes are themselves drawn from the true risk, both its observed and its redrawn outcomes are null outcomes. , number of risk groups of EDGE (10, or chosen automatically); CITL, calibration-in-the-large test. DeepGOF-1 is a pretrained neural goodness-of-fit test; its size comes from a separate size study. Panel l is the localization procedure, which names the part of the misfit (overall level, slope, curve shape or covariate subgroup) with the familywise error rate (FWER) controlled at 0.05; it shows the rate of naming any part.
Figure 5: Recalibration with a local sample. Mean true ICI at after each level of the ladder: R0, none; R1, intercept only; R2, logistic recalibration; R3, isotonic regression; fitted on a local sample of 500 (solid) or 100 (dashed) rows, for FLIC and the three TFMs. Rows are event rates and columns outcome shapes. Points are means MCSE; with 100 rows, only replicates in which a fit was possible (both outcome classes present) are included. The same figure at , 1,000 and 3,000 is in Supplementary Information.
Figure 6: Where each foundation model is better or worse calibrated than FLIC across random designs. Meta-model prediction of the paired difference in true ICI (TFM minus FLIC) over development size and event rate , with every other design factor at its median. Dark grey: the 95% simultaneous band lies below (TFM better). Hatched light grey: the band lies above (TFM worse; no such region occurred). White: within 0.01. Dashed and solid lines are contours of the predicted difference. Cross-hatched: Mitra-v2 not run because it does not fit in 6 GB of graphics memory. Based on 1,000 random designs with one replicate each.
Figure 7: Calibration of each model on the external real-data sets. Empirical calibration on the external evaluation set against development size , for FLIC, LightGBM with Platt scaling and the three TFMs, in the International Stroke Trial (IST; two outcomes; developed in the United Kingdom, evaluated in the other countries), CRASH-2 (74 held-out hospitals), the sepsis data (developed in admissions not meeting the Sepsis-3 definition, evaluated in Sepsis-3 admissions) and the PhysioNet 2012 data (sets A and B against set C) (columns). In IST death or dependency ( b , g , l ) and sepsis ( d , i , n ), the event rate differs between the two populations. a – e , Mean empirical ICI over 100 development subsamples at each , with 95% intervals ( MCSE). f – j , Median empirical calibration slope with its interquartile range; the dashed line marks 1. k – o , Median calibration-in-the-large (log-odds scale, slope fixed at 1; positive means predicted risks too low) with its interquartile range; the dashed line marks 0. Mitra-v2 was not run at in IST and PhysioNet 2012, where it does not fit in graphics memory. CRASH-2 results are aggregates only. The real-data results are exploratory.
More in this topic
Tabular Foundation Models
Tabular foundation models learn reusable representations or in-context prediction skills for structured data such as spreadsheets and databases.
See all 191 papers