Conformal Prediction and Conditional Coverage for Tabular Foundation Models

Sungwoo Park; Sunghee Park; Won Chang

stat.MLSep 28, 2026

Conformal Prediction and Conditional Coverage for Tabular Foundation Models

Authors: Sungwoo Park, Sunghee Park, Won Chang

Organizations: Department of Statistics Seoul National University Seoul, Republic of Korea

Abstract

Tabular foundation models (TFMs) provide predictive distributions for regression, but their prediction regions can exhibit undercoverage or overcoverage even when point predictions are accurate. We introduce C-USIM (Conditionally-Uniformized Score Integration Method), a lightweight application of highest predictive density split conformal prediction that accommodates multimodal predictions. Given calibration and test outputs, it requires no additional training or model inference. It provides finite-sample marginal validity under our assumptions. We bound conditional-marginal coverage gaps using distribution-estimation error and score discreteness, and examine coverage heterogeneity through percentile rank-score plots. Experiments with TabPFN and TabICL show improved marginal coverage accuracy and lower average conditional and group coverage errors. Under a fixed data budget, allocating more observations to calibration can reduce marginal coverage error despite less accurate point predictions.

Figures & tables

Figure 1: C-USIM overview with $n_{\mathrm{cal}}$ calibration scores and target coverage $1-\alpha$ . Using a pretrained TFM, the procedure predicts densities, computes density-rank scores, calibrates a threshold, and constructs prediction regions without fitting an additional model. See Section 4 for details.

Figure 2: Percentile rank–score plots over test covariates (plot (a)), obtained using TabPFN on the sine with shifted exponential noise function ( B.2.1 ). Plot (b) approximates $P_{t}(X_{i})$ at the nominal score threshold $t=1-\alpha$ (blue) and the ideal C-USIM threshold $t=q_{1-\alpha}$ (red), which is estimated by the empirical $(1-\alpha)$ -quantile of the pooled Monte Carlo scores. The histogram summarizes interpolated estimates of $P_{t}(X_{1}),\dots,P_{t}(X_{n_{test}})$ , approximating the distribution of $P_{t}(X)$ . See Appendices B.2 and B.3 for details.

Figure 3: Synthetic and Journal SJR overview. (a,b) Synthetic seed means, averaged equally over the selected mechanisms. (c,d) Journal SJR seed means of marginal coverage and CEC-X, respectively; (d) uses a representative covariate grouping. See Table 1 for experiment settings and Appendix B.6 for detailed results.

Figure 4: Split-ratio sensitivity on B.2.2 ; settings follow Table 1 . Rows correspond to TabPFN and TabICL. Columns show mean-function RMSE, absolute marginal coverage error, and CCAD; lower values are better. Standard boxplots summarize variation across seeds; connected diamonds mark the means.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Percentile rank–score diagnostics for B.2.1 , seed 2026: C-USIM ( $n_{\mathrm{cal}}\to\infty$ ) and plug-in HPD.

Panel of Figure 6 — Figure 6: Percentile rank–score diagnostics for B.2.1 , seed 2026: C-USIM ( $n_{\mathrm{cal}}\to\infty$ ) and plug-in HPD.

Panel of Figure 7 — Figure 7: Percentile rank–score diagnostics for B.2.1 , seed 2026: C-USIM ( $n_{\mathrm{cal}}\to\infty$ ) and plug-in HPD.

Panel of Figure 8 — Figure 8: Percentile rank–score diagnostics for B.2.2 , seed 2026: C-USIM ( $n_{\mathrm{cal}}\to\infty$ ) and plug-in HPD.

Panel of Figure 9 — Figure 9: Percentile rank–score diagnostics for B.2.2 , seed 2026: C-USIM ( $n_{\mathrm{cal}}\to\infty$ ) and plug-in HPD.

Panel of Figure 10 — Figure 10: Percentile rank–score diagnostics for B.2.2 , seed 2026: C-USIM ( $n_{\mathrm{cal}}\to\infty$ ) and plug-in HPD.

Setting	Synthetic	Journal SJR	Split ratio
Dataset size	1,792 per run	27,803 cleaned rows	1,792 per run
Total label budget	1,536	1,536	1,536
Input dimensions	1, 5, 10, 20	7 (categorical)	1, 5, 10, 20
Run seeds	100–109	12100–12299	25600–25649
Number of seeds	10	200	50
Plug-in context observations	1,536	1,536	–

Appendix

Table 1: Experimental settings. Synthetic and split-ratio sample counts are per mechanism and run; conditional response draws are additional Monte Carlo samples. Split-ratio coverage and CCAD use the known conditional CDF rather than the stored response draws. A dash denotes an unused setting.

Panel of Figure 11 — Figure 11: B.2.1 , seed 100. Plotting details: Appendix B.6.1 .

Panel of Figure 12 — Figure 12: B.2.1 , seed 100. Plotting details: Appendix B.6.1 .

Panel of Figure 13 — Figure 13: B.2.1 , seed 100. Plotting details: Appendix B.6.1 .

Panel of Figure 14 — Figure 14: B.2.2 , seed 100. Plotting details: Appendix B.6.1 .

Panel of Figure 15 — Figure 15: B.2.2 , seed 100. Plotting details: Appendix B.6.1 .

Panel of Figure 16 — Figure 16: B.2.2 , seed 100. Plotting details: Appendix B.6.1 .

Model	Example	Coverage (%)	CCAD (pp)	Mean length
		Plug-in HPD(1536) $\rightarrow$ C-USIM
TabPFN	B.2.1	$91.92\rightarrow 95.28$	$3.572\rightarrow 2.159$	$0.571\rightarrow 0.717$
	B.2.1	$89.77\rightarrow 94.74$	$5.384\rightarrow 2.186$	$0.754\rightarrow 1.290$
	B.2.1	$91.63\rightarrow 94.91$	$3.513\rightarrow 1.708$	$0.483\rightarrow 0.580$
	B.2.2	$89.05\rightarrow 95.33$	$5.986\rightarrow 2.201$	$0.915\rightarrow 2.100$
	B.2.2	$88.24\rightarrow 94.81$	$6.854\rightarrow 1.642$	$2.274\rightarrow 3.313$

Appendix

Table 2: Synthetic comparison under a fixed budget of 1,536 labels, averaged over ten seeds per model and case. Each arrow denotes Plug-in HPD(1536)

\rightarrow

C-USIM(512+1024). Coverage is in percent and CCAD in percentage points (pp). Cases are defined in Appendix B.2 .

Figure 17: Synthetic coverage (a,b) and CCAD (c,d) across ten seeds for Plug-in HPD(1536) and C-USIM(512+1024); methods and label budgets follow Table 2 . Boxes show Q1–Q3 and medians, with 1.5-IQR whiskers and all outliers. Dashed lines mark 95% coverage. CCAD is in percentage points on model-specific scales. These seed distributions are not confidence intervals.

Figure 18: Journal SJR across 200 seeds for Plug-in HPD(1536) and C-USIM(512+1024); methods and label budgets follow Table 3 . (a,b) CEC-X, weighted by group size; (c,d) group coverage; (e,f) absolute group error. Errors are computed within each seed. Boxes show Q1–Q3 and medians, with 1.5-IQR whiskers and all outliers, not confidence intervals. Red bands mark higher mean absolute group error with C-USIM; dashed lines mark 95% coverage. Errors are in percentage points. CEC-X shares a scale across models; group plots use model-specific scales with visual padding below zero.

Measure	TabPFN	TabICL
	Plug-in HPD(1536) $\rightarrow$ C-USIM
Marginal coverage (%)	$91.50\rightarrow 94.83$	$92.93\rightarrow 94.87$
Mean marginal gap (pp)	$3.499\rightarrow 0.553$	$2.241\rightarrow 0.611$
Mean group gap (pp)	$2.979\rightarrow 1.760$	$2.458\rightarrow 1.856$
CEC-X (pp)	$3.698\rightarrow 1.662$	$2.782\rightarrow 1.707$
Mean set length	$1.149\rightarrow 1.453$	$1.255\rightarrow 1.540$

Appendix

Table 3: Journal SJR under a fixed budget of 1,536 labels at 95% target coverage, averaged over 200 seeds. Each arrow denotes Plug-in HPD(1536)

\rightarrow

C-USIM(512+1024). Absolute marginal and group gaps and CEC-X are computed within each seed before averaging, and are reported in percentage points (pp).

Figure 19: Additional real-world examples under a total budget of 1,536 labels. Points are means over 50 seeds, not confidence intervals. (a,b) JP Anime; (c,d) Allstate Claims Severity. Dashed lines mark 95% coverage. CEC-X uses the representative covariate grouping and is computed within each seed before averaging. Settings and detailed results are in Table 4 and Appendix B.6.3 .

Dataset	Rows	Features	Categorical	Test rows	Response scale
JP Anime	15,351	10	8	5,372	$\ln(\mathrm{Score})$
Allstate	188,318	130	116	9,731	Claim loss

Appendix

Table 4: Additional real-world datasets.

Dataset	Measure	TabPFN	TabICL
		Plug-in HPD (1536) $\rightarrow$ C-USIM
JP Anime	Marginal coverage (%)	$88.54\rightarrow 94.96$	$91.40\rightarrow 94.96$
	Mean marginal gap (pp)	$6.461\rightarrow 0.590$	$3.598\rightarrow 0.540$
	Mean group gap (pp)	$6.224\rightarrow 2.770$	$4.235\rightarrow 2.837$
	CEC-X (pp)	$6.598\rightarrow 2.429$	$4.211\rightarrow 2.521$
	Mean set length	$0.447\rightarrow 0.565$	$0.494\rightarrow 0.539$

Appendix

Table 5: Additional real-world datasets under a fixed total budget of 1,536 labels, averaged over 50 seeds. Each arrow denotes Plug-in HPD(1536)

\rightarrow

C-USIM(512+1024). Marginal and group errors are computed within each seed before averaging; group measures use the representative grouping. Length uses the first 256 fixed test inputs.

	TabPFN		TabICL
Case	Gap (pp)	CCAD (pp)	Gap (pp)	CCAD (pp)
	1229:307 ( $\simeq 8{:}2$ ) $\rightarrow$ 512:1024 ( $1{:}2$ )
B.2.1	$1.033\rightarrow 0.554$	$2.110\rightarrow 2.152$	$0.911\rightarrow 0.440$	$2.830\rightarrow 3.381$
B.2.1	$0.964\rightarrow 0.497$	$1.998\rightarrow 2.093$	$1.013\rightarrow 0.561$	$2.370\rightarrow 2.541$
B.2.1	$0.857\rightarrow 0.546$	$1.231\rightarrow 1.379$	$1.098\rightarrow 0.552$	$2.486\rightarrow 3.104$
B.2.2	$1.026\rightarrow 0.583$	$2.271\rightarrow 2.218$	$1.119\rightarrow 0.650$	$2.569\rightarrow 2.634$

Appendix

Table 6: Split-ratio sensitivity across all six selected mechanisms with 1,536 total labels. Entries are means over the same 50 seeds. Gap is the per-seed absolute deviation of marginal coverage from 95%; gap and CCAD use percentage points (pp).

Model	Example	Coverage (%)	CCAD (pp)	Mean length
		Plug-in HPD(512) $\rightarrow$ C-USIM
TabPFN	B.2.1	$92.95\rightarrow 95.28$	$2.947\rightarrow 2.159$	$0.618\rightarrow 0.717$
	B.2.1	$90.24\rightarrow 94.74$	$5.074\rightarrow 2.186$	$0.805\rightarrow 1.290$
	B.2.1	$92.11\rightarrow 94.91$	$3.268\rightarrow 1.708$	$0.516\rightarrow 0.580$
	B.2.2	$88.49\rightarrow 95.33$	$6.569\rightarrow 2.201$	$0.841\rightarrow 2.100$
	B.2.2	$88.42\rightarrow 94.81$	$6.714\rightarrow 1.642$	$2.281\rightarrow 3.313$

Appendix

Table 7: Synthetic comparison using identical predictive densities from a 512-observation context. Each arrow denotes Plug-in HPD(512)

\rightarrow

C-USIM(512+1024), averaged over ten seeds. C-USIM additionally uses 1,024 calibration responses.

Figure 20: Synthetic coverage, CCAD, and prediction-set length for all three configurations. Points are means over ten seeds, not confidence intervals. Horizontal lines separate mechanisms. The two 512-context methods share predictive densities. Dashed lines mark 95% coverage. All interval components contribute to length, excluding gaps.

Figure 21: Journal SJR with both plug-in baselines and C-USIM, using the representative grouping. (a,d) CEC-X over 200 seeds: boxes show quartiles and medians, whiskers extend to 1.5 IQR, all outliers are retained, and colored markers denote means. (b,e) Mean coverage in each group. (c,f) Mean within-seed absolute group error. Horizontal lines in the group plots separate groups. These summaries are not confidence intervals. Dashed lines mark 95% coverage; all ten groups are retained.

Measure	TabPFN	TabICL
	Plug-in HPD(512) $\rightarrow$ C-USIM
Marginal coverage (%)	$91.06\rightarrow 94.83$	$91.72\rightarrow 94.87$
Mean marginal gap (pp)	$3.964\rightarrow 0.553$	$3.431\rightarrow 0.611$
Mean group gap (pp)	$3.846\rightarrow 1.760$	$3.630\rightarrow 1.856$
CEC-X (pp)	$4.333\rightarrow 1.662$	$3.863\rightarrow 1.707$
Mean set length	$1.261\rightarrow 1.453$	$1.359\rightarrow 1.540$

Appendix

Table 8: Journal SJR using identical predictive densities from a 512-observation context, averaged over 200 seeds. Each arrow denotes Plug-in HPD(512)

\rightarrow

C-USIM(512+1024). Both configurations use the same joint calibration/test query table; only C-USIM uses the 1,024 calibration responses. Group errors use the representative grouping. Coverage uses all 9,731 test rows; length uses the first 256 inputs in the fixed test order.

Dataset	Measure	TabPFN	TabICL
		Plug-in HPD (512) $\rightarrow$ C-USIM
JP Anime	Marginal coverage (%)	$91.05\rightarrow 94.96$	$94.01\rightarrow 94.96$
	Mean marginal gap (pp)	$3.974\rightarrow 0.590$	$1.853\rightarrow 0.540$
	Mean group gap (pp)	$4.697\rightarrow 2.770$	$3.330\rightarrow 2.837$
	CEC-X (pp)	$4.681\rightarrow 2.429$	$3.103\rightarrow 2.521$
	Mean set length	$0.486\rightarrow 0.565$	$0.522\rightarrow 0.539$

Appendix

Table 9: Additional real-world datasets using identical predictive densities from a 512-observation context, averaged over 50 seeds. Each arrow denotes Plug-in HPD(512)

\rightarrow

C-USIM(512+1024). Only C-USIM uses the additional 1,024 calibration responses. Group measures use the representative grouping; response scales and test sizes follow Table 4 .

Figure 22: JP Anime with both plug-in baselines and C-USIM, using the representative grouping. (a,d) CEC-X over 50 seeds: boxes show quartiles and medians, whiskers extend to 1.5 IQR, all outliers are retained, and colored markers denote means. (b,e) Mean group coverage. (c,f) Mean within-seed absolute group error. Horizontal lines separate groups; group 9 has 15 test observations. Dashed lines mark 95% coverage. These summaries are not confidence intervals.

Figure 23: Allstate Claims Severity with both plug-in baselines and C-USIM, using the representative grouping. (a,d) CEC-X over 50 seeds: boxes show quartiles and medians, whiskers extend to 1.5 IQR, all outliers are retained, and colored markers denote means. (b,e) Mean group coverage. (c,f) Mean within-seed absolute group error. Horizontal lines separate all ten groups; dashed lines mark 95% coverage. These summaries are not confidence intervals.

Explore similar work

Aug 6, 2026cs.LG

Do Tabular Foundation Models Agree with Themselves?

Tabular Foundation Models (TFMs) are currently the best approach to tabular prediction problems. They are constructed as transformers that approximate the Bayesian posterior predictive distribution based on a pre-training prior. These univariate predictors can be converted into multivariate ones autoregressively by sampling one target and adding it to the features. However, the faithfulness of the resulting joint has not been investigated. Furthermore, TFMs cannot be evaluated against the posterior itself, at least not on real-world datasets, because the ground-truth distribution is unknown. We therefore propose asking a different question: could a model's predictions result from any joint distribution? To answer this question, we pose two requirements that any such model must satisfy. The first is marginalization consistency, which demands that marginalized conditionals are equal to directly predicted marginals. The second is factorization consistency, which demands that different factorization orders result in equal joint distributions. Every TFM that we evaluate violates both of these requirements for both classification and regression across all datasets.

Christian Klötergens, Vijaya Krishna Yalavarthi, Lars Schmidt-Thieme +1

Institute of Computer Science & VWFS DARC, University of Hildesheim, Hildesheim, Germany

Tabular Foundation Models Time Series Foundation Models Posterior Predictive Distribution Factor

Ask PDF

May 27, 2026cs.LG

High Performance, Low Reliability: Uncertainty Benchmarking for Tabular Foundation Models

Recent Tabular Foundation Models (TFMs) have demonstrated state-of-the-art predictive performance, often surpassing Gradient-Boosted Decision Trees (GBDTs). However, the trustworthiness of these models, particularly their uncertainty quantification, has been largely overlooked. We investigate this gap through an extensive study comparing TFMs, GBDTs, and classical baselines on the 112 datasets of the TALENT benchmark. Our results reveal a performance-uncertainty trade-off: although TFMs achieve the highest predictive performance, measured by AUC, they exhibit lower conditional coverage under conformal prediction, measured by SSCS, compared to GBDTs. Complementary experiments on synthetic datasets further characterize the regimes in which this effect intensifies. We conclude that while TFMs advance predictive frontiers, achieving well-calibrated uncertainty remains a major open challenge for their reliable adoption. Code is available at: https://github.com/jose-melo/high-performance-low-reliability

José Lucas De Melo Costa, Fabrice Popineau, Arpad Rimmel +1

CentraleSup´elec, Universit´e Paris-Saclay Gif-sur-Yvette - France

Tabular Foundation Models Uncertainty Quantification Uncertainty Gradient Boosted Decision Tree

Ask PDF

Mar 9, 2026cs.LG

Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules

Modern tabular foundation models such as TabPFN and TabICL naturally produce full predictive distributions, while the benchmarks used to evaluate them (TabArena, TALENT, and others) still rely almost exclusively on point-estimate metrics (RMSE,

R^2

). This mismatch implicitly rewards machine learning models or pipelines that elicit a good conditional mean while ignoring the quality of the predictive distribution. We make the case for using proper scoring rules for training, fine-tuning, and benchmarking (ranking) of tabular foundation models. Although all strictly proper scoring rules are theoretically equivalent at the population level, they may differ on finite data: We demonstrate analytically and empirically that different scoring rules can induce different inductive biases during finite-sample optimization, leading to different model performance. We validate this finding by running fine-tuning experiments with TabPFN and TabICL using different scoring rules for various data sets, revealing non-trivial interactions between training objectives and evaluation metrics. Our results show that practitioners can adapt tabular foundation models to task-specific scoring objectives, and that the choice of scoring rule can influence model behavior in practice.

Jonas Landsgesell, Pascal Knoll, Tizian Wenzel

University of Stuttgart (Stuttgart, Germany) · Ludwig Maximilian University of Munich (Munich, Germany) · Munich Center for Machine Learning (Munich, Germany)

Tabular Foundation Models Distributional Information Exponential Family Inductive Bias

Ask PDF