stat.MLSep 28, 2026

Conformal Prediction and Conditional Coverage for Tabular Foundation Models

Authors: Sungwoo Park, Sunghee Park, Won Chang

Organizations: Department of Statistics Seoul National University Seoul, Republic of Korea

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Tabular foundation models (TFMs) provide predictive distributions for regression, but their prediction regions can exhibit undercoverage or overcoverage even when point predictions are accurate. We introduce C-USIM (Conditionally-Uniformized Score Integration Method), a lightweight application of highest predictive density split conformal prediction that accommodates multimodal predictions. Given calibration and test outputs, it requires no additional training or model inference. It provides finite-sample marginal validity under our assumptions. We bound conditional-marginal coverage gaps using distribution-estimation error and score discreteness, and examine coverage heterogeneity through percentile rank-score plots. Experiments with TabPFN and TabICL show improved marginal coverage accuracy and lower average conditional and group coverage errors. Under a fixed data budget, allocating more observations to calibration can reduce marginal coverage error despite less accurate point predictions.

Figures & tables

Figure 1: C-USIM overview with ncaln_{\mathrm{cal}} calibration scores and target coverage 1−α1-\alpha . Using a pretrained TFM, the procedure predicts densities, computes density-rank scores, calibrates a threshold, and constructs prediction regions without fitting an additional model. See Section 4 for details.
Figure 2: Percentile rank–score plots over test covariates (plot (a)), obtained using TabPFN on the sine with shifted exponential noise function ( B.2.1 ). Plot (b) approximates Pt(Xi)P_{t}(X_{i}) at the nominal score threshold t=1−αt=1-\alpha (blue) and the ideal C-USIM threshold t=q1−αt=q_{1-\alpha} (red), which is estimated by the empirical (1−α)(1-\alpha) -quantile of the pooled Monte Carlo scores. The histogram summarizes interpolated estimates of Pt(X1),…,Pt(Xntest)P_{t}(X_{1}),\dots,P_{t}(X_{n_{test}}) , approximating the distribution of Pt(X)P_{t}(X) . See Appendices B.2 and B.3 for details.
Figure 3: Synthetic and Journal SJR overview. (a,b) Synthetic seed means, averaged equally over the selected mechanisms. (c,d) Journal SJR seed means of marginal coverage and CEC-X, respectively; (d) uses a representative covariate grouping. See Table 1 for experiment settings and Appendix B.6 for detailed results.
Figure 4: Split-ratio sensitivity on B.2.2 ; settings follow Table 1 . Rows correspond to TabPFN and TabICL. Columns show mean-function RMSE, absolute marginal coverage error, and CCAD; lower values are better. Standard boxplots summarize variation across seeds; connected diamonds mark the means.
Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Percentile rank–score diagnostics for B.2.1 , seed 2026: C-USIM ( ncal→∞n_{\mathrm{cal}}\to\infty ) and plug-in HPD.
Figure 6: Percentile rank–score diagnostics for B.2.1 , seed 2026: C-USIM ( ncal→∞n_{\mathrm{cal}}\to\infty ) and plug-in HPD.
Figure 7: Percentile rank–score diagnostics for B.2.1 , seed 2026: C-USIM ( ncal→∞n_{\mathrm{cal}}\to\infty ) and plug-in HPD.
Figure 8: Percentile rank–score diagnostics for B.2.2 , seed 2026: C-USIM ( ncal→∞n_{\mathrm{cal}}\to\infty ) and plug-in HPD.
Figure 9: Percentile rank–score diagnostics for B.2.2 , seed 2026: C-USIM ( ncal→∞n_{\mathrm{cal}}\to\infty ) and plug-in HPD.
Figure 10: Percentile rank–score diagnostics for B.2.2 , seed 2026: C-USIM ( ncal→∞n_{\mathrm{cal}}\to\infty ) and plug-in HPD.
SettingSyntheticJournal SJRSplit ratio
Dataset size1,792 per run27,803 cleaned rows1,792 per run
Total label budget1,5361,5361,536
Input dimensions1, 5, 10, 207 (categorical)1, 5, 10, 20
Run seeds100–10912100–1229925600–25649
Number of seeds1020050
Plug-in context observations1,5361,536–
Appendix
Table 1: Experimental settings. Synthetic and split-ratio sample counts are per mechanism and run; conditional response draws are additional Monte Carlo samples. Split-ratio coverage and CCAD use the known conditional CDF rather than the stored response draws. A dash denotes an unused setting.
Figure 11: B.2.1 , seed 100. Plotting details: Appendix B.6.1 .
Figure 12: B.2.1 , seed 100. Plotting details: Appendix B.6.1 .
Figure 13: B.2.1 , seed 100. Plotting details: Appendix B.6.1 .
Figure 14: B.2.2 , seed 100. Plotting details: Appendix B.6.1 .
Figure 15: B.2.2 , seed 100. Plotting details: Appendix B.6.1 .
Figure 16: B.2.2 , seed 100. Plotting details: Appendix B.6.1 .
ModelExampleCoverage (%)CCAD (pp)Mean length
  Plug-in HPD(1536) →\rightarrow C-USIM
TabPFNB.2.191.92→95.2891.92\rightarrow 95.283.572→2.1593.572\rightarrow 2.1590.571→0.7170.571\rightarrow 0.717
 B.2.189.77→94.7489.77\rightarrow 94.745.384→2.1865.384\rightarrow 2.1860.754→1.2900.754\rightarrow 1.290
 B.2.191.63→94.9191.63\rightarrow 94.913.513→1.7083.513\rightarrow 1.7080.483→0.5800.483\rightarrow 0.580
 B.2.289.05→95.3389.05\rightarrow 95.335.986→2.2015.986\rightarrow 2.2010.915→2.1000.915\rightarrow 2.100
 B.2.288.24→94.8188.24\rightarrow 94.816.854→1.6426.854\rightarrow 1.6422.274→3.3132.274\rightarrow 3.313
Appendix
Table 2: Synthetic comparison under a fixed budget of 1,536 labels, averaged over ten seeds per model and case. Each arrow denotes Plug-in HPD(1536) →\rightarrow C-USIM(512+1024). Coverage is in percent and CCAD in percentage points (pp). Cases are defined in Appendix B.2 .
Figure 17: Synthetic coverage (a,b) and CCAD (c,d) across ten seeds for Plug-in HPD(1536) and C-USIM(512+1024); methods and label budgets follow Table 2 . Boxes show Q1–Q3 and medians, with 1.5-IQR whiskers and all outliers. Dashed lines mark 95% coverage. CCAD is in percentage points on model-specific scales. These seed distributions are not confidence intervals.
Figure 18: Journal SJR across 200 seeds for Plug-in HPD(1536) and C-USIM(512+1024); methods and label budgets follow Table 3 . (a,b) CEC-X, weighted by group size; (c,d) group coverage; (e,f) absolute group error. Errors are computed within each seed. Boxes show Q1–Q3 and medians, with 1.5-IQR whiskers and all outliers, not confidence intervals. Red bands mark higher mean absolute group error with C-USIM; dashed lines mark 95% coverage. Errors are in percentage points. CEC-X shares a scale across models; group plots use model-specific scales with visual padding below zero.
MeasureTabPFNTabICL
 Plug-in HPD(1536) →\rightarrow C-USIM
Marginal coverage (%)91.50→94.8391.50\rightarrow 94.8392.93→94.8792.93\rightarrow 94.87
Mean marginal gap (pp)3.499→0.5533.499\rightarrow 0.5532.241→0.6112.241\rightarrow 0.611
Mean group gap (pp)2.979→1.7602.979\rightarrow 1.7602.458→1.8562.458\rightarrow 1.856
CEC-X (pp)3.698→1.6623.698\rightarrow 1.6622.782→1.7072.782\rightarrow 1.707
Mean set length1.149→1.4531.149\rightarrow 1.4531.255→1.5401.255\rightarrow 1.540
Appendix
Table 3: Journal SJR under a fixed budget of 1,536 labels at 95% target coverage, averaged over 200 seeds. Each arrow denotes Plug-in HPD(1536) →\rightarrow C-USIM(512+1024). Absolute marginal and group gaps and CEC-X are computed within each seed before averaging, and are reported in percentage points (pp).
Figure 19: Additional real-world examples under a total budget of 1,536 labels. Points are means over 50 seeds, not confidence intervals. (a,b) JP Anime; (c,d) Allstate Claims Severity. Dashed lines mark 95% coverage. CEC-X uses the representative covariate grouping and is computed within each seed before averaging. Settings and detailed results are in Table 4 and Appendix B.6.3 .
DatasetRowsFeaturesCategoricalTest rowsResponse scale
JP Anime15,3511085,372ln⁡(Score)\ln(\mathrm{Score})
Allstate188,3181301169,731Claim loss
Appendix
Table 4: Additional real-world datasets.
DatasetMeasureTabPFNTabICL
  Plug-in HPD (1536) →\rightarrow C-USIM
JP AnimeMarginal coverage (%)88.54→94.9688.54\rightarrow 94.9691.40→94.9691.40\rightarrow 94.96
 Mean marginal gap (pp)6.461→0.5906.461\rightarrow 0.5903.598→0.5403.598\rightarrow 0.540
 Mean group gap (pp)6.224→2.7706.224\rightarrow 2.7704.235→2.8374.235\rightarrow 2.837
 CEC-X (pp)6.598→2.4296.598\rightarrow 2.4294.211→2.5214.211\rightarrow 2.521
 Mean set length0.447→0.5650.447\rightarrow 0.5650.494→0.5390.494\rightarrow 0.539
Appendix
Table 5: Additional real-world datasets under a fixed total budget of 1,536 labels, averaged over 50 seeds. Each arrow denotes Plug-in HPD(1536) →\rightarrow C-USIM(512+1024). Marginal and group errors are computed within each seed before averaging; group measures use the representative grouping. Length uses the first 256 fixed test inputs.
 TabPFNTabICL
CaseGap (pp)CCAD (pp)Gap (pp)CCAD (pp)
 1229:307 ( ≃8:2\simeq 8{:}2 ) →\rightarrow 512:1024 ( 1:21{:}2 )
B.2.11.033→0.5541.033\rightarrow 0.5542.110→2.1522.110\rightarrow 2.1520.911→0.4400.911\rightarrow 0.4402.830→3.3812.830\rightarrow 3.381
B.2.10.964→0.4970.964\rightarrow 0.4971.998→2.0931.998\rightarrow 2.0931.013→0.5611.013\rightarrow 0.5612.370→2.5412.370\rightarrow 2.541
B.2.10.857→0.5460.857\rightarrow 0.5461.231→1.3791.231\rightarrow 1.3791.098→0.5521.098\rightarrow 0.5522.486→3.1042.486\rightarrow 3.104
B.2.21.026→0.5831.026\rightarrow 0.5832.271→2.2182.271\rightarrow 2.2181.119→0.6501.119\rightarrow 0.6502.569→2.6342.569\rightarrow 2.634
Appendix
Table 6: Split-ratio sensitivity across all six selected mechanisms with 1,536 total labels. Entries are means over the same 50 seeds. Gap is the per-seed absolute deviation of marginal coverage from 95%; gap and CCAD use percentage points (pp).
ModelExampleCoverage (%)CCAD (pp)Mean length
  Plug-in HPD(512) →\rightarrow C-USIM
TabPFNB.2.192.95→95.2892.95\rightarrow 95.282.947→2.1592.947\rightarrow 2.1590.618→0.7170.618\rightarrow 0.717
 B.2.190.24→94.7490.24\rightarrow 94.745.074→2.1865.074\rightarrow 2.1860.805→1.2900.805\rightarrow 1.290
 B.2.192.11→94.9192.11\rightarrow 94.913.268→1.7083.268\rightarrow 1.7080.516→0.5800.516\rightarrow 0.580
 B.2.288.49→95.3388.49\rightarrow 95.336.569→2.2016.569\rightarrow 2.2010.841→2.1000.841\rightarrow 2.100
 B.2.288.42→94.8188.42\rightarrow 94.816.714→1.6426.714\rightarrow 1.6422.281→3.3132.281\rightarrow 3.313
Appendix
Table 7: Synthetic comparison using identical predictive densities from a 512-observation context. Each arrow denotes Plug-in HPD(512) →\rightarrow C-USIM(512+1024), averaged over ten seeds. C-USIM additionally uses 1,024 calibration responses.
Figure 20: Synthetic coverage, CCAD, and prediction-set length for all three configurations. Points are means over ten seeds, not confidence intervals. Horizontal lines separate mechanisms. The two 512-context methods share predictive densities. Dashed lines mark 95% coverage. All interval components contribute to length, excluding gaps.
Figure 21: Journal SJR with both plug-in baselines and C-USIM, using the representative grouping. (a,d) CEC-X over 200 seeds: boxes show quartiles and medians, whiskers extend to 1.5 IQR, all outliers are retained, and colored markers denote means. (b,e) Mean coverage in each group. (c,f) Mean within-seed absolute group error. Horizontal lines in the group plots separate groups. These summaries are not confidence intervals. Dashed lines mark 95% coverage; all ten groups are retained.
MeasureTabPFNTabICL
 Plug-in HPD(512) →\rightarrow C-USIM
Marginal coverage (%)91.06→94.8391.06\rightarrow 94.8391.72→94.8791.72\rightarrow 94.87
Mean marginal gap (pp)3.964→0.5533.964\rightarrow 0.5533.431→0.6113.431\rightarrow 0.611
Mean group gap (pp)3.846→1.7603.846\rightarrow 1.7603.630→1.8563.630\rightarrow 1.856
CEC-X (pp)4.333→1.6624.333\rightarrow 1.6623.863→1.7073.863\rightarrow 1.707
Mean set length1.261→1.4531.261\rightarrow 1.4531.359→1.5401.359\rightarrow 1.540
Appendix
Table 8: Journal SJR using identical predictive densities from a 512-observation context, averaged over 200 seeds. Each arrow denotes Plug-in HPD(512) →\rightarrow C-USIM(512+1024). Both configurations use the same joint calibration/test query table; only C-USIM uses the 1,024 calibration responses. Group errors use the representative grouping. Coverage uses all 9,731 test rows; length uses the first 256 inputs in the fixed test order.
DatasetMeasureTabPFNTabICL
  Plug-in HPD (512) →\rightarrow C-USIM
JP AnimeMarginal coverage (%)91.05→94.9691.05\rightarrow 94.9694.01→94.9694.01\rightarrow 94.96
 Mean marginal gap (pp)3.974→0.5903.974\rightarrow 0.5901.853→0.5401.853\rightarrow 0.540
 Mean group gap (pp)4.697→2.7704.697\rightarrow 2.7703.330→2.8373.330\rightarrow 2.837
 CEC-X (pp)4.681→2.4294.681\rightarrow 2.4293.103→2.5213.103\rightarrow 2.521
 Mean set length0.486→0.5650.486\rightarrow 0.5650.522→0.5390.522\rightarrow 0.539
Appendix
Table 9: Additional real-world datasets using identical predictive densities from a 512-observation context, averaged over 50 seeds. Each arrow denotes Plug-in HPD(512) →\rightarrow C-USIM(512+1024). Only C-USIM uses the additional 1,024 calibration responses. Group measures use the representative grouping; response scales and test sizes follow Table 4 .
Figure 22: JP Anime with both plug-in baselines and C-USIM, using the representative grouping. (a,d) CEC-X over 50 seeds: boxes show quartiles and medians, whiskers extend to 1.5 IQR, all outliers are retained, and colored markers denote means. (b,e) Mean group coverage. (c,f) Mean within-seed absolute group error. Horizontal lines separate groups; group 9 has 15 test observations. Dashed lines mark 95% coverage. These summaries are not confidence intervals.
Figure 23: Allstate Claims Severity with both plug-in baselines and C-USIM, using the representative grouping. (a,d) CEC-X over 50 seeds: boxes show quartiles and medians, whiskers extend to 1.5 IQR, all outliers are retained, and colored markers denote means. (b,e) Mean group coverage. (c,f) Mean within-seed absolute group error. Horizontal lines separate all ten groups; dashed lines mark 95% coverage. These summaries are not confidence intervals.

Explore similar work

CardsList

  1. Do Tabular Foundation Models Agree with Themselves?

    Aug 6, 2026Christian Klötergens, Vijaya Krishna Yalavarthi, Lars Schmidt-Thieme +1Tabular Foundation ModelsTime Series Foundation Models

    Institute of Computer Science & VWFS DARC, University of Hildesheim, Hildesheim, Germany

  2. High Performance, Low Reliability: Uncertainty Benchmarking for Tabular Foundation Models

    May 27, 2026José Lucas De Melo Costa, Fabrice Popineau, Arpad Rimmel +1Tabular Foundation ModelsUncertainty Quantification

    CentraleSup´elec, Universit´e Paris-Saclay Gif-sur-Yvette - France

  3. Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules

    Mar 9, 2026Jonas Landsgesell, Pascal Knoll, Tizian WenzelTabular Foundation ModelsDistributional Information

    University of Stuttgart (Stuttgart, Germany) · Ludwig Maximilian University of Munich (Munich, Germany) · Munich Center for Machine Learning (Munich, Germany)