Conformal Prediction Assessment: A Framework for Conditional Coverage Evaluation and Selection

Zheng Zhou; Xiangfei Zhang; Chongguang Tao; Yuhong Yang

stat.MEMar 28, 2026

Conformal Prediction Assessment: A Framework for Conditional Coverage Evaluation and Selection

Authors: Zheng Zhou, Xiangfei Zhang, Chongguang Tao, Yuhong Yang

Organizations: Qiuzhen College, Tsinghua University · Yau Mathematical Sciences Center, Tsinghua University

Abstract

Conformal prediction provides rigorous distribution-free finite-sample guarantees for marginal coverage under the assumption of exchangeability, but may exhibit systematic undercoverage or overcoverage for specific subpopulations. Assessing conditional validity is challenging, as standard stratification methods suffer from the curse of dimensionality. We propose Conformal Prediction Assessment (CPA), a framework that reframes the evaluation of conditional coverage as a supervised learning task by training a reliability estimator that predicts instance-level coverage probabilities. Building on this estimator, we introduce the Conditional Validity Index (CVI), which decomposes reliability into safety (undercoverage risk) and efficiency (overcoverage cost). We establish convergence rates for the reliability estimator and prove the consistency of CVI-based model selection. Extensive experiments on synthetic and real-world datasets demonstrate that CPA effectively diagnoses local failure modes and that CC-Select, our CVI-based model selection algorithm, consistently identifies predictors with superior conditional coverage performance.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the predictive reliability learning framework. The pipeline consists of four stages: data preparation, conformal model training, reliability estimation via $\hat{\eta}(x)$ , and downstream applications including evaluation, model selection, and deployment.

Figure 2: Conditional Validity Profile (CVP) Curve. This curve depicts conditional coverage for a standard conformal predictor. The red area reflects undercoverage risk (

\text{CVI}_{\text{U}}

), the blue area represents overcoverage cost (

\text{CVI}_{\text{O}}

), and their intersection marks the fraction of undercovered samples (

\pi_{-}

Setting	Mean Function $\mu(X)$	Noise Distribution	Challenge
A: Linear	Sparse Linear	$\mathcal{N}(0,1)$	Baseline Validity
B: Nonlinear	Nonlinear Interactions	$t_{2}$ (Heavy-tailed)	Model Misspecification
C: Heteroscedastic	Linear	$\mathcal{N}(0,\sigma(X)^{2})$	Conditional Coverage
D: Correlated Covariates	Linear (Dependent $X$ )	$\sigma(X)\cdot t_{2}$	Feature Dependence & Heavy-tails

Table 1: Summary of Data Generating Processes used in our experiments.

Setting	Kendall’s $\tau_{w}$	Spearman’s $\rho$	NDCG@1	NDCG@3	Hit@3
A (Linear)	0.902 (0.014)	0.771 (0.021)	0.809 (0.028)	0.836 (0.019)	0.580 (0.035)
B (Heavy-Tailed)	0.784 (0.024)	0.718 (0.029)	0.953 (0.015)	0.965 (0.005)	0.740 (0.026)
C (Heteroscedastic)	0.902 (0.009)	0.800 (0.019)	0.852 (0.018)	0.964 (0.004)	0.920 (0.020)
D (Correlated Covariates)	0.797 (0.025)	0.696 (0.034)	0.955 (0.015)	0.972 (0.004)	0.673 (0.033)

Table 2: Rank concordance between CPA-estimated and Oracle rankings. Results are averaged over independent experimental runs, with standard deviations given in parentheses.

Figure 3: Distributional recovery in Setting B (nonlinear, Heavy-tailed). The estimated CVP ( red dashed ) closely tracks the ground-truth Oracle distribution ( blue ), demonstrating that CPA effectively recovers the full reliability landscape. The annotated WD denotes the Wasserstein Distance, quantifying the discrepancy between the estimated and Oracle distributions (lower is better).

Figure 4: Robustness analysis of ranking fidelity (Left) and stability (Right). Heatmaps compare reliability estimators ( $y$ -axis) across DGP settings ( $x$ -axis). The Left Column measures fidelity against the ground-truth Oracle ( $\hat{\pi}$ vs. $\pi^{\text{oracle}}$ ), while the Right Column assesses stability relative to the Baseline ( $\hat{\pi}$ vs. $\pi^{\text{base}}$ ).

Dataset	$n$	$p$	Target Variable	Domain
Bike	$10,886$	18	Hourly rental count	Transportation
Computer	$8192$	21	CPU execution time	Computer Systems
Debutanizer	$2400$	7	Butane content	Industrial Control
Kin8nm	$8192$	8	Kinematic response	Physics
Meps_21	$15,656$	139	Health expenditure	Healthcare
Miami_2016	$13,932$	15	Sale price	Real Estate

Table 3: Summary of real-world regression tasks used in the empirical study.

Figure 5: Reliability diagrams for CPA across nine datasets. Each plot compares empirical coverage (y-axis) with predicted reliability (x-axis) using

B=10

equal-frequency bins.

Figure 6: Distribution of Worst-Slab Coverage (WSC) scores. The procedure selected by CC-Select (“Selected”, blue box) consistently achieves high WSC scores.

Panel of Figure 7 — Figure 7: Diagnostics for the baseline conformal predictor CP-Residual on the Bike dataset. Left: the Conditional Validity Profile (CVP) reveals substantial heterogeneity in conditional coverage despite near-nominal marginal coverage. Right: the hourly reliability pattern shows clear undercoverage during peak commuting hours, and the estimated curve broadly follows the empirical trend.

Panel of Figure 8 — Figure 8: Conditional Validity Profile (CVP) curves for two adaptive refinements on the Bike dataset. Compared with CP-Residual, both CP-Studentized and CQR exhibit improved conditional validity, with CVP curves closer to the nominal target.

Panel of Figure 9 — Figure 9: Conditional coverage by hour for two adaptive refinements on the Bike dataset. Compared with CP-Residual, both CP-Studentized and CQR reduce the undercoverage during peak commuting hours, with CQR exhibiting the most stable hourly pattern.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure 10: Illustration of the limiting conditional coverage function $\eta(x)$ . Panels (a)–(c) correspond to settings where the true regression function is linear with heteroscedastic noise, while panel (d) considers a nonlinear (sine) regression function with homoscedastic noise. In all cases, the fitted regression model is linear, demonstrating that $\eta(x)$ may deviate substantially from the nominal level $1-\alpha$ even asymptotically.

Figure 11: High-dimensional illustration of the feasibility claim. Although the ambient dimension is

d=200

, the conditional coverage surface induced by the fixed residual-score conformal predictor varies mainly with

x_{1}

, and a one-dimensional smoother recovers this pattern closely. The MAE and RMSE are computed against Monte Carlo approximations of the pointwise true conditional coverage.

$d$	20	50	100	200
MAE	0.026 (0.007)	0.024 (0.004)	0.022 (0.006)	0.026 (0.009)
RMSE	0.034 (0.009)	0.031 (0.005)	0.030 (0.009)	0.033 (0.012)

Appendix

Table 4: Estimation accuracy of the conditional coverage function under different ambient dimensions. Entries report mean (SD) over 10 repetitions.

ID	Model Architecture / Specification	Calibration
Baseline	AutoML (Best of LR, RF, GBM, XGB, KNN pool)	Isotonic
Sensitivity to Calibration Method
pert_1	Same as Baseline (AutoML)	None (Raw Output)
pert_2	Same as Baseline (AutoML)	Sigmoid (Platt)
pert_3	Same as Baseline (AutoML)	Histogram Binning
Sensitivity to Structural Misspecification

Appendix

Table 5: Detailed configurations of reliability estimators used in the robustness analysis. The Baseline employs the full AutoML pipeline coupled with isotonic calibration.

Figure 12: Distribution of pointwise estimation bias

\hat{\eta}(x)-\eta_{\text{oracle}}(x)

. While the Baseline and other high-capacity estimators remain robustly centered near zero, the Logistic Regression estimator exhibits a severe systemic downward bias in the nonlinear Setting B, corroborating the capacity mismatch hypothesis.

Figure 13: Impact of the data split ratio

\rho=n_{\text{train}}/N

on assessment accuracy (Weighted Kendall’s

\tau_{w}

). While the complex scenarios (Settings B, C, D) exhibit a characteristic concave profile peaking near

\rho=0.5

, the simple linear scenario (Setting A) achieves optimal performance with minimal training data (

\rho=0.1

Figure 14: Sample complexity analysis of CPA in heteroscedastic settings. Panel A illustrates the learning curve of ranking consistency (

\tau_{w}

). Panel B shows the probability of identifying the optimal model (Hit@1). Panel C demonstrates the robustness of the selection utility (NDCG@1). The lines represent the mean performance over 50 replications.

Figure 15: Reliability landscape: Temperature vs. Hour. Rows correspond to CP-Residual (top), CP-Studentized (middle), and CQR (bottom). Columns display the smoothed empirical coverage (left), the CPA-predicted reliability $\hat{\eta}(x)$ (middle), and the sample density (right). Red indicates undercoverage; blue indicates overcoverage.

Explore similar work

Beyond Marginal Validity: Finite-Sample Guarantees for Localized Conformal Prediction
Aug 6, 2026Anton Conrad, Rustam Isaev, Denis Belomestny +2Conformal Prediction Statistical Learning Theory
Laboratoire de Recherche d’EPITA, 14–16 rue Voltaire, 94276 Le Kremlin-Bicêtre CEDEX, France · Faculty of Computer Science, HSE University, 11 Pokrovsky Boulevard, Moscow 109028, Russia · Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University, 1 Leninskie Gory, Building 52, Moscow 119991, Russia +2
Co-optimization for Adaptive Conformal Prediction
Mar 2, 2026Xiaoyi Su, Zhixin Zhou, Rui LuoConformal Prediction Regression Prediction Intervals
Department of Systems Engineering, City University of Hong Kong, Hong Kong SAR, China · Alpha Benito Research, Los Angeles, USA
Conformal Prediction and Conditional Coverage for Tabular Foundation Models
Sep 28, 2026Sungwoo Park, Sunghee Park, Won ChangTabular Foundation Models Group-Conditional Conformal Prediction
Department of Statistics Seoul National University Seoul, Republic of Korea

Aug 6, 2026J/K move · Enter open · S save

Beyond Marginal Validity: Finite-Sample Guarantees for Localized Conformal Prediction

Anton Conrad, Rustam Isaev, Denis Belomestny, Eric Moulines+1

Laboratoire de Recherche d’EPITA, 14–16 rue Voltaire, 94276 Le Kremlin-Bicêtre CEDEX, France · Faculty of Computer Science, HSE University, 11 Pokrovsky Boulevard, Moscow 109028, Russia · Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University, 1 Leninskie Gory, Building 52, Moscow 119991, Russia+2

Conformal Prediction Statistical Learning Theory

Open paper AskPDF arXiv

Conformal prediction endows arbitrary black-box predictors with finite-sample, distribution-free marginal coverage, yet marginal validity can hide severe covariate-specific miscalibration, while exact distribution-free conditional coverage is finite-sample unattainable. Randomly localized conformal prediction (RLCP) mitigates this gap by calibrating near the test point while preserving marginal coverage. Existing theory, however, lacks finite-sample guarantees for the realized localized set that jointly control conditional validity and oracle efficiency. We provide such guarantees. For any fixed score, under Hölder regularity of the conditional score CDF and standard density and kernel assumptions, we prove high-probability bounds, uniform over a realized localization neighbourhood, for the conditional-coverage gap and the length error relative to the oracle. The bounds decompose into an

O(h^β)

localization bias and a calibration term decreasing with calibration size, clarifying the bandwidth bias-variance tradeoff and when RLCP tracks the oracle. We also analyze data-split learned scores: when the score targets a pivotal score, as in conformalized quantile regression, uniform local guarantees decompose into fixed-score calibration and uniform score-estimation errors, showing that improved learning sharpens localized guarantees.

Abstract

Explore similar work

Beyond Marginal Validity: Finite-Sample Guarantees for Localized Conformal Prediction

Co-optimization for Adaptive Conformal Prediction

Conformal Prediction and Conditional Coverage for Tabular Foundation Models