Rectifying Conformity Scores for Better Conditional Coverage
Organizations: Lagrange Mathematics and Computing Research Center · Mohamed bin Zayed University of Artificial Intelligence · Skolkovo Institute of Science and Technology · University of Mons ·
Abstract
We present a new method for generating confidence sets within the split conformal prediction framework. Our method performs a trainable transformation of any given conformity score to improve conditional coverage while ensuring exact marginal coverage. The transformation is based on an estimate of the conditional quantile of conformity scores. The resulting method is particularly beneficial for constructing adaptive confidence sets in multi-output problems where standard conformal quantile regression approaches have limited applicability. We develop a theoretical bound that captures the influence of the accuracy of the quantile estimate on the approximate conditional validity, unlike classical bounds for conformal prediction methods that only offer marginal coverage. We experimentally show that our method is highly adaptive to the local data structure and outperforms existing methods in terms of conditional coverage, improving the reliability of statistical inference in various applications.
Figures & tables
| Coverage |
|---|
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| dataset | PCP | RCP-PCP | DCP | RCP-DCP | ResCP | RCP-ResCP |
|---|---|---|---|---|---|---|
| households | 88.3 | 1.33e+02 | 47.4 | 1.02e+02 | 1.81e+02 | 4.51e+02 |
| scm20d | 4.26e+05 | 7.92e+06 | 1.11e+06 | 3.95e+07 | 5.22e+05 | 7.15e+12 |
| rf1 | 0.0274 | 0.190 | 0.000562 | 4.35e+04 | 0.0276 | 7.94e+08 |
| scm1d | 1.92e+04 | 1.30e+08 | 2.30e+04 | 1.67e+08 | 6.27e+04 | 2.04e+15 |
| meps_21 | 1.65 | 10.0 | 0.746 | 6.07 | 5.35 | 8.32e+12 |
| meps_19 | 90.0 | 3.27e+04 | 3.64 | 3.27e+04 | 5.56 | 3.56e+22 |
| dataset | PCP | RCP-PCP | DCP | RCP-DCP | ResCP | RCP-ResCP |
|---|---|---|---|---|---|---|
| households | 67.4 | 56.5 | 39.2 | 32.1 | 1.81e+02 | 1.67e+02 |
| scm20d | 3.11e+04 | 1.42e+04 | 7.03e+05 | 7.74e+04 | 5.22e+05 | 2.00e+05 |
| rf1 | 0.0110 | 0.00525 | 0.000583 | 33.1 | 0.0276 | 0.0231 |
| scm1d | 1.16e+02 | 2.05 | 1.01e+04 | 25.7 | 6.27e+04 | 1.70e+03 |
| meps_21 | 1.04 | 0.704 | 0.433 | 0.227 | 5.35 | 2.42 |
| meps_19 | 3.85 | 0.754 | 2.10 | 0.303 | 5.56 | 2.54 |
| PCP | RCP -PCP | RCP -PCP | CPCG-PCP | DCP | RCP -DCP | RCP -DCP | CPCG-DCP | |
|---|---|---|---|---|---|---|---|---|
| households | 0.825 | 0.905 | 0.899 | 0.888 | 0.853 | 0.891 | 0.900 | 0.900 |
| scm20d | 0.830 | 0.892 | 0.891 | 0.897 | 0.877 | 0.877 | 0.868 | 0.899 |
| rf1 | 0.731 | 0.830 | 0.877 | 0.838 | 0.715 | 0.863 | 0.827 | 0.872 |
| scm1d | 0.758 | 0.882 | 0.895 | 0.910 | 0.756 | 0.902 | 0.896 | 0.882 |
| meps_21 | 0.739 | 0.874 | 0.904 | 0.881 | 0.789 | 0.881 | 0.879 | 0.905 |
| meps_19 | 0.762 | 0.875 | 0.867 | 0.880 | 0.788 | 0.884 | 0.889 | 0.878 |
| PCP | RCP -PCP | RCP -PCP | CPCG-PCP | DCP | RCP -DCP | RCP -DCP | CPCG-DCP | |
|---|---|---|---|---|---|---|---|---|
| households | 0.258 | 0.604 | 104 | 8840 | 0.00759 | 0.531 | 104 | 8164 |
| scm20d | 2.63 | 4.22 | 772 | 6409 | 0.0182 | 0.852 | 766 | 6012 |
| rf1 | 0.667 | 1.35 | 340 | 10682 | 0.00836 | 0.348 | 339 | 9674 |
| scm1d | 2.27 | 3.57 | 1209 | 4971 | 0.0133 | 0.867 | 1205 | 4692 |
| meps_21 | 0.236 | 0.607 | 581 | 6283 | 0.0123 | 0.261 | 581 | 6031 |
| meps_19 | 0.272 | 0.515 | 493 | 6411 | 0.0123 | 0.184 | 492 | 6128 |
| Paper | Dataset | |||
|---|---|---|---|---|
| Tsoumakas et al. (2011) | scm20d | 8966 | 60 | 16 |
| rf1 | 9005 | 64 | 8 | |
| rf2 | 9005 | 64 | 8 | |
| scm1d | 9803 | 279 | 16 | |
| Feldman et al. (2023) | meps_21 | 15656 | 137 | 2 |
| meps_19 | 15785 | 137 | 2 |
Conformal Prediction
Conformal prediction wraps predictive models with calibrated sets or intervals that provide statistical coverage guarantees.
See all 393 papers