Hidden Tails: Certifying Tail-Risk Claims under Selective Labels
Organizations: School of Engineering, Institute of Science Tokyo Yokohama, Kanagawa 226-8501, Japan
Abstract
Historical decisions determine which outcomes are observed, potentially concealing a predictor's largest errors. Even unlimited data from the same observation process can therefore leave population tail risk unresolved. Here we derive sharp bounds on conditional value-at-risk, the average loss in a specified upper tail, for binary outcomes and bounded losses. These bounds span exactly the risks compatible with the observations and a stated limit on how outcome odds differ between observed and hidden cases at the same covariate values. Under stated regularity conditions, calibrated statistical certificates cover these ranges simultaneously across candidate predictors, supporting risk limits and model comparisons while separating statistical uncertainty from persistent ambiguity. For a fixed predictor and loss threshold, we derive a minimum-cost allocation for reducing excess-loss ambiguity by prioritizing hidden outcomes according to ambiguity removed per unit cost. This allocation treats audited population contributions as known and retains the original sensitivity bounds separately for audited and still-hidden outcomes. In controlled masking experiments with known loan repayments, recovering about 16% of hidden labels removes a median 61% of estimated fixed-threshold ambiguity, compared with 16% for random recovery. These results connect population tail-risk claims to the available evidence and identify where additional outcomes can sharpen their assessment.
Figures & tables
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Sharp width | Radius | Outer width | Coverage | RMSE upper | RMSE lower | |
|---|---|---|---|---|---|---|
| 1,000 | .0949 | .260 | .614 | 1.000 | .0361 | .0425 |
| 3,000 | .0949 | .150 | .395 | 1.000 | .0171 | .0160 |
| 10,000 | .0949 | .082 | .259 | 1.000 | .0087 | .0076 |
| 30,000 | .0949 | .047 | .190 | 1.000 | .0049 | .0043 |
| 100,000 | .0949 | .026 | .147 | 1.000 | .0027 | .0023 |
| Predictor | Accepted-only | ||||
|---|---|---|---|---|---|
| .95 | Gradient boosting | .795 | .733 | [.703,.759] | [.682,.773] |
| .95 | Logistic | .801 | .805 | [.770,.831] | [.745,.843] |
| .95 | Default-weighted | .661 | .813 | [.782,.833] | [.757,.841] |
| .95 | Constant | .250 | .250 | [.250,.250] | [.250,.250] |
| .90 | Gradient boosting | .740 | .679 | [.625,.715] | [.575,.732] |
| .90 | Logistic | .741 | .721 | [.669,.760] | [.631,.778] |
| Predictor | Plug-in | Enlarged | Mean width | Enlarged width | |
|---|---|---|---|---|---|
| .90 | Gradient boosting | 11/20 | 19/20 | .401 | .434 |
| .90 | Logistic | 13/20 | 20/20 | .401 | .434 |
| .90 | Default-weighted | 14/20 | 20/20 | .332 | .365 |
| .95 | Gradient boosting | 14/20 | 19/20 | .070 | .089 |
| .95 | Logistic | 12/20 | 20/20 | .066 | .084 |
| .95 | Default-weighted | 14/20 | 20/20 | .110 | .139 |
| Comparisons | Resolved | Incorrect | Unresolved | ||
|---|---|---|---|---|---|
| 3 | .90 | 60 | 0 | 0 | 60 |
| 3 | .95 | 60 | 37 | 0 | 23 |
| 1 | Both levels | 120 | 120 | 16 | 0 |
| Acquisition rule | Fixed-threshold gap reduction | Final median sequential interval width |
|---|---|---|
| TIV per label cost | 61% | .035 |
| Random | 16% | .067 |
| Lowest revelation propensity | 9% | .084 |
| Predicted loss | 0% | .116 |
| Predictive uncertainty | 0% | .116 |
| Sensitivity width | 0% | .116 |
| Acquisition rule | Estimated width | Mean Brier | CVaR |
|---|---|---|---|
| TIV | 0.0348 | 0.1362 | 0.8541 |
| Random | 0.0671 | 0.1347 | 0.8682 |
| Paired TIV Random | |||
| Exact predictor reference | |||
| Constant | 0.0000 | 0.2500 | 0.2500 |
| Rule | Observed CVaR | Full CVaR | Upper endpoint | Width | |
|---|---|---|---|---|---|
| Observed labels | .25 | 0.6447 | 0.7845 | 0.8877 | 0.2844 |
| IPW | .25 | 0.6447 | 0.7845 | 0.8877 | 0.2844 |
| Upper envelope | .65 | 0.6686 | 0.6582 | 0.8628 | 0.2286 |
| Full-label oracle | .65 | 0.6686 | 0.6582 | 0.8628 | 0.2286 |
| Labels | TIV | Random | Density | Uncertainty | Predicted loss |
|---|---|---|---|---|---|
| Fixed-threshold gap removed (%) | |||||
| 600 | 42.4 | 30.6 | 40.7 | 15.8 | 15.8 |
| 1,200 | 76.2 | 61.6 | 67.8 | 47.1 | 47.1 |
| 1,800 | 100.0 | 91.4 | 95.5 | 88.9 | 88.9 |
| Separately optimized empirical width removed (%) | |||||
| 600 | 29.4 | 30.3 | 28.7 | 21.3 | 21.3 |
| Specified forecast | Observed CVaR | Full CVaR | Upper endpoint |
|---|---|---|---|
| Synthetic populations: mean over 120 worlds | |||
| Selected-label probability | 0.8052 | 0.8703 | 0.8856 |
| Sensitivity upper probability | 0.8250 | 0.8102 | 0.8198 |
| Full-label probability | 0.7972 | 0.7464 | 0.8287 |
| Diabetes-derived finite population | |||
| Selected-label probability | 0.2665 | 0.9252 | 0.9320 |
| Synthetic median | Diabetes population | |||
|---|---|---|---|---|
| Full-cost budget | Grid width | Grid width | ||
| 0% | 0.2562 | 1.3309 | 0.7295 | 1.1502 |
| 25% | 0.0197 | 0.5393 | 0.2057 | 0.5857 |
| 50% | 0.0080 | 0.1989 | 0.0084 | 0.1484 |
| 75% | 0.0024 | 0.0599 | 0.0002 | 0.0025 |
| 100% | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| Allocation rule | First recorded cost to 25% gap |
|---|---|
| TIV/cost | 42% |
| Fractional-knapsack frontier | 42% |
| Random | 76% |
| High density | 76% |
| Lowest observation probability | 54% |
| Predicted absolute loss | 44% |
Conditional Value-at-Risk
Conditional Value-at-Risk (CVaR) measures and optimizes losses in the worst part of a distribution.
See all 32 papers