When Does an AUROC Comparison Survive a Change in Population Mix?
Organizations: University of Southern California · Independent Researcher · Islamic Azad University, Amol Branch
Abstract
Does an AUROC comparison survive a change in the proportions of tasks, populations, or image corruptions? A pooled-AUROC ranking of failure detectors or uncertainty scores describes one particular mix of groups. We study two fixed scores for a common binary target under mixtures of fixed group-conditional laws. Using the cross-group AUC decomposition, we prove that every rational symmetric quadratic comparison geometry is realizable by finite scalar-score distributions. Universal dominance is thus coNP-hard, and even exhaustive checks of every mixture on fewer than groups can miss a -group reversal. A grid bound gives a dimension-uniform approximation certificate with sharp order. Common score marginals across groups make the comparison numerator linear, with an error envelope for imperfect alignment. These results yield exact finite-bank audits, explicit reversal mixtures and distances, and simultaneous population statements. For failure detection with a frozen ResNet-50 on ImageNet-C, 15 of 18 confidence-score comparisons win in every corruption family on a discovery split, and all 15 are finite-bank certified over all family mixtures on a held-out split; over all 15 corruption types, of 14 discovery winners the corrected grid certifies 10, leaves 3 unresolved, and finds one confirmation reversal of at least 0.05 AUROC. An audit of all 11,019 released Stable LM 2 12B outputs and 35 uncertainty methods finds 66 confirmation-set comparisons that win within all four tasks yet reverse under mixing. Discovery-fitted marginal rank alignment removes all 66; an alignment-error envelope certifies 83 comparisons. Label checks also change several selected reversals. The findings distinguish robustness of frozen scores from robustness after calibration; they do not establish human-error detection, robustness to unseen corruptions, or generalization beyond the evaluated banks.
Figures & tables
| Pair | Reference | Minimum within | Stress: released | Stress: task check | Stress: AlignScore |
|---|---|---|---|---|---|
| Token entropy–Mahalanobis | |||||
| Sequence probability–NLI degree | |||||
| ROUGE-L–normalized entropy | |||||
| CCP–NLI eccentricity | |||||
| NLI eccentricity–Jaccard eigenvalue |
| Groups | Panel | Wins | Frozen rev. (material) | Cert. | Rev. witness | Unres. |
|---|---|---|---|---|---|---|
| Raw | 15/18 | 0 (0) | 15 | 3 | 0 | |
| Marginal rank | 15/18 | 0 (0) | 15 | 3 | 0 | |
| Raw | 14/18 | – (0) | 10 | 4 | 4 | |
| Marginal rank | 14/18 | – (0) | 10 | 4 | 4 |
| Scores | Mixed pairs | Within-win | Envelope |
|---|---|---|---|
| reversals | certificates | ||
| Raw | 422 | 66 | 0 |
| Marginal rank | 347 | 0 | 83 |
| Negative-class rank | 348 | 1 | 0 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Rows | Discovery | Confirmation | Disc. clusters | Conf. clusters | Disc. errors | Conf. errors |
|---|---|---|---|---|---|---|---|
| MMLU | 5700 | 2840 | 2860 | 2830 | 2856 | 1287 | 1305 |
| CoQA | 2000 | 1059 | 941 | 260 | 226 | 651 | 608 |
| GSM8K | 1319 | 671 | 648 | 671 | 648 | 300 | 290 |
| TriviaQA | 2000 | 1035 | 965 | 978 | 912 | 448 | 415 |
| Total | 11019 | 5605 | 5414 | 4739 | 4642 | 2686 | 2618 |
| ID | Estimator | ID | Estimator |
|---|---|---|---|
| 1 | MaximumSequenceProbability | 19 | DegMat_NLI_score_entail |
| 2 | Perplexity | 20 | DegMat_NLI_score_contra |
| 3 | MeanTokenEntropy | 21 | DegMat_Jaccard_score |
| 4 | MeanPointwiseMutualInformation | 22 | Eccentricity_NLI_score_entail |
| 5 | MeanConditionalPointwiseMutualInformation | 23 | Eccentricity_NLI_score_contra |
| 6 | CCP | 24 | Eccentricity_Jaccard_score |
| Task | Reproducing rule for released Accuracy | Parser changes |
|---|---|---|
| MMLU | Case-sensitive equality after stripping output and target letter. | 7/5700 |
| CoQA | Case-sensitive equality after stripping output and the single stored answer; no punctuation, article, or case normalization. | 676/2000 |
| GSM8K | Greedily remove target prefix through #### and output prefix through The answer is ; strip, lowercase, remove ASCII punctuation, and compare. | 7/1319 |
| TriviaQA | Strip, lowercase, remove ASCII punctuation, and compare with any provided target alias. | 28/2000 |
| Full simplex | Interior | |||||
|---|---|---|---|---|---|---|
| Endpoint | Discovery | Confirmation | Full bank | Discovery | Confirmation | Full bank |
| Released Accuracy | 419 | 422 | 419 | 350 | 358 | 355 |
| AlignScore threshold | 415 | 406 | 404 | 294 | 293 | 288 |
| Task-parser sensitivity | 388 | 396 | 387 | 287 | 297 | 292 |
| versus | Accuracy gap [interval] | AlignScore gap | Parser gap | |
|---|---|---|---|---|
| 3 vs 31 | ||||
| 1 vs 20 | ||||
| 13 vs 10 | ||||
| 6 vs 23 | ||||
| 22 vs 18 |
| Scores | Mixture-dependent | Interior | All-task winners | Winners that reverse |
|---|---|---|---|---|
| Raw | 422/595 | 358/595 | 239 | 66 |
| Marginal mid-CDF | 347/595 | 268/595 | 239 | 0 |
| Marginal strict rank | 347/595 | 268/595 | 239 | 0 |
| Correct-example mid-CDF | 348/595 | 264/595 | 239 | 1 |
| Correct-example strict rank | 348/595 | 264/595 | 239 | 1 |
| Pair | Discovery TV bracket | Confirmation TV bracket | Confirmation gap at discovery witness |
|---|---|---|---|
| 3 vs 31 | |||
| 1 vs 20 | |||
| 13 vs 10 | |||
| 6 vs 23 | |||
| 22 vs 18 |
| per group | Method | Joint coverage | Reversal certificates | 95% interval for power | Half-width |
|---|---|---|---|---|---|
| 500 | Hoeffding | 500 | 0 | .104696 | |
| 500 | Intersection | 500 | 0 | .106432 | |
| 2000 | Hoeffding | 500 | 0 | .052348 | |
| 2000 | Intersection | 500 | 0 | .037200 | |
| 8000 | Hoeffding | 500 | 0 | .026174 | |
| 8000 | Intersection | 500 | 32 | .014477 |
| Bank | Grid dominance | Grid reversal | Unresolved | Exact dominance | Median width | Maximum width |
|---|---|---|---|---|---|---|
| Discovery | 168 | 419 | 8 | 176 | .006837 | .018203 |
| Confirmation | 156 | 422 | 17 | 173 | .006948 | .017595 |
| Pair | Discovery witness domain | Confirmation gap | Hellinger interval | TV interval |
|---|---|---|---|---|
| 1 | Full simplex | |||
| 2 | Full simplex | |||
| 3 | Full simplex | |||
| 4 | Full simplex | |||
| 5 | Full simplex | |||
| 1 |
Benchmark Auditing
Benchmark auditing checks whether evaluation datasets, metrics, and infrastructure measure the capabilities and risks they claim to assess.
See all 109 papers