Coverage Hides Churn: Auditing and Preserving Conformal Decisions as VLM Label Registries Grow
Organizations: Virginia Tech · Accenture
Abstract
Vision-language classifiers can learn new labels from text alone, so the label lists they serve keep growing. We show that this growth silently changes conformal decisions about old labels. Common softmax-based conformal scores compare each label with an average over all labels; adding labels changes that average, and with it the score of every old label, even when the model, the image, and the old calibration data stay the same. Recalibrating after the update keeps the coverage guarantee, yet across eleven settings it changes at least one old decision on 53-90% of new images. Coverage cannot reveal this, because it checks only the true label: in five controlled settings that keep the old calibration, so that only the normalizer changes, old-class coverage moves by at most 0.01 while 60-67% of images change. We propose a lifecycle audit that measures these changes alongside coverage and set size, and a simple fix, PrefixCP: each class keeps the label registry that existed when it was added. Its old decisions then never change, each class keeps exact class-conditional coverage under exchangeability, and one prefix scan computes all historical normalizers at once. PrefixCP recovers 40.9-58.5% of the set-size gap between never updating the reference and recalibrating, which gives smaller sets but changes old decisions.
Figures & tables
| Dataset | Encoders | Calibration / evaluation pools | Processing on both pools | |
| Food-101 | SigLIP | Provided train / test | Blur severity 5 (radius 6) | |
| Stanford Cars | CLIP, SigLIP | Provided train / test | Blur severity 5 (radius 6) | |
| CUB | CLIP, SigLIP | Within-class identity-group halves | Blur severity 5 (radius 6) | |
| ImageNet-R | CLIP, SigLIP | Within-class identity-group halves | Natural renditions | |
| ImageNet-Sketch | CLIP, SigLIP | Within-class identity-group halves | Sketches |
| revocation rate (%) | addition rate (%) | affected-image rate (%) | worst coverage | valid coverage? | old classes refit? | |
| HPS-Frozen ‡ | 8.4–51.8 | 0.1–0.9 | 48–78 | 0.153–0.789 | no | none |
| HPS-Refresh | 11.6–52.7 | 0.2–2.5 | 53–90 | 0.504–0.833 | by refit | all |
| raw cosine | 0 | 0 | 0 | 0.503–0.823 | yes | none |
| PRE | 0 | 0 | 0 | 0.500–0.833 | yes | none |
| PrefixCP | 0 | 0 | 0 | 0.483–0.843 | yes | none |
| average number of labels returned | paired set-size difference | ||||||||
| stable | mutable | ||||||||
| Setting ( ) | raw | PRE | PrefixCP | HPS-Refresh | HPS-Frozen ‡ | gap recovered (%) | PrefixCP PRE | PrefixCP HPS-Refresh ( CI) | |
| Food-101 / SigLIP | 101 | 17.74 | 13.73 | 12.15 | 11.03 | 10.33 | 58.5 | ||
| Stanford Cars / CLIP | 196 | 44.85 | 30.09 | 29.55 | 28.78 | 29.02 | 40.9 | ||
| Stanford Cars / SigLIP | 196 | 30.13 | 17.60 | 16.51 | 15.24 | 15.08 | 46.4 | ||
| CUB / CLIP | 200 | 60.29 | 30.25 | 29.72 | 29.23 | 28.85 | 52.1 | ||
| PRE | PrefixCP | HPS- Refresh † | PrefixCP coverage | PrefixCP churn rate | |
| 0.80 | 17.55 | 16.54 | 15.36 | 0.898 | 0 |
| 1.00 | 17.60 | 16.51 | 15.24 | 0.898 | 0 |
| 1.20 | 17.85 | 16.69 | 15.40 | 0.899 | 0 |
| PrefixCP | |||||
| Setting | PRE | all at once | one class per release | advantage over PRE used up (%) | worst tested order |
| Food-101 / SigLIP | 13.73 | 12.15 | 13.00 | 53.9 | high-confusion-first |
| Stanford Cars / CLIP | 30.09 | 29.55 | 29.85 | 54.9 | near-base-first |
| Stanford Cars / SigLIP | 17.60 | 16.51 | 17.20 | 64.0 | near-base-first |
Conformal Prediction
Conformal prediction wraps predictive models with calibrated sets or intervals that provide statistical coverage guarantees.
See all 395 papers