Calibrate Once, Grow the Vocabulary: Candidate-Independent Scores for Conformal Risk Control in Open-Vocabulary Segmentation
Organizations: Bangladesh University of Engineering and Technology
Abstract
Open-vocabulary segmenters accept class names at inference. Conformal risk control can bound how often their prediction sets miss the true label, but we show that this bound breaks when names are added after calibration. On ADE20K, a threshold calibrated for 10% risk at 150 names reaches up to 15.4% risk at 845 names in released CAT-Seg models. The risk exceeds the target on every calibration split, yet the prediction sets grow far less than the target requires. In SED they instead grow larger than needed. The cause is that pruning, cross-label attention and vocabulary normalisation make each label's score depend on the other requested labels. Scores can move either way, so a threshold recalibrated at a larger vocabulary need not hold at smaller ones. We remove this dependence without training by scoring each label against a fixed reference bank of names. A label's score then stays the same when names are added, so one calibration keeps its guarantee as the vocabulary grows. The guarantee holds whatever names are added, even ones chosen per image after seeing it. On three benchmarks and five segmenters, the converted models meet the target at every vocabulary size we evaluate. For CAT-Seg and SED at alpha=10%, they use at most one more label per pixel than the released models recalibrated at the grown vocabulary. Treating names instead of images as calibration samples also bounds the miss rate of added names. If the annotated names are a random sample of the names in the calibration images, an added name's expected miss rate on those images is at most the target. For harder added names, the excess is at most their gap in mean miss rate. Applied to Grounding DINO, which splits large vocabularies into chunks, the conversion also makes inference about six times faster.
Figures & tables
| anchored | native | |||||
| once | reused | recal. | random | recal. | rules | |
| calibration needs | all | all | 17k | all rules | ||
| ADE20K, | ||||||
| CAT-Seg ViT-L | 9.9/4.2 | 14.9/2.2 | 9.5/3.6 | 10.4/3.3 | 4.1/8.9 | 9.9/3.4 |
| SED | 9.9/5.9 | 12.8/3.4 | 18.8/2.0 | 18.4/2.1 | 25.4/1.4 | 9.9/5.0 |
| FC-CLIP | 9.9/9.7 | 14.1/4.8 | 10.3/7.4 | 10.8/6.9 | 5.6/17.4 | 9.9/7.7 |
| anchored | native, label-level | ||||
| image-level | label-level (ours) | recal. at grown | reused from source | ||
| Source vocabulary | added-label risk (%) / set size | ||||
| random draw | |||||
| ADE20K | .10 | 23.7 / 6.8 | 9.1 / 28.9 | 9.2 / 35.8 | 20.8 / 9.7 |
| ADE20K | .05 | 14.3 / 15.0 | 4.5 / 74.5 | 4.3 / 228.1 | 11.8 / 24.3 |
| Pascal Context | .10 | 16.7 / 3.7 | 7.6 / 11.5 | 7.6 / 14.9 | – |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | risk | set size |
|---|---|---|
| SCLIP (CLIP ViT-B) | ||
| CLIPSeg [ 49 ] | ||
| anchored CAT-Seg ViT-B |
| Deployment | Native threshold | Anchored, calibrated once |
|---|---|---|
| One vocabulary, fixed before deployment | Recalibrate at it: valid, usually smaller sets ( Tab. S4 ); cheaper above the break-even | Valid; cheaper below the break-even |
| Vocabulary fixed per project before its images are seen | Recalibrate for each project: valid, one pass over the calibration images each time | Valid, no pass |
| Vocabularies chosen per image by a rule replayable on the calibration images, e.g . a fixed image tagger | Calibrate under the rule, or the worst case over a fixed set of rules: valid, smaller sets; repeated whenever a rule changes | Valid |
| Names from a known lexicon, rule not known | Recalibrate at the lexicon: meets the target for CAT-Seg and FC-CLIP, with sets about twice the anchored size at a large lexicon, not for SED, which needs more memory than a GB card from names ( Tab. S35 ). Calibrate on random vocabularies from the lexicon: exceeds the target under the tagger rule for all three models | Valid |
| Names listed per image by a language model, not limited to a list ( Tab. S17 ) | Calibrate under the model’s names on the calibration images: valid, larger sets than anchored; repeated whenever the model or prompt changes. Recalibrating at the names exceeds the target for all three models; a -name lexicon meets it for FC-CLIP at the anchored size, not for SED | Valid |
| Names written per image or per query by a user, or added during annotation | Not replayable; guaranteed if every vocabulary the process can produce is scored on every calibration image ( Proposition 2 ) | Valid, with appended to the names |
| native | anchored | ||
|---|---|---|---|
| Detector | reused | recal. | once |
| Grounding DINO T | 9.6/94.7 | 9.9/88.3 | 9.9/34.2 |
| MM-Grounding DINO T | 9.0/23.0 | 9.9/20.2 | 9.9/3.1 |
| Grounding DINO B | 9.8/3.7 | 9.9/3.7 | 9.9/3.6 |
| OWLv2 B (control) | 9.9/8.3 | – | – |
Conformal Risk Control
Conformal risk control uses calibration data to choose prediction or action thresholds with finite-sample guarantees on expected error or other risks.
See all 72 papers