Split-conformal prediction provides marginal coverage under exchangeability and is increasingly used as an abstention layer for zero-shot vision-language models (VLMs). We audit this practice under deployment shift for CLIP, OpenCLIP, and SigLIP across ImageNet and non-ImageNet settings. Marginal coverage can remain relatively high while class-conditional tail coverage collapses: on ImageNet-Sketch, worst-class coverage falls to ≈0 and 10-12% of classes lie below a finite-sample null floor, despite marginal coverage of about 0.86. The failure is aligned with target-domain class accuracy but is not predicted by the source-domain diagnostics we test. Source-side Mondrian calibration improves the in-distribution tail but does not transfer, while clustered conformal and Conf-OT improve marginal or average metrics without recovering the worst-class tail. Target-side class calibration substantially lifts the tail, but requires labels for every class and remains set-size-intensive. We further identify a 2-3× cross-family efficiency gap and show that native SigLIP sigmoid scores remove APS's probability-mass interpretation. The findings persist across the tested model scale, pretraining corpus, prompt, miscoverage level α, and shifted non-ImageNet settings. Marginal conformal coverage should therefore be treated as an average reliability statistic, not as a safety guarantee for the class tail.
Figures & tables
Figure 1: ImageNet-Sketch, APS calibrated on ImageNet-val. Marginal coverage remains near 0.86 for all three families (A; target 0.90), while 10–12% of classes fall below the finite-sample null floor of 0.70 (B) and the worst class receives zero coverage.
Figure 2: Mechanism (CLIP, Sketch): per-class coverage tracks per-class target accuracy ( ρ=0.70 , left) but is essentially uncorrelated with per-class source accuracy ( ρ=0.14 , right), i.e. accuracy-driven yet source-invisible (Sec. 4.3).
Figure 3: Why source-side class calibration does not transfer (CLIP, ImageNet-Sketch). (a) Per-class APS thresholds learned on ImageNet-val weakly predict the oracle thresholds required on Sketch (Spearman ρ=0.20 ; 55% of classes need a larger target threshold, points above y=x ). (b) ECDF of per-class coverage: marginal coverage is near 0.86 for all three families, yet a long left tail falls below the 0.70 null floor (10–12%) and the 0.90 target. Together the panels explain why source-Mondrian improves the in-distribution tail but fails under shift (Sec. 4.2–4.3).
Figure 4: Deployable repairs on ImageNet-Sketch (CLIP) improve different average metrics, and Conf-OT restores marginal coverage toward 0.90 (left), but none restores the worst-class tail (right, all at 0); only the target-label oracle substantially lifts the tail (Sec. 4.4).
family
ID
V2
Sketch
R(200)
CLIP
11.5
15.8
26.7
5.9
OpenCLIP
5.5
7.4
14.1
2.7
SigLIP(softmax)
5.0
7.2
9.3
2.7
Table 1: APS-marginal mean set size by family across shifts ( α =0.10). CLIP is consistently least efficient; softmax-normalized SigLIP is most efficient.
score
mean coverage
mean set size
mean CCV gap
SigLIP-sigmoid set size (V2/Sketch)
LAC/THR
0.698
1.99
0.220
4 / 2
APS
0.842
9.57
0.081
721 / 940
RAPS
0.818
4.32
0.103
42 / 64
Table 2: Conformity-score comparison averaged over three families and four natural-shift sets (V2/Sketch/R/A, α =0.10): mean coverage, set size, CCV gap, and SigLIP native-sigmoid set size (V2/Sketch). RAPS is the efficiency–score-geometry compromise, not a tail-safety solution. ImageNet-R/A use a restricted-200 label space, so the mean set size is a within-protocol summary and is not directly comparable across the 1000-way and 200-way shifts.
method
target information
evaluated population
coverage
worst-class cov
mean set size
plain split-conformal
none
all inputs
0.862
0.00
26.7
source-Mondrian
none
all inputs
0.873
0.00
45.3
clustered conformal
none
all inputs
0.830
0.00
29.6
Conf-OT (transductive)
unlabeled target
all inputs
0.898
0.00
27.9
selective (k ≤ 5)
none
retained 47%
0.880
N/A
2.3
oracle (target Mondrian)
labeled target
all inputs
0.911
0.75
39.2
Table 3: Methods on ImageNet-Sketch (CLIP, APS, α =0.10; all families in Appendix J). Coverage is full-input marginal coverage for the all-input methods and empirical retained coverage for selective prediction; the latter is computed on a retained population and is not directly comparable with full-input worst-class coverage. Worst-class is the minimum class-conditional coverage. The rows target different objectives (marginal, class-conditional, or retained-subset coverage), so the table summarizes the label-access trade-off rather than ranking methods on one criterion.
Figure 5: Scale improves top-1 accuracy but leaves the class-tail failure nearly unchanged on ImageNet-Sketch: across CLIP, OpenCLIP, and SigLIP the below-null class fraction stays ≈ 10–13% from base to XL.
Table S1: Claim map: each supporting result and the section that establishes it.
Figure S1 : Target-side Mondrian worst-class recovery (left) and mean set size (right, log scale) vs. per-class target-label budget (ImageNet-Sketch, APS; 5 seeds). At ≤5 labels/class the conformal rank exceeds the per-class sample, so thresholds are infinite and sets are full (shaded region); finite-threshold recovery begins at ≥10 labels/class and stays sub-nominal (Main Sec. 4.6).
Figure S2 : APS marginal coverage and mean set size across three representative ImageNet-C corruptions (Gaussian noise, JPEG, motion blur), averaged at each severity 1–5. Coverage decreases and mean set size grows with severity for every family; the dashed line marks the 0.90 target (Main Sec. 4.1).
Figure S3 : Coverage across prediction-set-size bins on ImageNet-Sketch: the largest-set bins fall below the 0.90 target for every family, though the relationship is not strictly monotone. Error bars are 95% binomial intervals and per-bin example counts are annotated for each family (in family colours); the largest-set bins hold relatively few examples (Main Sec. 4.6).
Figure S4 : Empirical retained coverage and top-1 accuracy under size-thresholded selection on ImageNet-Sketch, as a function of retention. Points correspond to discrete set-size caps k ; the k=1 and k=5 operating points are annotated. Coverage here is empirical because selection changes the evaluated population (Main Sec. 4.6).
Figure S5 : Marginal coverage vs. the val → shift true-class APS score shift Δ , across the tested ImageNet-A (circles) and ImageNet-R (squares) checkpoints. A larger shift is associated with lower marginal coverage (Spearman ρ=−0.97 over the pooled points); the dashed line marks the 0.90 target (Main Sec. 4.7).