We present a systematic benchmark of out-of-distribution (OOD) detection confidence scoring functions (CSFs) through a representation-centric lens. Our study spans CNN and ViT backbones, multiple training paradigms, four image-classification source datasets (CIFAR-10, CIFAR-100, SuperCIFAR-100, and TinyImageNet), and OOD datasets grouped into near, mid, and far regimes based on CLIP-derived semantic distances. To compare CSFs across these settings, we employ a rank-based pipeline of Friedman and Conover tests that identifies top cliques of statistically indistinguishable winners under threshold-free ranking metrics (AURC and AUGRC). The main empirical finding is that the family of CSFs that are competitive detectors correlates with characteristics of the learned representation. For both CNNs and ViTs, simple probabilistic scores dominate misclassification detection. On CNNs, margin-based scores are strongest in near-OOD regimes, while geometry-aware scores such as NNGuide, fDBD, and CTM become more competitive as shift severity increases. On fine-tuned ViTs, the top cliques are led mainly by reconstruction- and residual-based scores. To interpret these ranking shifts, we analyze the last-layer representation using Neural Collapse (NC) metrics. The resulting picture is consistent across architectures: prototype- and boundary-aware scores become stronger when the representation is more collapsed and better aligned with classifier weights, whereas weaker-collapse regimes favor gradient- and manifold-based scores. Building on these insights, we propose two contributions: a simple PCA-based projection-filtering procedure that improves detector performance, and an approach that uses only the NC measurements of a trained classifier to predict its competitive out-of-distribution detector shortlist, without OOD data for that classifier and without knowledge of the shift.
Figures & tables
Figure 1: Top-clique maps across four training settings (AURC/AUGRC, α=0.05 , 22 CSFs). Panels, left to right: VGG-13 with vanilla cross-entropy (the ConfidNet paradigm of Section 3.1 ), VGG-13 with DeVries, VGG-13 with Deep Gamblers, and a fine-tuned ViT. Columns within each panel are the four source datasets crossed with the four evaluation regimes ( test , near , mid , far ); connected colored dots within a column show the Conover top clique. Rows are the filtered set of CSFs, grouped by where the score reads from the network: head-side scores (logit/softmax statistics, blue band) on top and feature-side scores (penultimate-feature geometry, peach band) on the bottom. Within each band, rows are ordered by regime centre-of-mass (near-dominant at top, far-dominant at bottom). Dotted rectangles mark the CSFs with the three highest clique counts per paradigm, including ties.
Figure 2: Cross-architecture transfer (VGG-13 → ResNet-18, 22 CSFs). Mean set-regret (raw AUGRC, empty-set imputed) with 95% bootstrap CIs for the regime-free NC-based predictor (NC only, the headline; NC + source, an ablation) and the strongest fixed-CSF baseline of each cell (named above its bar; candidates include Always-Maha++ and Always-NCI). Panels: joint / feature-only / head-only side; bar groups: near, mid, far. Lower is better.
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
Table 1: CNN (VGG-13) hyperparameters: dropout setting used during training (1 = trained with dropout, 0 = without). N/A indicates the (CSF, projection variant) combination is not applicable.
Table 4
Table 3: ViT hyperparameters: dropout setting used during training (1 = trained with dropout, 0 = without). N/A indicates the (CSF, projection variant) combination is not applicable.
Table 4: Deep Gamblers: reservation reward o selected on the validation set, per (CSF, projection variant) and source dataset. N/A indicates the combination is not applicable.
Table 5: Neural Collapse metrics computed on penultimate-layer activations (subtables (a) and (b)) and on classifier weights (subtable (c)). (a) Per-dataset NC metrics for VGG-13 models trained from scratch. (b) Per-dataset NC metrics for fine-tuned ViT models. (c) NC metrics computed on the classifier weights W , reported per backbone and source dataset. Lower values indicate stronger collapse on each metric. The corresponding values under PCA projection filtering (Global and Class pred) are reported in Table 20 of Appendix F .
Source
Near
Mid
Far
CIFAR-10
CIFAR-100, TinyImageNet
iSUN, LSUN(r), LSUN(c), SVHN
Places365, Textures
SuperCIFAR-100
CIFAR-10, TinyImageNet
iSUN, LSUN(r), LSUN(c), SVHN
Places365, Textures
CIFAR-100
CIFAR-10, TinyImageNet
iSUN, LSUN(r), LSUN(c), SVHN
Places365, Textures
TinyImageNet
CIFAR-10, CIFAR-100, iSUN, LSUN(r), LSUN(c)
Places365, Textures
SVHN
Appendix
Table 7: OOD dataset clustering. For each source dataset, the corresponding OOD datasets are categorized in near, mid and far datasets based on the CLIP-derived distances.
Table 8: Per-source CLIP-based proximity metrics for every candidate OOD dataset, with the resulting near/mid/far group from k -means clustering on the standardized four-metric vector. Heat-map shading is per column (blue = closer to ID, red = farther). The four sub-tables report (a) CIFAR-10, (b) SuperCIFAR-100, (c) CIFAR-100, and (d) TinyImageNet as the source.
OOD set
Group
Distances that place it
Content
Source: CIFAR-10
CIFAR-100
Near
Lowest KID ( 0.0002 ) and FD ( 0.16 )
Same image source and format, related object classes
TinyImageNet
Near
Low KID ( 0.0009 ) and FD ( 0.32 )
Downsampled object photographs
iSUN, LSUN (both)
Mid
KID 0.0015 – 0.0016 ; image-centroid distance 0.82 – 0.86 , lower than for TinyImageNet ( 0.93 )
Scene photographs
SVHN
Mid
Largest FD ( 0.70 ), but text and centroid distances ( 0.77 , 0.86 ) like those of the scene sets
House-number digits
Places365, Textures
Far
Largest class-aware distances (centroid 1.06 – 1.15 , text 0.82 – 0.83 )
Scenes; texture patches without objects
Appendix
Table 9: Walk-through of the regime assignments: the distances of Table 8 that place each OOD set in its group, and the content of the set. SuperCIFAR-100 and CIFAR-100 have the same groups as CIFAR-10, with the same pattern (for example, SVHN has the largest FD, 0.62 , but an image-centroid distance of only 0.78 – 0.80 ).
Metric
Mean ρ (12 pairs)
Min ρ
All sig. ( p<0.05 )?
FD
0.92
0.83
Yes
KID
0.95
0.86
Yes
Image centroid dist.
0.91
0.83
Yes
Text alignment
0.58
− 0.33
No
Appendix
Table 10: Spearman rank correlation ( ρ ) of OOD proximity rankings across three CLIP backbones, aggregated over all 12 source–backbone pairs.
k
Mean silhouette
Configs with min-cluster sil. >0
2
0.45
12/12
3
0.46
9/12
4
0.43
0/12
5
0.36
0/12
6
0.25
0/12
Appendix
Table 11: Silhouette analysis for different values of k across 12 source–backbone configurations.
k
k -means vs Ward (ARI)
k -means vs Average (ARI)
2
1.0 in 11/12, mean = 0.96
1.0 in 8/12, mean = 0.86
3
1.0 in 12/12, mean = 1.00
1.0 in 8/12, mean = 0.90
4
1.0 in 12/12, mean = 1.00
1.0 in 8/12, mean = 0.88
Appendix
Table 12: Agreement (ARI) between k -means and hierarchical clustering across 12 configurations.
Source dataset
CLIP backbone
τ
p -value
CIFAR-10
ViT-B/16
0.716
0.026
CIFAR-10
ViT-B/32
0.689
0.029
CIFAR-10
ViT-L/14
0.798
0.011
CIFAR-100
ViT-B/16
0.592
0.058
CIFAR-100
ViT-B/32
0.676
0.030
CIFAR-100
ViT-L/14
0.840
0.008
Appendix
Table 13: Kendall τ between average metric rank and assigned group number.
Source
Near-OOD
Far-OOD
CIFAR-10
CIFAR-100, TinyImageNet
SVHN, Textures, Places365
CIFAR-100
CIFAR-10, TinyImageNet
SVHN, Textures, Places365
SuperCIFAR-100
CIFAR-10, TinyImageNet
SVHN, Textures, Places365
Appendix
Table 14: OpenOOD-style binary grouping used for the comparison in Figure 3 . Only the intersection of OOD datasets shared between OpenOOD and our benchmark is retained.
Figure 3: Top-clique maps under OpenOOD-style binary (near/far) grouping. Same panel layout as Figure 1 : VGG-13 with vanilla cross-entropy / ConfidNet (left), VGG-13 with DeVries, VGG-13 with Deep Gamblers, and a fine-tuned ViT (right). Rows are head-side (top, blue band) and feature-side (bottom, peach band) CSFs; columns are source datasets crossed with regimes ( test , near , far ) under the OpenOOD binary partition. TinyImageNet is excluded as a source because OpenOOD does not define a near/far split for it. Compared to the three-group maps in Figure 1 : for CIFAR-10 on both backbones, the binary near and far cliques are identical, hiding the distinct mid-regime winners visible in the three-group analysis; for CIFAR-100 and SuperCIFAR-100 on ViT, the binary far clique contains eight and six tied CSFs, respectively, whereas the three-group split resolves this into a focused mid clique (KPCA RecError) and a distinct far clique (Residual, ViM).
Table 15: Worked example of the rank pipeline on six CSFs evaluated on CIFAR-10 (AUGRC, misclassification regime), with N=15 blocks formed by 3 paradigms × 5 runs. (a) Within-block ranking of the six CSFs by AUGRC for one example block (lower score ⇒ better ⇒ smaller rank). (b) Average rank Rˉj per CSF across all 15 blocks.
Figure 4: Unadjusted Conover p -values
Layer
Average Rank
Members
1
1.933
Confidence, GEN
2
3.500
CTM, MSR
3
5.533
Energy
Appendix
Table 16: Layered cliques ranked from best to worst.
Blocks
Regime
Jaccard
Winner in
Size (AUGRC)
Size (other)
Feat. share (AUGRC)
Feat. share (other)
FPR@95
near
0.45
0.75
3.5
5.1
0.49
0.52
mid
0.39
0.88
1.6
3.9
0.64
0.42
far
0.47
0.88
2.5
3.4
0.81
0.61
AUROC + FPR@95
near
0.48
0.56
3.5
3.4
0.49
0.43
mid
0.53
0.88
1.6
2.9
0.64
0.46
far
0.53
0.81
2.5
2.9
0.81
0.54
Appendix
Table 17: Top cliques under threshold-based blocks against the AUGRC top cliques, per OOD regime ( 16 cells each). Jaccard = mean Jaccard index of the two cliques; Winner in = fraction of cells in which the AUGRC winner is in the other top clique; Size and Feature share = mean clique size and mean fraction of feature-side CSFs in the clique.
Base
Variant
Δ AUGRC
Median Δ
Win
Loss
n
p
PCA RecError
class pred
+38.6
+44.9
32
0
32
< 0.001 ✓
PCA RecError
class
+37.4
+43.0
32
0
32
< 0.001 ✓
PCA RecError
class avg
−116.1
−119.6
0
40
40
< 0.001
Maha
global
+24.1
+16.6
50
14
64
< 0.001 ✓
Maha
class pred
+18.7
+10.2
40
16
56
< 0.001 ✓
Maha
class avg
+5.22
+0.24
38
18
56
0.221
Appendix
Table 18: Full paired AUGRC comparison: base CSF vs. projection variants on VGG-13 (Conv) . Δ AUGRC = mean(base − variant); positive = variant better. Significant improvements ( p<0.05 , variant better) are marked with ✓ .
Base
Variant
Δ AUGRC
Median Δ
Win
Loss
n
p
PCA RecError
class pred
+11.6
+3.07
26
6
32
< 0.001 ✓
PCA RecError
class
+10.4
+2.97
25
7
32
< 0.001 ✓
PCA RecError
class avg
−41.5
−28.6
0
24
24
< 0.001
GradNorm
class pred
+5.91
+5.68
25
7
32
< 0.001 ✓
GradNorm
global
+4.92
+3.76
22
10
32
0.001 ✓
GradNorm
class avg
−40.7
−28.6
10
22
32
< 0.001
Appendix
Table 19: Full paired AUGRC comparison: base CSF vs. projection variants on ViT . Same format as Table 18 .
Table 20: NC metrics under projection filtering (Global and Class pred). Compare against the unfiltered (None) values in Table 5 .
VGG-13
ResNet-18
CSF
ReAct
ASH-S
ReAct
ASH-S
Energy
+6.45 ( 33% )
+0.04 ( 61% )
+8.95 ( 21% )
−0.03 ( 69% )
GE
+6.29 ( 32% )
+0.02 ( 60% )
+12.50 ( 21% )
−0.02 ( 68% )
GEN
+3.73 ( 33% )
+0.02 ( 55% )
+6.05 ( 28% )
+0.13 ( 56% )
GradNorm
+43.39 ( 21% )
+0.08 ( 54% )
+84.10 ( 1% )
+1.10 ( 44% )
MLS
+4.34 ( 30% )
+0.03 ( 59% )
+13.18 ( 21% )
−0.01 ( 65% )
Appendix
Table 21: ReAct and ASH-S as wrappers of the nine head-side CSFs. A row is one model on one OOD dataset ( 2,240 rows per CSF for the VGG-13 models, 448 for the ResNet-18 models). In each cell, the first number is the mean over the rows of the paired change in AUGRC ( ×103 , wrapper − base; positive = worse). The number in parentheses is the share of rows on which the wrapped CSF has a lower AUGRC than the base CSF. The mean base AUGRC is 229 – 252 on VGG-13 and 222 – 255 on ResNet-18.
Paradigm
r (rank dist.)
r (Jaccard top-1)
r (Jaccard top-3)
p
Conv/ConfidNet
0.63
0.55
0.47
< 0.001
Conv/DeVries
0.62
0.27
0.51
< 0.001
Conv/Deep Gamblers
0.37
0.44
0.39
0.001
ViT
0.70
0.40
0.56
< 0.001
Appendix
Table 22: Global Mantel test: correlation between NC distance and CSF-ranking distance across training paradigms. All tests use 9,999 permutations.
NC metric
ConfidNet
DeVries
Deep Gamblers
ViT
Var. collapse
0.34
0.49
0.46
0.46
Equiangularity ( μ )
0.65
0.66
0.15
0.58
Equiangularity ( W )
0.74
0.49
0.88
0.56
Equinormness ( μ )
0.51
0.45
0.12 ns
0.48
Equinormness ( W )
0.33
0.21
0.07 ns
0.40
Max. equiang. ( μ )
0.71
0.51
0.13 ns
0.51
Appendix
Table 23: Per-metric Mantel r (single NC metric vs. Spearman rank distance). The strongest predictor in each paradigm is bolded. ns marks the entries that are not significant ( p>0.05 ); all other entries are significant at p<0.05 .
Stratum
No control
ID perf.
ID perf. (mean)
Source
ID perf. + source
Conv/ConfidNet
0.63 ( 10−4 )
0.56 ( 10−4 )
0.46 ( 10−4 )
0.31 ( 4×10−4 )
0.40 ( 10−4 )
Conv/DeVries
0.62 ( 10−4 )
0.56 ( 10−4 )
0.57 ( 10−4 )
0.37 ( 2×10−4 )
0.51 ( 10−4 )
Conv/Deep Gamblers
0.37 ( 0.001 )
0.32 ( 8×10−4 )
0.34 ( 0.002 )
−0.08 ( 0.80 )
0.36 ( 0.001 )
ViT
0.70 ( 10−4 )
0.59 ( 10−4 )
0.59 ( 10−4 )
0.41 ( 2×10−4 )
0.41 ( 2×10−4 )
Appendix
Table 24: Partial Mantel tests (Spearman, 9,999 permutations): Mantel r between NC distance and CSF-ranking distance after controlling for the listed covariates, with one-sided p -values. p=10−4 is the smallest value the permutation test can give. ID perf. = ID-test AUGRC of MSR; ID perf. (mean) = mean ID-test AUGRC over all CSFs.
Hypothesis
Scope
ρ
p
n
Verdict
H1 (paradigm)
no dropout, pooled
0.61
6.4×10−6
79
Supported
CIFAR-10 / CIFAR-100
0.38 / 0.60
0.055 / 0.010
15 / 21
SuperCIFAR-100 / TinyImageNet
0.88 / 0.46
<0.001 / 0.11
28 / 15
H1 (paradigm)
dropout, pooled
0.49
1.8×10−4
79
Supported
CIFAR-10 / CIFAR-100
0.16 / 0.82
0.21 / 0.003
15 / 21
SuperCIFAR-100 / TinyImageNet
0.38 / 0.58
0.072 / 0.020
28 / 15
Appendix
Table 25: Matched-pair test on the VGG-13 pool (20-CSF roster). H1 per source: Mantel p with paradigm-entry permutations; pooled H1: Stouffer combination over the sources.
Configuration
Inputs
near
mid
far
Shortlist size
22 CSFs
NC only (headline)
8 NC
0.93
0.88
0.50
8.1 – 8.4
NC + source
8 NC, source one-hot
0.78
0.82
0.63
6.4 – 7.0
Always-CTM (best fixed CSF)
none
2.08
3.56
2.57
1
20 CSFs (without Maha++ and NCI)
NC only
8 NC
1.23
1.16
0.80
7.3 – 7.5
Appendix
Table 26: Joint-side mean set-regret (raw AUGRC, empty-set imputed) under VGG-13 → ResNet-18 transfer for each input configuration and roster, and mean shortlist size per regime. No configuration produces an empty joint-side shortlist. Regime-free configurations return one shortlist per model.
Figure 5: Coefficient heatmap of the headline NC-only predictor on the 22-CSF roster (8 NC metrics, no regime or source input; PCA RecError has no head). Red = positive weight (raises predicted top-clique probability); blue = negative. Columns grouped into feature-side (left) and head-side (right) CSFs. The same fitted models are applied without re-training to the ResNet-18 pool to obtain the regret numbers in Section 4.4 .
Figure 6: Coefficient heatmap of the NC + source ablation on the 22-CSF roster (8 NC metrics and the source-dataset one-hots, CIFAR-10 as reference; no regime input). Same axes as Figure 5 .
ResNet-18
Fine-tuned ViT
Self-supervised probes
Regime
Source
NC only
Best fixed
Random
NC only
Best fixed
Random
NC only
Best fixed
Random
near
C10
0.33 †
2.58 (NCI)
0.80
0.57
0.34 (ViM)
0.57
1.12
1.41 (ViM)
0.95
C100
0.66 †
1.54 (CTM)
0.73
0.49 †
0.73 (ViM)
0.53
2.47
1.75 (ViM)
2.29
SC100
1.42 †
4.19 (CTM)
1.49
5.70
9.35 (ViM)
2.67
16.18
5.20 (ViM)
2.40
TI
1.00 †
0.55 (CTM)
2.04
0.48 †
1.51 (GradNorm)
1.35
1.59 †
4.25 (NCI)
1.79
pooled
0.93 †
2.08 (CTM)
1.46
1.44
5.85 (GE)
1.30
4.32
6.03 (ViM)
1.84
Appendix
Table 27: Per-source transfer of the NC-only predictor (22-CSF heads trained on VGG-13): joint-side mean set-regret (AUGRC ×103 , empty-set imputed), the best fixed CSF of the cell with its regret, and a random shortlist of the same size. Bold: the predictor beats the best fixed CSF; † : it beats the random set. C10, C100, SC100, TI: the four source datasets; pooled: all rows of the regime, with the best fixed CSF recomputed on them.
Side
Regime
NC only
Random set
k
Wilcoxon pHolm
Mean test pHolm
joint
near
0.93
1.46
8.1
<10−4
0.008
joint
mid
0.88
1.38
8.4
<10−4
<10−3
joint
far
0.50
1.43
8.4
<10−4
<10−3
head
near
2.41
2.43
4.0
1.00
0.96
head
mid
3.64
3.65
3.6
1.00
0.96
head
far
2.75
2.95
3.6
0.37
0.63
Appendix
Table 30: NC-only predictor (22 CSFs) against a random shortlist of the same size: mean set-regret, mean shortlist size k , and Holm-corrected p -values over the nine cells (predictor better).
Cell
NC only
Best fixed CSF
Best CNN fixed
Random set
Oracle in set
p , best fixed
p , random
pooled
3.01
7.51 (KPCA RecError)
8.92 (NCI)
2.08
33% ( 40% )
<10−4 / <10−4
0.79 / 1.00
near
1.44
5.85 (GE)
6.26 (MLS)
1.30
45% ( 38% )
<10−4 / <10−4
0.007 / 1.00
mid
4.04
3.67 (KPCA RecError)
8.87 (NCI)
3.22
28% ( 41% )
0.50 / 0.64
0.98 / 1.00
far
3.43
5.90 (ViM)
10.62 (fDBD)
1.03
26% ( 41% )
0.074 / 0.25
0.60 / 1.00
Appendix
Table 31: CNN → ViT transfer: mean set-regret (AUGRC ×103 , empty-set imputed) of the VGG-13-trained NC-only predictor over the 40 fine-tuned ViTs. The shortlists hold 8.0 – 8.5 of the 21 CSFs. “Oracle in set” is the share of rows whose shortlist contains the oracle CSF, with the expected share for a random set of the same size in parentheses. Holm-corrected one-sided p -values for the predictor against the best fixed CSF and against the random set (Wilcoxon / mean test).
Held-out source
VGG-13 heads
ViT heads
VGG-13 + ViT heads
Base-rate set
Random set
Set size
CIFAR-10
0.40
0.13
0.57
0.10
0.55
6.0
CIFAR-100
0.79
0.74
1.30
0.52
1.82
6.1
SuperCIFAR-100
9.20
44.77
19.19
38.85
40.26
0.7
TinyImageNet
1.66
2.47
1.64
3.35
2.08
4.5
pooled
3.01
12.03
5.67
10.71
11.18
4.3
Appendix
Table 32: Leave-one-ViT-source-out test: mean set-regret (AUGRC ×103 , empty-set imputed) on the held-out source, for heads trained on VGG-13 (the predictor of Table 31 ), on the ViTs of the other three sources, and on both, against a base-rate shortlist (the CSFs most often competitive on the training ViTs; post hoc) and a random shortlist, both with the size of the ViT-head shortlist. Set size: mean size of the ViT-head shortlist.
Regime
Source
Best fixed CSF
NC + source
NC only
near
C10
1.41 (ViM)
10.55
1.12
near
C100
1.75 (ViM)
4.72
2.47
near
SC100
5.20 (ViM)
7.58
16.18
near
TI
4.25 (NCI)
3.90
1.59
near
all
6.03 (ViM)
5.93
4.32
mid
C10
0.37 (CTM)
1.02
0.21
Appendix
Table 33: Self-supervised probes: mean set-regret (raw AUGRC, empty-set imputed, joint side) of the VGG-13-trained 22-CSF heads against the best fixed CSF of each cell (name in parentheses). C10/C100/SC100/TI are the four sources; all pools every (probe, OOD dataset) row of the regime. Bold: NC only beats the best fixed CSF.
Cell
NC only
Best fixed CSF
Always-CTM
Random set
p , best fixed
p , random
pooled (8 sets)
1.05
4.64 (NNGuide)
7.82
1.25
<10−4 / <10−4
10−4 / 0.58
near (2 sets)
1.00
2.80 (PE)
14.93
1.26
<10−4 / <10−4
0.002 / 0.58
CLIP far (6 sets)
1.07
3.34 (fDBD)
5.45
1.25
<10−4 / <10−4
0.002 / 0.58
OpenOOD far (3 sets)
0.92
2.99 (fDBD)
5.18
1.31
<10−4 / <10−4
0.002 / 0.21
Appendix
Table 34: ImageNet transfer: mean set-regret (AUGRC ×103 , empty-set imputed) of the VGG-13-trained NC-only predictor over 47 models. The shortlists hold 7.8 of the 21 CSFs on average. Holm-corrected p -values for the predictor against the best fixed CSF and against the random set of the same size (Wilcoxon / mean test). The CLIP near cell equals OpenOOD’s near cell.
Family
Cell
NC only
Best fixed CSF
Random set
p vs best fixed
p vs random
ResNet-50 (16)
pooled
0.83
4.93 (fDBD)
1.42
<10−4
0.054
near
0.30
1.07 (REN)
1.03
0.006
0.004
far
1.01
2.31 (NCI)
1.55
0.019
0.10
ViT-B/16 (8)
pooled
0.60
1.12 (Maha++)
1.03
0.086
0.074
near
1.00
1.99 (Maha++)
1.43
0.093
0.17
far
0.47
0.83 (Maha++)
0.90
0.15
0.066
Appendix
Table 35: ImageNet transfer per family (exploratory): mean set-regret of the NC-only predictor, the best fixed CSF of the cell (chosen with hindsight), and the random set of the same size, with raw one-sided mean-test p -values (predictor better).