cs.LGAug 24, 2026

A Comparative Study of Label-free Representation Quality Metrics in Deep Learning

Authors: Daniel Richards Arputharaj, Daniel Jönsson, Gabriel Eilertsen

Organizations: Department of Science and Technology Linköping University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We present a comparative study of label-free metrics for assessing the quality of representations in deep neural networks to understand their reliability under a wide variety of configurations. We group existing label-free metrics into three families based on their construction and analytically establish connections between metrics within the same family. We then characterise the sensitivity of spectral metrics through controlled synthetic experiments. Finally, all label-free metrics are evaluated against downstream task accuracy across a diverse set of 260 vision models on six datasets spanning generic object classification, fine-grained object classification, scene recognition and geospatial task, stratifying results by architecture class and training objective. We find that intrinsic dimensionality (ID) is the most reliable predictor among the metrics considered. However, the reliability of all metrics, including ID, is moderated by architecture class and training objective. Our results provide a clearer understanding of what label-free representation quality metrics measure, when they are reliable, and how to interpret them in practice.

Figures & tables

Figure 1: Illustration of the representation extraction pipeline for self-supervised (SSL) and supervised models. In SSL models, the projector MLP is discarded after training, and the backbone output X∈RN×dX\in\mathbb{R}^{N\times d} is used as the representation. For supervised models, the penultimate layer activations are used as the representation, also denoted as XX . In both cases, representation quality metrics are computed directly on X, without access to the projector embeddings Z or classification heads.
MetricCategoryComplexityDescription
α\alpha -ReQ ( 1 )SpectralO(Nd2)\mathcal{O}(Nd^{2})Fits power law λi∝i−α\lambda_{i}\propto i^{-\alpha} to the eigenspectrum. Sensitive to fitting range in log-log space ( 13 ) .
RankMe ( 20 )SpectralO(Nd2)\mathcal{O}(Nd^{2})Entropy of normalised singular values. Originally computed on projector output; we compute on backbone output.
NE Sum ( 21 )SpectralO(Nd2)\mathcal{O}(Nd^{2})Measures degree of feature whitening. Equivalent to Stable Rank ( 39 ) for mean-centred representations (Section 3.1 ).
κ\kappa ( 39 )SpectralO(Nd2)\mathcal{O}(Nd^{2})Ratio of largest to smallest non-zero singular value. Measures sensitivity of linear system to perturbations.
Self-Cluster ( 39 )RelationalO(Nd2)†\mathcal{O}(Nd^{2})^{\dagger}Measures deviation of ℓ2\ell_{2} -normalised representations from a uniform spherical distribution. Reduced from O(N2d)\mathcal{O}(N^{2}d) via batched reformulation (Section 3.2 ).
DSE ( 30 )RelationalO(N′2d)\mathcal{O}(N^{\prime 2}d)Spectral entropy of anisotropic diffusion kernel matrix. Subsampled to N′ ⁣= ⁣10,000N^{\prime}\!=\!10{,}000 for scalability. Analytically related to Self-Cluster (Section 3.2 ).
Table 1: Summary of representation quality metrics considered in this work, grouped by construction. Metrics take the representation matrix, X∈RN×dX\in\mathbb{R}^{N\times d} , as input, where NN is the number of samples and dd is the representation dimension (we assume N≫dN\gg d ).
Figure 2: Comparison of real and synthetic eigenspectra in log-log scale. Real backbone representations with dimension d∈[1000,1024]d\in[1000,1024] (Fig. 2(b) ) exhibit spectral shapes consistent with the synthetically generated power-law family λi∝i−α\lambda_{i}\propto i^{-\alpha} , ∑i=1dλi=1\sum_{i=1}^{d}\lambda_{i}=1 , under negative values of α\alpha (Fig. 2(a) ). This similarity justifies including α<0\alpha<0 in the synthetic sensitivity analysis
Figure 3: Sensitivity of spectral metrics RankMe, NE Sum, and κ\kappa to spectral shape ( α\alpha ) and representation dimension ( dd ), where eigenspectra are synthetically generated as λi∝i−α\lambda_{i}\propto i^{-\alpha} with ∑i=1dλi=1\sum_{i=1}^{d}\lambda_{i}=1 . Each row shows the un-normalised metric (col. 1), the dimension-normalised surface (col. 2), and the normalised metric as a function of dd for fixed values of α\alpha (col. 3). Normalisation is performed by dividing by dd , following the normalisation proposed in 39 for RankMe, which we extend to all three metrics. In col. 3, dashed lines correspond to α<0\alpha<0 and solid lines to α≥0\alpha\geq 0 . Since κ\kappa is symmetric about α=0\alpha=0 (Fig. 3(h) ), only α≥0\alpha\geq 0 is shown in Fig. 3(i) .
Table 2: Spearman correlation ( ρ\rho ) between representation quality metrics and downstream accuracy across six evaluation datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out. – are cells with either no models or one model
Figure 4: Comparison of ID against OLS probe test accuracy across six datasets. The ρ\rho denotes the Spearman correlation and the corresponding pp -value is annotated in each subplot. Marker size is proportional to each model’s representation dimensionality, scaled linearly relative to the highest-dimensional model.
Figure 6: Comparison of DSE and Self-Cluster against OLS linear-probe test accuracy across six datasets. The ρ\rho denotes the Spearman correlation and the corresponding pp -value is annotated in each subplot. Marker size is proportional to each model’s representation dimensionality, scaled linearly relative to the highest-dimensional model.
  ConvNeXTRegNetResNetViT
CIFAR-100RankMe0.81890.8189 **0.89520.8952 **0.84250.8425 **0.59850.5985 **
RankMe ⋆0.2661−0.5361-0.5361 **-0.42390.1086
NE-Sum-0.20150.47310.4731 **0.52760.5276 **0.48090.4809 **
Self-Cluster0.1316−0.3331-0.3331 *−0.7623-0.7623 **-0.143
ID−0.8087-0.8087 **−0.5637-0.5637 **-0.06056−0.5633-0.5633 **
ImageNetRankMe0.38970.3897 *0.75380.7538 **0.60470.6047 **0.1835
Table 3: Spearman correlation ( ρ\rho ) between representation metrics and OLS test accuracy, grouped by architecture class, across the six main datasets. Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
Figure 7: Self-Cluster metric values plotted against test accuracy across datasets. The top row shows all models coloured by architecture class, revealing that the models with extreme values predominantly belong to the ViT architecture class. The bottom row shows only ViT models coloured by training objective, illustrating that the near unit Self-Cluster values originate predominantly from Supervised and SSL+Sup models rather than SSL models. This suggests that the Self-Cluster metric may be a reliable predictor for SSL models (though the sample size is small), but is unreliable for Supervised and SSL+Sup models.
  SSLSupervised
CIFAR-100RankMe0.22580.30440.3044 *
RankMe ⋆0.61030.6103 **−0.3593-0.3593 **
NE-Sum0.66170.6617 **0.1204
Self-Cluster−0.8134-0.8134 **-0.02742
ID−0.6855-0.6855 **−0.5586-0.5586 **
ImageNetRankMe0.3310.24110.2411 *
Table 4: Spearman correlation ( ρ\rho ) between representation metrics and OLS test accuracy, grouped by training objective, across the six main datasets. Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
Figure 9: Comparison of RankMe and RankMe ⋆ against OLS linear-probe test accuracy across six datasets. The ρ\rho denotes the Spearman correlation and the corresponding pp -value is annotated in each subplot. Marker size is proportional to each model’s representation dimensionality, scaled linearly relative to the highest-dimensional model.
Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: α\alpha -ReQ fitting-range sensitivity. Spearman correlation between α\alpha ( top ) and ∣α−1∣|\alpha-1| ( bottom ) with OLS test accuracy, across five fitting ranges and ten datasets. Green: significant positive; red: significant negative; grey: p≥0.05p\geq 0.05 .
Figure 11: TwoNN intrinsic dimensionality (ID) estimation noise across 260 models and 10 datasets (2,600 model–dataset pairs). Left : Intra-batch subsample noise which is the relative standard deviation of ID across 20 within-batch subsamples (Number of repetition = 20, 90% subsampling rate within each batch). Right : Across-batch noise which is the relative standard deviation of the per-batch mean ID across batches. Median intra-batch and across-batch noise are 4.5%4.5\% and 7.6%7.6\% respectively, with 95th percentiles at 8.2%8.2\% and 15.2%15.2\% .
Figure 12: DSE estimation noise across 260 models and 10 datasets. Relative standard deviation of DSE across batches.
Figure 13: ID vs OLS test accuracy, coloured by log⁡10\log_{10} parameter count, across six main datasets. Partial Spearman correlations controlling for log parameter count are annotated in each panel ( n=255n=255 ; 5 models excluded due to unavailable parameter counts). Asterisk (*) denotes p<0.05p<0.05 .
        
GitHub RepoModel NameddArch.ObjectiveParams (M)GMACsTraining data
facebookresearch/barlowtwins:mainbarlowtwins1000ResNetSSL25.64.11ImageNet-1k
facebookresearch/vicregl:mainresnet50_alpha0p92048ResNetSSL23.54.11ImageNet-1k
facebookresearch/vicregl:mainresnet50_alpha0p752048ResNetSSL23.54.11ImageNet-1k
facebookresearch/vicregl:mainconvnext_small_alpha0p9768ConvNeXTSSL49.48.70ImageNet-1k
facebookresearch/vicregl:mainconvnext_small_alpha0p75768ConvNeXTSSL49.48.70ImageNet-1k
Appendix
Table 5: Summary of torch hub models and their characteristics. Params/GMACs are for the frozen feature backbone (classifier head removed), computed with timm+fvcore; GMACs are at the listed input resolution. Arrows denote a pretrain →\rightarrow fine-tune chain. LVD-142M/1689M: DINOv2/v3 curated web data; WIT-400M: OpenAI CLIP data; LAION: web image--text; IG: Instagram; YFCC-100M: Flickr; MetaCLIP-2B: Web-SSL web pool.
       
Model NameddArch.ObjectiveParams (M)GMACsTraining data
facebook/dinov2-small384ViTSSL22.129.46LVD-142M
timm/vit_small_patch16_224.dino384ViTSSL21.74.25ImageNet-1k
timm/vit_small_patch16_dinov3_qkvb.lvd1689m384ViTSSL21.65.63LVD-1689M (web)
timm/vit_small_plus_patch16_dinov3_qkvb.lvd1689m384ViTSSL28.77.48LVD-1689M (web)
facebook/dinov2-base768ViTSSL86.6117.11LVD-142M
Appendix
Table 6: Summary of Hugging Face ( 41 ) and timm models and their characteristics. Params/GMACs are for the frozen feature backbone (classifier head removed), computed with timm+fvcore; GMACs are at the listed input resolution. Arrows denote a pretrain →\rightarrow fine-tune chain. LVD-142M/1689M: DINOv2/v3 curated web data; WIT-400M: OpenAI CLIP data; LAION: web image--text; IG: Instagram; YFCC-100M: Flickr; MetaCLIP-2B: Web-SSL web pool.
Table 7: Spearman correlation ( ρ\rho ) between representation quality metrics and OLS test accuracy across 10 evaluation datasets, grouped by task type. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  ConvNeXTRegNetResNetViT
CIFAR-10α\bm{\alpha} -ReQ−0.469-0.469 **-0.08933-0.1038-0.2193
RankMe0.79830.7983 **0.7790.779 **0.76680.7668 **0.56490.5649 **
RankMe ⋆0.1358−0.3692-0.3692 *-0.36690.1175
NE-Sum0.27070.48840.4884 **0.31440.36970.3697 *
κ\bm{\kappa}0.32920.64380.6438 **0.62040.6204 **0.4090.409 **
Self-Cluster0.181−0.5331-0.5331 **−0.7515-0.7515 **-0.1731
Appendix
Table 8: Spearman correlation ( ρ\rho ) between representation quality metrics and OLS test accuracy, stratified by architecture class, across natural image datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction: ↑\uparrow (higher is better), ↓\downarrow (lower is better), →1\rightarrow 1 (closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  ConvNeXTRegNetResNetViT
Food-101α\bm{\alpha} -ReQ−0.5099-0.5099 **-0.1406-0.1182-0.1822
RankMe0.81680.8168 **0.8740.874 **0.72190.7219 **0.62750.6275 **
RankMe ⋆0.2391−0.4341-0.4341 **−0.4802-0.4802 *0.08788
NE-Sum0.67550.6755 **0.55040.5504 **0.61680.6168 **0.60990.6099 **
κ\bm{\kappa}0.16340.64210.6421 **0.5850.585 *0.1812
Self-Cluster−0.3117-0.3117 *-0.272−0.6201-0.6201 **-0.2651
Appendix
Table 9: Spearman correlation ( ρ\rho ) between representation quality metrics and OLS test accuracy, stratified by architecture class, across fine-grained datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction: ↑\uparrow (higher is better), ↓\downarrow (lower is better), →1\rightarrow 1 (closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  ConvNeXTRegNetResNetViT
DTDα\bm{\alpha} -ReQ−0.4812-0.4812 **-0.181-0.1530.03269
RankMe0.60540.6054 **0.02952−0.6883-0.6883 **0.44780.4478 **
RankMe ⋆0.1516-0.12940.4217-0.1118
NE-Sum-0.1059-0.1169−0.4944-0.4944 *0.49430.4943 **
κ\bm{\kappa}0.33550.3355 *0.04234−0.5242-0.5242 *0.05048
Self-Cluster0.037950.25640.62520.6252 **-0.225
Appendix
Table 10: Spearman correlation ( ρ\rho ) between representation quality metrics and OLS test accuracy, stratified by architecture class, across specialised datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction: ↑\uparrow (higher is better), ↓\downarrow (lower is better), →1\rightarrow 1 (closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  SSLSupervised
CIFAR-10α\bm{\alpha} -ReQ ( →1\bm{\rightarrow 1} )−0.4703-0.4703 **0.18770.1877 *
RankMe ( ↑\bm{\uparrow} )0.090540.1503
RankMe ⋆ ( ↑\bm{\uparrow} )0.49850.4985 **−0.3683-0.3683 **
NE-Sum ( ↑\bm{\uparrow} )0.59420.5942 **0.0295
κ\bm{\kappa} ( ↑\bm{\uparrow} )-0.064520.294
Self-Cluster ( ↓\bm{\downarrow} )−0.7885-0.7885 **0.003576
Appendix
Table 11: Spearman correlation ( ρ\rho ) between representation quality metrics and OLS test accuracy, stratified by training objective, across natural image datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction: ↑\uparrow (higher is better), ↓\downarrow (lower is better), →1\rightarrow 1 (closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  SSLSupervised
Food-101α\bm{\alpha} -ReQ ( →1\bm{\rightarrow 1} )-0.07001-0.01501
RankMe ( ↑\bm{\uparrow} )0.14880.45720.4572 **
RankMe ⋆ ( ↑\bm{\uparrow} )0.2804-0.1865
NE-Sum ( ↑\bm{\uparrow} )0.74740.7474 **0.36050.3605 **
κ\bm{\kappa} ( ↑\bm{\uparrow} )-0.29290.33140.3314 *
Self-Cluster ( ↓\bm{\downarrow} )−0.9062-0.9062 **-0.1227
Appendix
Table 12: Spearman correlation ( ρ\rho ) between representation quality metrics and OLS test accuracy, stratified by training objective, across fine-grained datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction: ↑\uparrow (higher is better), ↓\downarrow (lower is better), →1\rightarrow 1 (closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  SSLSupervised
DTDα\bm{\alpha} -ReQ ( →1\bm{\rightarrow 1} )−0.5796-0.5796 **0.07191
RankMe ( ↑\bm{\uparrow} )-0.1747−0.4853-0.4853 **
RankMe ⋆ ( ↑\bm{\uparrow} )0.65990.6599 **0.3193
NE-Sum ( ↑\bm{\uparrow} )0.48830.4883 **−0.4066-0.4066 **
κ\bm{\kappa} ( ↑\bm{\uparrow} )−0.448-0.448 **-0.3383
Self-Cluster ( ↓\bm{\downarrow} )−0.6756-0.6756 **0.37180.3718 **
Appendix
Table 13: Spearman correlation ( ρ\rho ) between representation quality metrics and OLS test accuracy, stratified by training objective, across specialised datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction: ↑\uparrow (higher is better), ↓\downarrow (lower is better), →1\rightarrow 1 (closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  d≤520d\leq 520520<d≤960520<d\leq 960960<d≤1792960<d\leq 1792d>1792d>1792
CIFAR-10α\bm{\alpha} -ReQ ( →1\bm{\rightarrow 1} )0.1734-0.15710.047380.1315
RankMe ( ↑\bm{\uparrow} )-0.3555-0.02149-0.13270.4376
RankMe ⋆ ( ↑\bm{\uparrow} )-0.32260.08077-0.03157-0.2487
NE-Sum ( ↑\bm{\uparrow} )-0.27650.10830.41890.4189 *0.1896
κ\bm{\kappa} ( ↑\bm{\uparrow} )0.48290.4829 *0.13350.044470.04695
Self-Cluster ( ↓\bm{\downarrow} )0.34050.03767−0.583-0.583 *-0.1373
Appendix
Table 14: Spearman correlation ( ρ\rho ) between representation quality metrics and OLS test accuracy, stratified by representation dimension dd , across natural image datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction: ↑\uparrow (higher is better), ↓\downarrow (lower is better), →1\rightarrow 1 (closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  d≤520d\leq 520520<d≤960520<d\leq 960960<d≤1792960<d\leq 1792d>1792d>1792
Food-101α\bm{\alpha} -ReQ ( →1\bm{\rightarrow 1} )0.2028−0.446-0.446 *0.12160.2136
RankMe ( ↑\bm{\uparrow} )-0.053240.3067-0.11930.4383
RankMe ⋆ ( ↑\bm{\uparrow} )-0.18710.44530.4453 *-0.08247-0.2668
NE-Sum ( ↑\bm{\uparrow} )0.083910.48440.4844 *0.18240.3293
κ\bm{\kappa} ( ↑\bm{\uparrow} )0.45790.4579 *-0.0023270.1213-0.1018
Self-Cluster ( ↓\bm{\downarrow} )0.2196-0.1607−0.5652-0.5652 *-0.1326
Appendix
Table 15: Spearman correlation ( ρ\rho ) between representation quality metrics and OLS test accuracy, stratified by representation dimension dd , across fine-grained datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction: ↑\uparrow (higher is better), ↓\downarrow (lower is better), →1\rightarrow 1 (closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  d≤520d\leq 520520<d≤960520<d\leq 960960<d≤1792960<d\leq 1792d>1792d>1792
DTDα\bm{\alpha} -ReQ ( →1\bm{\rightarrow 1} )0.3104-0.2760.086730.2104
RankMe ( ↑\bm{\uparrow} )-0.1474-0.144−0.7875-0.7875 *0.06973
RankMe ⋆ ( ↑\bm{\uparrow} )-0.31890.33090.1021−0.8435-0.8435 *
NE-Sum ( ↑\bm{\uparrow} )-0.20040.01337−0.3821-0.3821 *-0.2098
κ\bm{\kappa} ( ↑\bm{\uparrow} )0.40920.4092 *-0.0831-0.058620.4448
Self-Cluster ( ↓\bm{\downarrow} )0.21620.09457-0.072270.2249
Appendix
Table 16: Spearman correlation ( ρ\rho ) between representation quality metrics and OLS test accuracy, stratified by representation dimension dd , across specialised datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction: ↑\uparrow (higher is better), ↓\downarrow (lower is better), →1\rightarrow 1 (closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
Table 17: Spearman correlation ( ρ\rho ) between representation quality metrics and KNN test accuracy ( k=1k=1 ) across 10 evaluation datasets, grouped by task type. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  ConvNeXTRegNetResNetViT
CIFAR-10α\bm{\alpha} -ReQ−0.5643-0.5643 **-0.227-0.2629-0.2075
RankMe0.77380.7738 **0.61020.6102 **0.53090.5309 *0.53690.5369 **
RankMe ⋆0.2589-0.1931-0.055980.2075
NE-Sum0.31570.3157 *0.62410.6241 **0.31010.67910.6791 **
κ\bm{\kappa}0.24080.46660.4666 **0.36150.2019
Self-Cluster0.1193−0.6115-0.6115 **−0.5767-0.5767 **−0.8651-0.8651 **
Appendix
Table 18: Spearman correlation ( ρ\rho ) between representation quality metrics and KNN test accuracy ( k=1k=1 ), stratified by architecture class, across natural image datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  ConvNeXTRegNetResNetViT
Food-101α\bm{\alpha} -ReQ−0.5981-0.5981 **−0.2761-0.2761 *-0.13460.1837
RankMe0.76320.7632 **0.84860.8486 **0.66840.6684 **0.52750.5275 **
RankMe ⋆0.37450.3745 **−0.3125-0.3125 *−0.4269-0.4269 *-0.1435
NE-Sum0.80390.8039 **0.64420.6442 **0.64260.6426 **0.57050.5705 **
κ\bm{\kappa}0.05780.63230.6323 **0.51630.5163 *-0.1739
Self-Cluster−0.3704-0.3704 **−0.3716-0.3716 *−0.6934-0.6934 **−0.914-0.914 **
Appendix
Table 19: Spearman correlation ( ρ\rho ) between representation quality metrics and KNN test accuracy ( k=1k=1 ), stratified by architecture class, across fine-grained datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  ConvNeXTRegNetResNetViT
DTDα\bm{\alpha} -ReQ−0.5196-0.5196 **0.039560.2612-0.2834
RankMe0.76140.7614 **0.82540.8254 **0.68130.6813 **0.46360.4636 **
RankMe ⋆0.1018−0.7366-0.7366 **−0.6649-0.6649 **0.1615
NE-Sum-0.25640.56820.5682 **0.53340.5334 **0.63930.6393 **
κ\bm{\kappa}0.45260.4526 **0.76140.7614 **0.62880.6288 *-0.09476
Self-Cluster0.01003-0.2031−0.6567-0.6567 **−0.9018-0.9018 **
Appendix
Table 20: Spearman correlation ( ρ\rho ) between representation quality metrics and KNN test accuracy ( k=1k=1 ), stratified by architecture class, across specialised datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  SSLSupervised
CIFAR-10α\bm{\alpha} -ReQ−0.5896-0.5896 **-0.05143
RankMe0.21960.36810.3681 **
RankMe ⋆0.67150.6715 **-0.1566
NE-Sum0.7540.754 **0.41380.4138 **
κ\bm{\kappa}-0.014510.03192
Self-Cluster−0.9059-0.9059 **−0.406-0.406 **
Appendix
Table 21: Spearman correlation ( ρ\rho ) between representation quality metrics and KNN test accuracy ( k=1k=1 ), stratified by training objective, across natural image datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  SSLSupervised
Food-101α\bm{\alpha} -ReQ-0.1068-0.08897
RankMe0.14190.61690.6169 **
RankMe ⋆0.36470.3647 *-0.1516
NE-Sum0.80260.8026 **0.57750.5775 **
κ\bm{\kappa}−0.4805-0.4805 **0.03732
Self-Cluster−0.915-0.915 **−0.4968-0.4968 **
Appendix
Table 22: Spearman correlation ( ρ\rho ) between representation quality metrics and KNN test accuracy ( k=1k=1 ), stratified by training objective, across fine-grained datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  SSLSupervised
DTDα\bm{\alpha} -ReQ−0.6826-0.6826 **−0.229-0.229 *
RankMe0.19680.67860.6786 **
RankMe ⋆0.6760.676 **−0.3815-0.3815 **
NE-Sum0.7010.701 **0.43930.4393 **
κ\bm{\kappa}−0.4032-0.4032 **0.39120.3912 **
Self-Cluster−0.849-0.849 **−0.4414-0.4414 **
Appendix
Table 23: Spearman correlation ( ρ\rho ) between representation quality metrics and KNN test accuracy ( k=1k=1 ), stratified by training objective, across specialised datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  d≤520d\leq 520520<d≤960520<d\leq 960960<d≤1792960<d\leq 1792d>1792d>1792
CIFAR-10α\bm{\alpha} -ReQ-0.2205−0.3899-0.3899 **-0.056140.04828
RankMe0.1650.19980.065650.3609
RankMe ⋆0.31510.33950.3395 *0.08716-0.1348
NE-Sum0.40710.4071 **0.50950.5095 **0.48230.4823 **0.2383
κ\bm{\kappa}-0.2296-0.16430.007536-0.02937
Self-Cluster−0.4417-0.4417 **−0.4743-0.4743 **−0.6552-0.6552 **-0.2042
Appendix
Table 24: Spearman correlation ( ρ\rho ) between representation quality metrics and KNN test accuracy ( k=1k=1 ), stratified by representation dimension dd , across natural image datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  d≤520d\leq 520520<d≤960520<d\leq 960960<d≤1792960<d\leq 1792d>1792d>1792
Food-101α\bm{\alpha} -ReQ-0.07838-0.23440.022270.1511
RankMe0.1580.27040.014130.43860.4386 *
RankMe ⋆0.2380.2670.04507-0.1973
NE-Sum0.53710.5371 **0.56560.5656 **0.39250.3925 **0.38310.3831 *
κ\bm{\kappa}−0.6333-0.6333 **-0.2241-0.002442-0.1096
Self-Cluster−0.4853-0.4853 *−0.6011-0.6011 **−0.6187-0.6187 **-0.2234
Appendix
Table 25: Spearman correlation ( ρ\rho ) between representation quality metrics and KNN test accuracy ( k=1k=1 ), stratified by representation dimension dd , across fine-grained datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
  d≤520d\leq 520520<d≤960520<d\leq 960960<d≤1792960<d\leq 1792d>1792d>1792
DTDα\bm{\alpha} -ReQ−0.3514-0.3514 *−0.4515-0.4515 **-0.106-0.1975
RankMe0.34030.29150.2915 *0.04540.35090.3509 *
RankMe ⋆0.40680.4068 **0.44180.4418 **0.1551-0.3448
NE-Sum0.37250.3725 *0.32890.3289 *0.0054490.1462
κ\bm{\kappa}−0.4265-0.4265 *-0.14250.010420.195
Self-Cluster−0.4378-0.4378 *−0.4512-0.4512 **−0.5059-0.5059 **-0.05875
Appendix
Table 26: Spearman correlation ( ρ\rho ) between representation quality metrics and KNN test accuracy ( k=1k=1 ), stratified by representation dimension dd , across specialised datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. * p<0.05p<0.05 , ** p<0.01p<0.01 , *** p<0.001p<0.001 . Non-significant cells are greyed out.
Figure 14: DSE metric values plotted against test accuracy across datasets. The top row shows all models coloured by architecture class, revealing that the models with extreme values predominantly belong to the ViT architecture class. The bottom row shows only ViT models coloured by training objective, illustrating that the near zero DSE values originate predominantly from Supervised and SSL+Sup models rather than SSL models. This suggests that the DSE metric may be a reliable predictor for SSL models (though the sample size is small), but is unreliable for Supervised and SSL+Sup models.
Figure 16: Comparison of DSE and Self-Cluster against KNN linear-probe test accuracy across six datasets. The ρ\rho denotes the Spearman correlation and the corresponding pp -value is annotated in each subplot. Marker size is proportional to each model’s representation dimensionality, scaled linearly relative to the highest-dimensional model.
Figure 18: Comparison of RankMe and RankMe⋆\text{RankMe}^{\star} against KNN probe test accuracy across six datasets. The ρ\rho denotes the Spearman correlation and the corresponding pp -value is annotated in each subplot. Marker size is proportional to each model’s representation dimensionality, scaled linearly relative to the highest-dimensional model.
Figure 19: Comparison of NE Sum against KNN linear-probe test accuracy across six datasets. The ρ\rho denotes the Spearman correlation and the corresponding pp -value is annotated in each subplot. Marker size is proportional to each model’s representation dimensionality, scaled linearly relative to the highest-dimensional model.
Figure 20: Comparison of ID against KNN probe test accuracy across six datasets. The ρ\rho denotes the Spearman correlation and the corresponding pp -value is annotated in each subplot. Marker size is proportional to each model’s representation dimensionality, scaled linearly relative to the highest-dimensional model.
 CIFAR-10CIFAR-100ImageNet-1k
Metricρ\rhoppρ\rhoppρ\rhopp
α>=1.0\alpha>=1.00.13570.13570.032 360.032\,360.081 670.081\,670.22980.22980.067 480.067\,480.56510.5651
α<1.0\alpha<1.0−0.4794-0.47940.060 240.060\,24−0.5287-0.52870.000 158 60.000\,158\,6−0.2734-0.27340.001 6470.001\,647
α\alpha -Req0.025 860.025\,860.67520.6752−0.014 64-0.014\,640.81290.8129−0.096 97-0.096\,970.16660.1666
RankMe0.11380.11380.064 380.064\,380.24720.24724.893\times10−054.893\text{\times}{10}^{-05}0.20090.20090.003 8680.003\,868
RankMe ⋆−0.1565-0.15650.010 720.010\,72−0.1378-0.13780.025 140.025\,14−0.022 45-0.022\,450.74930.7493
Appendix
Table 27: Spearman correlation ( ρ\rho ) between representation metrics and test accuracy on Linear Probe trained using SGD, across CIFAR-10, CIFAR-100, and ImageNet-1k. We analyze three regimes: α<1\alpha<1 , α>1\alpha>1 , and unconstrained (all α\alpha ). Bold underlined values indicate statistical significance ( p<0.05p<0.05 ).()
More in this topic

Reference-Free Evaluation

Reference-free evaluation measures AI outputs or models without relying on gold answers, using source-grounded checks, learned judges, or intrinsic signals.

See all 25 papers

Explore similar work

CardsList
May 7, 2026

Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuning

Models trained with deep learning often fail to signal when inputs fall outside their training data manifold, leading to unreliable predictions under distribution shift. Prior work suggests that effective out-of-distribution (OOD) detection often requires class-conditional modeling or specialized models obtained through supervised fine-tuning. We revisit this assumption in modern pretrained models and show that their frozen representations already encode sufficient geometric structure for accurate label-free OOD detection. Across 59 backbone-task pairings spanning vision and language, we compare two complementary label-free detectors: a global Mahalanobis estimator fit on unlabeled latent representations, and ReSCOPED, a lightweight, diffusion-based typicality estimator operating on the same features at a local level. Despite their different detection mechanisms, representation scaling reveals a consistent regime-dependent pattern: both local and global detectors' absolute performance improves with better representation quality, and performance gaps between the two detectors disappear across both language and vision tasks as representations scale. These results suggest that label-free OOD detection depends strongly on the geometry exposed by frozen pretrained backbones, reducing the importance of detector choice as backbone scale increases and enabling efficient deployment directly on frozen models.

Brett Barkley, Preston Culbertson, David Fridovich-Keil

Department of Electrical and Computer Engineering The University of Texas at Austin Austin, TX, USA · Department of Computer Science Cornell University Ithaca, NY, USA · Department of Aerospace Engineering and Engineering Mechanics The University of Texas at Austin Austin, TX, USA

OOD Detection
AskPDF
Feb 27, 2026

Who Guards the Guardians? The Challenges of Evaluating Identifiability of Learned Representations

Identifiability in representation learning is commonly evaluated using standard metrics (e.g., MCC, DCI, R^2) on synthetic benchmarks with known ground-truth factors. These metrics are assumed to reflect recovery up to the equivalence class guaranteed by identifiability theory. We show that this assumption holds only under specific structural conditions: each metric implicitly encodes assumptions about both the data-generating process (DGP) and the encoder. When these assumptions are violated, metrics become misspecified and can produce systematic false positives and false negatives. Such failures occur both within classical identifiability regimes and in post-hoc settings where identifiability is most needed. We introduce a taxonomy separating DGP assumptions from encoder geometry, use it to characterise the validity domains of existing metrics, and release an evaluation suite for reproducible stress testing and comparison.

Shruti Joshi, Théo Saulus, Wieland Brendel +3

Mila - Québec AI Institute & Université de Montréal · Max-Planck-Institute for Intelligent Systems, ELLIS Institute Tübingen, University of Tübingen

Representation IdentifiabilityDisentangled Representation Learning
AskPDF
Apr 8, 2026

Training Deep Visual Networks Beyond Loss and Accuracy Through a Dynamical Systems Approach

Deep visual recognition models are usually trained and evaluated using metrics such as loss and accuracy. While these measures show whether a model is improving, they reveal very little about how its internal representations change during training. This paper introduces a complementary way to study that process by examining training through the lens of dynamical systems. Drawing on ideas from signal analysis originally used to study biological neural activity, we define three measures from layer activations collected across training epochs: an integration score that reflects long-range coordination across layers, a metastability score that captures how flexibly the network shifts between more and less synchronised states, and a combined dynamical stability index. We apply this framework to nine combinations of model architecture and dataset, including several ResNet variants, DenseNet-121, MobileNetV2, VGG-16, and a pretrained Vision Transformer on CIFAR-10 and CIFAR-100. The results suggest three main patterns. First, the integration measure consistently distinguishes the easier CIFAR-10 setting from the more difficult CIFAR-100 setting. Second, changes in the volatility of the stability index may provide an early sign of convergence before accuracy fully plateaus. Third, the relationship between integration and metastability appears to reflect different styles of training behaviour. Overall, this study offers an exploratory but promising new way to understand deep visual training beyond loss and accuracy.

Hai La Quang, Hassan Ugail, Newton Howard +3

Posts and Telecommunications Institute of Technology Vietnam · Centre for Visual Computing and Intelligent Systems University of Bradford United Kingdom · School of Individualized Study Rochester Institute of Technology United States

Neural Network Training Dynamics
AskPDF