A Comparative Study of Label-free Representation Quality Metrics in Deep Learning

Daniel Richards Arputharaj; Daniel Jönsson; Gabriel Eilertsen

cs.LGAug 24, 2026

A Comparative Study of Label-free Representation Quality Metrics in Deep Learning

Authors: Daniel Richards Arputharaj, Daniel Jönsson, Gabriel Eilertsen

Organizations: Department of Science and Technology Linköping University

Abstract

We present a comparative study of label-free metrics for assessing the quality of representations in deep neural networks to understand their reliability under a wide variety of configurations. We group existing label-free metrics into three families based on their construction and analytically establish connections between metrics within the same family. We then characterise the sensitivity of spectral metrics through controlled synthetic experiments. Finally, all label-free metrics are evaluated against downstream task accuracy across a diverse set of 260 vision models on six datasets spanning generic object classification, fine-grained object classification, scene recognition and geospatial task, stratifying results by architecture class and training objective. We find that intrinsic dimensionality (ID) is the most reliable predictor among the metrics considered. However, the reliability of all metrics, including ID, is moderated by architecture class and training objective. Our results provide a clearer understanding of what label-free representation quality metrics measure, when they are reliable, and how to interpret them in practice.

Figures & tables

Figure 1: Illustration of the representation extraction pipeline for self-supervised (SSL) and supervised models. In SSL models, the projector MLP is discarded after training, and the backbone output $X\in\mathbb{R}^{N\times d}$ is used as the representation. For supervised models, the penultimate layer activations are used as the representation, also denoted as $X$ . In both cases, representation quality metrics are computed directly on X, without access to the projector embeddings Z or classification heads.

Metric	Category	Complexity	Description
$\alpha$ -ReQ ( 1 )	Spectral	$\mathcal{O}(Nd^{2})$	Fits power law $\lambda_{i}\propto i^{-\alpha}$ to the eigenspectrum. Sensitive to fitting range in log-log space ( 13 ) .
RankMe ( 20 )	Spectral	$\mathcal{O}(Nd^{2})$	Entropy of normalised singular values. Originally computed on projector output; we compute on backbone output.
NE Sum ( 21 )	Spectral	$\mathcal{O}(Nd^{2})$	Measures degree of feature whitening. Equivalent to Stable Rank ( 39 ) for mean-centred representations (Section 3.1 ).
$\kappa$ ( 39 )	Spectral	$\mathcal{O}(Nd^{2})$	Ratio of largest to smallest non-zero singular value. Measures sensitivity of linear system to perturbations.
Self-Cluster ( 39 )	Relational	$\mathcal{O}(Nd^{2})^{\dagger}$	Measures deviation of $\ell_{2}$ -normalised representations from a uniform spherical distribution. Reduced from $\mathcal{O}(N^{2}d)$ via batched reformulation (Section 3.2 ).
DSE ( 30 )	Relational	$\mathcal{O}(N^{\prime 2}d)$	Spectral entropy of anisotropic diffusion kernel matrix. Subsampled to $N^{\prime}\!=\!10{,}000$ for scalability. Analytically related to Self-Cluster (Section 3.2 ).

Table 1: Summary of representation quality metrics considered in this work, grouped by construction. Metrics take the representation matrix,

X\in\mathbb{R}^{N\times d}

, as input, where

N

is the number of samples and

d

is the representation dimension (we assume

N\gg d

Panel of Figure 2 — Figure 2: Comparison of real and synthetic eigenspectra in log-log scale. Real backbone representations with dimension $d\in[1000,1024]$ (Fig. 2(b) ) exhibit spectral shapes consistent with the synthetically generated power-law family $\lambda_{i}\propto i^{-\alpha}$ , $\sum_{i=1}^{d}\lambda_{i}=1$ , under negative values of $\alpha$ (Fig. 2(a) ). This similarity justifies including $\alpha<0$ in the synthetic sensitivity analysis

Panel of Figure 3 — Figure 3: Sensitivity of spectral metrics RankMe, NE Sum, and $\kappa$ to spectral shape ( $\alpha$ ) and representation dimension ( $d$ ), where eigenspectra are synthetically generated as $\lambda_{i}\propto i^{-\alpha}$ with $\sum_{i=1}^{d}\lambda_{i}=1$ . Each row shows the un-normalised metric (col. 1), the dimension-normalised surface (col. 2), and the normalised metric as a function of $d$ for fixed values of $\alpha$ (col. 3). Normalisation is performed by dividing by $d$ , following the normalisation proposed in 39 for RankMe, which we extend to all three metrics. In col. 3, dashed lines correspond to $\alpha<0$ and solid lines to $\alpha\geq 0$ . Since $\kappa$ is symmetric about $\alpha=0$ (Fig. 3(h) ), only $\alpha\geq 0$ is shown in Fig. 3(i) .

Table 2: Spearman correlation (

\rho

) between representation quality metrics and downstream accuracy across six evaluation datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out. – are cells with either no models or one model

Panel of Figure 4 — Figure 4: Comparison of ID against OLS probe test accuracy across six datasets. The $\rho$ denotes the Spearman correlation and the corresponding $p$ -value is annotated in each subplot. Marker size is proportional to each model’s representation dimensionality, scaled linearly relative to the highest-dimensional model.

Panel of Figure 6 — Figure 6: Comparison of DSE and Self-Cluster against OLS linear-probe test accuracy across six datasets. The $\rho$ denotes the Spearman correlation and the corresponding $p$ -value is annotated in each subplot. Marker size is proportional to each model’s representation dimensionality, scaled linearly relative to the highest-dimensional model.

		ConvNeXT	RegNet	ResNet	ViT
CIFAR-100	RankMe	$0.8189$ **	$0.8952$ **	$0.8425$ **	$0.5985$ **
	RankMe ⋆	0.2661	$-0.5361$ **	-0.4239	0.1086
	NE-Sum	-0.2015	$0.4731$ **	$0.5276$ **	$0.4809$ **
	Self-Cluster	0.1316	$-0.3331$ *	$-0.7623$ **	-0.143
	ID	$-0.8087$ **	$-0.5637$ **	-0.06056	$-0.5633$ **
ImageNet	RankMe	$0.3897$ *	$0.7538$ **	$0.6047$ **	0.1835

Table 3: Spearman correlation (

\rho

) between representation metrics and OLS test accuracy, grouped by architecture class, across the six main datasets. Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

Panel of Figure 7 — Figure 7: Self-Cluster metric values plotted against test accuracy across datasets. The top row shows all models coloured by architecture class, revealing that the models with extreme values predominantly belong to the ViT architecture class. The bottom row shows only ViT models coloured by training objective, illustrating that the near unit Self-Cluster values originate predominantly from Supervised and SSL+Sup models rather than SSL models. This suggests that the Self-Cluster metric may be a reliable predictor for SSL models (though the sample size is small), but is unreliable for Supervised and SSL+Sup models.

		SSL	Supervised
CIFAR-100	RankMe	0.2258	$0.3044$ *
	RankMe ⋆	$0.6103$ **	$-0.3593$ **
	NE-Sum	$0.6617$ **	0.1204
	Self-Cluster	$-0.8134$ **	-0.02742
	ID	$-0.6855$ **	$-0.5586$ **
ImageNet	RankMe	0.331	$0.2411$ *

Table 4: Spearman correlation (

\rho

) between representation metrics and OLS test accuracy, grouped by training objective, across the six main datasets. Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

Panel of Figure 9 — Figure 9: Comparison of RankMe and RankMe ⋆ against OLS linear-probe test accuracy across six datasets. The $\rho$ denotes the Spearman correlation and the corresponding $p$ -value is annotated in each subplot. Marker size is proportional to each model’s representation dimensionality, scaled linearly relative to the highest-dimensional model.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10:

\alpha

-ReQ fitting-range sensitivity. Spearman correlation between

\alpha

( top ) and

|\alpha-1|

( bottom ) with OLS test accuracy, across five fitting ranges and ten datasets. Green: significant positive; red: significant negative; grey:

p\geq 0.05

Figure 11: TwoNN intrinsic dimensionality (ID) estimation noise across 260 models and 10 datasets (2,600 model–dataset pairs). Left : Intra-batch subsample noise which is the relative standard deviation of ID across 20 within-batch subsamples (Number of repetition = 20, 90% subsampling rate within each batch). Right : Across-batch noise which is the relative standard deviation of the per-batch mean ID across batches. Median intra-batch and across-batch noise are

4.5\%

and

7.6\%

respectively, with 95th percentiles at

8.2\%

and

15.2\%

Figure 12: DSE estimation noise across 260 models and 10 datasets. Relative standard deviation of DSE across batches.

Figure 13: ID vs OLS test accuracy, coloured by $\log_{10}$ parameter count, across six main datasets. Partial Spearman correlations controlling for log parameter count are annotated in each panel ( $n=255$ ; 5 models excluded due to unavailable parameter counts). Asterisk (*) denotes $p<0.05$ .


GitHub Repo	Model Name	$d$	Arch.	Objective	Params (M)	GMACs	Training data
facebookresearch/barlowtwins:main	barlowtwins	1000	ResNet	SSL	25.6	4.11	ImageNet-1k
facebookresearch/vicregl:main	resnet50_alpha0p9	2048	ResNet	SSL	23.5	4.11	ImageNet-1k
facebookresearch/vicregl:main	resnet50_alpha0p75	2048	ResNet	SSL	23.5	4.11	ImageNet-1k
facebookresearch/vicregl:main	convnext_small_alpha0p9	768	ConvNeXT	SSL	49.4	8.70	ImageNet-1k
facebookresearch/vicregl:main	convnext_small_alpha0p75	768	ConvNeXT	SSL	49.4	8.70	ImageNet-1k

Appendix

Table 5: Summary of torch hub models and their characteristics. Params/GMACs are for the frozen feature backbone (classifier head removed), computed with timm+fvcore; GMACs are at the listed input resolution. Arrows denote a pretrain

\rightarrow

fine-tune chain. LVD-142M/1689M: DINOv2/v3 curated web data; WIT-400M: OpenAI CLIP data; LAION: web image--text; IG: Instagram; YFCC-100M: Flickr; MetaCLIP-2B: Web-SSL web pool.


Model Name	$d$	Arch.	Objective	Params (M)	GMACs	Training data
facebook/dinov2-small	384	ViT	SSL	22.1	29.46	LVD-142M
timm/vit_small_patch16_224.dino	384	ViT	SSL	21.7	4.25	ImageNet-1k
timm/vit_small_patch16_dinov3_qkvb.lvd1689m	384	ViT	SSL	21.6	5.63	LVD-1689M (web)
timm/vit_small_plus_patch16_dinov3_qkvb.lvd1689m	384	ViT	SSL	28.7	7.48	LVD-1689M (web)
facebook/dinov2-base	768	ViT	SSL	86.6	117.11	LVD-142M

Appendix

Table 6: Summary of Hugging Face ( 41 ) and timm models and their characteristics. Params/GMACs are for the frozen feature backbone (classifier head removed), computed with timm+fvcore; GMACs are at the listed input resolution. Arrows denote a pretrain

\rightarrow

fine-tune chain. LVD-142M/1689M: DINOv2/v3 curated web data; WIT-400M: OpenAI CLIP data; LAION: web image--text; IG: Instagram; YFCC-100M: Flickr; MetaCLIP-2B: Web-SSL web pool.

Table 7: Spearman correlation (

\rho

) between representation quality metrics and OLS test accuracy across 10 evaluation datasets, grouped by task type. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		ConvNeXT	RegNet	ResNet	ViT
CIFAR-10	$\bm{\alpha}$ -ReQ	$-0.469$ **	-0.08933	-0.1038	-0.2193
	RankMe	$0.7983$ **	$0.779$ **	$0.7668$ **	$0.5649$ **
	RankMe ⋆	0.1358	$-0.3692$ *	-0.3669	0.1175
	NE-Sum	0.2707	$0.4884$ **	0.3144	$0.3697$ *
	$\bm{\kappa}$	0.3292	$0.6438$ **	$0.6204$ **	$0.409$ **
	Self-Cluster	0.181	$-0.5331$ **	$-0.7515$ **	-0.1731

Appendix

Table 8: Spearman correlation (

\rho

) between representation quality metrics and OLS test accuracy, stratified by architecture class, across natural image datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction:

\uparrow

(higher is better),

\downarrow

(lower is better),

\rightarrow 1

(closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		ConvNeXT	RegNet	ResNet	ViT
Food-101	$\bm{\alpha}$ -ReQ	$-0.5099$ **	-0.1406	-0.1182	-0.1822
	RankMe	$0.8168$ **	$0.874$ **	$0.7219$ **	$0.6275$ **
	RankMe ⋆	0.2391	$-0.4341$ **	$-0.4802$ *	0.08788
	NE-Sum	$0.6755$ **	$0.5504$ **	$0.6168$ **	$0.6099$ **
	$\bm{\kappa}$	0.1634	$0.6421$ **	$0.585$ *	0.1812
	Self-Cluster	$-0.3117$ *	-0.272	$-0.6201$ **	-0.2651

Appendix

Table 9: Spearman correlation (

\rho

) between representation quality metrics and OLS test accuracy, stratified by architecture class, across fine-grained datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction:

\uparrow

(higher is better),

\downarrow

(lower is better),

\rightarrow 1

(closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		ConvNeXT	RegNet	ResNet	ViT
DTD	$\bm{\alpha}$ -ReQ	$-0.4812$ **	-0.181	-0.153	0.03269
	RankMe	$0.6054$ **	0.02952	$-0.6883$ **	$0.4478$ **
	RankMe ⋆	0.1516	-0.1294	0.4217	-0.1118
	NE-Sum	-0.1059	-0.1169	$-0.4944$ *	$0.4943$ **
	$\bm{\kappa}$	$0.3355$ *	0.04234	$-0.5242$ *	0.05048
	Self-Cluster	0.03795	0.2564	$0.6252$ **	-0.225

Appendix

Table 10: Spearman correlation (

\rho

) between representation quality metrics and OLS test accuracy, stratified by architecture class, across specialised datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction:

\uparrow

(higher is better),

\downarrow

(lower is better),

\rightarrow 1

(closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		SSL	Supervised
CIFAR-10	$\bm{\alpha}$ -ReQ ( $\bm{\rightarrow 1}$ )	$-0.4703$ **	$0.1877$ *
	RankMe ( $\bm{\uparrow}$ )	0.09054	0.1503
	RankMe ⋆ ( $\bm{\uparrow}$ )	$0.4985$ **	$-0.3683$ **
	NE-Sum ( $\bm{\uparrow}$ )	$0.5942$ **	0.0295
	$\bm{\kappa}$ ( $\bm{\uparrow}$ )	-0.06452	0.294
	Self-Cluster ( $\bm{\downarrow}$ )	$-0.7885$ **	0.003576

Appendix

Table 11: Spearman correlation (

\rho

) between representation quality metrics and OLS test accuracy, stratified by training objective, across natural image datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction:

\uparrow

(higher is better),

\downarrow

(lower is better),

\rightarrow 1

(closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		SSL	Supervised
Food-101	$\bm{\alpha}$ -ReQ ( $\bm{\rightarrow 1}$ )	-0.07001	-0.01501
	RankMe ( $\bm{\uparrow}$ )	0.1488	$0.4572$ **
	RankMe ⋆ ( $\bm{\uparrow}$ )	0.2804	-0.1865
	NE-Sum ( $\bm{\uparrow}$ )	$0.7474$ **	$0.3605$ **
	$\bm{\kappa}$ ( $\bm{\uparrow}$ )	-0.2929	$0.3314$ *
	Self-Cluster ( $\bm{\downarrow}$ )	$-0.9062$ **	-0.1227

Appendix

Table 12: Spearman correlation (

\rho

) between representation quality metrics and OLS test accuracy, stratified by training objective, across fine-grained datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction:

\uparrow

(higher is better),

\downarrow

(lower is better),

\rightarrow 1

(closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		SSL	Supervised
DTD	$\bm{\alpha}$ -ReQ ( $\bm{\rightarrow 1}$ )	$-0.5796$ **	0.07191
	RankMe ( $\bm{\uparrow}$ )	-0.1747	$-0.4853$ **
	RankMe ⋆ ( $\bm{\uparrow}$ )	$0.6599$ **	0.3193
	NE-Sum ( $\bm{\uparrow}$ )	$0.4883$ **	$-0.4066$ **
	$\bm{\kappa}$ ( $\bm{\uparrow}$ )	$-0.448$ **	-0.3383
	Self-Cluster ( $\bm{\downarrow}$ )	$-0.6756$ **	$0.3718$ **

Appendix

Table 13: Spearman correlation (

\rho

) between representation quality metrics and OLS test accuracy, stratified by training objective, across specialised datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction:

\uparrow

(higher is better),

\downarrow

(lower is better),

\rightarrow 1

(closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		$d\leq 520$	$520<d\leq 960$	$960<d\leq 1792$	$d>1792$
CIFAR-10	$\bm{\alpha}$ -ReQ ( $\bm{\rightarrow 1}$ )	0.1734	-0.1571	0.04738	0.1315
	RankMe ( $\bm{\uparrow}$ )	-0.3555	-0.02149	-0.1327	0.4376
	RankMe ⋆ ( $\bm{\uparrow}$ )	-0.3226	0.08077	-0.03157	-0.2487
	NE-Sum ( $\bm{\uparrow}$ )	-0.2765	0.1083	$0.4189$ *	0.1896
	$\bm{\kappa}$ ( $\bm{\uparrow}$ )	$0.4829$ *	0.1335	0.04447	0.04695
	Self-Cluster ( $\bm{\downarrow}$ )	0.3405	0.03767	$-0.583$ *	-0.1373

Appendix

Table 14: Spearman correlation (

\rho

) between representation quality metrics and OLS test accuracy, stratified by representation dimension

d

, across natural image datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction:

\uparrow

(higher is better),

\downarrow

(lower is better),

\rightarrow 1

(closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		$d\leq 520$	$520<d\leq 960$	$960<d\leq 1792$	$d>1792$
Food-101	$\bm{\alpha}$ -ReQ ( $\bm{\rightarrow 1}$ )	0.2028	$-0.446$ *	0.1216	0.2136
	RankMe ( $\bm{\uparrow}$ )	-0.05324	0.3067	-0.1193	0.4383
	RankMe ⋆ ( $\bm{\uparrow}$ )	-0.1871	$0.4453$ *	-0.08247	-0.2668
	NE-Sum ( $\bm{\uparrow}$ )	0.08391	$0.4844$ *	0.1824	0.3293
	$\bm{\kappa}$ ( $\bm{\uparrow}$ )	$0.4579$ *	-0.002327	0.1213	-0.1018
	Self-Cluster ( $\bm{\downarrow}$ )	0.2196	-0.1607	$-0.5652$ *	-0.1326

Appendix

Table 15: Spearman correlation (

\rho

) between representation quality metrics and OLS test accuracy, stratified by representation dimension

d

, across fine-grained datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction:

\uparrow

(higher is better),

\downarrow

(lower is better),

\rightarrow 1

(closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		$d\leq 520$	$520<d\leq 960$	$960<d\leq 1792$	$d>1792$
DTD	$\bm{\alpha}$ -ReQ ( $\bm{\rightarrow 1}$ )	0.3104	-0.276	0.08673	0.2104
	RankMe ( $\bm{\uparrow}$ )	-0.1474	-0.144	$-0.7875$ *	0.06973
	RankMe ⋆ ( $\bm{\uparrow}$ )	-0.3189	0.3309	0.1021	$-0.8435$ *
	NE-Sum ( $\bm{\uparrow}$ )	-0.2004	0.01337	$-0.3821$ *	-0.2098
	$\bm{\kappa}$ ( $\bm{\uparrow}$ )	$0.4092$ *	-0.0831	-0.05862	0.4448
	Self-Cluster ( $\bm{\downarrow}$ )	0.2162	0.09457	-0.07227	0.2249

Appendix

Table 16: Spearman correlation (

\rho

) between representation quality metrics and OLS test accuracy, stratified by representation dimension

d

, across specialised datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Arrows indicate the preferred direction:

\uparrow

(higher is better),

\downarrow

(lower is better),

\rightarrow 1

(closer to 1 is better). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

Table 17: Spearman correlation (

\rho

) between representation quality metrics and KNN test accuracy (

k=1

) across 10 evaluation datasets, grouped by task type. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		ConvNeXT	RegNet	ResNet	ViT
CIFAR-10	$\bm{\alpha}$ -ReQ	$-0.5643$ **	-0.227	-0.2629	-0.2075
	RankMe	$0.7738$ **	$0.6102$ **	$0.5309$ *	$0.5369$ **
	RankMe ⋆	0.2589	-0.1931	-0.05598	0.2075
	NE-Sum	$0.3157$ *	$0.6241$ **	0.3101	$0.6791$ **
	$\bm{\kappa}$	0.2408	$0.4666$ **	0.3615	0.2019
	Self-Cluster	0.1193	$-0.6115$ **	$-0.5767$ **	$-0.8651$ **

Appendix

Table 18: Spearman correlation (

\rho

) between representation quality metrics and KNN test accuracy (

k=1

), stratified by architecture class, across natural image datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		ConvNeXT	RegNet	ResNet	ViT
Food-101	$\bm{\alpha}$ -ReQ	$-0.5981$ **	$-0.2761$ *	-0.1346	0.1837
	RankMe	$0.7632$ **	$0.8486$ **	$0.6684$ **	$0.5275$ **
	RankMe ⋆	$0.3745$ **	$-0.3125$ *	$-0.4269$ *	-0.1435
	NE-Sum	$0.8039$ **	$0.6442$ **	$0.6426$ **	$0.5705$ **
	$\bm{\kappa}$	0.0578	$0.6323$ **	$0.5163$ *	-0.1739
	Self-Cluster	$-0.3704$ **	$-0.3716$ *	$-0.6934$ **	$-0.914$ **

Appendix

Table 19: Spearman correlation (

\rho

) between representation quality metrics and KNN test accuracy (

k=1

), stratified by architecture class, across fine-grained datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		ConvNeXT	RegNet	ResNet	ViT
DTD	$\bm{\alpha}$ -ReQ	$-0.5196$ **	0.03956	0.2612	-0.2834
	RankMe	$0.7614$ **	$0.8254$ **	$0.6813$ **	$0.4636$ **
	RankMe ⋆	0.1018	$-0.7366$ **	$-0.6649$ **	0.1615
	NE-Sum	-0.2564	$0.5682$ **	$0.5334$ **	$0.6393$ **
	$\bm{\kappa}$	$0.4526$ **	$0.7614$ **	$0.6288$ *	-0.09476
	Self-Cluster	0.01003	-0.2031	$-0.6567$ **	$-0.9018$ **

Appendix

Table 20: Spearman correlation (

\rho

) between representation quality metrics and KNN test accuracy (

k=1

), stratified by architecture class, across specialised datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		SSL	Supervised
CIFAR-10	$\bm{\alpha}$ -ReQ	$-0.5896$ **	-0.05143
	RankMe	0.2196	$0.3681$ **
	RankMe ⋆	$0.6715$ **	-0.1566
	NE-Sum	$0.754$ **	$0.4138$ **
	$\bm{\kappa}$	-0.01451	0.03192
	Self-Cluster	$-0.9059$ **	$-0.406$ **

Appendix

Table 21: Spearman correlation (

\rho

) between representation quality metrics and KNN test accuracy (

k=1

), stratified by training objective, across natural image datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		SSL	Supervised
Food-101	$\bm{\alpha}$ -ReQ	-0.1068	-0.08897
	RankMe	0.1419	$0.6169$ **
	RankMe ⋆	$0.3647$ *	-0.1516
	NE-Sum	$0.8026$ **	$0.5775$ **
	$\bm{\kappa}$	$-0.4805$ **	0.03732
	Self-Cluster	$-0.915$ **	$-0.4968$ **

Appendix

Table 22: Spearman correlation (

\rho

) between representation quality metrics and KNN test accuracy (

k=1

), stratified by training objective, across fine-grained datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		SSL	Supervised
DTD	$\bm{\alpha}$ -ReQ	$-0.6826$ **	$-0.229$ *
	RankMe	0.1968	$0.6786$ **
	RankMe ⋆	$0.676$ **	$-0.3815$ **
	NE-Sum	$0.701$ **	$0.4393$ **
	$\bm{\kappa}$	$-0.4032$ **	$0.3912$ **
	Self-Cluster	$-0.849$ **	$-0.4414$ **

Appendix

Table 23: Spearman correlation (

\rho

) between representation quality metrics and KNN test accuracy (

k=1

), stratified by training objective, across specialised datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		$d\leq 520$	$520<d\leq 960$	$960<d\leq 1792$	$d>1792$
CIFAR-10	$\bm{\alpha}$ -ReQ	-0.2205	$-0.3899$ **	-0.05614	0.04828
	RankMe	0.165	0.1998	0.06565	0.3609
	RankMe ⋆	0.3151	$0.3395$ *	0.08716	-0.1348
	NE-Sum	$0.4071$ **	$0.5095$ **	$0.4823$ **	0.2383
	$\bm{\kappa}$	-0.2296	-0.1643	0.007536	-0.02937
	Self-Cluster	$-0.4417$ **	$-0.4743$ **	$-0.6552$ **	-0.2042

Appendix

Table 24: Spearman correlation (

\rho

) between representation quality metrics and KNN test accuracy (

k=1

), stratified by representation dimension

d

, across natural image datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		$d\leq 520$	$520<d\leq 960$	$960<d\leq 1792$	$d>1792$
Food-101	$\bm{\alpha}$ -ReQ	-0.07838	-0.2344	0.02227	0.1511
	RankMe	0.158	0.2704	0.01413	$0.4386$ *
	RankMe ⋆	0.238	0.267	0.04507	-0.1973
	NE-Sum	$0.5371$ **	$0.5656$ **	$0.3925$ **	$0.3831$ *
	$\bm{\kappa}$	$-0.6333$ **	-0.2241	-0.002442	-0.1096
	Self-Cluster	$-0.4853$ *	$-0.6011$ **	$-0.6187$ **	-0.2234

Appendix

Table 25: Spearman correlation (

\rho

) between representation quality metrics and KNN test accuracy (

k=1

), stratified by representation dimension

d

, across fine-grained datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

		$d\leq 520$	$520<d\leq 960$	$960<d\leq 1792$	$d>1792$
DTD	$\bm{\alpha}$ -ReQ	$-0.3514$ *	$-0.4515$ **	-0.106	-0.1975
	RankMe	0.3403	$0.2915$ *	0.0454	$0.3509$ *
	RankMe ⋆	$0.4068$ **	$0.4418$ **	0.1551	-0.3448
	NE-Sum	$0.3725$ *	$0.3289$ *	0.005449	0.1462
	$\bm{\kappa}$	$-0.4265$ *	-0.1425	0.01042	0.195
	Self-Cluster	$-0.4378$ *	$-0.4512$ **	$-0.5059$ **	-0.05875

Appendix

Table 26: Spearman correlation (

\rho

) between representation quality metrics and KNN test accuracy (

k=1

), stratified by representation dimension

d

, across specialised datasets. Metrics are organised by construction: spectral (top), relation-based (middle), and manifold-based (bottom). Statistical significance is considered after Benjamini–Hochberg correction. *

p<0.05

, **

p<0.01

, ***

p<0.001

. Non-significant cells are greyed out.

Panel of Figure 14 — Figure 14: DSE metric values plotted against test accuracy across datasets. The top row shows all models coloured by architecture class, revealing that the models with extreme values predominantly belong to the ViT architecture class. The bottom row shows only ViT models coloured by training objective, illustrating that the near zero DSE values originate predominantly from Supervised and SSL+Sup models rather than SSL models. This suggests that the DSE metric may be a reliable predictor for SSL models (though the sample size is small), but is unreliable for Supervised and SSL+Sup models.

Panel of Figure 16 — Figure 16: Comparison of DSE and Self-Cluster against KNN linear-probe test accuracy across six datasets. The $\rho$ denotes the Spearman correlation and the corresponding $p$ -value is annotated in each subplot. Marker size is proportional to each model’s representation dimensionality, scaled linearly relative to the highest-dimensional model.

Panel of Figure 18 — Figure 18: Comparison of RankMe and $\text{RankMe}^{\star}$ against KNN probe test accuracy across six datasets. The $\rho$ denotes the Spearman correlation and the corresponding $p$ -value is annotated in each subplot. Marker size is proportional to each model’s representation dimensionality, scaled linearly relative to the highest-dimensional model.

Panel of Figure 19 — Figure 19: Comparison of NE Sum against KNN linear-probe test accuracy across six datasets. The $\rho$ denotes the Spearman correlation and the corresponding $p$ -value is annotated in each subplot. Marker size is proportional to each model’s representation dimensionality, scaled linearly relative to the highest-dimensional model.

Panel of Figure 20 — Figure 20: Comparison of ID against KNN probe test accuracy across six datasets. The $\rho$ denotes the Spearman correlation and the corresponding $p$ -value is annotated in each subplot. Marker size is proportional to each model’s representation dimensionality, scaled linearly relative to the highest-dimensional model.

	CIFAR-10		CIFAR-100		ImageNet-1k
Metric	$\rho$	$p$	$\rho$	$p$	$\rho$	$p$
$\alpha>=1.0$	$0.1357$	$0.032\,36$	$0.081\,67$	$0.2298$	$0.067\,48$	$0.5651$
$\alpha<1.0$	$-0.4794$	$0.060\,24$	$-0.5287$	$0.000\,158\,6$	$-0.2734$	$0.001\,647$
$\alpha$ -Req	$0.025\,86$	$0.6752$	$-0.014\,64$	$0.8129$	$-0.096\,97$	$0.1666$
RankMe	$0.1138$	$0.064\,38$	$0.2472$	$4.893\text{\times}{10}^{-05}$	$0.2009$	$0.003\,868$
RankMe ⋆	$-0.1565$	$0.010\,72$	$-0.1378$	$0.025\,14$	$-0.022\,45$	$0.7493$

Appendix

Table 27: Spearman correlation (

\rho

) between representation metrics and test accuracy on Linear Probe trained using SGD, across CIFAR-10, CIFAR-100, and ImageNet-1k. We analyze three regimes:

\alpha<1

\alpha>1

, and unconstrained (all

\alpha

). Bold underlined values indicate statistical significance (

p<0.05

).()

Reference-Free Evaluation

Reference-free evaluation measures AI outputs or models without relying on gold answers, using source-grounded checks, learned judges, or intrinsic signals.

See all 25 papers

Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuning
May 7, 2026Brett Barkley, Preston Culbertson, David Fridovich-KeilOOD Detection
Department of Electrical and Computer Engineering The University of Texas at Austin Austin, TX, USA · Department of Computer Science Cornell University Ithaca, NY, USA · Department of Aerospace Engineering and Engineering Mechanics The University of Texas at Austin Austin, TX, USA
Who Guards the Guardians? The Challenges of Evaluating Identifiability of Learned Representations
Feb 27, 2026Shruti Joshi, Théo Saulus, Wieland Brendel +3Representation Identifiability Disentangled Representation Learning
Mila - Québec AI Institute & Université de Montréal · Max-Planck-Institute for Intelligent Systems, ELLIS Institute Tübingen, University of Tübingen
Training Deep Visual Networks Beyond Loss and Accuracy Through a Dynamical Systems Approach
Apr 8, 2026Hai La Quang, Hassan Ugail, Newton Howard +3Neural Network Training Dynamics
Posts and Telecommunications Institute of Technology Vietnam · Centre for Visual Computing and Intelligent Systems University of Bradford United Kingdom · School of Individualized Study Rochester Institute of Technology United States

May 7, 2026J/K move · Enter open · S save

Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuning

Brett Barkley, Preston Culbertson, David Fridovich-Keil

Department of Electrical and Computer Engineering The University of Texas at Austin Austin, TX, USA · Department of Computer Science Cornell University Ithaca, NY, USA · Department of Aerospace Engineering and Engineering Mechanics The University of Texas at Austin Austin, TX, USA

OOD Detection

Open paper AskPDF arXiv

Models trained with deep learning often fail to signal when inputs fall outside their training data manifold, leading to unreliable predictions under distribution shift. Prior work suggests that effective out-of-distribution (OOD) detection often requires class-conditional modeling or specialized models obtained through supervised fine-tuning. We revisit this assumption in modern pretrained models and show that their frozen representations already encode sufficient geometric structure for accurate label-free OOD detection. Across 59 backbone-task pairings spanning vision and language, we compare two complementary label-free detectors: a global Mahalanobis estimator fit on unlabeled latent representations, and ReSCOPED, a lightweight, diffusion-based typicality estimator operating on the same features at a local level. Despite their different detection mechanisms, representation scaling reveals a consistent regime-dependent pattern: both local and global detectors' absolute performance improves with better representation quality, and performance gaps between the two detectors disappear across both language and vision tasks as representations scale. These results suggest that label-free OOD detection depends strongly on the geometry exposed by frozen pretrained backbones, reducing the importance of detector choice as backbone scale increases and enabling efficient deployment directly on frozen models.

Abstract

Reference-Free Evaluation

Explore similar work

Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuning

Who Guards the Guardians? The Challenges of Evaluating Identifiability of Learned Representations

Training Deep Visual Networks Beyond Loss and Accuracy Through a Dynamical Systems Approach