Which Tasks Survive Self-Supervised Learning?

Achleshwar Luthra; Lucas Bryant; Tracy Zhu; Tomer Galanti

cs.LGSep 29, 2026

Which Tasks Survive Self-Supervised Learning?

Authors: Achleshwar Luthra, Lucas Bryant, Tracy Zhu, Tomer Galanti

Organizations: Department of Computer Science and Engineering Texas A&M University

Abstract

Same-instance self-supervised learning (SSL) learns representations by enforcing consistency across two views of the same underlying instance. This principle alone, however, does not determine which downstream tasks remain recoverable from the learned representation. We study this question through \emph{semantic recoverability}, defined as the amount of a task's posterior score captured by the represented function space. We show that, for centered and whitened representations, recoverability exactly determines directional class-distance-normalized variance (CDNV), controls few-shot nearest-centroid classification, and governs the strength of task-relevant semantic directions. The population linear probe and centroid axis coincide, and multiple well-recovered tasks approach a factorial centroid geometry. We then analyze a canonical two-view SSL objective and show that its population optimum spans the leading cross-view-stable modes of the associated two-view operator. This yields a closed-form spectral characterization of semantic recoverability: a downstream task is preserved to the extent that its posterior lies in the selected spectral subspace. We validate these predictions on synthetic and real datasets across several SSL methods, testing the predicted relationships among recoverability, directional geometry, spectral structure, and few-shot transfer. Together, these results give a task-level account of what information survives same-instance SSL and how the retained information appears in downstream geometry and transfer.

Figures & tables

Panel of Figure 1 — Figure 1: Conceptual flow of the theory. Same-instance SSL retains the most cross-view-stable modes of the two-view operator $T$ (Prop. 4.1 ), so a task survives to the extent that its posterior overlaps that subspace (Cor. 4.2 ). Under centered whitening, $B_{r}$ fixes the directional collapse law (Prop. 3.1 ) and few-shot nearest-centroid guarantee (Thm. 3.4 ); the task-specific $B_{r}^{(t)}$ determine the multitask centroid geometry (Thm. 3.5 ).

Figure 2: Geometric interpretation of semantic recoverability. The shaded plane is the SSL-selected subspace

\mathcal{U}_{r}=\operatorname{span}\{\psi_{1},\psi_{2}\}

of leading two-view eigenfunctions. The task posterior

\eta

decomposes into its projection

\Pi_{r}\eta

onto this subspace and an orthogonal residual. From left to right, the posterior is fully captured, partially captured, or discarded. The captured posterior energy

B_{r}=\|\Pi_{r}\eta\|_{L^{2}(P_{X})}^{2}

measures the retained task-relevant information.

Figure 3: Captured posterior energy on CelebA. Mean

B_{r}^{(t)}

over the 1, 5, and 10 most recoverable CelebA attributes and for all 40 attributes, shown across spectral ranks

r

. Shaded regions denote the corresponding interquartile ranges. The dashed curve is the mean for a matched randomly initialized encoder, with dotted curves showing its interquartile range.

Figure 4: Predicted versus observed directional CDNV on CelebA. Each point is one CelebA attribute at a given rank

r

. The prediction

\widehat{\tilde{V}}_{\mathrm{pred}}=(1-\widehat{B}_{A}^{(t)})/(2\widehat{B}_{A}^{(t)})

is computed on split A. Class centroids, the semantic direction, and observed directional CDNV are estimated independently on held-out split B. The dashed line is

y=x

, and each panel contains all 40 attributes.

Figure 5: Captured posterior energy controls few-shot NCC error. Empirical

m

-shot NCC error versus task-specific

B_{r}^{(t)}

across CelebA attributes, for several ranks

r

(panels) and support sizes

m

(colors). Each point corresponds to a downstream attribute, and dashed curves are the corresponding upper bounds from Thm. 3.4 .

Figure 6: Multitask centroid geometry on CelebA. (a–b) Illustrative held-out joint centroids for three attributes, compared with the hyperrectangle predicted by Thm. 3.5 ; triplets are selected on the training split. (c–d) Results over all 1646 eligible triplets per encoder, without recoverability or orthogonality screening. Larger mean task recoverability is associated with lower normalized centroid RMSE and smaller maximum pairwise task-axis overlap. Curves show bin means and shaded regions show interquartile ranges; selection and normalization details are in App. B.1 .

Figure 7: Recoverability from the paired-view spectral subspace. Directly measured task recoverability

\widehat{B}^{(t)}(F)

versus its reconstruction

\widehat{B}_{r^{\star}}^{(t)}

from the label-free paired-view spectral modes. Each point is one CelebA attribute; the dashed line is perfect agreement. For ImageNet-pretrained encoders, the retained subspace is reduced-rank and the median absolute discrepancy is below

0.05

across models; additional encoders are shown in App. B.1 .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Additional recoverability results for VICReg-IM1K on CelebA. (a) Captured posterior energy across representation ranks, using the same attribute groups and random-initialization reference as Fig. 3 . (b) Directional CDNV predicted from recoverability estimated on split A versus directional CDNV measured independently on held-out split B; the dashed line is

y=x

. (c) Recoverability measured directly from the learned representation versus its reconstruction from the paired-view spectral subspace; the dashed line denotes perfect agreement.

Figure 9: Captured posterior energy and few-shot NCC error. Results for VICReg-CelebA, VICReg-IM1K, and Barlow Twins-IM1K across ranks

r

and support sizes

m

. Dashed curves are the upper bound from Thm. 3.4 .

Figure 10: Multitask centroid geometry on CelebA. Held-out joint centroids and the hyperrectangles predicted by Thm. 3.5 for ImageNet-1K-pretrained VICReg and Barlow Twins. Experiment setup is same as Fig. 6

Bound	$\boldsymbol{m=1}$	$\boldsymbol{m=5}$	$\boldsymbol{m=10}$	$\boldsymbol{m=100}$	$\boldsymbol{m=500}$
( Galanti et al., 2023 )	2923.1	709.9	433.2	184.3	162.1
( Luthra et al., 2025 )	138.1	32.1	20.6	6.70	3.53
( Luthra et al., 2026 )	1544.4	282.5	126.7	10.27	2.05
Our bound	0.490	0.253	0.223	0.190	0.161

Appendix

Table 1: Comparison of few-shot transfer bounds on CelebA. Bounds on NCC classification error for W-MSE trained on CelebA (“Male/Female” attribute) . Our bound is minimized over the evaluated ranks separately for each shot count

m

. Prior bounds use

\ell_{2}

-normalized full-dimensional features, whereas ours uses whitened top-

r

features. Values below

0.5

are non-vacuous relative to chance classification.

Figure 11: Semantic recoverability on dSprites and 3DShapes. Task-specific captured posterior energy

B_{r}^{(t)}

across representation ranks for ImageNet-1K-pretrained VICReg, I-JEPA, and Barlow Twins.

Figure 12: Predicted versus observed directional CDNV on synthetic datasets. Results for VICReg, I-JEPA, and Barlow Twins on dSprites and 3DShapes. Each point is one attribute at one rank

r

; the prediction is computed on split A and the observed directional CDNV on held-out split B. Dashed lines show the theoretical prediction

y=x

Figure 13: Multitask centroid geometry on synthetic datasets. Held-out joint centroids compared with the hyperrectangles predicted by Thm. 3.5 .

Explore similar work

Sharp Rates and a One-Line Correction for Spectral Representation Learning
Sep 14, 2026Dier Tang, Jing Yee Tan, Guangyue HanSelf-Supervised Representations Subspace
Department of Mathematics, The University of Hong Kong
Three Necessary Principles for Self-Supervised Visual Representation Learning
Aug 8, 2026Nikos Giakoumoglou, Paschalis Giakoumoglou, Tania StathakiSelf-Supervised Representations Self-Supervised Learning
Imperial College London · CERTH, ITI
Learning to Perceive "Where": Spatial Pretext Tasks for Robust Self-Supervised Learning
May 11, 2026Yang Shen, Yusen Cai, Weronika Hryniewska-Guzik +2Self-Supervised Learning Spatial Supervision
Nanyang Technological University, Singapore · Warsaw University of Technology, Poland

Sep 14, 2026, cs.LGJ/K move · Enter open · S save

Sharp Rates and a One-Line Correction for Spectral Representation Learning

Dier Tang, Jing Yee Tan, Guangyue Han

Department of Mathematics, The University of Hong Kong

Self-Supervised Representations Subspace Spurious Correlations Correction

Open paper AskPDF arXiv

A self-supervised encoder is trained once, frozen, and reused through lightweight probes on tasks nobody named at training time; the practitioner's question is when the off-the-shelf features are good enough and when they need fixing. Canonical correlation analysis, HGR maximal correlation, and the population optimum of the spectral contrastive loss all return the top-

k

singular subspace of a cross-view dependence operator, justified by isotropy: if the task prior has no directional preference, that subspace is universally optimal. We show isotropy is the wrong hypothesis. The prior enters the transfer risk only through the task covariance

Λ=\mathbb{E}[ΔΔ^\top]

, and only through its compression onto the operator's leading singular directions; what matters is not whether

Λ

is isotropic but whether its preferred directions are ordered consistently with the operator's spectrum. We prove matching two-sided rates---worst-case regret is exactly

1-1/κ(Λ)

, refines to

1-A_k

for an alignment coefficient

A_k

, localizes to the top-

2k

subspace, becomes second order under a spectral gap, and is improvable by no task-agnostic representation---and show why alignment is generic: incoherent preferences cancel in high dimension, and

T

diverse tasks force

α=\widetilde O(\sqrt{d_x/T})

, a quantitative account of why task diversity, not symmetry, makes self-supervised features transfer. The governing statistics cost

O(kd_x^2)

, and when they signal misalignment a one-line reweighting of the positive-pair term provably restores exact optimality. The result is a diagnostic that answers the practitioner's question from a small labelled budget and refuses when the task bank cannot support the width requested; on controlled data it takes a regret of

0.86

down to

0.003

, and on a CIFAR-100 encoder it correctly predicts that no correction is needed.

Abstract

Explore similar work

Sharp Rates and a One-Line Correction for Spectral Representation Learning

Three Necessary Principles for Self-Supervised Visual Representation Learning

Learning to Perceive "Where": Spatial Pretext Tasks for Robust Self-Supervised Learning