cs.RONov 16, 2026

SADL: On the Demand-Supply Geometry of Representation Collapse

Authors: Junhan Sun, Yigui Feng, Mingzhe Liu, Haoyan Xu, Lei Ye, Ziqi Han, Hanru Bai, Yanzhao Zhang, 

Organizations: State Key Laboratory of CAD&CG, Zhejiang University · RoboParty Lab · Alibaba Group · AIR, Tsinghua University · 

Abstract

In Joint-Embedding Predictive Architectures (JEPAs), a well-constrained projector output does not by itself prevent representation collapse. We find that, from initialization, the encoder representation is dominated by a component shared by different inputs. A batch-normalized projector removes this component, so regularization after the projector cannot correct it, and the differences between representations shrink until they collapse. We propose SADL (Supply And Demand Loss), which keeps the original training objective and constrains the encoder directly when needed. SADL compares the representation differences that the projector output demands with the differences the encoder can retain and pass on to the projector, and adds a constraint on the encoder only when supply falls short of demand. It works with different Demand Regularizers and introduces no auxiliary networks, additional trainable parameters, or architectural changes. In single-task and jointly trained four-task LeWorldModel (LeWM) and INTACT experiments, SADL lowers the collapse rate over ten random seeds from 50% to 0%. After one epoch of training, encoder effective rank is 3.4-6.4 times that of the corresponding baseline without SADL, and latent prediction error falls to 1/20-1/303 of it. Without additional tuning, SADL stays robust as its loss weight spans orders of magnitude, and with several Demand Regularizers INTACT exceeds 95% mean search-free control success. On four image datasets, SADL also consistently improves accuracy and stability across random seeds. These results show that matching the representation differences a regularizer demands with the differences the encoder can supply leads to faster and more stable representation learning in control and image classification.

Figures & tables

Appendix figures & tables49 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

May 31, 2026

UR-JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architectures

A central difficulty in training Joint-Embedding Predictive Architectures (JEPAs) is preventing representation collapse. LeJEPA addresses this by enforcing an isotropic Gaussian target on the embeddings via Sketched Isotropic Gaussian Regularization (SIGReg). This target is in tension with the manifold hypothesis, which expects embeddings to concentrate on a low-dimensional subset of the ambient space. We propose UR-JEPA, which targets a uniformly nn-rectifiable measure of local tangent dimension nn at small scales, realized through a Gaussian-kernel smoothed Carleson-type square function LCGLT\mathcal{L}^{\text{CGLT}}, with a complementary Jones ββ-number formulation. On Inet10, UR-JEPA(LCGLT\mathcal{L}^{\text{CGLT}}) attains 0.9141±0.00140.9141 \pm 0.0014 for a +0.83+0.83,pp gain over LeJEPA(LSIGReg\mathcal{L}^{\text{SIGReg}}) with ∼30%\sim 30\% lower seed standard deviation; on matched-recipe Galaxy10~SDSS, a single-seed ImageNet-100100 run, and a 33-seed EuroSAT remote-sensing run, the two methods lie in the same peak-accuracy band at convergence, with UR-JEPA retaining its lower-seed-variance signature. On EuroSAT the in-domain pair is competitive at 96.096.0 to 96.1%96.1\% with large remote-sensing foundation-model transfer at a 25×25\times smaller backbone. The distinction is geometric: direct visualization of the projector output distribution shows that on all four datasets UR--JEPA(LCGLT\mathcal{L}^{\text{CGLT}}) produces a global PCA spectrum with a 44 to 55 order-of-magnitude drop at index ∼20\sim 20 to 2525 out of D=32D = 32, while LeJEPA's spectrum is near-flat (top-to-bottom ratio at most 3.63.6). Per-dimension marginals are simultaneously near-Gaussian for both methods (mean Shapiro-Wilk W∈[0.992,0.996]W \in [0.992, 0.996]) as a Diaconis-Freedman consequence. At matched accuracy the two regularizers therefore yield structurally distinct projected representations.
Oct 1, 2026

Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures

Joint-Embedding Predictive Architectures (JEPAs) are prone to representation collapse, typically mitigated through empirical heuristics. We develop an early-training stability theory that unifies these heuristics. Linearising the coupled JEPA gradient flow around the trivial fixed point reveals two competing effects: a driving force (γγ) and a decay effect (σσ). Under approximate spectral decoupling, a per-mode stability ratio μi=γi/σiμ_i = γ_i / σ_i factorises into independent data-side and predictor-side terms and the count of unstable modes tracks the rank of representations that can emerge. The framework predicts a phase boundary, which we confirm empirically across more than 800 Tabular-JEPA configurations. It also unifies predictor scaling, masking ratio, and EMA as distinct mechanisms for shifting μμ. Guided by this analysis, we introduce ResidualPred, a transformer predictor whose attention is biased toward the identity at initialisation; it improves both effective rank and downstream accuracy on tabular benchmarks and in I-JEPA pretraining on CIFAR-10, CIFAR-100, STL-10, and ImageNet. Our framework connects empirical collapse-avoidance heuristics to an explicit dynamical picture, yielding theory-driven stabilizers. Code is available at https://github.com/jose-melo/drive-vs-decay.
Sep 28, 2026

λλ-JEPA Spectral Anti-Collapse Regularization for Self-Supervised Learning

Joint-embedding self-supervised learning typically combines an invariance objective across augmented views with additional mechanisms to prevent representational collapse. These objectives are often applied after a projection head, while downstream tasks use the backbone representation before the projector. We find that this mismatch does not necessarily prevent dimensional collapse in the backbone, which can retain low effective rank and potentially limit downstream transfer. To address this, we introduce SACReg, a spectral anti-collapse regularizer motivated by an analysis of λλ-balance, which captures the relative scale of weight matrices across layers. In a two-layer linear network, we show that (i) λλ-balance prevents collapse, and (ii) our regularizer applied to the backbone induces λλ-balance. In the nonlinear case, this regularizer leads to anti-collapse as well and, in realistic architectures on ImageNet100, it empirically increases the representations' ranks. We apply SACReg to JEPA and propose λλ-JEPA, which improves over LeJEPA and VISReg on ImageNet-1k classification and in average linear-probe transfer performance across eight downstream image datasets. On video self-supervised learning, λλ-JEPA improves over LeVJEPA and V-JEPA 2 on the Something-Something-v2 and Kinetics-400 benchmarks. Code is available at https://github.com/berkerdemirel/lambda-jepa.