Representation Collapse

Follow this topic Ask about this topic

Representation collapse occurs when a model's features become too similar or occupy too few dimensions, making it harder to distinguish inputs or preserve useful information. It is studied in self-supervised vision and JEPA world models, as well as in transformers, recommendation, reinforcement learning, and continual learning. Researchers diagnose it with effective rank, feature variance, angular dispersion, and spectral or geometric analyses, and try to prevent it using covariance or isotropy regularization, contrastive objectives, and architectural changes to routing and residual connections. A central challenge is balancing diversity with useful invariance: high rank or good probe accuracy alone does not ensure that representations retain information needed for retrieval, transfer, or control.

Latest papers

47 papers8 papers in the last four weeks, up 167% on the four weeks before. 0.0% of all new papers.

LocaLeJEPA: Maximum Entropy Representations under Geometric Priors
Oct 20, 2026Kartik Tandon, Alessio Borgi, Pavel Snopov +8Representation Learning Joint-Embedding Predictive Architecture
University of Pennsylvania · Sapienza University of Rome · University of Cambridge +6
Representations Have Trajectories: Characterizing Temporal Representation Dynamics in Neural Networks
Oct 17, 2026Shivaprasad B. Gowda, Sparsh MittalLow-Rank Representation Learning Neural Representation Geometry
Indian Institute of Information Technology, Nagpur · Indian Institute of Technology, Roorkee
Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures
Oct 1, 2026José Lucas De Melo Costa, Seong Woo Ahn, Fabrice Popineau +2Joint-Embedding Predictive Architecture Representation Collapse
Université Paris-Saclay, CNRS, CentraleSupélec, LISN, Orsay, France
CF-JEPA: Improving Robustness of JEPA World Models via Controllability Factorization
Sep 30, 2026Morgan Byrd, Robert Wright, Sehoon HaRepresentation Learning Representation Collapse
Georgia Institute of Technology, Atlanta, GA, 30308, USA · Georgia Tech Research Institute, Atlanta, GA, 30308, USA
Collapse, Not Invariance: Diagnosing Auxiliary Objectives in Speech Anti-Spoofing
Sep 30, 2026Ksenia Lysikova, Kirill Borodin, Maxim Maslov +1Audio Deepfake Detection Representation Collapse
Lab260, Yerevan, Armenia · BitmanagerAI, Dubai, UAE · MTUCI, Moscow, Russia
Stable Transformers for Graph Generation
Sep 30, 2026Luca Miglior, Alessio Gravina, Davide BacciuRepresentation Collapse Graph Generation
Department of Computer Science, University of Pisa, Italy
Learning Macroscopic Dynamics without Reconstructing Microscopic States
Sep 29, 2026Zhichao Han, Yue Zhao, Qianxiao LiLatent Dynamics Modeling State Representation Learning
National University of Singapore
Generative Residual Factorization
Sep 28, 2026Letian Gong, Yuzhou HongRepresentation Collapse Latent Variable Models
Zhejiang University of Science and Technology · Zhejiang Sci-Tech University
The Price of Locality: Why Forward-Forward Underperforms Backpropagation?
Sep 27, 2026Zhaoxian Wu, Haichuan Liu, Tianyi ChenForward-Forward Learning Neural Network Optimization
Cornell Tech and Cornell University
Rank Collapse Is Recoverable, Growing $|Q|$ Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Critic
Sep 26, 2026Tianqi Bu, YuXuan Peng, Junteng Tu +1Off-Policy RL Soft Actor-Critic
AeroLat: Channel-Aware Latent Space Semantic Communication for Decentralized UAV Swarms
Sep 15, 2026Rajdeep Ghosh, Goparaju Venkata Seshachala Sree Vatsava, Sudip MisraSemantic Communication Wireless Communications
Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, Kharagpur 721302, India
When Graph-JEPA Learns the Wrong Thing: Diagnosing and Repairing Category-Conditional Collapse
Aug 20, 2026Gollam Rabby, Sören AuerRepresentation Collapse Joint-Embedding Predictive Architecture
L3S Research Centre, Leibniz University Hannover, Hannover, Germany · TIB Leibniz Information Centre for Science and Technology, Hannover, Germany
No Gaussian Required: Contrastive Inverse Dynamics for JEPA World Models
Aug 18, 2026Jack Boylan, Chris HokampJoint-Embedding Predictive Architecture Representation Collapse
Quantexa
Three Necessary Principles for Self-Supervised Visual Representation Learning
Aug 8, 2026Nikos Giakoumoglou, Paschalis Giakoumoglou, Tania StathakiContrastive Learning Representation Collapse
Imperial College London · CERTH, ITI
SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation
Jul 27, 2026Yu Cui, Yi Xu, Jiahao Wang +6Representation Collapse Transformer
Zhejiang University Hangzhou, China · Alibaba Group Beijing, China
The JEPA Paradox in Language: The Geometry of Linguistic Alternatives
Jul 26, 2026Anh Trac Duc Dinh, Khang Nhat Hoang VoRepresentation Collapse Masked Language Modeling
Center for AI Research (CAIR), VinUniversity, Hanoi, Vietnam · Faculty of Computer Science and Engineering Ho Chi Minh City University of Technology (HCMUT), VNU-HCM Ho Chi Minh City, Vietnam · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates
Depth-Dependent Hidden-State Collapse in Dynamical System Autoencoders for LiDAR Point-Cloud Classification
Jul 16, 2026Patricia Medina, Hy P. G. LamRepresentation Collapse Latent Dynamics Modeling
Department of Mathematics New York City College of Technology, CUNY Brooklyn, NY, USA · The Graduate Center, CUNY New York, NY, USA · Department of Mathematical Sciences Worcester Polytechnic Institute Worcester, MA, USA
How to Tame Grokking: Representation Geometry as a Control Signal
Jul 13, 2026Maksim A KazanskiiNeural Network Generalization Spectral Regularization
Independent Researcher
Escaping the Procrustean Bed: Groupwise Orthogonal Connectors for Audio-Language Models
Jul 7, 2026Ho-Lam Chung, Ke-Han Lu, Yi-Cheng Lin +3Audio Representation Learning Representation Learning
Graduate Institute of Communication Engineering, National Taiwan University, Taipei, Taiwan · ASUS AICS, Taipei, Taiwan
All Routes Lead to Collapse
Jun 21, 2026K. R. BalasubramanianRepresentation Collapse Attention Sinks
Independent Research
Template Collapse and Information-Theoretic Limits in Camera rPPG Pulse Morphology Restoration
Jun 2, 2026Achraf Ben AhmedRemote Photoplethysmography Representation Collapse
PlesmoSense SARL
VISReg: Variance-Invariance-Sketching Regularization for JEPA training
Jun 1, 2026Haiyu Wu, Randall Balestriero, Morgan LevineRepresentation Collapse Joint-Embedding Predictive Architecture
Altos Labs · Brown University
Representation Collapse in Sequential Post-Training of Large Language Models
May 28, 2026Yichen Liu, Mingyu Chen, Hao Wang +7Continual Learning for LLMs Representation Collapse
Hangzhou Dianzi University · Zhejiang Gongshang University · Ningbo University +1
Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation
May 22, 2026Guoming Li, Shangyu Zhang, Junwei Pan +7Representation Collapse Recommender Systems
The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China · Tencent Inc. Shenzhen, China
How Well Do Large Language Models Capture Human Personality?
May 12, 2026Aanisha Bhattacharyya, Yaman Kumar Singla, Rajiv Ratn Shah +2Persona-Conditioned Generation Representation Collapse
Adobe Media and Data Science Research (MDSR) · IIIT-Delhi · SUNY at Buffalo
Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets
May 11, 2026Thuan Nguyen, Shuchin Aeron, D. Richard Brown +1Class-Imbalanced Learning Contrastive Learning
Department of Engineering, Engineering Technology East Tennessee State University · Department of Electrical and Computer Engineering Tufts University · Department of Electrical and Computer Engineering Worcester Polytechnic Institute +1
Sub-JEPA: Subspace Gaussian Regularization for Stable End-to-End World Models
May 10, 2026Kai Zhao, Dongliang Nie, Yuchen Lin +4Representation Learning Representation Collapse
1Shanghai University · 2The University of Manchester · 3Nankai University
Enhancing Federated Quadruplet Learning: Stochastic Client Selection and Embedding Stability Analysis
May 8, 2026Ozgu Goksu, Nicolas PugeaultRepresentation Collapse Non-IID Federated Learning
School of Computing Science University of Glasgow
Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers
May 7, 2026Pengqi LuDeep Learning Optimization Representation Collapse
Beijing, China
Lost in State Space: Probing Frozen Mamba Representations
Apr 30, 2026Bhagyashree Wagh, Akash SinghRepresentation Geometry Representation Collapse
University of Washington · University of Florida

Oct 20, 2026J/K move · Enter open · S save

LocaLeJEPA: Maximum Entropy Representations under Geometric Priors

Kartik Tandon, Alessio Borgi, Pavel Snopov, Inés García-Redondo+7

University of Pennsylvania · Sapienza University of Rome · University of Cambridge+6

Representation Learning Joint-Embedding Predictive Architecture Self-Supervised Visual Representation Learning Geometric Representation Learning Representation Collapse

Open paper AskPDF arXiv

Joint-Embedding Predictive Architectures (JEPA) offer an appealing approach to self-supervised learning, but are typically prone to representation collapse. LeJEPA resolves this by introducing a theoretically grounded distribution-matching objective that drives the embedding towards an isotropic Gaussian. However, this approach is fundamentally tied to Euclidean assumptions, and its global target does not reflect domain-specific geometric priors, such as the use of non-Euclidean embedding geometries, e.g., equivariant CNNs or hyperbolic ViTs. We thus introduce LocaLeJEPA, a framework motivated by a manifold-theoretic reformulation of LeJEPA's entropy-maximization property. The accompanying LocaLeReg, a novel locally Euclidean distribution-matching regularizer, enables a stable and scalable implementation of the LocaLeJEPA framework. We then perform extensive experiments in the vision setting on models trained on ImageNet-1K. LocaLeJEPA significantly outperforms LeJEPA when paired with hyperbolic ViT-S/8 and equivariant ResNet-50 backbones, and also with ordinary Euclidean ViT-S/8 and ViT-L/14 backbones. This holds both in terms of linear and k-NN classification performance on ImageNet and on a series of downstream tasks tested. We view the LocaLeJEPA formalism as a principled approach to the JEPA paradigm that better respects intrinsic data geometry while also accommodating generalizations to exotic embedding geometries. Code available at https://github.com/alessioborgi/LocaLeJEPA.