Representation Collapse

Ask about this topic

Overview

Representation collapse occurs when a model's features become too similar or occupy too few dimensions, making it harder to distinguish inputs or preserve useful information. It is studied in self-supervised vision and JEPA world models, as well as in transformers, recommendation, reinforcement learning, and continual learning. Researchers diagnose it with effective rank, feature variance, angular dispersion, and spectral or geometric analyses, and try to prevent it using covariance or isotropy regularization, contrastive objectives, and architectural changes to routing and residual connections. A central challenge is balancing diversity with useful invariance: high rank or good probe accuracy alone does not ensure that representations retain information needed for retrieval, transfer, or control.

Latest papers

CardsList

47 papers8 papers in the last four weeks, up 167% on the four weeks before. 0.0% of all new papers.

  1. Why Self-Supervised Encoders Want to Be Normal

    Apr 30, 2026Yuval DombRepresentation CollapseSelf-Supervised Learning

  2. Monitoring Neural Training with Topology: A Footprint-Predictable Collapse Index

    Apr 28, 2026Alexander KalinowskiRepresentation CollapseNeural Representation Geometry

    SUNY Empire

  3. Complexity of Linear Regions in Self-supervised Deep ReLU Networks

    Apr 27, 2026Mufhumudzi Muthivhi, Terence L. van ZylRepresentation CollapseSelf-Supervised Visual Representation Learning

    University of Johannesburg, Johannesburg, South Africa

  4. Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers

    Apr 26, 2026Giansalvo CirrincioneTransformer InterpretabilityRepresentation Collapse

    aLaboratoire LTI, Université de Picardie Jules Verne, Chemin du Thil, 80025 Amiens, France

  5. A Minimal Model of Representation Collapse: Frustration, Stop-Gradient, and Dynamics

    Apr 11, 2026Louie Hong Yao, Yuhao Li, Shengchao LiuRepresentation CollapseSelf-Supervised Learning

    Center for Soft Matter and Biological Physics, Virginia Tech, Blacksburg, VA 24061, USA · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China

  6. Homogenized Transformers

    Apr 2, 2026Hugo Koubbi, Borjan Geshkovski, Philippe RigolletTransformerMulti-Head Attention

  7. Pooling and Semantic Shift: The Fundamental Challenges in Long Text Embedding and Retrieval

    Mar 22, 2026Hang Gao, Wujiang Xu, Kai Mei +1Text EmbeddingsText Embedding Models

    Rutgers University

  8. Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization

    Mar 17, 2026Wenhao Zhao, Qiran Zou, Rushi Shah +3Vector QuantizationRepresentation Collapse

    National University of Singapore · Zhejiang University · Shanghai Jiao Tong University

  9. IConE: Batch Independent Collapse Prevention for Self-Supervised Representation Learning

    Mar 16, 2026Konstantinos Almpanakis, Anna KreshukSelf-Supervised LearningRepresentation Collapse

    European Molecular Biology Laboratory, Heidelberg, Germany

  10. Anchored Alignment: Preventing Positional Collapse in Multimodal Recommender Systems

    Mar 13, 2026Yonghun Jeong, David Yoon Suk Kang, Yeon-Chang LeeMultimodal RecommendationCross-Modal Alignment

    UNIST, Ulsan, Korea · Chungbuk National University, Cheongju, Korea

  11. Weak-SIGReg: Covariance Regularization for Stable Deep Learning

    Mar 6, 2026Habibullah AkbarNeural Network OptimizationRepresentation Collapse

    Kreasof AI, Jakarta, Indonesia

  12. Why Do Neural Networks Forget: A Study of Collapse in Continual Learning

    Mar 4, 2026Yunqin Zhu, Jun JinCatastrophic ForgettingIncremental Learning

    Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada

  13. Stable Deep Reinforcement Learning via Isotropic Gaussian Representations

    Feb 22, 2026Ali Saheb Pasand, Johan Obando-Ceron, Aaron Courville +2RL StabilityRepresentation Learning for RL

    Mila – Quebec AI Institute · McGill · Université de Montréal +1

  14. Representation Collapse in Machine Translation Through the Lens of Angular Dispersion

    Feb 19, 2026Evgeniia Tokarchuk, Maya K. Nachesa, Sergey Troshin +1Representation CollapseMachine Translation

    Language Technology Lab, University of Amsterdam

  15. The Geometric Anatomy of Capability Acquisition in Transformers

    Feb 17, 2026Jayadev BillaRepresentation Geometry in Language ModelsRepresentation Collapse

    San Jose, CA, USA

  16. Hypersolid: Emergent Vision Representations via Short-Range Repulsion

    Jan 29, 2026Esteban Rodríguez-Betancourt, Edgar Casasola-MurilloUnsupervised LearningRepresentation Learning

    Posgrado en Computación e Informática, Universidad de Costa Rica, San José, Costa Rica · Escuela de Ciencias de la Computación e Informática, Universidad de Costa Rica, San José, Costa Rica

  17. You Do Not Fully Utilize Transformer's Representation Capacity

    Feb 13, 2025Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky +2Representation CollapseMemory-Augmented Language Models

    1T-Tech · 2Moscow Institute of Physics and Technology · 3HSE University