Representation Collapse
Overview
Representation collapse occurs when a model's features become too similar or occupy too few dimensions, making it harder to distinguish inputs or preserve useful information. It is studied in self-supervised vision and JEPA world models, as well as in transformers, recommendation, reinforcement learning, and continual learning. Researchers diagnose it with effective rank, feature variance, angular dispersion, and spectral or geometric analyses, and try to prevent it using covariance or isotropy regularization, contrastive objectives, and architectural changes to routing and residual connections. A central challenge is balancing diversity with useful invariance: high rank or good probe accuracy alone does not ensure that representations retain information needed for retrieval, transfer, or control.
Latest papers
54 papers5 papers in the last four weeks, down 44% on the four weeks before. 0.0% of all new papers.