Dimensional Collapse

Dimensional collapse, a phenomenon where learned feature representations in machine learning models occupy a lower-dimensional subspace than expected, hinders model performance and generalization. Current research focuses on mitigating this issue across various model types, including contrastive learning, deep metric learning, and self-supervised learning, employing techniques like anti-collapse loss functions, stain normalization, and feature decorrelation to improve representation quality. Addressing dimensional collapse is crucial for enhancing the performance and robustness of machine learning models in diverse applications, ranging from medical image analysis and object detection to natural language processing and graph representation learning.

Papers

November 1, 2024

Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization
Junlin He, Jinxiao Du, Wei Ma
Self Supervised Learning Self Supervised Hidden Knowledge Weight Matrix Dimensional Collapse Orthogonality Regularization

October 30, 2024

HEX: Hierarchical Emergence Exploitation in Self-Supervised Algorithms
Kiran Kokilepersaud, Seulgi Kim, Mohit Prabhushankar, Ghassan AlRegib
Supervised ImageNet Dimensional Collapse Emergence Dynamic

September 26, 2024

Task-recency bias strikes back: Adapting covariances in Exemplar-Free Class Incremental Learning
Grzegorz Rypeść, Sebastian Cygert, Tomasz Trzciński, Bartłomiej Twardowski
Feature Extractor Covariance Conditioning Event Collapse Exemplar Free Class Incremental Learning Dimensional Collapse Recency Bias

August 1, 2024

Enhancing Whole Slide Pathology Foundation Models through Stain Normalization
Juseung Yun, Yi Hu, Jinhyung Kim, Jongseong Jang, Soonyoung Lee
Digital Pathology Self Supervised Model Stain Normalization Dimensional Collapse Pathology Foundation Model

July 3, 2024

Anti-Collapse Loss for Deep Metric Learning Based on Coding Rate Metric
Xiruo Jiang, Yazhou Yao, Xili Dai, Fumin Shen, Xian-Sheng Hua, Heng-Tao Shen
Loss Function Deep Metric Learning Event Collapse Dimensional Collapse Rate Reduction

April 18, 2024

Measuring Feature Dependency of Neural Networks by Collapsing Feature Dimensions in the Data Manifold
Yinzhu Jin, Matthew B. Dwyer, P. Thomas Fletcher
Neural Network Data Manifold Dimensional Collapse Question Based Dependency Hippocampus Segmentation Feature Dimension

March 20, 2024

Bounding Box Stability against Feature Dropout Reflects Detector Generalization across Environments
Yang Yang, Wenhai Wang, Zhe Chen, Jifeng Dai, Liang Zheng
Environment Feature Bounding Box Dimensional Collapse Canonical Image

February 22, 2024

Overcoming Dimensional Collapse in Self-supervised Contrastive Learning for Medical Image Segmentation
Jamshid Hassanpour, Vinkle Srivastav, Didier Mutter, Nicolas Padoy
Contrastive Learning Self Supervised Medical Image Segmentation Local Feature Self Supervised Contrastive Learning Medical Segmentation Dimensional Collapse

February 14, 2024

WERank: Towards Rank Degradation Prevention for Self-Supervised Learning Using Weight Regularization
Ali Saheb Pasand, Reza Moravej, Mahdi Biparva, Ali Ghodsi
Self Supervised Learning Representation Space Dimensional Collapse Weight Regularization

November 9, 2023

Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapse
Ruijie Jiang, Thuan Nguyen, Shuchin Aeron, Prakash Ishwar
Contrastive Learning Negative Sampling Neural Collapse Supervised Contrastive Learning Adam Optimizer unsupeRvised Contrastive Learning Feature Normalization Dimensional Collapse

October 23, 2023

Air-Decoding: Attribute Distribution Reconstruction for Decoding-Time Controllable Text Generation
Tianqi Zhong, Quan Wang, Jingxuan Han, Yongdong Zhang, Zhendong Mao
Language Model Text Generation Accurate Decoding Dimensional Collapse Attribute Distribution

May 26, 2023

Modulate Your Spectrum in Self-Supervised Learning
Xi Weng, Yunhao Ni, Tengwei Song, Jie Luo, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan, Lei Huang
Self Supervised Learning Structured Output Time to Spectrum Dimensional Collapse Hard Whitening Approach

May 25, 2023

Feature Collapse
Thomas Laurent, James H. von Brecht, Xavier Bresson
Strong Generalization NLP Task Local Feature Similar Representation Dimensional Collapse

March 27, 2023

Dimensionality Collapse: Optimal Measurement Selection for Low-Error Infinite-Horizon Forecasting
Helmuth Naumer, Farzad Kamalabadi
State Estimation Language Correction Dimensional Collapse Optimal Measurement Sequential Experimental Design Forecasting Algorithm

March 22, 2023

Preventing Dimensional Collapse of Incomplete Multi-View Clustering via Direct Contrastive Learning
Kaiwu Zhang, Shiqiang Du, Baokai Liu, Shengxia Gao
Contrastive Learning Multi View Clustering Dimensional Collapse Uncertainty Induced Incomplete Incomplete Multi View Clustering Latent Subspace Cross View Prediction

March 4, 2023

Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism
Zhijian Zhuo, Yifei Wang, Jinwen Ma, Yisen Wang
Theoretical Understanding Unified Alignment Dimensional Collapse Non Contrastive Learning

December 23, 2022

Generalization Bounds for Few-Shot Transfer Learning with Pretrained Classifiers
Tomer Galanti, András György, Marcus Hutter
Transfer Learning Generalization Bound Feature Embeddings Unseen Class Dimensional Collapse Pre Trained Classifier

December 16, 2022

Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learning
Alex Tamkin, Margalit Glasgow, Xiluo He, Noah Goodman
Contrastive Learning Integral Role Soft Augmentation Dimensional Collapse Label Invariant Augmentation General Purpose Representation

October 7, 2022

An Investigation into Whitening Loss for Self-supervised Learning
Xi Weng, Lei Huang, Lei Zhao, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan
Self Supervised Learning Jina Embeddings Dimensional Collapse ImageNet Classification Hard Whitening Approach

October 2, 2022

What shapes the loss landscape of self-supervised learning?
Liu Ziyin, Ekdeep Singh Lubana, Masahito Ueda, Hidenori Tanaka
Native Robustness Self Supervised Learning Inherent Interpretability Loss Landscape Dimensional Collapse

Dimensional Collapse

Papers

Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization

HEX: Hierarchical Emergence Exploitation in Self-Supervised Algorithms

Task-recency bias strikes back: Adapting covariances in Exemplar-Free Class Incremental Learning

Enhancing Whole Slide Pathology Foundation Models through Stain Normalization

Anti-Collapse Loss for Deep Metric Learning Based on Coding Rate Metric

Measuring Feature Dependency of Neural Networks by Collapsing Feature Dimensions in the Data Manifold

Bounding Box Stability against Feature Dropout Reflects Detector Generalization across Environments

Overcoming Dimensional Collapse in Self-supervised Contrastive Learning for Medical Image Segmentation

WERank: Towards Rank Degradation Prevention for Self-Supervised Learning Using Weight Regularization

Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapse

Air-Decoding: Attribute Distribution Reconstruction for Decoding-Time Controllable Text Generation

Modulate Your Spectrum in Self-Supervised Learning

Feature Collapse

Dimensionality Collapse: Optimal Measurement Selection for Low-Error Infinite-Horizon Forecasting

Preventing Dimensional Collapse of Incomplete Multi-View Clustering via Direct Contrastive Learning

Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism

Generalization Bounds for Few-Shot Transfer Learning with Pretrained Classifiers

Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learning

An Investigation into Whitening Loss for Self-supervised Learning

What shapes the loss landscape of self-supervised learning?