Neural Collapse

Neural collapse (NC) describes a surprising geometric structure consistently observed in the final layers of deep neural networks during training, where features cluster and align with classifier weights in a highly symmetric manner. Current research focuses on understanding NC's emergence across various architectures and tasks, including classification, regression, and even language modeling, often employing unconstrained feature models for theoretical analysis. This phenomenon has implications for improving model generalization, fairness, and robustness, as well as for developing novel training strategies and out-of-distribution detection methods.

Papers

June 8, 2022

Neural Collapse: A Review on Modelling Principles and Generalization
Vignesh Kothapalli
Neural Network Strong Generalization Narrative Review Neural Collapse Neural Network Classifier Layer Output Nearest Centroid

May 9, 2022

Posterior Collapse of a Linear Latent Variable Model
Zihao Wang, Liu Ziyin
Variational Autoencoders Neural Collapse Bayesian Deep Learning Dimensional Collapse Posterior Collapse

April 19, 2022

Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learning
Liang Xie, Yibo Yang, Deng Cai, Xiaofei He
Deep Learning Model Large Scale Class Imbalance Neural Collapse Imbalanced Learning

April 17, 2022

Turing's cascade instability supports the coordination of the mind, brain, and behavior
Damian G. Kelty-Stephen, Madhur Mangalam
Human Mind Brain Function Neural Collapse Prior Coordination TF Cascade Turing Machine Metaphor Generation

March 17, 2022

Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?
Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, Dacheng Tao
Deep Neural Network Neural Collapse Linear Classifier Annotated End State Imbalanced Learning Learning Classifier Frame Classification

March 2, 2022

On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, Zhihui Zhu
Cross Entropy Loss Neural Collapse Global Optimality Mean Squared Error Optimization Landscape Last Layer Unconstrained Feature Model

February 18, 2022

On the Implicit Bias Towards Minimal Depth of Deep Neural Networks
Tomer Galanti, Liane Galanti, Ido Ben-Shaul
Deep Neural Network Stochastic Gradient Descent Implicit Bias Neural Collapse Layer Representation

February 17, 2022

Limitations of Neural Collapse for Understanding Generalization in Deep Learning
Like Hui, Mikhail Belkin, Preetum Nakkiran
Deep Learning Strong Generalization Fundamental Limitation Neural Collapse Generalization Property

February 16, 2022

Extended Unconstrained Features Model for Exploring Deep Neural Collapse
Tom Tirer, Joan Bruna
Deep Neural Network Cross Entropy Loss Neural Collapse Cross Entropy Unconstrained Feature Model

February 14, 2022

NeuroMixGDP: A Neural Collapse-Inspired Random Mixup for Private Data Release
Donghao Li, Yang Cao, Yuan Yao
Differential Privacy Neural Topic Neural Collapse Mixup Method Data Release Local Mixup

January 21, 2022

Nearest Class-Center Simplification through Intermediate Layers
Ido Ben-Shaul, Shai Dekel
Deep Network Neural Collapse Geometric Feature Sentence Simplification Intermediate Layer Deep Learning Theory Variance Reduced Stochastic

December 30, 2021

On the Role of Neural Collapse in Transfer Learning
Tomer Galanti, András György, Marcus Hutter
Transfer Learning Integral Role Meaningful Representation Neural Collapse Classification Network