Neural Collapse

Neural collapse (NC) describes a surprising geometric structure consistently observed in the final layers of deep neural networks during training, where features cluster and align with classifier weights in a highly symmetric manner. Current research focuses on understanding NC's emergence across various architectures and tasks, including classification, regression, and even language modeling, often employing unconstrained feature models for theoretical analysis. This phenomenon has implications for improving model generalization, fairness, and robustness, as well as for developing novel training strategies and out-of-distribution detection methods.

Papers

May 19, 2023

Towards understanding neural collapse in supervised contrastive learning with the information bottleneck method
Siwei Wang, Stephanie E Palmer
Deep Neural Network Contrastive Learning Strong Generalization Contrastive Loss Information Bottleneck Neural Collapse Supervised Contrastive Learning

April 18, 2023

A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry and Generalization
Peifeng Gao, Qianqian Xu, Peisong Wen, Huiyang Shao, Zhiyong Yang, Qingming Huang
Strong Generalization Study Feature Neural Collapse Cyclic Symmetry Grassmann Manifold

March 27, 2023

Neural Collapse Inspired Federated Learning with Non-iid Data
Chenxi Huang, Liang Xie, Yibo Yang, Wenxiao Wang, Binbin Lin, Deng Cai
Deep Learning Model Neural Collapse Non IID Data Global Structure Faster Convergence Speed

March 17, 2023

No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier
Zexi Li, Xinyi Shang, Rui He, Tao Lin, Chao Wu
Neural Collapse Optimal Classifier FEAR Speech Biased Classifier

March 11, 2023

Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap
Weiyang Liu, Longhui Yu, Adrian Weller, Bernhard Schölkopf
Strong Generalization Neural Collapse Class Separability HyperSpherical Energy

March 10, 2023

Inducing Neural Collapse to a Fixed Hierarchy-Aware Frame for Reducing Mistake Severity
Tong Liang, Jim Davis
Deep Neural Network Neural Collapse Error Mitigation Intra Class

February 24, 2023

Inducing Neural Collapse in Deep Long-tailed Learning
Xuantong Liu, Jianfeng Zhang, Tianyang Hu, He Cao, Lujia Pan, Yuan Yao
Neural Collapse Balanced Dataset Class Imbalanced Data Balanced Distribution

February 6, 2023

Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning
Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, Dacheng Tao
Class Incremental Learning Neural Collapse Prototype Classifier Classifier Alignment

January 3, 2023

Understanding Imbalanced Semantic Segmentation Through Neural Collapse
Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi, Xiangyu Zhang, Jiaya Jia
Semantic Segmentation Neural Collapse Semantic Segmentation Benchmark Last Layer Class Imbalanced Feature Richness Enhancement

January 1, 2023

Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data
Hien Dang, Tho Tran, Stanley Osher, Hung Tran-The, Nhat Ho, Tan Nguyen
Neural Collapse Imbalanced Data Deep Linear Global Minimum Balanced Approach Layer Output

December 23, 2022

Principled and Efficient Transfer Learning of Deep Models via Neural Collapse
Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda, Zhihui Zhu, Qing Qu
Fine Tuning Transfer Learning Deep Model Large Pre Trained Model Neural Collapse Efficient Transfer Learning Variability Collapse

October 29, 2022

Perturbation Analysis of Neural Collapse
Tom Tirer, Haoxiang Huang, Jonathan Niles-Weed
Deep Neural Network Gradient Flow Neural Collapse Deep Layer Perturbation Analysis Practical Deep

October 27, 2022

Class Based Thresholding in Early Exit Semantic Segmentation Networks
Alperen Görmez, Erdem Koyuncu
Semantic Segmentation Model Neural Collapse True Class Automatic Thresholding Early Exit Early Exit Neural Network Masking Rate

October 4, 2022

Are All Losses Created Equal: A Neural Collapse Perspective
Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, Zhihui Zhu
Loss Function Neural Collapse Cross Entropy Planning Loss

September 27, 2022

FG-UAP: Feature-Gathering Universal Adversarial Perturbation
Zhixing Ye, Xinwen Cheng, Xiaolin Huang
Adversarial Example Neural Collapse Universal Adversarial Perturbation

September 19, 2022

Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold
Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, Qing Qu
Deep Network Geometric Analysis Neural Collapse Feature Normalization Riemannian Geometry Approach

September 17, 2022

Linking Neural Collapse and L2 Normalization with Improved Out-of-Distribution Detection in Deep Neural Networks
Jarrod Haas, William Yolland, Bernhard Rabus
Deep Neural Network Distribution Detection Neural Collapse Normalization Dictionary ResNet Based Deep Uncertainty

August 10, 2022

Imbalance Trouble: Revisiting Neural-Collapse Geometry
Christos Thrampoulidis, Ganesh R. Kini, Vala Vakilian, Tina Behnia
Cross Entropy Loss Neural Collapse Group Imbalance Class Embeddings

August 4, 2022

Feature selection with gradient descent on two-layer networks in low-rotation regimes
Matus Telgarsky
Gradient Descent Feature Selection Gradient Flow Neural Collapse Margin Maximization Two Layer ReLU Low Data Regime Two Layer Network

June 11, 2022

Memorization-Dilation: Modeling Neural Collapse Under Label Noise
Duc Anh Nguyen, Ron Levie, Julian Lienen, Gitta Kutyniok, Eyke Hüllermeier
Deep Neural Network Label Noise Neural Collapse Cross Entropy Limited Memorization Regularization Effect Unconstrained Feature Model