Variability Collapse

Variability collapse in neural networks refers to the phenomenon where the variability of learned features within classes diminishes significantly during training, leading to a highly structured, often simplified, representation of data. Current research focuses on understanding the mechanisms driving this collapse, particularly in deep neural networks, using models like recursive feature machines and investigating algorithms such as Stein Variational Gradient Descent (SVGD) and its variants to mitigate issues like variance collapse and improve model robustness. This research is crucial for improving the generalizability, transferability, and robustness of deep learning models, with implications for applications ranging from machine unlearning to brain-machine interfaces. Addressing variability collapse is key to building more reliable and efficient AI systems.

Papers

October 7, 2024

Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse
Arthur Jacot, Peter Súkeník, Zihan Wang, Marco Mondelli
Deep Neural Network Neural Collapse Wide Neural Network DNN Training Unconstrained Feature Model Variability Collapse

September 23, 2024

Distribution-Level Feature Distancing for Machine Unlearning: Towards a Better Trade-off Between Model Utility and Forgetting
Dasol Choi, Dongbin Na
Machine Unlearning Deep Learning Application Model Improvement Face Image Datasets Variability Collapse

June 17, 2024

Long-time asymptotics of noisy SVGD outside the population limit
Victor Priser, Pascal Bianchi, Adil Salim
Stein Variational Gradient Descent Population Size Variability Collapse Feature Noise

February 21, 2024

Average gradient outer product as a mechanism for deep neural collapse
Daniel Beaglehole, Peter Súkeník, Marco Mondelli, Mikhail Belkin
Deep Neural Network Neural Collapse Functional Mechanism Auto Encoder Input Gradient Variability Collapse

January 1, 2024

Learn to integrate parts for whole through correlated neural variability
Zhichao Zhu, Yang Qi, Wenlian Lu, Jianfeng Feng
Computational Neuroscience Part Identification Perceptual Information Covariance Neural Network Neural Code Variability Collapse

June 6, 2023

Quantifying the Variability Collapse of Neural Networks
Jing Xu, Haoxiong Liu
Neural Network Neural Collapse Variability Collapse

May 23, 2023

Toward stochastic neural computing
Yang Qi, Zhichao Zhu, Yiming Wei, Lu Cao, Zhigang Wang, Jie Zhang, Wenlian Lu, Jianfeng Feng
Neuromorphic Hardware Neural Activity Bayesian Computation Stochastic Neural Network Variability Collapse

May 18, 2023

Augmented Message Passing Stein Variational Gradient Descent
Jiankui Zhou, Yue Qiu
Stein Variational Gradient Descent Particle Method Variability Collapse

December 23, 2022

Principled and Efficient Transfer Learning of Deep Models via Neural Collapse
Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda, Zhihui Zhu, Qing Qu
Fine Tuning Transfer Learning Deep Model Large Pre Trained Model Neural Collapse Efficient Transfer Learning Variability Collapse

November 15, 2022

REPAIR: REnormalizing Permuted Activations for Interpolation Repair
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, Behnam Neyshabur
Third Position Repair Variability Collapse Linear Connectivity

August 23, 2022

A Stochastic Variance Reduced Gradient using Barzilai-Borwein Techniques as Second Order Information
Hardik Tankaria, Nobuo Yamashita
Natural Gradient Stochastic Gradient Second Order Barzilai Borwein Technique Variability Collapse Efficient Expert

April 22, 2022

Dynamic Ensemble Bayesian Filter for Robust Control of a Human Brain-machine Interface
Yu Qi, Xinyun Zhu, Kedi Xu, Feixiao Ren, Hongjie Jiang, Junming Zhu, Jianmin Zhang, Gang Pan, Yueming Wang
Neural Network Controller Robust Control Ensemble Kalman Filter Neural Signal Brain Machine Interface Variability Collapse