Unconstrained Feature Model

Unconstrained feature models (UFMs) are simplified analytical frameworks used to understand "neural collapse" (NC), a phenomenon where deep neural networks, during training, exhibit highly structured feature representations at convergence. Current research focuses on extending UFMs to analyze NC in deeper, non-linear networks and under various loss functions (e.g., mean squared error, cross-entropy), investigating the influence of data properties (e.g., imbalance, noise) and network architecture on NC's emergence. Understanding NC through UFMs offers valuable insights into the optimization landscape of deep learning, potentially leading to improved training strategies and a deeper understanding of generalization capabilities.

Papers

October 30, 2024

The Persistence of Neural Collapse Despite Low-Rank Bias: An Analytic Perspective Through Unconstrained Features
Connall Garrod, Jonathan P. Keating
Absolute Stance Bias Visual Perspective Cross Entropy Loss Neural Collapse Unconstrained Feature Model

October 7, 2024

Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse
Arthur Jacot, Peter Súkeník, Zihan Wang, Marco Mondelli
Deep Neural Network Neural Collapse Wide Neural Network DNN Training Unconstrained Feature Model Variability Collapse

September 3, 2024

Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data
Wanli Hong, Shuyang Ling
Deep Neural Network Raw Data Class Imbalance Neural Collapse Shallow Neural Network Three Layer Neural Network Unconstrained Feature Model

July 15, 2024

Geometric Analysis of Unconstrained Feature Models with $d=K$
Yi Shen, Shao Gu
Geometric Analysis Neural Collapse Saddle Point Unconstrained Feature Model

April 9, 2024

Unifying Low Dimensional Observations in Deep Learning Through the Deep Linear Unconstrained Feature Model
Connall Garrod, Jonathan P. Keating
Deep Learning High Dimensional Neural Collapse Deep Linear Low Dimensional Structure Unconstrained Feature Model

February 29, 2024

Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features
Tina Behnia, Christos Thrampoulidis
Neural Network Cross Entropy Loss Neural Collapse Contrastive Representation Learning Landscape Analysis Layer Output Unconstrained Feature Model

September 18, 2023

Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data
Wanli Hong, Shuyang Ling
Class Imbalance Cross Entropy Loss Neural Collapse Imbalanced Data Unconstrained Feature Model Class Relevant

July 17, 2023

RAYEN: Imposition of Hard Convex Constraints on Neural Networks
Jesus Tordesillas, Jonathan P. How, Marco Hutter
Neural Network Constrained Optimization Convex Constraint Unconstrained Feature Model Exact Enforcement Quadratic Constraint

May 22, 2023

Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model
Peter Súkeník, Marco Mondelli, Christoph Lampert
Deep Neural Network Neural Collapse Unconstrained Feature Model

January 26, 2023

Neural networks learn to magnify areas near decision boundaries
Jacob A. Zavatone-Veth, Sheng Yang, Julian A. Rubinfien, Cengiz Pehlevan
Neural Network Inductive Bias Feature Learning Decision Boundary Area Usage Geometric Prior Adapter Learning Unconstrained Feature Model

August 9, 2022

An Unconstrained Symmetric Nonnegative Latent Factor Analysis for Large-scale Undirected Weighted Networks
Zhe Xie, Weiling Li, Yurong Zhong
Stochastic Gradient Descent Mapping Function Undirected Graph Unconstrained Feature Model Latent Factor Analysis

June 11, 2022

Memorization-Dilation: Modeling Neural Collapse Under Label Noise
Duc Anh Nguyen, Ron Levie, Julian Lienen, Gitta Kutyniok, Eyke Hüllermeier
Deep Neural Network Label Noise Neural Collapse Cross Entropy Limited Memorization Regularization Effect Unconstrained Feature Model

March 2, 2022

On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, Zhihui Zhu
Cross Entropy Loss Neural Collapse Global Optimality Mean Squared Error Last Layer Optimization Landscape Unconstrained Feature Model

February 16, 2022

Extended Unconstrained Features Model for Exploring Deep Neural Collapse
Tom Tirer, Joan Bruna
Deep Neural Network Cross Entropy Loss Neural Collapse Cross Entropy Unconstrained Feature Model

Unconstrained Feature Model

Papers

The Persistence of Neural Collapse Despite Low-Rank Bias: An Analytic Perspective Through Unconstrained Features

Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse

Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data

Geometric Analysis of Unconstrained Feature Models with $d=K$

Unifying Low Dimensional Observations in Deep Learning Through the Deep Linear Unconstrained Feature Model

Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features

Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data

RAYEN: Imposition of Hard Convex Constraints on Neural Networks

Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model

Neural networks learn to magnify areas near decision boundaries

An Unconstrained Symmetric Nonnegative Latent Factor Analysis for Large-scale Undirected Weighted Networks

Memorization-Dilation: Modeling Neural Collapse Under Label Noise

On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features

Extended Unconstrained Features Model for Exploring Deep Neural Collapse