Rank Collapse

Rank collapse, a phenomenon where the representational capacity of neural networks diminishes with increasing depth, is a significant challenge hindering the performance of various deep learning models, including message-passing neural networks (MPNNs) and transformers. Current research focuses on understanding the underlying causes of rank collapse in different architectures, exploring mitigation strategies such as modifying message-passing schemes, incorporating attention mechanisms and LayerNorm, and analyzing the impact of architectural choices like activation functions and weight decay. Addressing rank collapse is crucial for improving the efficiency and effectiveness of deep learning models across diverse applications, from graph-structured data analysis to natural language processing.

Papers

October 14, 2024

Lambda-Skip Connections: the architectural component that prevents Rank Collapse
Federico Arangath Joseph, Jerome Sieber, Melanie N. Zeilinger, Carmen Amo Alonso
Stable Rank Sequence Model Event Collapse Skip Connection Connected Component Rank Collapse

October 10, 2024

Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Transformers
Alireza Naderi, Thiziri Nait Saada, Jared Tanner
Transformer Megatron Decepticons Attention Layer Propagation Environment Attention Matrix Random Matrix Layer Transformer Spectral Analysis Rank Collapse

September 17, 2024

Preventing Representational Rank Collapse in MPNNs by Splitting the Computational Graph
Andreas Roth, Franka Bause, Nils M. Kriege, Thomas Liebig
Node Representation Message Passing Neural Network Split and Fit Multi Relational Rank Collapse Computational Graph

July 16, 2024

Simplifying the Theory on Over-Smoothing
Andreas Roth
Theoretical Understanding Graph Convolution Undisciplined Over Smoothing Rank Collapse Power Iteration

May 29, 2024

On the Role of Attention Masks and LayerNorm in Transformers
Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, Ali Jadbabaie
Transformer Megatron Decepticons Self Attention Integral Role Attention Mask Deeper Model Tuning LayerNorm Rank Collapse

February 14, 2024

Understanding team collapse via probabilistic graphical models
Iasonas Nikolaou, Konstantinos Pelechrinis, Evimaria Terzi
Probabilistic Graphical Model Graphical Model Team Dynamic Team Affiliation Rank Collapse Resilient Robot

February 9, 2024

Low-Rank Learning by Design: the Role of Network Architecture and Activation Linearity in Gradient Rank Collapse
Bradley T. Baker, Barak A. Pearlmutter, Robyn Miller, Vince D. Calhoun, Sergey M. Plis
LeArning Abstract Deep Neural Network Product Design Neural Collapse DNN Architecture Low Rank Structure Network Architecture Nonlinear Activation Rank Collapse

February 6, 2024

Neural Rank Collapse: Weight Decay and Small Within-Class Variability Yield Low-Rank Bias
Emanuele Zangrando, Piero Deidda, Simone Brugiapaglia, Nicola Guglielmi, Francesco Tudisco
Absolute Stance Bias Neural Collapse Nonlinear Activation Weight Decay Rank Collapse Class Variability

October 2, 2023

An Investigation of Representation and Allocation Harms in Contrastive Learning
Subha Maity, Mayank Agarwal, Mikhail Yurochkin, Yuekai Sun
Contrastive Learning Meaningful Representation Individual Representation Representational Harm Rank Collapse

August 31, 2023

Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networks
Andreas Roth, Thomas Liebig
Graph Neural Network Node Representation Undisciplined Over Smoothing Deep Graph Rank Collapse Invariant Subspace

June 13, 2023

Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching
Ganesh Ramachandra Kini, Vala Vakilian, Tina Behnia, Jaidev Gill, Christos Thrampoulidis
Global Impact Contrastive Loss Cross Entropy Loss ReLU Layer Supervised Contrastive Loss Rank Collapse Representational Geometry

June 7, 2022

Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse
Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, Aurelien Lucchi
Transformer Megatron Decepticons Integral Role Natural Gradient Gradient Norm Self Attention Layer Propagation Environment Rank Collapse Theoretical Consideration Effective Depth Up Scaling

Rank Collapse

Papers

Lambda-Skip Connections: the architectural component that prevents Rank Collapse

Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Transformers

Preventing Representational Rank Collapse in MPNNs by Splitting the Computational Graph

Simplifying the Theory on Over-Smoothing

On the Role of Attention Masks and LayerNorm in Transformers

Understanding team collapse via probabilistic graphical models

Low-Rank Learning by Design: the Role of Network Architecture and Activation Linearity in Gradient Rank Collapse

Neural Rank Collapse: Weight Decay and Small Within-Class Variability Yield Low-Rank Bias

An Investigation of Representation and Allocation Harms in Contrastive Learning

Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networks

Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching

Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse