Event Collapse

Event collapse, a phenomenon where model representations or embeddings lose dimensionality and structure, is a significant challenge across various machine learning domains. Current research focuses on understanding and mitigating this collapse in diverse contexts, including deep metric learning, large language models, and reinforcement learning, often employing techniques like anti-collapse loss functions, regularized training, and architectural modifications to promote representational diversity. Addressing event collapse is crucial for improving the robustness, generalization ability, and overall performance of machine learning models, impacting fields ranging from computer vision and natural language processing to robotics and climate modeling.

Papers

November 4, 2024

Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning
Md Rifat Arefin, Gopeshh Subbaraj, Nicolas Gontier, Yann LeCun, Irina Rish, Ravid Shwartz-Ziv, Christopher Pal
Intermediate Representation Event Collapse Decoder Only Transformer Representation Collapse Cognitive Reasoning Transformer Representation

October 31, 2024

Length-Induced Embedding Collapse in Transformer-based Models
Yuqi Zhou, Sunhao Dai, Zhanshuo Cao, Xiao Zhang, Jun Xu
Transformer Based Model Long Context Text Embeddings Long Text Event Collapse

October 22, 2024

Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World
Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David L. Donoho, Sanmi Koyejo
Generative Model Synthetic Data Generated Content Promise Event Collapse Model Collapse Possible World

October 18, 2024

Mitigating Embedding Collapse in Diffusion Models for Categorical Data
Bac Nguyen, and Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Stefano Ermon, Yuki Mitsufuji
Diffusion Model Latent Diffusion Model Categorical Data Event Collapse Diffusion Embeddings

October 16, 2024

The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse
Ekansh Sharma, Daniel M. Roy, Gintare Karolina Dziugaite
Model Merging Expert Model Variance Information Event Collapse Permutation Symmetry Task Expert

October 14, 2024

Lambda-Skip Connections: the architectural component that prevents Rank Collapse
Federico Arangath Joseph, Jerome Sieber, Melanie N. Zeilinger, Carmen Amo Alonso
Stable Rank Sequence Model Event Collapse Skip Connection Connected Component Rank Collapse

October 9, 2024

Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling
Yingfa Chen, Xinrong Zhang, Shengding Hu, Xu Han, Zhiyuan Liu, Maosong Sun
Recurrent Neural Network Long Context Mamba in Mamba Capacity Loss Event Collapse Recurrent Dynamic

September 26, 2024

Task-recency bias strikes back: Adapting covariances in Exemplar-Free Class Incremental Learning
Grzegorz Rypeść, Sebastian Cygert, Tomasz Trzciński, Bartłomiej Twardowski
Feature Extractor Covariance Conditioning Event Collapse Exemplar Free Class Incremental Learning Dimensional Collapse Recency Bias

July 3, 2024

Anti-Collapse Loss for Deep Metric Learning Based on Coding Rate Metric
Xiruo Jiang, Yazhou Yao, Xili Dai, Fumin Shen, Xian-Sheng Hua, Heng-Tao Shen
Loss Function Deep Metric Learning Event Collapse Dimensional Collapse Rate Reduction

June 17, 2024

The Fall of ROME: Understanding the Collapse of LLMs in Model Editing
Wanli Yang, Fei Sun, Jiajun Tan, Xinyu Ma, Du Su, Dawei Yin, Huawei Shen
Large Language Model Medical LLM Model Editing Autoregressive Transformer Parameter Update Event Collapse Model Collapse Rank One Model Editing

May 30, 2024

Separation and Collapse of Equilibria Inequalities on AND-OR Trees without Shape Constraints
Fuki Ito, Toshio Suzuki
Practical Algorithm Tree Specie Separation Performance Event Collapse Efficient Equilibrium Shape Constraint Recursive Thinking

May 2, 2024

Progressive Feedforward Collapse of ResNet Training
Sicong Wang, Kuo Gai, Shihua Zhang
Deep Neural Network Neural Collapse Backward Propagation Event Collapse ResNet Model Feed Forward

May 1, 2024

No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO
Skander Moalla, Andrea Miele, Razvan Pascanu, Caglar Gulcehre
Reinforcement Learning Proximal Policy Optimization Individual Representation Appropriate Trust Policy Optimization Event Collapse Dynamic Representation Policy Deep Reinforcement Learning Representation Collapse

April 25, 2024

Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents
Giorgio Piatti, Zhijing Jin, Max Kleiman-Weiner, Bernhard Schölkopf, Mrinmaya Sachan, Rada Mihalcea
AI Agent Path Breaking Emergence LLM Agent Long Term Cooperation Event Collapse Equilibrium State Agent Framework Social Simulation Sustainable Consumption

April 8, 2024

Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
Ruiqi Zhang, Licong Lin, Yu Bai, Song Mei
Unlearning Method Event Collapse LLM Unlearning Unlearning Model

April 2, 2024

Collapse of Self-trained Language Models
David Herel, Tomas Mikolov
Language Model Self Training Event Collapse GPT 2

February 22, 2024

Ad Recommendation in a Collapsed and Entangled World
Junwei Pan, Wei Xue, Ximei Wang, Haibin Yu, Xun Liu, Shijie Quan, Xueming Qiu, Dapeng Liu, Lei Xiao, Jie Jiang
Jina Embeddings Recommendation System Feature Representation Event Collapse Sparse Feature

February 21, 2024

Machine-learning prediction of tipping with applications to the Atlantic Meridional Overturning Circulation
Shirin Panahi, Ling-Wei Kong, Mohammadamin Moradi, Zheng-Meng Zhai, Bryan Glaz, Mulugeta Haile, Ying-Cheng Lai
Machine Learning Climate Model Event Collapse TIP Generation Tipping Point Atlantic Meridional Overturning Circulation

January 16, 2024

Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data
Yuhui Zhang, Elaine Sui, Serena Yeung-Levy
Cross Modal New Connection Modality Gap Event Collapse Modality Data Cross Modal Task Multi Modal Contrastive Representation

October 6, 2023

On the Embedding Collapse when Scaling up Recommendation Models
Xingzhuo Guo, Junwei Pan, Ximei Wang, Baixu Chen, Jie Jiang, Mingsheng Long
Recommendation Model Event Collapse Multiple Embeddings Large Scale Recommender System Scalable Model

Event Collapse

Papers

Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning

Length-Induced Embedding Collapse in Transformer-based Models

Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World

Mitigating Embedding Collapse in Diffusion Models for Categorical Data

The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse

Lambda-Skip Connections: the architectural component that prevents Rank Collapse

Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling

Task-recency bias strikes back: Adapting covariances in Exemplar-Free Class Incremental Learning

Anti-Collapse Loss for Deep Metric Learning Based on Coding Rate Metric

The Fall of ROME: Understanding the Collapse of LLMs in Model Editing

Separation and Collapse of Equilibria Inequalities on AND-OR Trees without Shape Constraints

Progressive Feedforward Collapse of ResNet Training

No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO

Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents

Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning

Collapse of Self-trained Language Models

Ad Recommendation in a Collapsed and Entangled World

Machine-learning prediction of tipping with applications to the Atlantic Meridional Overturning Circulation

Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data

On the Embedding Collapse when Scaling up Recommendation Models