Catastrophic Forgetting

Catastrophic forgetting describes the phenomenon where artificial neural networks, upon learning new tasks, lose previously acquired knowledge. Current research focuses on mitigating this issue through various strategies, including parameter-efficient fine-tuning methods (like LoRA), generative model-based data replay, and novel optimization algorithms that constrain gradient updates or leverage hierarchical task structures. Addressing catastrophic forgetting is crucial for developing robust and adaptable AI systems capable of continuous learning in real-world applications, particularly in domains like medical imaging, robotics, and natural language processing where data streams are constantly evolving.

Papers

November 4, 2022

A Theoretical Study on Solving Continual Learning
Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, Bing Liu
Continual LEArning Catastrophic Forgetting Incremental Learning Class Incremental Learning Theoretical Analysis Within Task Prediction

November 3, 2022

Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions
Shuhao Gu, Bojie Hu, Yang Feng
Domain Adaptation Continual LEArning Catastrophic Forgetting Neural Machine Translation Anti Forgetting Neural Machine Translation Model Language Adaptation

October 31, 2022

Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives
Si Sun, Chenyan Xiong, Yue Yu, Arnold Overwijk, Zhiyuan Liu, Jie Bao
Catastrophic Forgetting Dense Retrieval Sparse Retrieval Training Iteration Symmetry Teleportation

October 27, 2022

Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition
Steven Vander Eeckt, Hugo Van hamme
Automatic Speech Recognition Catastrophic Forgetting Multilingual Automatic Speech Recognition Effective Method WEight AVERaging Knowledge Distillation Loss

October 25, 2022

On Robust Incremental Learning over Many Multilingual Steps
Karan Praharaj, Irina Matveeva
Data Augmentation Catastrophic Forgetting Incremental Learning Multilingual Instruction

October 24, 2022

Correlation of the importances of neural network weights calculated by modern methods of overcoming catastrophic forgetting
Alexey Kutalev
Neural Network Catastrophic Forgetting Importance Aware Total Correlation Neural Network Weight Current Method Different Calculation Method

October 18, 2022

Exclusive Supermask Subnetwork Training for Continual Learning
Prateek Yadav, Mohit Bansal
Continual LEArning Catastrophic Forgetting Knowledge Transfer

October 17, 2022

Review Learning: Alleviating Catastrophic Forgetting with Generative Replay without Generator
Jaesung Yoo, Sunghyuk Choi, Ye Seul Yang, Suhyeon Kim, Jieun Choi, Dongkyeong Lim, Yaeji Lim, Hyung Joon Joo, Dae Jung Kim, Rae Woong Park, Hyeong-Jin Yoon, Kwangsoo Kim
Deep Learning Model Transfer Learning Catastrophic Forgetting Literature Review Synchronous Generator Generative Replay

October 16, 2022

Navigating Memory Construction by Global Pseudo-Task Simulation for Continual Learning
Yejia Liu, Wang Zhu, Shaolei Ren
Continual LEArning Catastrophic Forgetting Continual Learning Experience Replay Task Distribution Memory and Communication

October 11, 2022

Continual Learning by Modeling Intra-Class Variation
Longhui Yu, Tianyang Hu, Lanqing Hong, Zhen Liu, Adrian Weller, Weiyang Liu
Continual LEArning Catastrophic Forgetting Continual Learning Joint Representation Intra Class Variation

October 10, 2022

October 8, 2022

Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition
Junhao Zheng, Zhanxian Liang, Haibin Chen, Qianli Ma
Continual LEArning Entity Recognition Catastrophic Forgetting Causal Inference Causal Feature Distillation General Class

October 6, 2022

Topological Continual Learning with Wasserstein Distance and Barycenter
Tananun Songdechakraiwut, Xiaoshuang Yin, Barry D. Van Veen
Continual LEArning Catastrophic Forgetting Wasserstein Distance Persistent Homology Barycenter Driven Localization Topological Regularization

October 3, 2022

September 16, 2022

Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentation
Tobias Kalb, Jürgen Beyerer
Semantic Segmentation Catastrophic Forgetting Class Incremental Learning Semantic Segmentation Model Actual Cause Incremental Semantic Segmentation Semantic Class

September 12, 2022

Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation
Ya-nan Han, Jian-wei Liu
Knowledge Distillation Data Augmentation Continual LEArning Catastrophic Forgetting Meta Learning Online Continual Learning

September 9, 2022

Catastrophic Forgetting

Papers

A Theoretical Study on Solving Continual Learning

Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions

Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives

Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition

On Robust Incremental Learning over Many Multilingual Steps

Correlation of the importances of neural network weights calculated by modern methods of overcoming catastrophic forgetting

Exclusive Supermask Subnetwork Training for Continual Learning

Review Learning: Alleviating Catastrophic Forgetting with Generative Replay without Generator

Navigating Memory Construction by Global Pseudo-Task Simulation for Continual Learning

Continual Learning by Modeling Intra-Class Variation

Learning Robust Representations for Continual Relation Extraction via Adversarial Class Augmentation

A Memory Transformer Network for Incremental Learning

Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition

Topological Continual Learning with Wasserstein Distance and Barycenter

How Relevant is Selective Memory Population in Lifelong Language Learning?

Efficient Meta-Learning for Continual Learning with Taylor Expansion Approximation

Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentation

Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation

Continual learning benefits from multiple sleep mechanisms: NREM, REM, and Synaptic Downscaling

Selecting Related Knowledge via Efficient Channel Attention for Online Continual Learning