Catastrophic Forgetting

Catastrophic forgetting describes the phenomenon where artificial neural networks, upon learning new tasks, lose previously acquired knowledge. Current research focuses on mitigating this issue through various strategies, including parameter-efficient fine-tuning methods (like LoRA), generative model-based data replay, and novel optimization algorithms that constrain gradient updates or leverage hierarchical task structures. Addressing catastrophic forgetting is crucial for developing robust and adaptable AI systems capable of continuous learning in real-world applications, particularly in domains like medical imaging, robotics, and natural language processing where data streams are constantly evolving.

Papers

March 14, 2023

Is forgetting less a good inductive bias for forward transfer?
Jiefeng Chen, Timothy Nguyen, Dilan Gorur, Arslan Chaudhry
Continual LEArning Catastrophic Forgetting Continual Learning Inductive Bias Continual Learning Algorithm Continual Learner Backward Transfer

March 13, 2023

March 7, 2023

Robustness-preserving Lifelong Learning via Dataset Condensation
Jinghan Jia, Yihua Zhang, Dogyoon Song, Sijia Liu, Alfred Hero
Adversarial Attack Catastrophic Forgetting Lifelong Learning Model Robustness Dataset Condensation

March 3, 2023

EcoTTA: Memory-Efficient Continual Test-time Adaptation via Self-distilled Regularization
Junha Song, Jungsoo Lee, In So Kweon, Sungha Choi
Catastrophic Forgetting Test Time Adaptation lightWeight Network Continual Test Time Adaptation

March 2, 2023

Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Study
Mingxu Tao, Yansong Feng, Dongyan Zhao
Pre Trained Language Model Catastrophic Forgetting Ticket BERT Large Pre Trained Language Model Linear Probing Memory Replay Sequential Task

February 28, 2023

GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgetting
Kangyang Luo, Xiang Li, Yunshi Lan, Ming Gao
Continual LEArning Federated Learning Catastrophic Forgetting Memory Reduction

February 22, 2023

Preventing Catastrophic Forgetting in Continual Learning of New Natural Language Tasks
Sudipta Kar, Giuseppe Castellucci, Simone Filice, Shervin Malmasi, Oleg Rokhlenko
Natural Language Processing Training Data Continual LEArning Multi Task Learning Catastrophic Forgetting New Task N Task

February 18, 2023

On Handling Catastrophic Forgetting for Incremental Learning of Human Physical Activity on the Edge
Jingwei Zuo, George Arvanitakis, Hakim Hacid
Machine Learning Catastrophic Forgetting Incremental Learning Activity Recognition Extreme Edge Physical Activity

February 14, 2023

February 12, 2023

Theory on Forgetting and Generalization of Continual Learning
Sen Lin, Peizhong Ju, Yingbin Liang, Ness Shroff
Strong Generalization Continual LEArning Catastrophic Forgetting Theoretical Understanding Generalization Performance Anti Forgetting Generalization Error

February 7, 2023

February 2, 2023

January 31, 2023

A Comprehensive Survey of Continual Learning: Theory, Method and Application
Liyuan Wang, Xingxing Zhang, Hang Su, Jun Zhu
Application Proficiency Continual LEArning Real World Catastrophic Forgetting Comprehensive Survey Theoretical Understanding Practical Method Stability Plasticity

January 27, 2023

Streaming LifeLong Learning With Any-Time Inference
Soumya Banerjee, Vinay Kumar Verma, Vinay P. Namboodiri
Continual LEArning Catastrophic Forgetting Scientific Inference Self Distillation Lifelong Learning

January 18, 2023

Adaptively Integrated Knowledge Distillation and Prediction Uncertainty for Continual Learning
Kanghao Chen, Sijia Liu, Ruixuan Wang, Wei-Shi Zheng
Knowledge Distillation Continual LEArning Catastrophic Forgetting Prediction Uncertainty Stability Plasticity