Catastrophic Forgetting

Catastrophic forgetting describes the phenomenon where artificial neural networks, upon learning new tasks, lose previously acquired knowledge. Current research focuses on mitigating this issue through various strategies, including parameter-efficient fine-tuning methods (like LoRA), generative model-based data replay, and novel optimization algorithms that constrain gradient updates or leverage hierarchical task structures. Addressing catastrophic forgetting is crucial for developing robust and adaptable AI systems capable of continuous learning in real-world applications, particularly in domains like medical imaging, robotics, and natural language processing where data streams are constantly evolving.

Papers

May 12, 2022

KASAM: Spline Additive Models for Function Approximation
Heinrich van Deventer, Pieter Janse van Rensburg, Anna Bosman
Catastrophic Forgetting Function Approximation

May 11, 2022

Contrastive Supervised Distillation for Continual Representation Learning
Tommaso Barletti, Niccolo' Biondi, Federico Pernici, Matteo Bruni, Alberto Del Bimbo
Catastrophic Forgetting Image Retrieval Visual Search Contrastive Distillation Continual Representation Learning

May 6, 2022

Forget Less, Count Better: A Domain-Incremental Self-Distillation Learning Benchmark for Lifelong Crowd Counting
Jiaqi Gao, Jingqi Li, Hongming Shan, Yanyun Qu, James Z. Wang, Fei-Yue Wang, Junping Zhang
Catastrophic Forgetting Anti Forgetting Self Distillation Crowd Counting Borda Counting

May 4, 2022

Explain to Not Forget: Defending Against Catastrophic Forgetting with XAI
Sami Ede, Serop Baghdadlian, Leander Weber, An Nguyen, Dario Zanca, Wojciech Samek, Sebastian Lapuschkin
Neural Network Catastrophic Forgetting xAI Community Layer Wise Relevance Propagation

April 26, 2022

Theoretical Understanding of the Information Flow on Continual Learning Performance
Josh Andle, Salimeh Yasaei Sekeh
Continual LEArning Catastrophic Forgetting Theoretical Understanding Information Flow Task Specific Knowledge Continual Learning Performance Learning Multi Phase Incremental Task

April 21, 2022

CNLL: A Semi-supervised Approach For Continual Noisy Label Learning
Nazmul Karim, Umar Khalid, Ashkan Esmaeili, Nazanin Rahnavard
Fine Tuning Continual LEArning Semi Supervised Catastrophic Forgetting Noisy Label Noisy Label Learning Negative Log Likelihood

April 19, 2022

An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditions
M. Jehanzeb Mirza, Marc Masana, Horst Possegger, Horst Bischof
Autonomous Driving Catastrophic Forgetting Incremental Learning Anti Forgetting Extreme Event Domain Incremental Learning Challenging Domain

April 17, 2022

Continual Hippocampus Segmentation with Transformers
Amin Ranem, Camila González, Anirban Mukhopadhyay
Transformer Megatron Decepticons Continual LEArning Catastrophic Forgetting Continual Learning Convolutional Architecture Hippocampus Segmentation

April 13, 2022

Improving generalization to new environments and removing catastrophic forgetting in Reinforcement Learning by using an eco-system of agents
Olivier Moulin, Vincent Francois-Lavet, Paul Elbers, Mark Hoogendoorn
Reinforcement Learning Strong Generalization Catastrophic Forgetting Individual Agent New Environment

April 10, 2022

FOSTER: Feature Boosting and Compression for Class-Incremental Learning
Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan
Deep Neural Network Catastrophic Forgetting Class Incremental Learning Linear Compression Ensemble Model ImageNet LT

April 8, 2022

Learning to Modulate Random Weights: Neuromodulation-inspired Neural Networks For Efficient Continual Learning
Jinyung Hong, Theodore P. Pavlic
LeArning Abstract Continual LEArning Catastrophic Forgetting Efficient Continual Learning Random Weight Synaptic Dynamic

April 5, 2022

Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation
Tao Feng, Mang Wang, Hangjie Yuan
Catastrophic Forgetting Incremental Learning Self Distillation Progressive Distillation Incremental Object Detection

April 2, 2022

Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation
Minsoo Kang, Jaeyoo Park, Bohyung Han
Deep Neural Network Knowledge Distillation Catastrophic Forgetting Incremental Learning Class Incremental Learning Accuracy Improvement

March 30, 2022

Rainbow Keywords: Efficient Incremental Learning for Online Spoken Keyword Spotting
Yang Xiao, Nana Hou, Eng Siong Chng
Catastrophic Forgetting Class Incremental Learning Keyword Spotting Keyword Enrollment Sample Diversity

March 27, 2022

Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions
Binghui Peng, Andrej Risteski
Machine Learning Continual LEArning Catastrophic Forgetting Feature Extraction Efficient Algorithm Gradient Based Algorithm

March 26, 2022

Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation
Guanglei Yang, Enrico Fini, Dan Xu, Paolo Rota, Mingli Ding, Moin Nabi, Xavier Alameda-Pineda, Elisa Ricci
Knowledge Distillation Catastrophic Forgetting Incremental Semantic Segmentation

March 25, 2022

Class-Incremental Learning for Action Recognition in Videos
Jaeyoo Park, Minsoo Kang, Bohyung Han
Continual LEArning Action Recognition Catastrophic Forgetting Gameplay Video Class Incremental Learning Video Recognition Action Recognition Benchmark

March 24, 2022

Probing Representation Forgetting in Supervised and Unsupervised Continual Learning
MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, Eugene Belilovsky
Continual LEArning Catastrophic Forgetting Continual Learning Unsupervised Continual Learning Representation Forgetting

March 22, 2022

Federated Class-Incremental Learning
Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, Qi Zhu
Catastrophic Forgetting Class Incremental Learning Anti Forgetting Federated Prompt Cooperation Relation Distillation

March 12, 2022

Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activations
Ali Abbasi, Parsa Nooralinejad, Vladimir Braverman, Hamed Pirsiavash, Soheil Kolouri
Continual LEArning Catastrophic Forgetting Sparsity Increase Structured Dropout Continual Learning Benchmark Continual Learner Null Space Neuron Activation Gradient Episodic Memory

Catastrophic Forgetting

Papers

KASAM: Spline Additive Models for Function Approximation

Contrastive Supervised Distillation for Continual Representation Learning

Forget Less, Count Better: A Domain-Incremental Self-Distillation Learning Benchmark for Lifelong Crowd Counting

Explain to Not Forget: Defending Against Catastrophic Forgetting with XAI

Theoretical Understanding of the Information Flow on Continual Learning Performance

CNLL: A Semi-supervised Approach For Continual Noisy Label Learning

An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditions

Continual Hippocampus Segmentation with Transformers

Improving generalization to new environments and removing catastrophic forgetting in Reinforcement Learning by using an eco-system of agents

FOSTER: Feature Boosting and Compression for Class-Incremental Learning

Learning to Modulate Random Weights: Neuromodulation-inspired Neural Networks For Efficient Continual Learning

Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation

Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation

Rainbow Keywords: Efficient Incremental Learning for Online Spoken Keyword Spotting

Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions

Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation

Class-Incremental Learning for Action Recognition in Videos

Probing Representation Forgetting in Supervised and Unsupervised Continual Learning

Federated Class-Incremental Learning

Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activations