Catastrophic Forgetting

Catastrophic forgetting describes the phenomenon where artificial neural networks, upon learning new tasks, lose previously acquired knowledge. Current research focuses on mitigating this issue through various strategies, including parameter-efficient fine-tuning methods (like LoRA), generative model-based data replay, and novel optimization algorithms that constrain gradient updates or leverage hierarchical task structures. Addressing catastrophic forgetting is crucial for developing robust and adaptable AI systems capable of continuous learning in real-world applications, particularly in domains like medical imaging, robotics, and natural language processing where data streams are constantly evolving.

Papers

August 25, 2024

CF-KAN: Kolmogorov-Arnold Network-based Collaborative Filtering to Mitigate Catastrophic Forgetting in Recommender Systems
Jin-Duk Park, Kyung-Min Kim, Won-Yong Shin
Continual LEArning Recommender System Catastrophic Forgetting Kolmogorov Arnold Network Collaborative Filtering Recommendation Accuracy

August 20, 2024

Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper
Tianyi Xu, Kaixun Huang, Pengcheng Guo, Yu Zhou, Longtao Huang, Hui Xue, Lei Xie
Training Data Catastrophic Forgetting Efficient Training Multilingual Automatic Speech Recognition State of the Art Whisper LoRA Fine Tuning Multilingual Speech Model

August 18, 2024

E-CGL: An Efficient Continual Graph Learner
Jianhao Guo, Zixuan Ni, Yun Zhu, Siliang Tang
Continual LEArning Catastrophic Forgetting Continual Training Continual Graph

August 15, 2024

An Efficient Replay for Class-Incremental Learning with Pre-trained Models
Weimin Yin, Bin Chen adn Chunzhao Xie, Zhenhao Tan
Pre Trained Model Catastrophic Forgetting Incremental Learning Class Incremental Learning Continuous Learning Efficient Generated Object Replay

August 6, 2024

Learning to Learn without Forgetting using Attention
Anna Vettoruzzo, Joaquin Vanschoren, Mohamed-Rafik Bouguelia, Thorsteinn Rögnvaldsson
LeArning Abstract Continual LEArning Transformer Based Catastrophic Forgetting Human Attention COntinual Learning Framework Meta Learned

July 31, 2024

PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning
Min Jae Jung, JooHee Kim
Continual LEArning Catastrophic Forgetting Expert Knowledge Deep Layer Stick Breaking Process Mixture

July 30, 2024

An Effective Dynamic Gradient Calibration Method for Continual Learning
Weichen Lin, Jiaxiang Chen, Ruomin Huang, Hu Ding
Continual LEArning Stochastic Gradient Descent Catastrophic Forgetting Gradient Estimation

July 24, 2024

Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective
Jingren Liu, Zhong Ji, YunLong Yu, Jiale Cao, Yanwei Pang, Jungong Han, Xuelong Li
Continual LEArning Catastrophic Forgetting Parameter Efficient Fine Tuning Neural Tangent Kernel Generalization Gap Efficient Continual Learning Continuous Improvement

July 19, 2024

An Attention-based Representation Distillation Baseline for Multi-Label Continual Learning
Martin Menabue, Emanuele Frascaroli, Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello, Simone Calderara
Continual LEArning Catastrophic Forgetting Continual Learning Multi Label Attention Distillation

July 16, 2024

July 14, 2024

Order parameters and phase transitions of continual learning in deep neural networks
Haozhe Shan, Qianyi Li, Haim Sompolinsky
Neural Network Deep Neural Network Continual LEArning Catastrophic Forgetting Anti Forgetting Order Matter Phase Transition Negative Learning

July 13, 2024

Overcoming Catastrophic Forgetting in Federated Class-Incremental Learning via Federated Global Twin Generator
Thinh Nguyen, Khoa D Doan, Binh T. Nguyen, Danh Le-Phuoc, Kok-Seng Wong
Generative Model Federated Learning Catastrophic Forgetting Class Incremental Learning Federated Prompt Cooperation Generative Training

July 12, 2024

Overcoming Catastrophic Forgetting in Tabular Data Classification: A Pseudorehearsal-based approach
Pablo García-Santaclara, Bruno Fernández-Castro, Rebeca P. Díaz-Redondo
Continual LEArning Catastrophic Forgetting Rehearsal Based Tabular Data Classification Rehearsal Based Continual Learning

July 11, 2024

July 10, 2024

How to Leverage Predictive Uncertainty Estimates for Reducing Catastrophic Forgetting in Online Continual Learning
Giuseppe Serra, Ben Werner, Florian Buettner
Catastrophic Forgetting Memory Trace Predictive Uncertainty Online Continual Learning

July 9, 2024

Neuromimetic metaplasticity for adaptive continual learning
Suhee Cho, Hyeonsu Lee, Seungdae Baek, Se-Bum Paik
Catastrophic Forgetting Continual Learning Working Memory Neural Plasticity Adaptive Continual Learning

July 7, 2024

HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning
Liyuan Wang, Jingyi Xie, Xingxing Zhang, Hang Su, Jun Zhu
Continual LEArning Pre Trained Model Catastrophic Forgetting Pre Trained Representation Parameter Efficient Tuning Hierarchical Decomposition Within Task Prediction

Catastrophic Forgetting

Papers

CF-KAN: Kolmogorov-Arnold Network-based Collaborative Filtering to Mitigate Catastrophic Forgetting in Recommender Systems

Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper

E-CGL: An Efficient Continual Graph Learner

An Efficient Replay for Class-Incremental Learning with Pre-trained Models

Learning to Learn without Forgetting using Attention

PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning

An Effective Dynamic Gradient Calibration Method for Continual Learning

Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective

An Attention-based Representation Distillation Baseline for Multi-Label Continual Learning

CroMo-Mixup: Augmenting Cross-Model Representations for Continual Self-Supervised Learning

SwitchCIT: Switching for Continual Instruction Tuning

Bridge Past and Future: Overcoming Information Asymmetry in Incremental Object Detection

Order parameters and phase transitions of continual learning in deep neural networks

Overcoming Catastrophic Forgetting in Federated Class-Incremental Learning via Federated Global Twin Generator

Overcoming Catastrophic Forgetting in Tabular Data Classification: A Pseudorehearsal-based approach

Mitigating Catastrophic Forgetting in Language Transfer via Model Merging

BiasPruner: Debiased Continual Learning for Medical Image Classification

How to Leverage Predictive Uncertainty Estimates for Reducing Catastrophic Forgetting in Online Continual Learning

Neuromimetic metaplasticity for adaptive continual learning

HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning