Catastrophic Forgetting

Follow this topic Ask about this topic

Catastrophic forgetting occurs when learning new tasks or adapting to shifting data degrades performance on earlier tasks, a central challenge for systems expected to learn over time. Common approaches include replaying examples, distilling prior outputs, regularizing important weights or gradients, and isolating updates with adapters or modular experts. Researchers study these methods in sequential vision and language benchmarks as well as foundation-model fine-tuning, multimodal systems, and robotics, measuring both new-task performance and retention. No single approach works best everywhere: stronger protection can limit adaptation, while memory, compute, and benchmark choices affect results; apparent forgetting may also reflect knowledge becoming inaccessible rather than erased.

Latest papers

351 papers59 papers in the last four weeks, up 119% on the four weeks before. 0.3% of all new papers.

TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning
May 29, 2026Xiaosong Han, Ke Chen, Xindi Dai +7Continual Learning for LLMs LLM Fine-Tuning
Key Laboratory of Symbolic Computation and Knowledge Engineering of the Ministry of Education · College of Computer Science and Technology, Jilin University Changchun, China · College of Software, Jilin University Changchun, China +3
RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting
May 29, 2026Yuduo Li, Xiaofeng Shi, Qian Kou +2Supervised Fine-Tuning Fine-Tuning
Beijing Academy of Artificial Intelligence (BAAI) · Beijing Jiaotong University (BJTU)
Foundation-Preserving Optimization in Generalized Eigenspace
May 28, 2026Dongjun Kim, Adrian de Wynter, Huancheng Chen +2Low-Rank Adaptation Parameter-Efficient Fine-Tuning
The University of Texas at Austin · Microsoft · Microsoft AI +1
Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies
May 28, 2026Kajetan Schweighofer, Conor F. Hayes, Roberto Dailey +2Continual Learning for LLMs Evolutionary Optimization
Cognizant AI Lab · UT Austin & Cognizant AI Lab
Non-Forgetting Knowledge Allocation with Bi-level Competition for Class-Incremental Learning
May 28, 2026Xiang Tan, Run He, Yawen Cui +6Class-Incremental Learning Incremental Learning
South China University of Technology · Hong Kong Polytechnic University · Agency for Science, Technology and Research +3
Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting
May 28, 2026Runze Xu, Arpit Garg, Hemanth Saratchandran +1Continual Learning for LLMs LLM Fine-Tuning
Australian Institute for Machine Learning Adelaide University Adelaide SA 5000
On-Policy Replay for Continual Supervised Fine-Tuning
May 28, 2026Yan Chen, Taojie Zhu, Meng Zhang +4Continual Learning for LLMs Replay-Based Continual Learning
Tsinghua University · Alibaba Group
PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
May 27, 2026Yangyi Huang, Ruotian Peng, Zeju Qiu +4LLM Evaluation Fine-Tuning
The Chinese University of Hong Kong · Westlake University · MPI for Intelligent Systems
AREA: Attribute Extraction and Aggregation for CLIP-Based Class-Incremental Learning
May 27, 2026Zhen-Hao Xie, Yu-Cheng Shi, Da-Wei ZhouClass-Incremental Learning Vision-Language Models
School of Artificial Intelligence, Nanjing University, China · State Key Laboratory of Novel Software Technology, Nanjing University, China
Continual Learning in Modern Hopfield Networks with an Application to Diffusion Models
May 27, 2026Ken Takeda, Masafumi Oizumi, Ryo KarakidaHopfield Networks Replay-Based Continual Learning
Graduate School of Arts and Science, The University of Tokyo, Tokyo, Japan · Artificial Intelligence Research Center, AIST, Tokyo, Japan
Can VLA Models Learn from Real-World Data Continually without Forgetting?
May 26, 2026Jiarun Zhu, Yijun Hong, Xiaoquan Sun +5Continual Robot Learning Continual Learning for VLA Models
1HKU · 5SUSTech · 2INFIFORCE +3
Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay
May 25, 2026Martin Marek, Dongkyu Cho, Shikai Qiu +3Continual Learning for LLMs LLM Fine-Tuning
New York University
Max-Window Scale Estimation for Near-Lossless HiF8 W8A8 Quantization-Aware Training
May 25, 2026Yingying Cheng, Jinquan Shi, Li Zhou +4LLM Quantization Quantization-Aware Training
X-Edit: Exact, Explicit, and Explainable Null-Space Editing for Medical Vision Transformers
May 24, 2026Yuanye Liu, Siyuan Zhou, Ke Zhang +3Vision Transformer Medical Image Classification
Fudan University, Shanghai, China · Johns Hopkins University, Baltimore, USA · National University of Singapore, Singapore +1
Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?
May 21, 2026Jeanmely Rojas Nunez, Viraj Sawant, Nathan Allen +4Supervised Fine-Tuning Mechanistic Interpretability
Algoverse AI Research · Independent
Don't Forget the Critic: Value-Based Data Rehearsal for Multi-Cyclic Continual Reinforcement Learning
May 21, 2026Benjamin Poole, Andrew Quinn, Li Yang +1Reinforcement Learning Deep Q-Learning
Department of Computer Science University of North Carolina at Charlotte
Dynamic Mixture of Latent Memories for Self-Evolving Agents
May 21, 2026Dianzhi Yu, Vireo Zhang, Hongru Wang +7Continual Learning for LLMs Mixture-of-Experts Language Models
The Chinese University of Hong Kong · Nanyang Technological University · University of Edinburgh +2
Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates
May 19, 2026Parjanya Prajakta Prashant, Jiongli Zhu, Aldan Creo +1Learning Rate Scheduling LLM Fine-Tuning
University of California San Diego
CPC-VAR:Continual Personalized and Compositional Generation in Visual Autoregressive Models
May 19, 2026Junhao Li, Xinhao Zhong, Yi sun +4Visual Autoregressive Models Subject-Driven Image Generation
Harbin Institute of Technology, Shenzhen · South China University of Technology · Peng Cheng Laboratory +1
Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining
May 19, 2026Aarash Abro, Muhammad TahirLLM Fine-Tuning Spectral Methods
Zeta Labs · Lahore University of Management Sciences
PMF-CL: Pareto-Minimal-Forgetting Continual Learner for Conflicting Tasks
May 18, 2026Srijith Nair, Atilla Eryilmaz, Jia LiuContinual Learning Catastrophic Forgetting
Department of Electrical and Computer Engineering The Ohio State University Columbus, OH 43201
MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning
May 18, 2026Ankita Awasthi, Marco Apolinario, Kaushik RoyClass-Incremental Learning Meta-Learning
Purdue University, USA · TU Delft, Netherlands
Learning When to Adapt
May 18, 2026Ali Zindari, Xiaowen Jiang, Rotem Mulayoff +1LLM Fine-Tuning Low-Rank Adaptation
1Universität des Saarlandes · 2CISPA Helmholtz Center for Information Security
CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
May 18, 2026Yang Liu, Toan Nguyen, Flora D. SalimContinual Learning for LLMs Vision-Language Models
School of Computer Science and Engineering University of New South Wales Australia
MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation
May 18, 2026Ronyu Zhang, Aosong Cheng, Gaole Dai +8Activation Sparsity Mixture of Experts
Nanjing University · The Hong Kong Polytechnic University · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University +3
MasFACT: Continual Multi-Agent Topology Learning via Geometry-Aware Posterior Transfer
May 17, 2026Xuefei Wang, Jialu Wang, Fengbo Zhang +6Continual Learning for LLM Agents Multi-Agent LLM Systems
Beihang University · Independent Researcher · Beijing University of Posts and Telecommunications
MixSD: Mixed Contextual Self-Distillation for Knowledge Injection
May 16, 2026Jiarui Liu, Lechen Zhang, Yongjin Yang +5Knowledge Augmentation for Language Models Factual Knowledge in Language Models
Carnegie Mellon University · Jinesis Lab, University of Toronto & Vector Institute · University of Illinois Urbana-Champaign +6
Shapley Neuron Values for Continual Learning: Which Neurons Matter Most?
May 15, 2026Mohammad Ali Vahedifar, Abhisek Ray, Qi ZhangTask-Incremental Learning Class-Incremental Learning
DIGIT and Department of Electrical and Computer Engineering, Aarhus University, Denmark
UAM: A Dual-Stream Perspective on Forgetting in VLA Training
May 15, 2026Jianke Zhang, Yuanfei Luo, Yucheng Hu +6Robotic Manipulation Vision-Language-Action Models
Tsinghua University · ByteDance Seed
Is One Score Enough? Rethinking the Evaluation of Sequentially Evolving LLM Memory
May 14, 2026Songwei Dong, Zihan Chen, Chengshuai Shi +3Continual Learning for LLMs LLM Evaluation
University of Virginia · Princeton University

May 29, 2026J/K move · Enter open · S save

TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning

Xiaosong Han, Ke Chen, Xindi Dai, Di Liang+6

Key Laboratory of Symbolic Computation and Knowledge Engineering of the Ministry of Education · College of Computer Science and Technology, Jilin University Changchun, China · College of Software, Jilin University Changchun, China+3

Continual Learning for LLMs LLM Fine-Tuning Selective Fine-Tuning Parameter-Efficient Fine-Tuning Catastrophic Forgetting

Open paper AskPDF arXiv

In real-world deployment, LLMs are often adapted continually across tasks to keep LLMs up-to-date in production, where new fine-tuning should preserve previously learned skills. However, indiscriminately mixing tasks can dilute task specialization, while sequential fine-tuning (full-parameter or low rank adaptation) often causes catastrophic forgetting due to destructive overwriting. Replay-based continual tuning and maintaining separate task-specific adapters can mitigate forgetting, but introduce additional compute, storage, and management overhead. Recognizing the redundancy of LLM parameters for any single task, we reframe continual task adaptation as task-specific parameter discovery via adaptation-aware probing: a short warm-start probe exposes a task's adaptation trace, enabling us to identify and isolate the small subset of parameters essential for each task to mitigate catastrophic forgetting. Building on this view, we introduce TRACE, a novel approach for discovering Task-specific paRameters via Adaptation-aware probing for Continual finE-tuning. We perform a short warm-start fine-tune to derive task-specific core parameters by comparing the warm-started and pre-trained models. Core parameters are identified via two strategies: importance scoring (L

_2

norm and Fisher Information) and specificity analysis (cosine similarity of parameter updates). In continual fine-tuning settings, only the active task's core parameters are updated while others remain frozen, preserving prior knowledge. We conduct extensive experiments across multiple standard benchmarks to demonstrate the superior performance of our proposed method. Additionally, we validate the generalization of our method through a cross-model and scale transferability study, demonstrating a "small-to-large" paradigm that guides the fine-tuning of large-scale models under resource constraints.