Catastrophic Forgetting

Follow this topic Ask about this topic

Catastrophic forgetting occurs when learning new tasks or adapting to shifting data degrades performance on earlier tasks, a central challenge for systems expected to learn over time. Common approaches include replaying examples, distilling prior outputs, regularizing important weights or gradients, and isolating updates with adapters or modular experts. Researchers study these methods in sequential vision and language benchmarks as well as foundation-model fine-tuning, multimodal systems, and robotics, measuring both new-task performance and retention. No single approach works best everywhere: stronger protection can limit adaptation, while memory, compute, and benchmark choices affect results; apparent forgetting may also reflect knowledge becoming inaccessible rather than erased.

Latest papers

351 papers59 papers in the last four weeks, up 119% on the four weeks before. 0.3% of all new papers.

TFGN: Task-Free, Replay-Free Continual Pre-Training Without Catastrophic Forgetting at LLM Scale
May 14, 2026Anurup GanguliContinual Learning for LLMs Continued Pretraining
Independent Researcher
InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting
May 14, 2026Mahdi Sabbaghi, George Pappas, Adel Javanmard +1Supervised Fine-Tuning Language Model Generalization
University of Pennsylvania · University of Southern California
Understanding Imbalanced Forgetting in Rehearsal-Based Class-Incremental Learning
May 14, 2026Alberto Tamajo, Srinandan Dasmahapatra, Rahman AttarClass-Incremental Learning Gradient Interference
School of Electronics and Computer Science, University of Southampton, SO17 1BJ Southampton, United Kingdom
Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax
May 14, 2026Zeli Su, Ziyin Zhang, Zhou Liu +7Reinforcement Learning Low-Resource MT
Minzu University of China · Ant Group · Shanghai Jiao Tong University +4
Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning
May 12, 2026Joana Pasquali, Ramiro N. Barros, Arthur S. Bianchessi +7Gradient Interference Catastrophic Forgetting
MALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, Brazil · Kunumi Institute, Brazil
Early Data Exposure Improves Robustness to Subsequent Fine-Tuning
May 12, 2026Lawrence Feng, Gaurav R. Ghosal, Jacob Mitchell Springer +2Language Model Pretraining Continual Learning for LLMs
Carnegie Mellon University
Learning, Fast and Slow: Towards LLMs That Adapt Continually
May 12, 2026Rishabh Tiwari, Kusha Sareen, Lakshya A Agrawal +6Continual Learning for LLMs RL for Language Model Reasoning
UC Berkeley · Eragon · Mila +3
ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging
May 12, 2026Neha Verma, Nikhil Mehta, Shao-Chuan Wang +7Continual Learning for LLMs LLM Fine-Tuning
Johns Hopkins University · Google DeepMind · Google
KAN-CL: Per-Knot Importance Regularization for Continual Learning with Kolmogorov-Arnold Networks
May 12, 2026Minjong CheonRegularization-Based Continual Learning Continual Learning
Sejong University Department of Computer Science and Engineering
Beyond Point-wise Neural Collapse: A Topology-Aware Hierarchical Classifier for Class-Incremental Learning
May 12, 2026Huiyu Yi, Zhiming Xu, Dunwei Tu +3Class-Incremental Learning Prototype-Based Classification
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · School of Computer Science, Nanjing University, Nanjing, China
More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing
May 12, 2026Xin Ma, Wei Chen, Qi Liu +4Knowledge Editing Continual Learning
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China, Anhui, China.
Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory
May 11, 2026Yao Shu, Jian Mu, Zhongxiang DaiZeroth-Order Optimization Continual Learning
HKUST(GZ)
Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm
May 11, 2026Haoyu Wang, Yifan Shang, Zhongxiang Sun +3Continual Learning for LLMs Generative Replay
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · School of Artificial Intelligence and Data Science, University of International Business and Economics, Beijing, China
Joint sparse coding and temporal dynamics support context reconfiguration
May 11, 2026Qianqian Shi, Yue Che, Faqiang Liu +6Representation Learning Continual Learning
Center for Brain-Inspired Computing Research (CBICR), Department of Precision Instrument, Tsinghua University, Beijing, China · Guangdong Institute of Intelligence Science and Technology, Hengqin, China · Optical Memory National Engineering Research Center, Tsinghua University, Beijing, China +3
Lost or Hidden? A Concept-Level Forgetting in Supervised Continual Learning
May 10, 2026Katarzyna Filus, Kamil Faber, Roberto Corizzo +1Visual Representation Learning Continual Learning
Institute of Theoretical and Applied Informatics Polish Academy of Sciences, Gliwice, Poland · AGH University of Krakow Krakow, Poland · American University Washington DC, USA +1
Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning
May 10, 2026Meng Lou, Hanzhong Guo, Linwei Chen +1Class-Incremental Learning Domain-Incremental Learning
The University of Hong Kong · The Hong Kong University of Science and Technology · Hong Kong Generative AI Research and Development Center
Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training
May 10, 2026Yuanyi Wang, Yifan Yang, Su Lu +9Continual Learning for LLMs Model Merging
The Hong Kong Polytechnic University, PolyU · The Hong Kong University of Science and Technology, HKUST · PolyU-Daya Bay Technology and Innovation Research Institute
Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning
May 9, 2026Binghang Lu, Zheyuan Deng, Runyu Zhang +6Continual Learning for LLMs Gradient Descent
Purdue University · Brown University · Massachusetts Institute of Technology +4
Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFT
May 9, 2026Tianyi Zhang, Shaopeng Zhai, Haoran Zhang +2Supervised Fine-Tuning Continual Learning for VLA Models
Shanghai Artificial Intelligence Laboratory
SR $^2$ -LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning
May 8, 2026Fengqiang Wan, Yipeng Lin, Kan Lv +1Representation Drift Low-Rank Adaptation
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less
May 7, 2026Yuxing Liu, Jianyu Wang, Tong ZhangSupervised Fine-Tuning LLM Fine-Tuning
1UIUC∗ · 2Apple
GeoStack: A Framework for Quasi-Abelian Knowledge Composition in VLMs
May 7, 2026Pranav Mantini, Shishir K. ShahVision-Language Models VLM Adaptation
Department of Computer Science University of Houston · The University of Oklahoma School of Computer Science
HEDP: A Hybrid Energy-Distance Prompt-based Framework for Domain Incremental Learning
May 7, 2026Yu Feng, Zhen Tian, Haoran Luo +8Prompt Learning Domain-Incremental Learning
China Mobile Research Institute, China · Beijing University of Posts and Telecommunications, China · Nanyang Technological University, Singapore +1
CRAFT: Forgetting-Aware Intervention-Based Adaptation for Continual Learning
May 7, 2026Md Anwar Hossen, Fatema Siddika, Juan Pablo Munoz +2Continual Learning for LLMs Low-Rank Adaptation
Iowa State University · Maro Systems, USA · University of California, Berkeley +1
Balancing Stability and Plasticity in Sequentially Trained Early-Exiting Neural Networks
May 6, 2026Alaa Zniber, Ouassim Karrakchou, Mounir GhoghoAdaptive Inference Dynamic Neural Networks
⋆TICLab, International University of Rabat, Morocco · ‡ School of Electronic and Electrical Engineering, University of Leeds, UK
Attribution-Guided Continual Learning for Large Language Models
May 6, 2026Yazheng Liu, Yuxuan Wan, Rui Xu +3Continual Learning for LLMs LLM Fine-Tuning
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The Beijing University of Posts and Telecommunications, Beijing, China
Stabilizing LLM Supervised Fine-Tuning via Explicit Distributional Control
May 6, 2026Xinyu Wang, Changzhi Sun, Yuanbin Wu +1Supervised Fine-Tuning Continual Learning for LLMs
School of Computer Science and Technology, East China Normal University, Shanghai, China · TeleAI, Shanghai, China
Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mapping
May 5, 2026Kaustubh Pethkar, Ziyang Xiong, Zuofeng Shang +1Continual Learning for LLMs Memorization in Language Models
New Jersey Institute of Technology · University of California, Berkeley
Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
May 4, 2026Prakhar Gupta, Garv Shah, Satyam Goyal +1Memory-Augmented Language Models LLM Fine-Tuning
University of Michigan
Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting
May 4, 2026Ishaan Watts, Catherine Li, Sachin Goyal +2Language Model Pretraining Sharpness-Aware Minimization
Carnegie Mellon University

May 14, 2026J/K move · Enter open · S save

TFGN: Task-Free, Replay-Free Continual Pre-Training Without Catastrophic Forgetting at LLM Scale

Anurup Ganguli

Independent Researcher

Continual Learning for LLMs Continued Pretraining Catastrophic Forgetting

Open paper AskPDF arXiv

Continually pre-training a large language model on heterogeneous text domains, without replay or task labels, has remained an unsolved architectural problem at LLM scale. Existing methods rely on replay buffers, task identifiers, regularization penalties that scale poorly, or sentence-classification-scale evaluation. We introduce TFGN, an architectural overlay for transformer language models that produces input-conditioned, parameter-efficient updates while leaving the rest of the transformer unchanged. On six heterogeneous text domains (Prose, Python, Math, Biomedical, Chinese, JavaScript) at 1B tokens per phase across three model scales (~398M, ~739M, ~9B) and two regimes (From-Scratch and Retrofit), TFGN achieves backward transfer of -0.007 at LLaMA 3.1 8B Retrofit, HellaSwag retention 0.506/0.504/0.510, and >=99.59% L2-orthogonal gradient separation between domain pairs - with no replay, no task IDs, no Fisher penalty. The same matrices show positive cross-domain forward transfer: held-out JavaScript PPL drops 26.8% at LLaMA-8B Retrofit and 62.0% at GPT-2 Medium From-Scratch purely from Python training. Two extensions on the same substrate close further open problems. A closed-loop meta-control layer (Extension A) reduces forgetting by an additional 81% at ~398M, mapping onto the System A and System M roles of Dupoux et al. (arXiv:2603.15381). An operator-level plan vector (Extension B) reshapes forward-pass behavior at 99.96% cosine fidelity over 30 source->target pairs. The architectural insight is a Read/Write decomposition: the forward pass is fully dense, while cross-domain parameter updates are structured so prior-domain subspaces are not written to. To our knowledge, TFGN is the first architecture that simultaneously closes catastrophic forgetting at LLM scale, realizes a closed-loop autonomous-learning meta-controller, and carries an operator-level latent planner.