Regularization-Based Continual Learning

Follow this topic Ask about this topic

Latest papers

24 papers10 papers in the last four weeks, with none the four weeks before. 0.1% of all new papers.

Continual Classification Beyond the Kernel Regime: Harnessing the Sparsity Bias of L2 Regularization in Diagonal Linear Networks
Nov 4, 2026Gilad Karpel, Itay Evron, Ron Meir +1Regularization-Based Continual Learning Statistical Learning Theory
Technion · NVIDIA
Drift-adaptive Representation Preservation for Domain-Incremental Learning
Nov 4, 2026Hyunmin Kong, Gyutae Oh, Jitae ShinDomain-Incremental Learning Regularization-Based Continual Learning
Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419, Republic of Korea
Beyond Parameter Protection: Fisher-Shaped EWC for Replay-Free Continual Learning
Oct 27, 2026Liying Wang, Simone Magistri, Joost van de WeijerElastic Weight Consolidation Regularization-Based Continual Learning
Department of Computer Science, Universitat Aut`onoma de Barcelona, Spain · Computer Vision Center, Barcelona, Spain · Media Integration and Communication Center, University of Florence, Italy
Semantic Concept Steering Breaks the Explanation Drift Loop in Continual Learning
Oct 24, 2026Yehonatan Elisha, Oren Barkan, Noam KoenigsteinRegularization-Based Continual Learning Spurious Correlation Robustness
Tel Aviv University · The Open University
Score-Preserving Continual Lasso under Task Shift
Oct 23, 2026Jingyi Guo, Runze Li, Lingzhou XueContinual Learning Regularization-Based Continual Learning
Pennsylvania State University
DRIVE: Fine-tuning via Data Contribution- and Diversity-aware Weighting with Prior Regularization
Oct 23, 2026Qing Liu, Xinrui Chen, Yi Du +2Fine-Tuning Instance Reweighting
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China · Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · School of Mathematics, Tianjin University, Tianjin, China
REDAR: Reinforcement-Driven Adaptive Regularisation for Class Incremental Learning
Oct 22, 2026Akila Ayanthi, Darshana Priyasad, Tharindu Fernando +3RL HPO Class-Incremental Learning
SAIVT, Queensland University of Technology, Australia · CSIRO Robotics, CSIRO, Australia
Why Constrained Updates Still Forget: A Functional-Path Analysis of Continual SFT
Oct 21, 2026Qingsen Ma, Dianyun Wang, Jiahong Liu +1Supervised Fine-Tuning Continual Learning for LLMs
The Chinese University of Hong Kong · Beijing University of Posts and Telecommunications
Sparse Autoencoders Can Induce Local Geometry for Mitigating Catastrophic Forgetting
Oct 15, 2026Kamil Faber, Katarzyna Filus, Łukasz PawelaCatastrophic Forgetting Regularization-Based Continual Learning
Faculty of Computer Science, AGH University of Krakow, Krakow, Poland · Institute of Theoretical and Applied Informatics, Polish Academy of Sciences, Gliwice, Poland
Optimal Explicit and Implicit Regularization Strengths in High-Dimensional Continual Linear Regression
Oct 14, 2026Gilad Karpel, Daniel SoudryHigh-Dimensional Linear Regression Regularization-Based Continual Learning
Department of Data and Decisions Sciences, Technion · Department of Electrical and Computer Engineering, Technion
SPARS: A Stability-Plasticity-Aware Region Selection Framework for Continual Learning
Oct 14, 2026Teif Aldaajani, Huda Hakami, Maha A. ThafarRegularization-Based Continual Learning Parameter Isolation
Department of Computer Science, College of Computers and Information Technology, Taif University, Taif, 21944, Saudi Arabia .
Do Artificial Neural Networks Need Dendritic Nanotubes? Two Reasons for Shrinking Importance Estimates in Continual Learning
Oct 9, 2026Saurav K. AryalRegularization-Based Continual Learning Incremental Learning
Howard University, Institute for Human Centered Artificial Intelligence, Washington, DC, USA
An Inclusive and Lightweight Approach to Federated Continual Learning for Cultural Heritage
Aug 20, 2026Ioannis Theologitis, Debin Meng, Stylianos Eleftheriadis +2Regularization-Based Continual Learning Federated Continual Learning
Information Technologies Institute, Centre for Research and Technology Hellas, Thessaloniki, Greece · School of Electronic Engineering and Computer Science, Queen Mary University of London, UK
Layers Matter: Why Continual Learning Regularization Should Be Layer-Adaptive
Aug 16, 2026Brian B. Moser, Ahmed Anwar, Tobias Christian Nauen +5Regularization-Based Continual Learning Continual Learning
German Research Center for Artificial Intelligence (DFKI) · RPTU University Kaiserslautern-Landau
Omega-S: A Functional Resilience Index for LLM Fine-Tuning
Aug 4, 2026Alberto AcedoRegularization-Based Continual Learning Continual Learning for LLMs
Biome Makers Inc.
Relative Parameter Importance in Task-Agnostic Replay-Free Continual Learning
Aug 1, 2026Malavika Suresh, Ikechukwu Nkisi-Orji, Nirmalie WiratungaClass-Incremental Learning Regularization-Based Continual Learning
Robert Gordon University
Theory of Continual Learning Against Data Poisoning Attacks
Jun 29, 2026Yiting Hu, Lingjie DuanRegularization-Based Continual Learning Continual Learning
Singapore University of Technology and Design, Singapore · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.
From Weights to Features: SAE-Guided Activation Regularization for LLM Continual Learning
Jun 25, 2026Evan Ning, Wei Xue, Dong Lou +1Continual Learning for LLMs Catastrophic Forgetting
1The Hong Kong University of Science and Technology
SAE-FD: Sparse Autoencoder Feature Distillation for Continual Learning of Large Language Models
May 25, 2026Mingxu Zhang, Yuhan Li, Lujundong Li +3Regularization-Based Continual Learning Continual Learning for LLMs
The Hong Kong University of Science and Technology (Guangzhou) · Nanjing University of Aeronautics and Astronautics · The 63rd Research Institute, National University of Defense Technology, Nanjing
KAN-CL: Per-Knot Importance Regularization for Continual Learning with Kolmogorov-Arnold Networks
May 12, 2026Minjong CheonRegularization-Based Continual Learning Continual Learning
Sejong University Department of Computer Science and Engineering
Elastic Weight Consolidation Done Right for Continual Learning
Mar 19, 2026Xuan Liu, Xiaobin ChangRegularization-Based Continual Learning Continual Learning
School of Artificial Intelligence, Sun Yat-sen University, China · Key Laboratory of Intelligent Assessment Technology for Sustainable Tourism, Ministry of Culture and Tourism, Sun Yat-sen University · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China
Trust Region Continual Learning as an Implicit Meta-Learner
Feb 2, 2026Zekun Wang, Anant Gupta, Christopher J. MacLellanReplay-Based Continual Learning Task-Incremental Learning
Georgia Institute of Technology
Optimal L2 Regularization in High-dimensional Continual Linear Regression
Jan 20, 2026Gilad Karpel, Edward Moroshko, Ran Levinstein +3Regularization-Based Continual Learning High-Dimensional Linear Regression
Department of Data and Decisions Sciences, Technion · School of Engineering, University of Edinburgh · Department of Computer Science, Technion +2
InTAct: Interval-based Task Activation Consolidation for Continual Learning
Nov 21, 2025Patryk Krukowski, Jan Miksa, Piotr Helm +3Prompt-Based Continual Learning Continual Learning
Jagiellonian University · AKCES NCBR · IDEAS Research Institute.

Nov 4, 2026J/K move · Enter open · S save

Continual Classification Beyond the Kernel Regime: Harnessing the Sparsity Bias of L2 Regularization in Diagonal Linear Networks

Gilad Karpel, Itay Evron, Ron Meir, Daniel Soudry

Technion · NVIDIA

Regularization-Based Continual Learning Statistical Learning Theory

Open paper AskPDF arXiv

We study logistic regression for continual learning with linear predictors in a high-dimensional statistical setting. Existing theoretical work has largely focused on continual linear regression with variants of L2 regularization, motivated by their analytical simplicity and connection to the kernel regime of neural networks. We first derive the optimal L2 regularization strength for continual linear classification and characterize its generalization performance, recovering behavior analogous to prior results for continual linear regression. Next, we examine the effect of L2 regularization in deeper models using a minimal two-layer diagonal linear network. In the single-task setting, L2 regularization on the weights of such models is known to promote sparsity in the resulting predictor, yielding a rich, sparse feature-learning regime beyond the kernel regime (a similar effect arises in two-layer ReLU networks). However, in continual learning, where L2 regularization is typically centered on previous-task weights, this sparsity bias is no longer preserved. We therefore propose a weighted variant of L2 regularization that extends this beneficial sparsity bias to continual learning. We show theoretically and empirically that, for a sufficiently sparse teacher, our variant converges substantially faster than directly learning a linear predictor with conventional continual L2 or L1 regularization, trading some long-horizon bias for markedly improved performance over moderate task horizons.