Convergence of Continual Learning in Homogeneous Deep Networks

Matan Schliserman; Gon Buzaglo; Itay Evron; Daniel Soudry

cs.LGJun 29, 2026

Convergence of Continual Learning in Homogeneous Deep Networks

Authors: Matan Schliserman, Gon Buzaglo, Itay Evron, Daniel Soudry

Organizations: Blavatnik School of Computer Science and AI, Tel Aviv University · Princeton University · Department of Electrical and Computing Engineering, Technion

Abstract

We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets. This result generalizes prior analyses restricted to either stationary (single-task) deep models or continual linear models. We show that global convergence generally fails, even for simple models linear in data but nonlinear in parameters. Nevertheless, by leveraging results from nonconvex projection theory, we identify regularity properties of homogeneous deep networks that guarantee local linear convergence under random and cyclic task sequences. Finally, we extend our analysis to continual regression, unifying the framework for homogeneous models.

Explore similar work

Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective
May 9, 2026Jiuqi Wang, Jayanth Srinivasa, Claire Chen +3Plasticity Efficient Continual Learning
Fine-Tuning Regimes Define Distinct Continual Learning Problems
Apr 23, 2026Paul-Tiberiu Iordache, Elena BurceanuEfficient Continual Learning Continual Fine-Tuning
The Bayesian Approach to Continual Learning: An Overview
Jul 11, 2025Tameem AdelEfficient Continual Learning Continual Learning Methods

May 9, 2026 · cs.LGJ/K move · Enter open · S save

Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective

Jiuqi Wang, Jayanth Srinivasa, Claire Chen, Shuze Daniel Liu+2

University of Virginia · Cisco Research · California Institute of Technology · Purdue University

Deep continual learning requires models to adapt to new tasks without retraining from scratch. However, neural networks can lose their ability to adapt to new tasks after training on previous ones, a phenomenon known as loss of plasticity. There have been several explanations and diagnostics proposed for plasticity loss. Motivated by the philosophy that "all models are wrong, but some are useful", we ask: can existing diagnostics predict a neural network's plasticity? In this work, we take a practical view to interpret plasticity as trainability, i.e., a neural network's future optimization gain on a target task. We first take a theoretical approach, showing, by constructing a few counterexamples, that some widely adopted diagnostics of plasticity, including representation rank and neural tangent kernel rank, can fail to predict the loss of trainability in both regression and classification settings. We instead propose a novel metric, called optimization readiness, which combines gradient strength and gradient reliability. We prove that optimization readiness lower bounds one-step optimization gain under standard smoothness assumptions, providing a theoretical guarantee for its predictive power. Empirically, we show that across commonly used deep continual learning settings, such as Slowly-Changing Regression and Permuted MNIST, optimization readiness more reliably ranks checkpoints by trainability than prior diagnostics, even with substantially fewer samples.

Plasticity Efficient Continual Learning Continual Learning Methods Trainability Theoretical Foundations

Open paper Ask about it PDF

Abstract

Explore similar work

Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective

Fine-Tuning Regimes Define Distinct Continual Learning Problems

The Bayesian Approach to Continual Learning: An Overview