Learning in the Null Space: Small Singular Values for Continual Learning

Cuong Anh Pham; Praneeth Vepakomma; Samuel Horváth

cs.LGFeb 25, 2026

Learning in the Null Space: Small Singular Values for Continual Learning

Authors: Cuong Anh Pham, Praneeth Vepakomma, Samuel Horváth

Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · Massachusetts Institute of Technology (MIT), USA

Abstract

Alleviating catastrophic forgetting while enabling further learning is a primary challenge in continual learning (CL). Orthogonal-based training methods have gained attention for their efficiency and strong theoretical properties, and many existing approaches enforce orthogonality through gradient projection. In this paper, we revisit orthogonality and exploit the fact that small singular values correspond to directions that are nearly orthogonal to the input space of previous tasks. Building on this principle, we introduce NESS (Null-space Estimated from Small Singular values), a CL method that applies orthogonality directly in the weight space rather than through gradient manipulation. Specifically, NESS constructs an approximate null space using the smallest singular values of each layer's input representation and parameterizes task-specific updates via a compact low-rank adaptation (LoRA-style) formulation constrained to this subspace. The subspace basis is fixed to preserve the null-space constraint, and only a single trainable matrix is learned for each task. This design ensures that the resulting updates remain approximately in the null space of previous inputs while enabling adaptation to new tasks. Our theoretical analysis and experiments on three benchmark datasets demonstrate competitive performance, low forgetting, and stable accuracy across tasks, highlighting the role of small singular values in continual learning. The code is available at https://github.com/pacman-ctm/NESS.

Figures & tables

Figure 1: Overview of NESS. During continual training (left), each task updates the network sequentially. For every task

t

and layer

l

, we collect the concatenated inputs from previous tasks and compute the SVD of the corresponding covariance matrix (right). The update is parameterized as a low-rank decomposition

\Delta W_{t}^{(l)}=U_{t}^{(l)}V_{t}^{(l)}

, where

U_{t}^{(l)}

is a frozen orthogonal basis constructed from singular vectors associated with small singular values, and

V_{t}^{(l)}

is a trainable matrix initialized to zero. This structured update constrains learning to an approximate null subspace of previous inputs, limiting interference while enabling adaptation to the current task.

Method	CIFAR-100 (10 Tasks)		5-datasets (5 Tasks)		MiniImageNet (20 Tasks)
Method	ACC(%)	BWT(%) $\uparrow$	ACC(%)	BWT(%) $\uparrow$	ACC(%)	BWT(%) $\uparrow$
OWM ‡ [ 13 ]	50.94 $\pm$ 0.60	-30 $\pm$ 1	–	–	–	–
EWC ‡ [ 11 ]	68.88 $\pm$ 0.80	-2 $\pm$ 1	88.64 $\pm$ 0.26	-4 $\pm$ 1	52.01 $\pm$ 2.53	-12 $\pm$ 3
HAT ‡ [ 10 ]	72.06 $\pm$ 0.50	0 $\pm$ 0	91.32 $\pm$ 0.18	-1 $\pm$ 0	59.78 $\pm$ 0.57	-3 $\pm$ 0
A-GEM ‡ [ 7 ]	63.98 $\pm$ 1.22	-15 $\pm$ 2	84.04 $\pm$ 0.33	-12 $\pm$ 1	57.24 $\pm$ 0.72	-12 $\pm$ 1
GPM [ 8 ]	71.63 $\pm$ 0.67	-0.28 $\pm$ 0.54	90.61 $\pm$ 0.57	-1.17 $\pm$ 0.26	63.56 $\pm$ 2.42	-1.39 $\pm$ 1.37

Table 1: The averaged accuracy (ACC) and backward transfer (BWT) over all tasks on different datasets (with the best learning rate scenario). Accuracy and BWT (mean

\pm

std) are reported based on 5 different seeds. The best BWTs are in bold, and the second-best BWTs are underlined, and all the BWTs above

-1\%

have a yellow background . For notations: † denotes that the baseline has no previous code/result for all datasets, and ‡ denotes the results taken from [ 23 ] . For a more detailed result, please see Appendix A.2 .

Panel of Figure 2 — Figure 2: Performance (best setting of each method, focusing on forgetting rate, with green color denotes improving performance and red color denotes losing performance, compared to the blue color cell for the performance when the task first trained) of (a) NESS compared to (b) SGP; (c) TRGP; (d) DFGP on miniImageNet experiment (seed 3).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	CIFAR-100	MiniImageNet
Number of tasks ( $T$ )	10	20
Number of classes per task	10	5
Training samples per task	4750	2375
Validation samples per task	250	125
Test samples per task	1000	500

Appendix

Table 2: Split CIFAR-100 and split MiniImageNet datasets statistics.

Dataset	CIFAR10	MNIST	SVHN	Fashion MNIST	notMNIST
Number of classes	10	10	10	10	10
Training samples	47500	57000	69595	57000	16011
Validation samples	2500	3000	3662	3000	842
Test samples	10000	10000	26032	10000	1873

Appendix

Table 3: 5-Datasets statistics (notice that each dataset is one task).

Dataset	CIFAR-100	5-Datasets	MiniImageNet
Batch size ( $b$ )	64	64	64
Learning rate decay factor	1/2	1/3	1/2
Patience	6	5	6
Number of epochs	200	100	100
Momentum (for SGD with momentum)	0.9	0.9	0.9

Appendix

Table 4: List of (general) hyperparameters used in our experiments.

Setting	Seed 1		Seed 2		Seed 3		Seed 4		Seed 37
	ACC	BWT	ACC	BWT	ACC	BWT	ACC	BWT	ACC	BWT
GPM	71.85	-0.67	72.00	-0.41	72.07	-0.44	71.78	0.67	70.45	-0.56
SGP	75.89	-0.63	75.78	-1.63	76.01	-1.03	76.18	-1.33	76.07	-1.26
TRGP	75.10	-0.01	75.62	-0.09	74.92	-0.06	75.49	0.32	74.94	0.15
FS-DGPM	73.95	-3.30	74.16	-3.13	74.14	-2.87	74.09	-2.56	74.16	-3.28
DFGP (lr=0.01, mixup=0.01)	73.09	-0.80	73.39	-1.07	72.90	-1.14	73.49	-0.71	73.34	-1.02

Appendix

Table 5: Performance across 5 random seeds (detailed results) for CIFAR-100 experiment. For SGD with momentum (SGDm), we used momentum=0.9 and we denoted

\lambda

= weight decay.

Setting	Mean $\pm$ std
	ACC	BWT
GPM	71.63 $\pm$ 0.67	-0.28 $\pm$ 0.54
SGP	75.99 $\pm$ 0.16	-1.18 $\pm$ 0.37
TRGP	75.21 $\pm$ 0.32	0.06 $\pm$ 0.17
FS-DGPM	74.10 $\pm$ 0.09	-3.03 $\pm$ 0.31
DFGP (lr=0.01, mixup=0.01)	73.24 $\pm$ 0.24	-0.95 $\pm$ 0.18

Appendix

Table 6: Performance across 5 random seeds (average performance) for CIFAR-100 experiment. For SGD with momentum (SGDm), we used momentum=0.9 and we denoted

\lambda

= weight decay.

Setting	Seed 1		Seed 2		Seed 3		Seed 4		Seed 37
	ACC	BWT	ACC	BWT	ACC	BWT	ACC	BWT	ACC	BWT
GPM	91.27	-0.81	89.98	-1.52	91.1	-1.08	90.12	-1.14	90.56	-1.31
SGP	91.20	-1.70	90.16	-1.88	90.72	-1.98	90.03	-1.88	90.30	-1.67
TRGP	93.59	0.03	92.24	-0.12	93.32	-0.08	92.12	-0.18	92.61	-0.12
DFGP (lr=0.1, mixup=0.01)	91.12	-2.67	91.48	-2.47	91.48	-2.22	91.55	-2.15	91.72	-1.86
DFGP (lr=0.1, mixup=0.05)	89.84	-4.54	90.70	-3.28	89.65	-4.30	90.34	-3.72	90.57	-3.50

Appendix

Table 7: Performance across 5 random seeds (detailed results) for 5-datasets experiment. For SGD with momentum (SGDm), we used momentum=0.9 and we denoted

\lambda

= weight decay.

Setting	Mean $\pm$ std
	ACC	BWT
GPM	90.61 $\pm$ 0.57	-1.17 $\pm$ 0.26
SGP	90.48 $\pm$ 0.48	-1.82 $\pm$ 0.13
TRGP	92.78 $\pm$ 0.65	-0.09 $\pm$ 0.08
DFGP (lr=0.1, mixup=0.01)	91.47 $\pm$ 0.22	-2.27 $\pm$ 0.31
DFGP (lr=0.1, mixup=0.05)	90.22 $\pm$ 0.46	-3.87 $\pm$ 0.53

Appendix

Table 8: Performance across 5 random seeds (average performance) for 5-datasets experiment. For SGD with momentum, we used momentum=0.9 and we denoted

\lambda

= weight decay.

Setting	Seed 1		Seed 2		Seed 3		Seed 4		Seed 37
	ACC	BWT	ACC	BWT	ACC	BWT	ACC	BWT	ACC	BWT
GPM	61.14	-3.54	66.59	-0.41	61.52	-1.95	65.53	-0.82	63.03	-0.25
SGP	65.42	-0.96	66.71	-0.14	61.41	-1.16	65.71	0.48	63.00	-0.88
TRGP	61.70	-1.34	65.06	-0.38	59.59	-1.99	63.74	-0.50	63.61	-1.96
DFGP (lr=0.1, mixup=0.1)	67.18	-0.33	68.82	-0.53	68.25	-0.87	69.53	-0.74	68.76	0.51
DFGP (lr=0.1, mixup=0.05)	68.55	0.58	70.49	1.56	66.25	-1.79	68.63	-1.20	68.56	0.28

Appendix

Table 9: Performance across 5 random seeds (detailed result) for MiniImageNet experiment. For SGD with momentum (SGDm), we used momentum=0.9 and we denoted

\lambda

= weight decay.

Setting	Mean $\pm$ std
	ACC	BWT
GPM	63.56 $\pm$ 2.42	-1.39 $\pm$ 1.37
SGP	64.45 $\pm$ 2.18	-0.53 $\pm$ 0.68
TRGP	62.74 $\pm$ 2.13	-1.23 $\pm$ 0.77
DFGP (lr=0.1, mixup=0.1)	68.51 $\pm$ 0.87	-0.39 $\pm$ 0.54
DFGP (lr=0.1, mixup=0.05)	68.50 $\pm$ 1.50	-0.11 $\pm$ 1.36

Appendix

Table 10: Performance across 5 random seeds (average performance) for MiniImageNet experiment. For SGD with momentum, we used momentum=0.9 and we denoted

\lambda

= weight decay.

Panel of Figure 3 — Figure 3: Performance (best setting of each method, focusing on forgetting rate, with green color denotes improving performance and red color denotes losing performance, compared to the time the task firstly trained) of (a) NESS compared to (b) SGP; (c) TRGP; (d) DFGP on CIFAR-100 experiment (seed 2).

Panel of Figure 4 — Figure 4: Performance (best setting of each method, focusing on forgetting rate, with green color denotes improving performance and red color denotes losing performance, compared to the time the task firstly trained) of (a) NESS compared to (b) SGP; (c) TRGP; (d) DFGP on CIFAR-100 experiment (seed 3).

Panel of Figure 5 — Figure 5: Performance (best setting of each method, focusing on forgetting rate, with green color denotes improving performance and red color denotes losing performance, compared to the time the task firstly trained) of (a) NESS compared to (b) SGP; (c) TRGP; (d) DFGP on miniImageNet experiment (seed 37).

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 294 papers

Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation
Jan 31, 2026Hao Gu, Mao-Lin Luo, Zi-Hao Zhou +3VLM Adaptation Continual Learning
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China
Energy-Structured Low-Rank Adaptation for Continual Learning
May 26, 2026Longhua Li, Lei Qi, Qi Tian +1Continual Learning Low-Rank Adaptation
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China
Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
Jul 26, 2026Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2Continual Learning for LLMs Continual LoRA
College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1

Jan 31, 2026J/K move · Enter open · S save

Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation

Hao Gu, Mao-Lin Luo, Zi-Hao Zhou, Han-Chen Zhang+2

School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China

VLM Adaptation Continual Learning Low-Rank Adaptation Catastrophic Forgetting Continual LoRA

Open paper AskPDF arXiv

Parameter-efficient continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge. Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge. From a knowledge-decomposition perspective, we observe that low-rank adaptations exhibit highly imbalanced singular value spectra: a few dominant components absorb most of the adaptation energy, thereby (i) more likely to disrupt previously acquired knowledge and (ii) making the update more vulnerable to interference from subsequent tasks. To enable explicit balance among components, we decouple the magnitude of the task update from its directional structure and formulate it as a constrained optimization problem on a restricted Stiefel manifold. We address this problem using a projected first-order method compatible with standard deep-learning optimizers used in vision-language models. Our method mitigates both backward and forward forgetting, consistently outperforming continual learning baselines. The implementation code is available at https://github.com/haodotgu/EBLoRA.

Abstract

Continual Learning

Explore similar work

Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation

Energy-Structured Low-Rank Adaptation for Continual Learning

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning