Learning in the Null Space: Small Singular Values for Continual Learning

Cuong Anh Pham; Praneeth Vepakomma; Samuel Horváth

cs.LGFeb 25, 2026

Learning in the Null Space: Small Singular Values for Continual Learning

Authors: Cuong Anh Pham, Praneeth Vepakomma, Samuel Horváth

Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · Massachusetts Institute of Technology (MIT), USA

Abstract

Alleviating catastrophic forgetting while enabling further learning is a primary challenge in continual learning (CL). Orthogonal-based training methods have gained attention for their efficiency and strong theoretical properties, and many existing approaches enforce orthogonality through gradient projection. In this paper, we revisit orthogonality and exploit the fact that small singular values correspond to directions that are nearly orthogonal to the input space of previous tasks. Building on this principle, we introduce NESS (Null-space Estimated from Small Singular values), a CL method that applies orthogonality directly in the weight space rather than through gradient manipulation. Specifically, NESS constructs an approximate null space using the smallest singular values of each layer's input representation and parameterizes task-specific updates via a compact low-rank adaptation (LoRA-style) formulation constrained to this subspace. The subspace basis is fixed to preserve the null-space constraint, and only a single trainable matrix is learned for each task. This design ensures that the resulting updates remain approximately in the null space of previous inputs while enabling adaptation to new tasks. Our theoretical analysis and experiments on three benchmark datasets demonstrate competitive performance, low forgetting, and stable accuracy across tasks, highlighting the role of small singular values in continual learning. The code is available at https://github.com/pacman-ctm/NESS.

Figures & tables

Figure 1: Overview of NESS. During continual training (left), each task updates the network sequentially. For every task

t

and layer

l

, we collect the concatenated inputs from previous tasks and compute the SVD of the corresponding covariance matrix (right). The update is parameterized as a low-rank decomposition

\Delta W_{t}^{(l)}=U_{t}^{(l)}V_{t}^{(l)}

, where

U_{t}^{(l)}

is a frozen orthogonal basis constructed from singular vectors associated with small singular values, and

V_{t}^{(l)}

is a trainable matrix initialized to zero. This structured update constrains learning to an approximate null subspace of previous inputs, limiting interference while enabling adaptation to the current task.

Method	CIFAR-100 (10 Tasks)		5-datasets (5 Tasks)		MiniImageNet (20 Tasks)
Method	ACC(%)	BWT(%) $\uparrow$	ACC(%)	BWT(%) $\uparrow$	ACC(%)	BWT(%) $\uparrow$
OWM ‡ [ 13 ]	50.94 $\pm$ 0.60	-30 $\pm$ 1	–	–	–	–
EWC ‡ [ 11 ]	68.88 $\pm$ 0.80	-2 $\pm$ 1	88.64 $\pm$ 0.26	-4 $\pm$ 1	52.01 $\pm$ 2.53	-12 $\pm$ 3
HAT ‡ [ 10 ]	72.06 $\pm$ 0.50	0 $\pm$ 0	91.32 $\pm$ 0.18	-1 $\pm$ 0	59.78 $\pm$ 0.57	-3 $\pm$ 0
A-GEM ‡ [ 7 ]	63.98 $\pm$ 1.22	-15 $\pm$ 2	84.04 $\pm$ 0.33	-12 $\pm$ 1	57.24 $\pm$ 0.72	-12 $\pm$ 1
GPM [ 8 ]	71.63 $\pm$ 0.67	-0.28 $\pm$ 0.54	90.61 $\pm$ 0.57	-1.17 $\pm$ 0.26	63.56 $\pm$ 2.42	-1.39 $\pm$ 1.37

Table 1: The averaged accuracy (ACC) and backward transfer (BWT) over all tasks on different datasets (with the best learning rate scenario). Accuracy and BWT (mean

\pm

std) are reported based on 5 different seeds. The best BWTs are in bold, and the second-best BWTs are underlined, and all the BWTs above

-1\%

have a yellow background . For notations: † denotes that the baseline has no previous code/result for all datasets, and ‡ denotes the results taken from [ 23 ] . For a more detailed result, please see Appendix A.2 .

Panel of Figure 2 — Figure 2: Performance (best setting of each method, focusing on forgetting rate, with green color denotes improving performance and red color denotes losing performance, compared to the blue color cell for the performance when the task first trained) of (a) NESS compared to (b) SGP; (c) TRGP; (d) DFGP on miniImageNet experiment (seed 3).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	CIFAR-100	MiniImageNet
Number of tasks ( $T$ )	10	20
Number of classes per task	10	5
Training samples per task	4750	2375
Validation samples per task	250	125
Test samples per task	1000	500

Appendix

Table 2: Split CIFAR-100 and split MiniImageNet datasets statistics.

Dataset	CIFAR10	MNIST	SVHN	Fashion MNIST	notMNIST
Number of classes	10	10	10	10	10
Training samples	47500	57000	69595	57000	16011
Validation samples	2500	3000	3662	3000	842
Test samples	10000	10000	26032	10000	1873

Appendix

Table 3: 5-Datasets statistics (notice that each dataset is one task).

Dataset	CIFAR-100	5-Datasets	MiniImageNet
Batch size ( $b$ )	64	64	64
Learning rate decay factor	1/2	1/3	1/2
Patience	6	5	6
Number of epochs	200	100	100
Momentum (for SGD with momentum)	0.9	0.9	0.9

Appendix

Table 4: List of (general) hyperparameters used in our experiments.

Setting	Seed 1		Seed 2		Seed 3		Seed 4		Seed 37
	ACC	BWT	ACC	BWT	ACC	BWT	ACC	BWT	ACC	BWT
GPM	71.85	-0.67	72.00	-0.41	72.07	-0.44	71.78	0.67	70.45	-0.56
SGP	75.89	-0.63	75.78	-1.63	76.01	-1.03	76.18	-1.33	76.07	-1.26
TRGP	75.10	-0.01	75.62	-0.09	74.92	-0.06	75.49	0.32	74.94	0.15
FS-DGPM	73.95	-3.30	74.16	-3.13	74.14	-2.87	74.09	-2.56	74.16	-3.28
DFGP (lr=0.01, mixup=0.01)	73.09	-0.80	73.39	-1.07	72.90	-1.14	73.49	-0.71	73.34	-1.02

Appendix

Table 5: Performance across 5 random seeds (detailed results) for CIFAR-100 experiment. For SGD with momentum (SGDm), we used momentum=0.9 and we denoted

\lambda

= weight decay.

Setting	Mean $\pm$ std
	ACC	BWT
GPM	71.63 $\pm$ 0.67	-0.28 $\pm$ 0.54
SGP	75.99 $\pm$ 0.16	-1.18 $\pm$ 0.37
TRGP	75.21 $\pm$ 0.32	0.06 $\pm$ 0.17
FS-DGPM	74.10 $\pm$ 0.09	-3.03 $\pm$ 0.31
DFGP (lr=0.01, mixup=0.01)	73.24 $\pm$ 0.24	-0.95 $\pm$ 0.18

Appendix

Table 6: Performance across 5 random seeds (average performance) for CIFAR-100 experiment. For SGD with momentum (SGDm), we used momentum=0.9 and we denoted

\lambda

= weight decay.

Setting	Seed 1		Seed 2		Seed 3		Seed 4		Seed 37
	ACC	BWT	ACC	BWT	ACC	BWT	ACC	BWT	ACC	BWT
GPM	91.27	-0.81	89.98	-1.52	91.1	-1.08	90.12	-1.14	90.56	-1.31
SGP	91.20	-1.70	90.16	-1.88	90.72	-1.98	90.03	-1.88	90.30	-1.67
TRGP	93.59	0.03	92.24	-0.12	93.32	-0.08	92.12	-0.18	92.61	-0.12
DFGP (lr=0.1, mixup=0.01)	91.12	-2.67	91.48	-2.47	91.48	-2.22	91.55	-2.15	91.72	-1.86
DFGP (lr=0.1, mixup=0.05)	89.84	-4.54	90.70	-3.28	89.65	-4.30	90.34	-3.72	90.57	-3.50

Appendix

Table 7: Performance across 5 random seeds (detailed results) for 5-datasets experiment. For SGD with momentum (SGDm), we used momentum=0.9 and we denoted

\lambda

= weight decay.

Setting	Mean $\pm$ std
	ACC	BWT
GPM	90.61 $\pm$ 0.57	-1.17 $\pm$ 0.26
SGP	90.48 $\pm$ 0.48	-1.82 $\pm$ 0.13
TRGP	92.78 $\pm$ 0.65	-0.09 $\pm$ 0.08
DFGP (lr=0.1, mixup=0.01)	91.47 $\pm$ 0.22	-2.27 $\pm$ 0.31
DFGP (lr=0.1, mixup=0.05)	90.22 $\pm$ 0.46	-3.87 $\pm$ 0.53

Appendix

Table 8: Performance across 5 random seeds (average performance) for 5-datasets experiment. For SGD with momentum, we used momentum=0.9 and we denoted

\lambda

= weight decay.

Setting	Seed 1		Seed 2		Seed 3		Seed 4		Seed 37
	ACC	BWT	ACC	BWT	ACC	BWT	ACC	BWT	ACC	BWT
GPM	61.14	-3.54	66.59	-0.41	61.52	-1.95	65.53	-0.82	63.03	-0.25
SGP	65.42	-0.96	66.71	-0.14	61.41	-1.16	65.71	0.48	63.00	-0.88
TRGP	61.70	-1.34	65.06	-0.38	59.59	-1.99	63.74	-0.50	63.61	-1.96
DFGP (lr=0.1, mixup=0.1)	67.18	-0.33	68.82	-0.53	68.25	-0.87	69.53	-0.74	68.76	0.51
DFGP (lr=0.1, mixup=0.05)	68.55	0.58	70.49	1.56	66.25	-1.79	68.63	-1.20	68.56	0.28

Appendix

Table 9: Performance across 5 random seeds (detailed result) for MiniImageNet experiment. For SGD with momentum (SGDm), we used momentum=0.9 and we denoted

\lambda

= weight decay.

Setting	Mean $\pm$ std
	ACC	BWT
GPM	63.56 $\pm$ 2.42	-1.39 $\pm$ 1.37
SGP	64.45 $\pm$ 2.18	-0.53 $\pm$ 0.68
TRGP	62.74 $\pm$ 2.13	-1.23 $\pm$ 0.77
DFGP (lr=0.1, mixup=0.1)	68.51 $\pm$ 0.87	-0.39 $\pm$ 0.54
DFGP (lr=0.1, mixup=0.05)	68.50 $\pm$ 1.50	-0.11 $\pm$ 1.36

Appendix

Table 10: Performance across 5 random seeds (average performance) for MiniImageNet experiment. For SGD with momentum, we used momentum=0.9 and we denoted

\lambda

= weight decay.

Panel of Figure 3 — Figure 3: Performance (best setting of each method, focusing on forgetting rate, with green color denotes improving performance and red color denotes losing performance, compared to the time the task firstly trained) of (a) NESS compared to (b) SGP; (c) TRGP; (d) DFGP on CIFAR-100 experiment (seed 2).

Panel of Figure 4 — Figure 4: Performance (best setting of each method, focusing on forgetting rate, with green color denotes improving performance and red color denotes losing performance, compared to the time the task firstly trained) of (a) NESS compared to (b) SGP; (c) TRGP; (d) DFGP on CIFAR-100 experiment (seed 3).

Panel of Figure 5 — Figure 5: Performance (best setting of each method, focusing on forgetting rate, with green color denotes improving performance and red color denotes losing performance, compared to the time the task firstly trained) of (a) NESS compared to (b) SGP; (c) TRGP; (d) DFGP on miniImageNet experiment (seed 37).

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 294 papers

Jan 31, 2026

Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation

Parameter-efficient continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge. Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge. From a knowledge-decomposition perspective, we observe that low-rank adaptations exhibit highly imbalanced singular value spectra: a few dominant components absorb most of the adaptation energy, thereby (i) more likely to disrupt previously acquired knowledge and (ii) making the update more vulnerable to interference from subsequent tasks. To enable explicit balance among components, we decouple the magnitude of the task update from its directional structure and formulate it as a constrained optimization problem on a restricted Stiefel manifold. We address this problem using a projected first-order method compatible with standard deep-learning optimizers used in vision-language models. Our method mitigates both backward and forward forgetting, consistently outperforming continual learning baselines. The implementation code is available at https://github.com/haodotgu/EBLoRA.

Hao Gu, Mao-Lin Luo, Zi-Hao Zhou +3

School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China

VLM Adaptation Continual Learning Low-Rank Adaptation Catastrophic Forgetting Continual LoRA

Ask PDF

May 26, 2026

Energy-Structured Low-Rank Adaptation for Continual Learning

While orthogonal subspace methods try to mitigate task interference in Continual Learning (CL), they often suffer from energy diffusion across the basis, hindering knowledge compaction and exhausting capacity for future tasks. We observe that output feature drift induced by parameter updates is inherently low-rank, and theoretically prove that preserving parameters along the principal directions of this drift minimizes the output reconstruction error. Motivated by this, we propose Energy-Concentrated and Energy-Ordered Low-Rank Adaptation (E

^2

-LoRA). By explicitly ordering and concentrating knowledge into leading ranks, E

^2

-LoRA frees capacity for subsequent tasks. Furthermore, we design a dynamic rank allocation strategy to balance stability and plasticity by jointly optimizing energy retention and model plasticity. Extensive experiments across multiple benchmarks demonstrate that E

^2

-LoRA achieves state-of-the-art performance. Code is available at https://github.com/kiddo127/E2-LoRA.

Longhua Li, Lei Qi, Qi Tian +1

School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China

Continual Learning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning Continual LoRA

Ask PDF

Jul 26, 2026

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially, leading to catastrophic forgetting. To mitigate this, LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately, letting irrelevant branches distort the output. Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting. In this work, we observe that pooled token embeddings from a frozen LLM embedding layer already separate task distributions throughout the learning sequence. A Gaussian mixture model fitted on these embeddings, without any gradient-based training, is sufficient for task-agnostic adapter selection at test time. This eliminates the need for a learned gating module. On the adapter side, constraining each task's parameters to the principal subspace of the pretrained weights via SVD yields a compact latent-space parameterization. Within this subspace, orthogonal regularization directly controls inter-task interference. The resulting system, Latent-LoRA, is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task. Experiments across five model scales and two established continual learning benchmarks show state-of-the-art performance with near-zero forgetting.

Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2

College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1

Continual Learning for LLMs Continual LoRA Incremental Learning Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Continual Learning

Explore similar work

Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation

Energy-Structured Low-Rank Adaptation for Continual Learning

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning