Stable-Plastic Subspace Construction for Differentially Private Continual LoRA

Jin Liu; Ning Xi; Yinbin Miao; Junkang Liu; Ke Cheng; Jianfeng Ma

cs.LGOct 13, 2026

Stable-Plastic Subspace Construction for Differentially Private Continual LoRA

Authors: Jin Liu, Ning Xi, Yinbin Miao, Junkang Liu, Ke Cheng, Jianfeng Ma

Organizations: Xidian University · Tianjin University

Abstract

Continual learning (CL) adapts pretrained models to a sequence of tasks while retaining knowledge acquired from earlier tasks. When task data are sensitive, differential privacy (DP) provides record-level protection, but maintaining both adaptation and retention under privacy constraints remains challenging. We study this problem in continual low-rank adaptation (LoRA) and observe that: 1) private training increases forgetting primarily during subsequent adaptation rather than initial task acquisition; 2) jointly perturbing both LoRA factors introduces a non-negligible bilinear noise interaction; and 3) DP aggravates the stability-plasticity dilemma, hindering both knowledge retention and adaptation to new tasks. Based on these observations, we propose DP-SPA, which constructs a stable subspace from previously privatized updates and a plastic subspace from pretrained weights within its orthogonal complement. The method fixes these bases and privately optimizes only the LoRA factor, eliminating the bilinear noise interaction within each step. Before merging each task update, it rescales stable-direction coefficients using historical and current update energies. We establish a privacy guarantee and show that recalibration does not increase historical overlap and tightens a conditional upper bound on old-task loss increase. Experiments across five datasets, eight settings, and multiple privacy budgets show that DP-SPA improves average accuracy and reduces forgetting compared with the evaluated baselines.

Figures & tables

Panel of Figure 1 — Figure 1: DP exacerbates forgetting during continual adaptation. (a) Average forgetting across tasks. (b) Early-task forgetting over subsequent adaptations. (c) Acquisition vs. retention after 15 subsequent adaptations. Light lines denote individual tasks; bold lines denote their mean.

Panel of Figure 3 — Figure 3: Stability–plasticity trade-off under differential privacy. (a) Stage-wise stability–plasticity trajectories for Non-DP and DP ( $\epsilon=3$ ). (b) Stability gap between Non-DP and DP at matched plasticity. (c) Trade-off degradation across privacy budgets, where markers denote stability gaps at individual matched plasticity levels $P$ and bars denote their mean.

Figure 4: Overview of DP-SPA . (A) Construct the stable subspace $\mathcal{S}^{t}$ from privatized update history $\bm{H}^{t-1}$ and the plastic subspace $\mathcal{P}^{t}$ from pretrained weights $\bm{W}^{0}$ within $(\mathcal{S}^{t})^{\perp}$ . (B) Within task $t$ , fix $\bm{A}_{S}^{t},\bm{A}_{P}^{t}$ and optimize only $\bm{B}_{S}^{t},\bm{B}_{P}^{t}$ with DP-SGD. (C) Recalibrate stable-branch updates using historical and current update energies, merge $\Delta\bm{W}^{t}$ into the model, and update $\bm{H}^{t}$ for the next task.

Method	ImageNet-R ( $T{=}10$ )		ImageNet-A ( $T{=}10$ )		CIFAR-100 ( $T{=}10$ )		CUB ( $T{=}10$ )
Method	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
AB-LoRA	$56.86_{\pm 1.37}$	$63.83_{\pm 1.54}$	$51.69_{\pm 1.22}$	$62.23_{\pm 0.92}$	$85.42_{\pm 0.36}$	$90.40_{\pm 0.42}$	$72.37_{\pm 0.90}$	$79.33_{\pm 0.67}$
FA-LoRA	$68.55_{\pm 0.77}$	$77.46_{\pm 0.89}$	$54.01_{\pm 0.78}$	$64.42_{\pm 0.71}$	$86.56_{\pm 0.30}$	$90.63_{\pm 0.41}$	$74.02_{\pm 0.85}$	$80.26_{\pm 0.71}$
DP-SPA	$\mathbf{71.85_{\pm 0.62}}$	$\mathbf{79.90_{\pm 0.57}}$	$\mathbf{57.21_{\pm 0.58}}$	$\mathbf{66.65_{\pm 0.63}}$	$\mathbf{87.07_{\pm 0.22}}$	$\mathbf{91.19_{\pm 0.35}}$	$\mathbf{75.78_{\pm 0.64}}$	$\mathbf{82.67_{\pm 0.59}}$

Table 1: Experimental results for 10 incremental tasks on ImageNet-R, ImageNet-A, CIFAR-100, and CUB. The best results are highlighted in bold .

Method	ImageNet-R ( $T{=}5$ )		ImageNet-R ( $T{=}20$ )		ImageNet-A ( $T{=}20$ )		DomainNet ( $T{=}10$ )
Method	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
AB-LoRA	$62.48_{\pm 0.86}$	$70.73_{\pm 0.74}$	$45.83_{\pm 2.96}$	$56.12_{\pm 2.41}$	$41.44_{\pm 2.23}$	$54.11_{\pm 1.80}$	$76.53_{\pm 1.12}$	$83.22_{\pm 1.05}$
FA-LoRA	$71.66_{\pm 0.59}$	$79.20_{\pm 0.61}$	$64.54_{\pm 0.93}$	$75.64_{\pm 0.79}$	$45.25_{\pm 1.42}$	$58.12_{\pm 1.04}$	$78.07_{\pm 0.64}$	$84.83_{\pm 0.65}$
DP-SPA	$\mathbf{75.45_{\pm 0.51}}$	$\mathbf{82.06_{\pm 0.42}}$	$\mathbf{68.14_{\pm 0.75}}$	$\mathbf{78.34_{\pm 0.67}}$	$\mathbf{48.44_{\pm 0.84}}$	$\mathbf{60.89_{\pm 0.78}}$	$\mathbf{79.45_{\pm 0.59}}$	$\mathbf{85.61_{\pm 0.72}}$

Table 2: Experimental results for 5 and 20 incremental tasks on ImageNet-R, 20 tasks on ImageNet-A, and 10 tasks on DomainNet. The best results are highlighted in bold .

Panel of Figure 5 — Figure 5: Overall, old task and new task accuracy on ImageNet-R across 20 incremental tasks.

Dataset	Method	$\bm{\epsilon=1}$		$\bm{\epsilon=3}$		$\bm{\epsilon=5}$		$\bm{\epsilon=7}$
Dataset	Method	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
ImageNet-R	AB-LoRA	$3.26_{\pm 4.77}$	$11.37_{\pm 3.25}$	$45.83_{\pm 2.96}$	$56.12_{\pm 2.41}$	$48.73_{\pm 2.66}$	$57.42_{\pm 1.95}$	$50.82_{\pm 1.23}$	$60.28_{\pm 0.96}$
	FA-LoRA	$34.45_{\pm 2.52}$	$55.73_{\pm 2.70}$	$64.54_{\pm 0.93}$	$75.64_{\pm 0.79}$	$67.25_{\pm 0.88}$	$78.36_{\pm 0.73}$	$70.76_{\pm 0.61}$	$78.64_{\pm 0.84}$
	DP-SPA	$\mathbf{48.69_{\pm 1.98}}$	$\mathbf{60.44_{\pm 1.94}}$	$\mathbf{68.14_{\pm 0.75}}$	$\mathbf{78.34_{\pm 0.67}}$	$\mathbf{69.85_{\pm 0.70}}$	$\mathbf{79.65_{\pm 0.72}}$	$\mathbf{72.46_{\pm 0.55}}$	$\mathbf{80.27_{\pm 0.54}}$
ImageNet-A	AB-LoRA	$31.03_{\pm 3.35}$	$45.12_{\pm 2.87}$	$41.44_{\pm 2.23}$	$54.11_{\pm 1.80}$	$43.33_{\pm 0.66}$	$55.08_{\pm 0.73}$	$44.08_{\pm 0.68}$	$55.46_{\pm 0.49}$
	FA-LoRA	$43.30_{\pm 2.95}$	$56.07_{\pm 2.96}$	$45.25_{\pm 1.42}$	$58.12_{\pm 1.04}$	$47.61_{\pm 0.63}$	$59.75_{\pm 0.75}$	$48.24_{\pm 0.49}$	$60.07_{\pm 0.47}$

Table 3: Experimental results under different

\epsilon

on ImageNet-A and ImageNet-R with

T=20

Variant	ImageNet-R ( $T{=}10$ )		ImageNet-R ( $T{=}20$ )		ImageNet-A ( $T{=}10$ )		ImageNet-A ( $T{=}20$ )
Variant	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
AB-LoRA	$56.86_{\pm 1.37}$	$63.83_{\pm 1.54}$	$45.83_{\pm 2.96}$	$56.12_{\pm 2.41}$	$51.69_{\pm 1.22}$	$62.23_{\pm 0.92}$	$41.44_{\pm 2.23}$	$54.11_{\pm 1.80}$
$\bm{A}_{S}^{t}$ -only	$70.53_{\pm 0.36}$	$78.05_{\pm 0.48}$	$67.76_{\pm 0.60}$	$77.15_{\pm 0.46}$	$56.17_{\pm 0.54}$	$65.13_{\pm 0.70}$	$47.05_{\pm 0.65}$	$59.43_{\pm 0.52}$
$\bm{A}_{P}^{t}$ -only	$68.81_{\pm 0.52}$	$77.94_{\pm 0.57}$	$65.80_{\pm 0.74}$	$76.32_{\pm 0.81}$	$54.82_{\pm 0.61}$	$64.96_{\pm 0.52}$	$45.19_{\pm 0.90}$	$58.24_{\pm 0.68}$
$\bm{A}_{S}^{t}{+}\bm{A}_{P}^{t}$	$\mathbf{71.85_{\pm 0.62}}$	$\mathbf{79.90_{\pm 0.57}}$	$\mathbf{68.14_{\pm 0.75}}$	$\mathbf{78.34_{\pm 0.67}}$	$\mathbf{57.21_{\pm 0.58}}$	$\mathbf{66.65_{\pm 0.63}}$	$\mathbf{48.44_{\pm 0.84}}$	$\mathbf{60.89_{\pm 0.78}}$

Table 4: Ablation studies on the stable and plastic bases on ImageNet-R and ImageNet-A.

Variant	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
DP-SPA w/o recal.	$67.25_{\pm 0.69}$	$77.49_{\pm 0.72}$
DP-SPA	$\mathbf{68.14_{\pm 0.75}}$	$\mathbf{78.34_{\pm 0.67}}$

Table 5: Effect of recalibration on ImageNet-R with

T=20

r=32

, and

\epsilon=3

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$T,\ t$	Number of tasks and current task index
$\mathcal{D}^{t},\ \mathcal{C}^{t}$	Private dataset and class set of task $t$
$\bm{W}^{0}\in\mathbb{R}^{m\times n}$	Pretrained weight matrix
$\bm{W}^{t}$	Weight matrix after completing task $t$
$r_{S},\ r_{P},\ r=r_{S}+r_{P}$	Stable, plastic, and total LoRA ranks
$\mathcal{S}^{t},\ \mathcal{P}^{t}$	Stable and plastic input-side subspaces

Appendix

Table 6: Summary of the main notations.

Strategy	ImageNet-R ( $T{=}20$ )		ImageNet-A ( $T{=}20$ )
Strategy	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
No recalibration	$67.25_{\pm 0.69}$	$77.49_{\pm 0.72}$	$47.56_{\pm 0.63}$	$59.68_{\pm 0.55}$
Constant scaling	$65.30_{\pm 0.54}$	$76.83_{\pm 0.61}$	$44.97_{\pm 0.67}$	$58.18_{\pm 0.52}$
Norm-matched scaling	$67.57_{\pm 0.59}$	$77.65_{\pm 0.42}$	$47.80_{\pm 0.49}$	$59.32_{\pm 0.60}$
DP-SPA	$68.14_{\pm 0.75}$	$78.34_{\pm 0.67}$	$48.44_{\pm 0.84}$	$60.89_{\pm 0.78}$

Appendix

Table 7: Comparison of recalibration strategies on ImageNet-R and ImageNet-A with

T=20

. Constant scaling uses

\gamma=0.5

, while norm-matched scaling matches the stable-branch update norm of direction-wise recalibration at the same pre-merge state.

$(r_{S},r_{P})$	ImageNet-R ( $T{=}20$ )		ImageNet-A ( $T{=}20$ )
$(r_{S},r_{P})$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
$(0,32)$	$65.80_{\pm 0.74}$	$76.32_{\pm 0.81}$	$45.19_{\pm 0.90}$	$58.24_{\pm 0.68}$
$(8,24)$	$67.91_{\pm 0.82}$	$77.63_{\pm 0.78}$	$49.12_{\pm 0.95}$	$61.06_{\pm 0.77}$
$(16,16)$	$68.14_{\pm 0.75}$	$78.34_{\pm 0.67}$	$48.44_{\pm 0.84}$	$60.89_{\pm 0.78}$
$(24,8)$	$67.93_{\pm 0.30}$	$77.83_{\pm 0.28}$	$47.56_{\pm 0.46}$	$59.97_{\pm 0.34}$
$(32,0)$	$67.76_{\pm 0.60}$	$77.15_{\pm 0.46}$	$47.05_{\pm 0.65}$	$59.43_{\pm 0.52}$

Appendix

Table 8: Sensitivity to stable–plastic rank allocation on ImageNet-R and ImageNet-A with

T=20

\epsilon=3

, and fixed total rank

r_{S}+r_{P}=32

Panel of Figure 6 — Figure 6: Overall, old-task, and new-task accuracy across 20 incremental tasks on ImageNet-A.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 301 papers

May 31, 2026

PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA

Applying differential privacy (DP) via DP-SGD to Low-Rank Adaptation (LoRA) is a natural approach for privacy-preserving fine-tuning. However, LoRA's low-rank parameterization poses a fundamental challenge. In LoRA, each trainable update is represented as a low-rank matrix

Z = AB^\top

, but this factorization is inherently non-identifiable: many factor pairs

(A,B)

represent the same update

Z

. As a result, applying DP-SGD directly to the factors induces gauge-dependent perturbations on

Z

, and we show that this naive DP-LoRA can lead to unbounded noise amplification. We propose PRISM, an intrinsic DP mechanism for LoRA that is gauge invariant by construction, avoids bilinear noise amplification, and admits an efficient low-dimensional noise sampler. Moreover, PRISM yields a closed-form characterization of the effective intrinsic noise induced on

Z

, enabling stable privacy-utility trade-offs through bounded, gauge-invariant perturbations. We establish standard

(ε,δ)

-DP guarantees for PRISM and introduce a DP-aware, gauge-invariant adaptive update rule that prevents adaptive optimization from amplifying injected privacy noise, improving numerical stability in practice.

Shihao Wang, Xueru Zhang

Department of Computer Science and Engineering, The Ohio State University, Columbus, OH, USA.

Differentially Private Stochastic Gradient Descent Low-Rank Adaptation Privacy-Preserving ML Differential Privacy

Ask PDF

Feb 27, 2026

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. LoDA fixes LoRA down-projections on two subspaces and learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. After each task, before integrating the LoRA updates into the backbone, LoDA derives a closed-form recalibration for the general update, approximating a feature-level joint optimum along this task-shared direction. Experiments indicate that LoDA outperforms existing CL methods. Our code is available at https://github.com/HHHLF/LoDA_ICML2026.

Lingfeng He, De Cheng, Huaijie Wang +3

State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi’an, China · School of Electronic Engineering, Xidian University, Xi’an, China · AUTHORERR: Missing \icmlaffiliation.

Continual Learning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Sep 28, 2026

SPACE-LoRA: Allocating Activation-Subspace Protection for Continual Learning

This study addresses the catastrophic forgetting problem that occurs when sequentially learning successive tasks using Low-Rank Adaptation (LoRA) from a lifelong learning perspective. While existing approaches have primarily constrained parameter updates or learning subspaces to reduce interference with past knowledge, they have not fully considered additive interference. This occurs when a newly added residual adapter on top of a fixed past model generates non-zero responses along input directions important for old tasks, thereby altering previous predictions. To this end, we propose Subspace Protection with Allocated Capacity for Efficient Continual Adaptation (SPACE-LoRA). SPACE-LoRA directly suppresses the responses of the new residual branch along input activation directions that are important for old tasks and adaptively determines the protection coverage for each module based on past-task sensitivity estimated via a common Fisher sensitivity-based coverage target. Under a fixed LoRA rank, this approach adaptively adjusts module-specific protection coverage while suppressing interference along input directions sensitive to old tasks. We assess the effectiveness of activation-subspace protection in mitigating catastrophic forgetting and examine the role of sensitivity-guided protection in continual learning across diverse tasks. Code is available at https://anonymous.4open.science/r/SPACE-LoRA-7864.

Seunghyun Yoo, Kiseok Kim, Hyeontae Joo +2

School of Electrical Engineering Korea University Seoul, Republic of Korea

Continual Learning Low-Rank Adaptation Catastrophic Forgetting

Ask PDF

Abstract

Continual Learning

Explore similar work

PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

SPACE-LoRA: Allocating Activation-Subspace Protection for Continual Learning