Stable-Plastic Subspace Construction for Differentially Private Continual LoRA

Jin Liu; Ning Xi; Yinbin Miao; Junkang Liu; Ke Cheng; Jianfeng Ma

cs.LGOct 13, 2026

Stable-Plastic Subspace Construction for Differentially Private Continual LoRA

Authors: Jin Liu, Ning Xi, Yinbin Miao, Junkang Liu, Ke Cheng, Jianfeng Ma

Organizations: Xidian University · Tianjin University

Abstract

Continual learning (CL) adapts pretrained models to a sequence of tasks while retaining knowledge acquired from earlier tasks. When task data are sensitive, differential privacy (DP) provides record-level protection, but maintaining both adaptation and retention under privacy constraints remains challenging. We study this problem in continual low-rank adaptation (LoRA) and observe that: 1) private training increases forgetting primarily during subsequent adaptation rather than initial task acquisition; 2) jointly perturbing both LoRA factors introduces a non-negligible bilinear noise interaction; and 3) DP aggravates the stability-plasticity dilemma, hindering both knowledge retention and adaptation to new tasks. Based on these observations, we propose DP-SPA, which constructs a stable subspace from previously privatized updates and a plastic subspace from pretrained weights within its orthogonal complement. The method fixes these bases and privately optimizes only the LoRA factor, eliminating the bilinear noise interaction within each step. Before merging each task update, it rescales stable-direction coefficients using historical and current update energies. We establish a privacy guarantee and show that recalibration does not increase historical overlap and tightens a conditional upper bound on old-task loss increase. Experiments across five datasets, eight settings, and multiple privacy budgets show that DP-SPA improves average accuracy and reduces forgetting compared with the evaluated baselines.

Figures & tables

Panel of Figure 1 — Figure 1: DP exacerbates forgetting during continual adaptation. (a) Average forgetting across tasks. (b) Early-task forgetting over subsequent adaptations. (c) Acquisition vs. retention after 15 subsequent adaptations. Light lines denote individual tasks; bold lines denote their mean.

Panel of Figure 3 — Figure 3: Stability–plasticity trade-off under differential privacy. (a) Stage-wise stability–plasticity trajectories for Non-DP and DP ( $\epsilon=3$ ). (b) Stability gap between Non-DP and DP at matched plasticity. (c) Trade-off degradation across privacy budgets, where markers denote stability gaps at individual matched plasticity levels $P$ and bars denote their mean.

Figure 4: Overview of DP-SPA . (A) Construct the stable subspace $\mathcal{S}^{t}$ from privatized update history $\bm{H}^{t-1}$ and the plastic subspace $\mathcal{P}^{t}$ from pretrained weights $\bm{W}^{0}$ within $(\mathcal{S}^{t})^{\perp}$ . (B) Within task $t$ , fix $\bm{A}_{S}^{t},\bm{A}_{P}^{t}$ and optimize only $\bm{B}_{S}^{t},\bm{B}_{P}^{t}$ with DP-SGD. (C) Recalibrate stable-branch updates using historical and current update energies, merge $\Delta\bm{W}^{t}$ into the model, and update $\bm{H}^{t}$ for the next task.

Method	ImageNet-R ( $T{=}10$ )		ImageNet-A ( $T{=}10$ )		CIFAR-100 ( $T{=}10$ )		CUB ( $T{=}10$ )
Method	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
AB-LoRA	$56.86_{\pm 1.37}$	$63.83_{\pm 1.54}$	$51.69_{\pm 1.22}$	$62.23_{\pm 0.92}$	$85.42_{\pm 0.36}$	$90.40_{\pm 0.42}$	$72.37_{\pm 0.90}$	$79.33_{\pm 0.67}$
FA-LoRA	$68.55_{\pm 0.77}$	$77.46_{\pm 0.89}$	$54.01_{\pm 0.78}$	$64.42_{\pm 0.71}$	$86.56_{\pm 0.30}$	$90.63_{\pm 0.41}$	$74.02_{\pm 0.85}$	$80.26_{\pm 0.71}$
DP-SPA	$\mathbf{71.85_{\pm 0.62}}$	$\mathbf{79.90_{\pm 0.57}}$	$\mathbf{57.21_{\pm 0.58}}$	$\mathbf{66.65_{\pm 0.63}}$	$\mathbf{87.07_{\pm 0.22}}$	$\mathbf{91.19_{\pm 0.35}}$	$\mathbf{75.78_{\pm 0.64}}$	$\mathbf{82.67_{\pm 0.59}}$

Table 1: Experimental results for 10 incremental tasks on ImageNet-R, ImageNet-A, CIFAR-100, and CUB. The best results are highlighted in bold .

Method	ImageNet-R ( $T{=}5$ )		ImageNet-R ( $T{=}20$ )		ImageNet-A ( $T{=}20$ )		DomainNet ( $T{=}10$ )
Method	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
AB-LoRA	$62.48_{\pm 0.86}$	$70.73_{\pm 0.74}$	$45.83_{\pm 2.96}$	$56.12_{\pm 2.41}$	$41.44_{\pm 2.23}$	$54.11_{\pm 1.80}$	$76.53_{\pm 1.12}$	$83.22_{\pm 1.05}$
FA-LoRA	$71.66_{\pm 0.59}$	$79.20_{\pm 0.61}$	$64.54_{\pm 0.93}$	$75.64_{\pm 0.79}$	$45.25_{\pm 1.42}$	$58.12_{\pm 1.04}$	$78.07_{\pm 0.64}$	$84.83_{\pm 0.65}$
DP-SPA	$\mathbf{75.45_{\pm 0.51}}$	$\mathbf{82.06_{\pm 0.42}}$	$\mathbf{68.14_{\pm 0.75}}$	$\mathbf{78.34_{\pm 0.67}}$	$\mathbf{48.44_{\pm 0.84}}$	$\mathbf{60.89_{\pm 0.78}}$	$\mathbf{79.45_{\pm 0.59}}$	$\mathbf{85.61_{\pm 0.72}}$

Table 2: Experimental results for 5 and 20 incremental tasks on ImageNet-R, 20 tasks on ImageNet-A, and 10 tasks on DomainNet. The best results are highlighted in bold .

Panel of Figure 5 — Figure 5: Overall, old task and new task accuracy on ImageNet-R across 20 incremental tasks.

Dataset	Method	$\bm{\epsilon=1}$		$\bm{\epsilon=3}$		$\bm{\epsilon=5}$		$\bm{\epsilon=7}$
Dataset	Method	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
ImageNet-R	AB-LoRA	$3.26_{\pm 4.77}$	$11.37_{\pm 3.25}$	$45.83_{\pm 2.96}$	$56.12_{\pm 2.41}$	$48.73_{\pm 2.66}$	$57.42_{\pm 1.95}$	$50.82_{\pm 1.23}$	$60.28_{\pm 0.96}$
	FA-LoRA	$34.45_{\pm 2.52}$	$55.73_{\pm 2.70}$	$64.54_{\pm 0.93}$	$75.64_{\pm 0.79}$	$67.25_{\pm 0.88}$	$78.36_{\pm 0.73}$	$70.76_{\pm 0.61}$	$78.64_{\pm 0.84}$
	DP-SPA	$\mathbf{48.69_{\pm 1.98}}$	$\mathbf{60.44_{\pm 1.94}}$	$\mathbf{68.14_{\pm 0.75}}$	$\mathbf{78.34_{\pm 0.67}}$	$\mathbf{69.85_{\pm 0.70}}$	$\mathbf{79.65_{\pm 0.72}}$	$\mathbf{72.46_{\pm 0.55}}$	$\mathbf{80.27_{\pm 0.54}}$
ImageNet-A	AB-LoRA	$31.03_{\pm 3.35}$	$45.12_{\pm 2.87}$	$41.44_{\pm 2.23}$	$54.11_{\pm 1.80}$	$43.33_{\pm 0.66}$	$55.08_{\pm 0.73}$	$44.08_{\pm 0.68}$	$55.46_{\pm 0.49}$
	FA-LoRA	$43.30_{\pm 2.95}$	$56.07_{\pm 2.96}$	$45.25_{\pm 1.42}$	$58.12_{\pm 1.04}$	$47.61_{\pm 0.63}$	$59.75_{\pm 0.75}$	$48.24_{\pm 0.49}$	$60.07_{\pm 0.47}$

Table 3: Experimental results under different

\epsilon

on ImageNet-A and ImageNet-R with

T=20

Variant	ImageNet-R ( $T{=}10$ )		ImageNet-R ( $T{=}20$ )		ImageNet-A ( $T{=}10$ )		ImageNet-A ( $T{=}20$ )
Variant	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
AB-LoRA	$56.86_{\pm 1.37}$	$63.83_{\pm 1.54}$	$45.83_{\pm 2.96}$	$56.12_{\pm 2.41}$	$51.69_{\pm 1.22}$	$62.23_{\pm 0.92}$	$41.44_{\pm 2.23}$	$54.11_{\pm 1.80}$
$\bm{A}_{S}^{t}$ -only	$70.53_{\pm 0.36}$	$78.05_{\pm 0.48}$	$67.76_{\pm 0.60}$	$77.15_{\pm 0.46}$	$56.17_{\pm 0.54}$	$65.13_{\pm 0.70}$	$47.05_{\pm 0.65}$	$59.43_{\pm 0.52}$
$\bm{A}_{P}^{t}$ -only	$68.81_{\pm 0.52}$	$77.94_{\pm 0.57}$	$65.80_{\pm 0.74}$	$76.32_{\pm 0.81}$	$54.82_{\pm 0.61}$	$64.96_{\pm 0.52}$	$45.19_{\pm 0.90}$	$58.24_{\pm 0.68}$
$\bm{A}_{S}^{t}{+}\bm{A}_{P}^{t}$	$\mathbf{71.85_{\pm 0.62}}$	$\mathbf{79.90_{\pm 0.57}}$	$\mathbf{68.14_{\pm 0.75}}$	$\mathbf{78.34_{\pm 0.67}}$	$\mathbf{57.21_{\pm 0.58}}$	$\mathbf{66.65_{\pm 0.63}}$	$\mathbf{48.44_{\pm 0.84}}$	$\mathbf{60.89_{\pm 0.78}}$

Table 4: Ablation studies on the stable and plastic bases on ImageNet-R and ImageNet-A.

Variant	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
DP-SPA w/o recal.	$67.25_{\pm 0.69}$	$77.49_{\pm 0.72}$
DP-SPA	$\mathbf{68.14_{\pm 0.75}}$	$\mathbf{78.34_{\pm 0.67}}$

Table 5: Effect of recalibration on ImageNet-R with

T=20

r=32

, and

\epsilon=3

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$T,\ t$	Number of tasks and current task index
$\mathcal{D}^{t},\ \mathcal{C}^{t}$	Private dataset and class set of task $t$
$\bm{W}^{0}\in\mathbb{R}^{m\times n}$	Pretrained weight matrix
$\bm{W}^{t}$	Weight matrix after completing task $t$
$r_{S},\ r_{P},\ r=r_{S}+r_{P}$	Stable, plastic, and total LoRA ranks
$\mathcal{S}^{t},\ \mathcal{P}^{t}$	Stable and plastic input-side subspaces

Appendix

Table 6: Summary of the main notations.

Strategy	ImageNet-R ( $T{=}20$ )		ImageNet-A ( $T{=}20$ )
Strategy	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
No recalibration	$67.25_{\pm 0.69}$	$77.49_{\pm 0.72}$	$47.56_{\pm 0.63}$	$59.68_{\pm 0.55}$
Constant scaling	$65.30_{\pm 0.54}$	$76.83_{\pm 0.61}$	$44.97_{\pm 0.67}$	$58.18_{\pm 0.52}$
Norm-matched scaling	$67.57_{\pm 0.59}$	$77.65_{\pm 0.42}$	$47.80_{\pm 0.49}$	$59.32_{\pm 0.60}$
DP-SPA	$68.14_{\pm 0.75}$	$78.34_{\pm 0.67}$	$48.44_{\pm 0.84}$	$60.89_{\pm 0.78}$

Appendix

Table 7: Comparison of recalibration strategies on ImageNet-R and ImageNet-A with

T=20

. Constant scaling uses

\gamma=0.5

, while norm-matched scaling matches the stable-branch update norm of direction-wise recalibration at the same pre-merge state.

$(r_{S},r_{P})$	ImageNet-R ( $T{=}20$ )		ImageNet-A ( $T{=}20$ )
$(r_{S},r_{P})$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
$(0,32)$	$65.80_{\pm 0.74}$	$76.32_{\pm 0.81}$	$45.19_{\pm 0.90}$	$58.24_{\pm 0.68}$
$(8,24)$	$67.91_{\pm 0.82}$	$77.63_{\pm 0.78}$	$49.12_{\pm 0.95}$	$61.06_{\pm 0.77}$
$(16,16)$	$68.14_{\pm 0.75}$	$78.34_{\pm 0.67}$	$48.44_{\pm 0.84}$	$60.89_{\pm 0.78}$
$(24,8)$	$67.93_{\pm 0.30}$	$77.83_{\pm 0.28}$	$47.56_{\pm 0.46}$	$59.97_{\pm 0.34}$
$(32,0)$	$67.76_{\pm 0.60}$	$77.15_{\pm 0.46}$	$47.05_{\pm 0.65}$	$59.43_{\pm 0.52}$

Appendix

Table 8: Sensitivity to stable–plastic rank allocation on ImageNet-R and ImageNet-A with

T=20

\epsilon=3

, and fixed total rank

r_{S}+r_{P}=32

Panel of Figure 6 — Figure 6: Overall, old-task, and new-task accuracy across 20 incremental tasks on ImageNet-A.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 301 papers

PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA
May 31, 2026Shihao Wang, Xueru ZhangDifferentially Private Stochastic Gradient Descent Low-Rank Adaptation
Department of Computer Science and Engineering, The Ohio State University, Columbus, OH, USA.
Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning
Feb 27, 2026Lingfeng He, De Cheng, Huaijie Wang +3Continual Learning Low-Rank Adaptation
State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi’an, China · School of Electronic Engineering, Xidian University, Xi’an, China · AUTHORERR: Missing \icmlaffiliation.
SPACE-LoRA: Allocating Activation-Subspace Protection for Continual Learning
Sep 28, 2026Seunghyun Yoo, Kiseok Kim, Hyeontae Joo +2Continual Learning Low-Rank Adaptation
School of Electrical Engineering Korea University Seoul, Republic of Korea

May 31, 2026J/K move · Enter open · S save

PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA

Shihao Wang, Xueru Zhang

Department of Computer Science and Engineering, The Ohio State University, Columbus, OH, USA.

Differentially Private Stochastic Gradient Descent Low-Rank Adaptation Privacy-Preserving ML Differential Privacy

Open paper AskPDF arXiv

Applying differential privacy (DP) via DP-SGD to Low-Rank Adaptation (LoRA) is a natural approach for privacy-preserving fine-tuning. However, LoRA's low-rank parameterization poses a fundamental challenge. In LoRA, each trainable update is represented as a low-rank matrix

Z = AB^\top

, but this factorization is inherently non-identifiable: many factor pairs

(A,B)

represent the same update

Z

. As a result, applying DP-SGD directly to the factors induces gauge-dependent perturbations on

Z

, and we show that this naive DP-LoRA can lead to unbounded noise amplification. We propose PRISM, an intrinsic DP mechanism for LoRA that is gauge invariant by construction, avoids bilinear noise amplification, and admits an efficient low-dimensional noise sampler. Moreover, PRISM yields a closed-form characterization of the effective intrinsic noise induced on

Z

, enabling stable privacy-utility trade-offs through bounded, gauge-invariant perturbations. We establish standard

(ε,δ)

-DP guarantees for PRISM and introduce a DP-aware, gauge-invariant adaptive update rule that prevents adaptive optimization from amplifying injected privacy noise, improving numerical stability in practice.

Abstract

Continual Learning

Explore similar work

PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

SPACE-LoRA: Allocating Activation-Subspace Protection for Continual Learning