DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editing

Gyanendra Das; Sai Satyam Jena

cs.CVApr 9, 2026

DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editing

Authors: Gyanendra Das, Sai Satyam Jena

Organizations: Zynix AI, FL, USA

Abstract

Model editing aims to update knowledge to add new concepts and change relevant information without retraining. Lifelong editing is a challenging task, prone to disrupting previously learned concepts, especially for Vision Language Models (VLMs), because sequential edits can lead to degraded reasoning and cross modal misalignment. Existing VLM knowledge editing methods based on gated adapters, activation edits, and parameter merging techniques address catastrophic forgetting seen in full fine tuning; however, they still operate in the shared representation space of the VLM, where concepts are entangled, so edits interfere with other non relevant concepts. We hypothesize that this instability persists because current methods algorithmically control edits via optimization rather than structurally separating knowledge. We introduce Dynamic Subspace Concept Alignment (DSCA) which by design mitigates this limitation by decomposing the representation space into a set of orthogonal semantic subspaces and proposing edits only in those transformed spaces. These subspaces are obtained through incremental clustering and PCA on joint vision language representations. This process structurally isolates concepts, enabling precise, non interfering edits by turning isolation from a soft training objective into an architectural property. The surgical edits are guided by a multi term loss function for maintaining task fidelity, edit locality, and cross modal alignment. With the base model frozen, our method achieves 98 percent single edit success, remains over 95 percent after 1000 sequential edits, lowers hallucination by 3 to 5 percent, and achieves the best backward transfer (BWT) scores on continual instruction tuning benchmarks. Extensive experiments demonstrate DSCA state of the art stability and knowledge retention capability in continual lifelong editing across various datasets and benchmarks.

Figures & tables

Figure 1: Conceptual comparison of knowledge-editing paradigms. (a) The initial concept space where concepts are well-separated. (b) Global fine-tuning perturbs the entire representation space, distorting unrelated concepts. (c) LoRA / local adapters constrain edits but still produce coupled interference. (d) DSCA performs subspace-confined, concept-specific interventions, maintaining isolation and preserving all other concepts.

Figure 2

Dataset	Method	Rel.	T-Gen.	V-Gen.	T-Loc.	M-Loc.	Avg.
E-VQA	DualEdit [ 30 ]	96.94	96.43	96.20	100.00	99.61	97.84
	VisEdit [ 4 ]	95.78	94.21	94.37	100.00	91.11	95.09
	SERAC [ 25 ]	82.51	81.60	80.05	100.00	57.48	80.33
	LTE [ 13 ]	94.16	93.54	93.06	83.76	81.65	89.23
	MEND [ 26 ]	92.30	92.16	92.10	90.30	81.13	89.60
	DSCA (ours)	98.12	97.30	97.25	100.00	99.83	98.50

Table 1: Single-edit results on E-VQA [ 5 ] and E-IC [ 5 ] (LLaVA-1.5-7B [ 15 ] ). Baselines from DualEdit [ 30 ] .

Dataset	Method	Rel.	T-Gen.	M-Gen.	T-Loc.	M-Loc.	Avg.
E-VQA [ 5 ]	LiveEdit [ 3 ]	92.93	90.16	84.30	100.00	96.43	92.76
	LTE [ 13 ]	83.93	82.55	81.34	83.97	73.09	80.98
	MEND [ 26 ]	0.04	0.05	0.05	0.08	0.09	0.06
	SERAC [ 25 ]	85.57	75.58	82.01	62.46	15.69	64.26
	DSCA (ours)	96.85	93.10	88.00	100.00	98.20	95.23
VLKEB [ 11 ]	LiveEdit [ 3 ]	92.22	83.97	82.75	100.00	100.00	91.79

Table 2: Lifelong editing results (

t=1000

edits) on LLaVA-1.5-7B [ 15 ] . Baselines from LiveEdit [ 3 ] .

METHOD	ACC	BWT	FWT	$A_{t}$
ZERO-SHOT	24.74	-	-	-
INDEPENDENT	76.46	-	-	-
MULTITASK	73.93	-	-	-
MoELoRA [ 2 ]	46.59 $\pm$ 9.98	-36.40 $\pm$ 11.97	7.79 $\pm$ 2.24	76.93 $\pm$ 0.27
MagMax [ 22 ]	45.74 $\pm$ 0.88	-22.68 $\pm$ 6.51	4.75 $\pm$ 3.36	76.29 $\pm$ 0.18
PAM [ 32 ]	49.89 $\pm$ 1.66	-19.45 $\pm$ 0.95	11.11 $\pm$ 0.09	76.31 $\pm$ 0.03

Table 3: Performance on the CoIN benchmark across four metrics (mean

\pm

std over three task orders). Higher is better for all metrics; less negative BWT indicates less forgetting. Baselines from PAM [ 32 ]

Method	MME [ 7 ]	MM-Vet [ 38 ]	VQA-v2 [ 9 ]	TextVQA [ 31 ]	COCO CIDEr [ 14 ] [ 34 ]
LiveEdit [ 3 ]	73.4	54.2	82.1	61.5	120.6
DualEdit [ 30 ]	74.1	55.6	83.5	62.9	121.4
MRT [ 17 ]	70.9	50.8	80.2	59.3	118.2
DSCA (ours)	76.3	57.8	84.9	64.7	123.8

Table 4: Capability retention on standard LVLM benchmarks.

Method	CHAIR-H $\downarrow$	Faithfulness $\uparrow$	Richness $\approx$
Gen-Anchor Rep-Edit [ 29 ]	18.5	92.3	98.1
DualEdit [ 30 ]	20.8	90.6	97.2
LiveEdit [ 3 ]	21.1	91.0	96.8
DSCA (ours)	15.9	94.5	98.4

Table 5: Hallucination stress test (CHAIR) [ 28 ] . Lower CHAIR-H is better.

Figure 2: Diagnostic analysis of DSCA. (a) Mean pairwise subspace overlap(

\varepsilon=\|\mathbf{R}_{i}^{\top}\mathbf{R}_{j}\|_{F}^{2}

) as a function of the number of sequential edits. DSCA with residualized Incremental PCA keeps the overlap essentially flat at

\approx 3\times 10^{-3}

across

1{,}000

edits, comparable to a globally orthonormal baseline, whereas a variant without orthogonalization drifts to more than

10^{-1}

. (b) Relationship between mean subspace overlap and forgetting, measured as

1-\text{BWT}

. The scatter plot and fitted line (Pearson

r\!\approx\!0.94

) show a strong monotonic dependence, as overlap

\varepsilon

increases, forgetting grows almost linearly, consistent with the bounded-interference result in Supplementary. (c) Ablation over subspace construction strategies (No orthogonalization, residualized PCA, global orthonormalization). For each metric (mean overlap, BWT, reliability) scores are normalized per metric to

[0,1]

for comparability. Residualized PCA attains nearly the same BWT and reliability as global orthonormalization while dramatically reducing overlap relative to the non-orthogonal baseline, giving the best overall trade-off.

Figure 3: Routing sparsity analysis. (a) Histogram of routing weights shows that over 95% are below 0.05, indicating highly selective module activation. (b) Trade-off between sparsity coefficient $\lambda_{\text{sparse}}$ and average number of active DSAMs. The chosen operating point (blue dot) yields $\approx 3$ active DSAMs per input without degrading performance.

Variant	Orthogonality in $R_{k}$	Gate Sparsity	Multi-stage Routing	Basis Residual intervention	Full Subspace	Full Rank	ES $\uparrow$	Locality $\Delta\downarrow$	GEN $\uparrow$
Full DSCA	✓	✓	✓	✓	✓	✓	98.0	0.5	97.3
w/o orthogonality	✗	✓	✓	✓	✓	✓	95.8	2.8	93.4
w/o gate sparsity	✓	✗	✓	✓	✓	✓	96.1	2.1	94.7
single-stage routing	✓	✓	✗	✓	✓	✓	96.9	1.9	95.0
no basis-residual	✓	✓	✓	✗	✓	✓	97.1	1.5	95.8
fewer subspaces (K/2)	✓	✓	✓	✓	✗	✓	97.0	1.7	95.5

Table 6: Ablation study on DSCA components. A checkmark (

\checkmark

) indicates the component is present; a cross (

\times

) indicates it has been ablated or modified.

Figure 4: Concept-wise subspace visualization. t-SNE of fused representations projected through their assigned semantic subspaces

\mathbf{R}_{k}^{\top}\mathbf{h}_{f}

for a subset of concepts (e.g., “car”, “truck”, “bicycle”). DSCA yields compact, well-separated clusters, indicating that edits remain confined to localized regions of the representation space.This empirically validates the conceptual illustration provided in Figure 1 (d) of the main paper.

Setting	$r$	$N_{\text{min}}$	$N_{\text{refine}}$	$\tau$	$\lambda_{\text{align}}$	$\lambda_{\text{distill}}$	$\lambda_{\text{sparse}}$
LLaVA-1.5-7B [ 15 ] (editing: E-VQA / E-IC / VLKEB)	128	32	500	0.07	0.5	1.0	$1\times 10^{-2}$
PaliGemma-3B [ 1 ] (CoIN continual learning [ 2 ] )	128	32	500	0.07	0.5	1.0	$1\times 10^{-2}$

Table 7: DSCA-specific hyperparameters used in all experiments. Values are kept fixed across editing and continual learning unless otherwise noted.

Methods	E-VQA [ 5 ]						E-IC [ 5 ]
Methods	Rel.	T-Gen.	V-Gen.	T-Loc.	M-Loc.	Avg.	Rel.	T-Gen.	V-Gen.	T-Loc.	M-Loc.	Avg.
LLaVA-V1.5 [ 15 ]
FT-V	31.68	29.96	26.68	100.00	91.23	55.91	52.85	51.57	48.63	100.00	92.55	69.12
FT-L	31.78	30.02	26.91	99.94	92.03	56.14	53.00	51.02	49.29	98.91	94.89	69.42
KE	85.86	84.00	82.23	93.57	73.06	83.74	83.54	82.15	81.12	92.46	73.83	82.62
IKE	91.35	90.84	91.08	60.18	51.08	76.91	93.72	88.37	76.99	76.60	64.90	80.12

Table 8: Extended Single Edit success comparision. Baselines from DualEdit [ 30 ]

Methods	E-VQA [ 5 ]						VLKEB [ 11 ]
Methods	Rel.	T-Gen.	V-Gen.	T-Loc.	M-Loc.	Avg.	Rel.	T-Gen.	V-Gen.	T-Loc.	M-Loc.	Avg.
LLaVA-V1.5 [ 15 ]
LiveEdit [ 3 ]	92.93	90.16	84.30	100.00	96.43	92.76	92.22	83.97	82.75	100.00	100.00	91.79
LTE [ 13 ]	83.93	82.55	81.34	83.97	73.09	80.98	64.51	56.26	64.80	80.85	76.52	68.59
MEND [ 26 ]	0.04	0.05	0.05	0.08	0.09	0.06	0.03	0.05	0.07	0.06	0.08	0.06
SERAC [ 25 ]	85.57	75.58	82.01	62.46	15.69	64.26	60.93	56.49	60.06	52.94	15.04	49.09

Table 9: Expanded Lifelong editing results (

t=1000

edits) on LLaVA-1.5-7B [ 15 ] . Baselines from LiveEdit [ 3 ] .

METHOD	ACC	BWT	FWT	$A_{t}$
ZERO-SHOT	24.74	-	-	-
INDEPENDENT	76.46	-	-	-
MULTITASK	73.93	-	-	-
FINE-TUNE	43.36 $\pm$ 8.18	-39.51 $\pm$ 9.87	7.71 $\pm$ 2.51	76.29 $\pm$ 0.18
LWF	47.15 $\pm$ 3.52	-33.66 $\pm$ 3.84	9.67 $\pm$ 1.22	75.20 $\pm$ 0.32
I-LoRA	42.11 $\pm$ 6.55	-40.95 $\pm$ 8.13	7.89 $\pm$ 2.60	76.24 $\pm$ 0.35

Table 10: Expanded table for perfomance on COIN benchmark [ 2 ] . Higher is better for all metrics; less negative BWT indicates less forgetting.Baselines are taken from PAM [ 32 ]

Continual Learning for VLMs

Continual learning for vision-language models studies how to adapt them to sequentially arriving data and tasks without losing earlier capabilities.

See all 82 papers

SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing
Mar 16, 2026Yuhuan Liu, Haitian Zhong, Xinyuan Xia +3Knowledge Editing Language Model Editing
New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · Cuiying Honors College, Lanzhou University · Zhongguancun Academy +1
Towards Scalable Lifelong Knowledge Editing with Selective Knowledge Suppression
Apr 21, 2026Dahyun Jung, Jaewook Lee, Heuiseok LimKnowledge Editing Continual Learning for LLMs
Department of Computer Science and Engineering, Korea University
Generalizable Lifelong Model Editing via Preference Optimization
Sep 29, 2026Dahyun Jung, Suhyune Son, Heuiseok LimContinual Learning for LLMs Continual Learning
Department of Computer Science and Engineering, Korea University

Mar 16, 2026J/K move · Enter open · S save

SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing

Yuhuan Liu, Haitian Zhong, Xinyuan Xia, Qiang Liu+2

New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · Cuiying Honors College, Lanzhou University · Zhongguancun Academy+1

Knowledge Editing Language Model Editing Catastrophic Forgetting Continual Learning for LLMs

Open paper AskPDF arXiv

Large Language Models (LLMs) often suffer from catastrophic forgetting and collapse during sequential knowledge editing. This vulnerability stems from the prevailing dense editing paradigm, which treats models as black boxes and relies on coarse-grained parameter interventions that inevitably disrupt preserved knowledge. To address this, we propose SCAN (a sparse editing framework based on Sparse Circuit Anchored Neuron) which transforms editing into a mechanism-aware manipulation by constructing a knowledge circuit via Sparse Transcoders. Experiments on Gemma2, Qwen3, and Llama3.1 across CounterFact, ZsRE and WikiFactDiff demonstrate that SCAN achieves a superior performance, maintaining model integrity on benchmarks like MMLU and GSM8K even after 3,000 sequential edits, whereas other existing methods deteriorate progressively as editing accumulates, eventually resulting in model collapse.

Abstract

Continual Learning for VLMs

Explore similar work

SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing

Towards Scalable Lifelong Knowledge Editing with Selective Knowledge Suppression

Generalizable Lifelong Model Editing via Preference Optimization