Model editing aims to update knowledge to add new concepts and change relevant information without retraining. Lifelong editing is a challenging task, prone to disrupting previously learned concepts, especially for Vision Language Models (VLMs), because sequential edits can lead to degraded reasoning and cross modal misalignment. Existing VLM knowledge editing methods based on gated adapters, activation edits, and parameter merging techniques address catastrophic forgetting seen in full fine tuning; however, they still operate in the shared representation space of the VLM, where concepts are entangled, so edits interfere with other non relevant concepts. We hypothesize that this instability persists because current methods algorithmically control edits via optimization rather than structurally separating knowledge. We introduce Dynamic Subspace Concept Alignment (DSCA) which by design mitigates this limitation by decomposing the representation space into a set of orthogonal semantic subspaces and proposing edits only in those transformed spaces. These subspaces are obtained through incremental clustering and PCA on joint vision language representations. This process structurally isolates concepts, enabling precise, non interfering edits by turning isolation from a soft training objective into an architectural property. The surgical edits are guided by a multi term loss function for maintaining task fidelity, edit locality, and cross modal alignment. With the base model frozen, our method achieves 98 percent single edit success, remains over 95 percent after 1000 sequential edits, lowers hallucination by 3 to 5 percent, and achieves the best backward transfer (BWT) scores on continual instruction tuning benchmarks. Extensive experiments demonstrate DSCA state of the art stability and knowledge retention capability in continual lifelong editing across various datasets and benchmarks.
Figures & tables
Figure 1: Conceptual comparison of knowledge-editing paradigms. (a) The initial concept space where concepts are well-separated. (b) Global fine-tuning perturbs the entire representation space, distorting unrelated concepts. (c) LoRA / local adapters constrain edits but still produce coupled interference. (d) DSCA performs subspace-confined, concept-specific interventions, maintaining isolation and preserving all other concepts.
Figure 2
Dataset
Method
Rel.
T-Gen.
V-Gen.
T-Loc.
M-Loc.
Avg.
E-VQA
DualEdit [ 30 ]
96.94
96.43
96.20
100.00
99.61
97.84
VisEdit [ 4 ]
95.78
94.21
94.37
100.00
91.11
95.09
SERAC [ 25 ]
82.51
81.60
80.05
100.00
57.48
80.33
LTE [ 13 ]
94.16
93.54
93.06
83.76
81.65
89.23
MEND [ 26 ]
92.30
92.16
92.10
90.30
81.13
89.60
DSCA (ours)
98.12
97.30
97.25
100.00
99.83
98.50
Table 1: Single-edit results on E-VQA [ 5 ] and E-IC [ 5 ] (LLaVA-1.5-7B [ 15 ] ). Baselines from DualEdit [ 30 ] .
Table 3: Performance on the CoIN benchmark across four metrics (mean ± std over three task orders). Higher is better for all metrics; less negative BWT indicates less forgetting. Baselines from PAM [ 32 ]
Method
MME [ 7 ]
MM-Vet [ 38 ]
VQA-v2 [ 9 ]
TextVQA [ 31 ]
COCO CIDEr [ 14 ] [ 34 ]
LiveEdit [ 3 ]
73.4
54.2
82.1
61.5
120.6
DualEdit [ 30 ]
74.1
55.6
83.5
62.9
121.4
MRT [ 17 ]
70.9
50.8
80.2
59.3
118.2
DSCA (ours)
76.3
57.8
84.9
64.7
123.8
Table 4: Capability retention on standard LVLM benchmarks.
Method
CHAIR-H ↓
Faithfulness ↑
Richness ≈
Gen-Anchor Rep-Edit [ 29 ]
18.5
92.3
98.1
DualEdit [ 30 ]
20.8
90.6
97.2
LiveEdit [ 3 ]
21.1
91.0
96.8
DSCA (ours)
15.9
94.5
98.4
Table 5: Hallucination stress test (CHAIR) [ 28 ] . Lower CHAIR-H is better.
Figure 2: Diagnostic analysis of DSCA. (a) Mean pairwise subspace overlap( ε=∥Ri⊤Rj∥F2 ) as a function of the number of sequential edits. DSCA with residualized Incremental PCA keeps the overlap essentially flat at ≈3×10−3 across 1,000 edits, comparable to a globally orthonormal baseline, whereas a variant without orthogonalization drifts to more than 10−1 . (b) Relationship between mean subspace overlap and forgetting, measured as 1−BWT . The scatter plot and fitted line (Pearson r≈0.94 ) show a strong monotonic dependence, as overlap ε increases, forgetting grows almost linearly, consistent with the bounded-interference result in Supplementary. (c) Ablation over subspace construction strategies (No orthogonalization, residualized PCA, global orthonormalization). For each metric (mean overlap, BWT, reliability) scores are normalized per metric to [0,1] for comparability. Residualized PCA attains nearly the same BWT and reliability as global orthonormalization while dramatically reducing overlap relative to the non-orthogonal baseline, giving the best overall trade-off.
Figure 3: Routing sparsity analysis. (a) Histogram of routing weights shows that over 95% are below 0.05, indicating highly selective module activation. (b) Trade-off between sparsity coefficient λsparse and average number of active DSAMs. The chosen operating point (blue dot) yields ≈3 active DSAMs per input without degrading performance.
Variant
Orthogonality in Rk
Gate Sparsity
Multi-stage Routing
Basis Residual intervention
Full Subspace
Full Rank
ES ↑
Locality Δ↓
GEN ↑
Full DSCA
✓
✓
✓
✓
✓
✓
98.0
0.5
97.3
w/o orthogonality
✗
✓
✓
✓
✓
✓
95.8
2.8
93.4
w/o gate sparsity
✓
✗
✓
✓
✓
✓
96.1
2.1
94.7
single-stage routing
✓
✓
✗
✓
✓
✓
96.9
1.9
95.0
no basis-residual
✓
✓
✓
✗
✓
✓
97.1
1.5
95.8
fewer subspaces (K/2)
✓
✓
✓
✓
✗
✓
97.0
1.7
95.5
Table 6: Ablation study on DSCA components. A checkmark ( ✓ ) indicates the component is present; a cross ( × ) indicates it has been ablated or modified.
Figure 4: Concept-wise subspace visualization. t-SNE of fused representations projected through their assigned semantic subspaces Rk⊤hf for a subset of concepts (e.g., “car”, “truck”, “bicycle”). DSCA yields compact, well-separated clusters, indicating that edits remain confined to localized regions of the representation space.This empirically validates the conceptual illustration provided in Figure 1 (d) of the main paper.
Table 10: Expanded table for perfomance on COIN benchmark [ 2 ] . Higher is better for all metrics; less negative BWT indicates less forgetting.Baselines are taken from PAM [ 32 ]
Large Language Models (LLMs) often suffer from catastrophic forgetting and collapse during sequential knowledge editing. This vulnerability stems from the prevailing dense editing paradigm, which treats models as black boxes and relies on coarse-grained parameter interventions that inevitably disrupt preserved knowledge. To address this, we propose SCAN (a sparse editing framework based on Sparse Circuit Anchored Neuron) which transforms editing into a mechanism-aware manipulation by constructing a knowledge circuit via Sparse Transcoders. Experiments on Gemma2, Qwen3, and Llama3.1 across CounterFact, ZsRE and WikiFactDiff demonstrate that SCAN achieves a superior performance, maintaining model integrity on benchmarks like MMLU and GSM8K even after 3,000 sequential edits, whereas other existing methods deteriorate progressively as editing accumulates, eventually resulting in model collapse.
Yuhuan Liu, Haitian Zhong, Xinyuan Xia +3
New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · Cuiying Honors College, Lanzhou University · Zhongguancun Academy +1
Large language models (LLMs) require frequent knowledge updates to reflect changing facts and mitigate hallucinations. To meet this demand, lifelong knowledge editing has emerged as a continual approach to modify specific pieces of knowledge without retraining the entire model. Existing parameter editing methods struggle with stability during sequential edits due to catastrophic forgetting. While retrieval-based approaches are proposed to alleviate this issue, their applicability remains limited across various datasets because of high training costs. To address these limitations and enhance scalability in lifelong settings, we propose LightEdit. Our framework first selects relevant knowledge from retrieved information to modify the query effectively. It then incorporates a decoding strategy to suppress the model's original knowledge probabilities, thereby enabling efficient edits based on the selected information. Extensive experiments on ZSRE, Counterfact, and RIPE benchmarks demonstrate that LightEdit outperforms existing lifelong knowledge editing methods. Furthermore, by minimizing training costs, LightEdit achieves cost-effective scalability, enabling easy adaptation to various datasets.
Dahyun Jung, Jaewook Lee, Heuiseok Lim
Department of Computer Science and Engineering, Korea University
Knowledge editing enables rapid updates of specific factual knowledge in large language models (LLMs) without full retraining. However, more realistic scenarios call for a lifelong framework that handles continual updates rather than one-off modifications. In such settings, existing editing methods often overfit to target prompts, significantly degrading both the generalization of the edited knowledge and the model's general capabilities. To address this issue, we propose GLIME (Generalizable Lifelong Model Editing), which combines knowledge editing with preference optimization over generation behavior. GLIME further incorporates replay-based editing and a gradient constraint to preserve previously edited knowledge. Experimental results show that GLIME significantly improves knowledge generalization in lifelong editing settings while maintaining both editing performance and general capabilities.
Dahyun Jung, Suhyune Son, Heuiseok Lim
Department of Computer Science and Engineering, Korea University