SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing
Authors: Yuhuan Liu, Haitian Zhong, Xinyuan Xia, Qiang Liu, Shu Wu, Liang Wang
Organizations: New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · Cuiying Honors College, Lanzhou University · Zhongguancun Academy · Research Institute of Intelligent Complex Systems, Fudan University
Large Language Models (LLMs) often suffer from catastrophic forgetting and collapse during sequential knowledge editing. This vulnerability stems from the prevailing dense editing paradigm, which treats models as black boxes and relies on coarse-grained parameter interventions that inevitably disrupt preserved knowledge. To address this, we propose SCAN (a sparse editing framework based on Sparse Circuit Anchored Neuron) which transforms editing into a mechanism-aware manipulation by constructing a knowledge circuit via Sparse Transcoders. Experiments on Gemma2, Qwen3, and Llama3.1 across CounterFact, ZsRE and WikiFactDiff demonstrate that SCAN achieves a superior performance, maintaining model integrity on benchmarks like MMLU and GSM8K even after 3,000 sequential edits, whereas other existing methods deteriorate progressively as editing accumulates, eventually resulting in model collapse.
Figures & tables
Figure 1: Comparison of current methods and ours. Current methods (a) modify the entire dense MLP weight matrix. Our approach (b) isolates factual features, editing knowledge-relevant vectors.
Method
Model
CounterFact
ZsRE
WikiFactDiff
Rel ↑
Gen ↑
Loc ↑
Avg ↑
Rel ↑
Gen ↑
Loc ↑
Avg ↑
Rel ↑
Gen ↑
Loc ↑
Avg ↑
FT
Gemma2-2B
44.28
11.10
11.77
16.52
66.14
57.20
64.67
62.43
73.07
69.53
41.24
56.97
RECT
7.23
1.63
20.90
3.86
15.32
11.58
18.38
14.56
35.78
31.50
31.64
32.97
AlphaEdit
55.38
18.35
32.63
28.91
73.88
60.22
47.71
59.11
76.92
67.26
55.73
65.88
MEMIT
5.00
2.80
4.40
3.77
10.91
9.16
7.61
9.07
5.09
4.68
6.30
5.32
GRACE
100
0.37
99.80
1.10
98.80
23.56
100
46.55
98.30
51.20
99.07
75.50
Table 1: Sequential editing task performance comparison of our method and other methods after 1000 edits. The Avg column is calculated using the Harmonic Mean: Avg=3/(Rel−1+Gen−1+Loc−1) , to illustrate balanced performance across metrics. Bold and underline denote the best and second-best results per column, respectively.
Figure 2: Cumulative proportion of selected feature across different token positions. (a) and (b) represent the distribution for Gemma2-2B and Qwen3-8B on CounterFact dataset, respectively.
Figure 3: Distribution of selected feature across layers. Both models exhibit a characteristic dual-peak pattern, indicating functional localization in shallow and middle-to-deep layers.
Figure 4: Heatmap of selected feature distribution across layers at special token position. The dark regions indicate that the early-layer peaks in Figure 3 align with the subject tokens, while the later-layer peaks correspond to the last token position on both models.
Method
Rel-Gen
Rel-Loc
J-scr
J-acc
F1-scr
F1-acc
J-scr
J-acc
F1-scr
F1-acc
Qwen3-8B
All
0.258
0.421
0.394
0.423
0.297
0.409
0.445
0.394
Sub
0.594
0.992
0.731
0.994
0.162
0.931
0.272
0.928
No_last
0.346
0.689
0.496
0.694
0.151
0.880
0.249
0.877
Both
0.574
0.978
0.714
0.979
0.094
0.955
0.161
0.954
Table 2: Comparison of feature discriminability across different selection methods using Qwen3-8B and Gemma2-2B on the CF dataset.
Figure 5: Activation visualization of identified features on the specific prompts. The left column shows Feature #13366 at Layer 19, and the right column shows Feature #410 at Layer 24. Darker colors indicate higher activation values.
Figure 6: Comparison of SCAN against other methods across six benchmarks. The results show the ability to preserve the model’s general performance as the number of edit cases grows.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
V=Vembed∪Vfeature∪Verror∪Vlogit
Appendix
Algorithm 1 Knowledge-Specific Circuit Construction and Pruning
Figure 7: Attribution Graph for the target token “Joe” on Gemma2-2B. Nodes represent features across layers. Blue edges denote promoting effects (positive weights), while orange edges denote suppressing effects (negative weights). The color opacity represents the relative magnitude of the attribution weight, with uniform line thickness for visual clarity.
Method
Model
CounterFact
ZsRE
WikiFactDiff
Rel ↑
Gen ↑
Loc ↑
Avg ↑
Rel ↑
Gen ↑
Loc ↑
Avg ↑
Rel ↑
Gen ↑
Loc ↑
Avg ↑
FT
Llama3.2-1B
32.25
11.05
1.90
5.23
55.97
48.16
6.01
14.86
68.17
64.81
16.95
30.98
RECT
2.30
4.10
0
0
1.50
1.33
1.56
1.46
0.44
0.37
0
0
AlphaEdit
86.30
42.95
23.70
39.61
80.66
66.36
45.70
61.21
48.06
41.45
24.60
35.15
MEMIT
0
0
0.30
0
0.23
0.20
4.50
0.55
0.15
0.15
0.76
0.24
GRACE
100
0.80
99.80
2.37
99.40
24.20
100
42.02
99.82
48.30
98.84
70.38
Appendix
Table 3: Sequential editing task performance comparison of our method and other methods after 1000 edits. Bold and underline denote the best and second-best results per column, respectively.
Null-space-based methods have garnered considerable attention in model editing by constraining updates to the null space of the pre-existing knowledge representation, thereby preserving the model's original behavior. However, in practice these methods rely on an approximate null space--leading to knowledge leakage--and further suffer from severe performance degradation during sequential editing. Recent work shows that history-aware editing strategies can empirically mitigate this decline, yet the underlying reason remains unclear. In this paper, we first expose the knowledge leakage inherent in existing null-space approaches and then analyze why history-aware updates effectively preserve both editing performance and general capabilities during long-horizon editing. Building on these insights, we propose BetaEdit, a refined framework that effectively controls the knowledge leakage and integrates history-aware updates into the null-space paradigm. Extensive experiments on three large language models across two standard benchmarks show that BetaEdit consistently outperforms prior methods in the challenging regime of massive-scale sequential editing. Code is available at: https://github.com/lbq8942/BetaEdit.
Bingqing Liu, Wei Liu, Yuhua Li
Huazhong University of Science and Technology, Wuhan, China
Sequential editing of structured knowledge in large language models allows targeted factual updates without retraining, yet existing methods often rely on complex regularization or constraint mechanisms whose necessity remains unclear. In this work, we systematically investigate the mechanisms underlying effective and stable sequential editing. Specifically, we first analyze the empirical success of AlphaEdit and establish, via a rigorous optimization analysis, the formal equivalence between one-time and sequential editing. Building on this insight, we generalize the equivalence to a broader class of editing objectives, demonstrating that stability emerges naturally from properly accounting for accumulated editing constraints, rather than from specialized regularization or null-space operations. We empirically confirm that many commonly used regularization strategies are unnecessary for reliable sequential updates. Furthermore, we extend our framework to handle conflicting edits, ensuring robust and consistent behavior under contradictory updates. Ultimately, our work provides Ariadne's thread through the labyrinth of sequential editing, charting a path toward simpler, more interpretable, and dependable knowledge updates. Our code is available at https://github.com/Wangzzzzzzzz/OTE-SE-Alignment.
Zheng Wang, Kaixuan Zhang, Wanfang Chen +2
Bosch Center for Artificial Intelligence (BCAI) · Bosch (China) Investment Ltd. · School of Statistics, East China Normal University.
Large language models (LLMs) inevitably generate some incorrect or outdated content, necessitating efficient and precise mechanisms for continual knowledge updates. However, existing model editing methods struggle to sequentially edit unstructured long-form knowledge, suffering from severe edit forgetting and degradation of general capabilities. To address these challenges, we reframe knowledge editing from a manifold perspective, viewing it as a localized displacement of an edit sub-manifold within the global knowledge manifold. Under this formulation, the problem can be decomposed into two key questions: (i) how to identify representative edit points that effectively anchor the edit sub-manifold, and (ii) how to preserve the remaining manifold structure during the sub-manifold displacement process. Based on this perspective, we propose ManiEdit, a novel manifold-aware autoregressive editing framework consisting of two core components. Pivot Localization addresses the mediocre-point dilemma by identifying high-leverage pivots to anchor the edit sub-manifold. Manifold-Aware Preservation preserves different knowledge types through an energy-weighted penalty combined with recursive null-space alignment. Experiments on two base LLMs and four unstructured editing benchmarks demonstrate that ManiEdit achieves state-of-the-art performance, outperforming the strongest baseline by up to +27.81 BERTScore and +8.50 ROUGE-L, while maintaining near-original general capabilities across six representative downstream tasks. Our code is available at: https://github.com/Areyliu/ManiEdit
Rui Liu, Chenheng Zhang, Haoxuan Li +1
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University