SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing

Yuhuan Liu; Haitian Zhong; Xinyuan Xia; Qiang Liu; Shu Wu; Liang Wang

cs.AIMar 16, 2026

SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing

Authors: Yuhuan Liu, Haitian Zhong, Xinyuan Xia, Qiang Liu, Shu Wu, Liang Wang

Organizations: New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · Cuiying Honors College, Lanzhou University · Zhongguancun Academy · Research Institute of Intelligent Complex Systems, Fudan University

Abstract

Large Language Models (LLMs) often suffer from catastrophic forgetting and collapse during sequential knowledge editing. This vulnerability stems from the prevailing dense editing paradigm, which treats models as black boxes and relies on coarse-grained parameter interventions that inevitably disrupt preserved knowledge. To address this, we propose SCAN (a sparse editing framework based on Sparse Circuit Anchored Neuron) which transforms editing into a mechanism-aware manipulation by constructing a knowledge circuit via Sparse Transcoders. Experiments on Gemma2, Qwen3, and Llama3.1 across CounterFact, ZsRE and WikiFactDiff demonstrate that SCAN achieves a superior performance, maintaining model integrity on benchmarks like MMLU and GSM8K even after 3,000 sequential edits, whereas other existing methods deteriorate progressively as editing accumulates, eventually resulting in model collapse.

Figures & tables

Figure 1: Comparison of current methods and ours. Current methods (a) modify the entire dense MLP weight matrix. Our approach (b) isolates factual features, editing knowledge-relevant vectors.

Method	Model	CounterFact				ZsRE				WikiFactDiff
Method	Model	Rel $\uparrow$	Gen $\uparrow$	Loc $\uparrow$	Avg $\uparrow$	Rel $\uparrow$	Gen $\uparrow$	Loc $\uparrow$	Avg $\uparrow$	Rel $\uparrow$	Gen $\uparrow$	Loc $\uparrow$	Avg $\uparrow$
FT	Gemma2-2B	44.28	11.10	11.77	16.52	66.14	57.20	64.67	62.43	73.07	69.53	41.24	56.97
RECT		7.23	1.63	20.90	3.86	15.32	11.58	18.38	14.56	35.78	31.50	31.64	32.97
AlphaEdit		55.38	18.35	32.63	28.91	73.88	60.22	47.71	59.11	76.92	67.26	55.73	65.88
MEMIT		5.00	2.80	4.40	3.77	10.91	9.16	7.61	9.07	5.09	4.68	6.30	5.32
GRACE		100	0.37	99.80	1.10	98.80	23.56	100	46.55	98.30	51.20	99.07	75.50

Table 1: Sequential editing task performance comparison of our method and other methods after 1000 edits. The Avg column is calculated using the Harmonic Mean:

\text{Avg}=3/(\text{Rel}^{-1}+\text{Gen}^{-1}+\text{Loc}^{-1})

, to illustrate balanced performance across metrics. Bold and underline denote the best and second-best results per column, respectively.

Panel of Figure 2 — Figure 2: Cumulative proportion of selected feature across different token positions. (a) and (b) represent the distribution for Gemma2-2B and Qwen3-8B on CounterFact dataset, respectively.

Panel of Figure 3 — Figure 3: Distribution of selected feature across layers. Both models exhibit a characteristic dual-peak pattern, indicating functional localization in shallow and middle-to-deep layers.

Panel of Figure 4 — Figure 4: Heatmap of selected feature distribution across layers at special token position. The dark regions indicate that the early-layer peaks in Figure 3 align with the subject tokens, while the later-layer peaks correspond to the last token position on both models.

Method	Rel-Gen				Rel-Loc
Method	J-scr	J-acc	F1-scr	F1-acc	J-scr	J-acc	F1-scr	F1-acc
Qwen3-8B
All	0.258	0.421	0.394	0.423	0.297	0.409	0.445	0.394
Sub	0.594	0.992	0.731	0.994	0.162	0.931	0.272	0.928
No_last	0.346	0.689	0.496	0.694	0.151	0.880	0.249	0.877
Both	0.574	0.978	0.714	0.979	0.094	0.955	0.161	0.954

Table 2: Comparison of feature discriminability across different selection methods using Qwen3-8B and Gemma2-2B on the CF dataset.

Panel of Figure 5 — Figure 5: Activation visualization of identified features on the specific prompts. The left column shows Feature #13366 at Layer 19, and the right column shows Feature #410 at Layer 24. Darker colors indicate higher activation values.

Panel of Figure 6 — Figure 6: Comparison of SCAN against other methods across six benchmarks. The results show the ability to preserve the model’s general performance as the number of edit cases grows.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

V=V_{\text{embed}}\cup V_{\text{feature}}\cup V_{\text{error}}\cup V_{\text{logit}}

Appendix

Algorithm 1 Knowledge-Specific Circuit Construction and Pruning

Figure 7: Attribution Graph for the target token “Joe” on Gemma2-2B. Nodes represent features across layers. Blue edges denote promoting effects (positive weights), while orange edges denote suppressing effects (negative weights). The color opacity represents the relative magnitude of the attribution weight, with uniform line thickness for visual clarity.

Method	Model	CounterFact				ZsRE				WikiFactDiff
Method	Model	Rel $\uparrow$	Gen $\uparrow$	Loc $\uparrow$	Avg $\uparrow$	Rel $\uparrow$	Gen $\uparrow$	Loc $\uparrow$	Avg $\uparrow$	Rel $\uparrow$	Gen $\uparrow$	Loc $\uparrow$	Avg $\uparrow$
FT	Llama3.2-1B	32.25	11.05	1.90	5.23	55.97	48.16	6.01	14.86	68.17	64.81	16.95	30.98
RECT		2.30	4.10	0	0	1.50	1.33	1.56	1.46	0.44	0.37	0	0
AlphaEdit		86.30	42.95	23.70	39.61	80.66	66.36	45.70	61.21	48.06	41.45	24.60	35.15
MEMIT		0	0	0.30	0	0.23	0.20	4.50	0.55	0.15	0.15	0.76	0.24
GRACE		100	0.80	99.80	2.37	99.40	24.20	100	42.02	99.82	48.30	98.84	70.38

Appendix

Table 3: Sequential editing task performance comparison of our method and other methods after 1000 edits. Bold and underline denote the best and second-best results per column, respectively.

Explore similar work

BetaEdit: Null-Space Constrained Sequential Model Editing
May 10, 2026Bingqing Liu, Wei Liu, Yuhua LiKnowledge Editing Model Editing
Huazhong University of Science and Technology, Wuhan, China
The Labyrinth and the Thread: Rethinking Regularizations in Sequential Knowledge Editing for Large Language Models
May 26, 2026Zheng Wang, Kaixuan Zhang, Wanfang Chen +2Knowledge Editing Factual Knowledge in Language Models
Bosch Center for Artificial Intelligence (BCAI) · Bosch (China) Investment Ltd. · School of Statistics, East China Normal University.
ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspective
Sep 27, 2026Rui Liu, Chenheng Zhang, Haoxuan Li +1Knowledge Editing Continual Learning for LLMs
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University

May 10, 2026J/K move · Enter open · S save

BetaEdit: Null-Space Constrained Sequential Model Editing

Bingqing Liu, Wei Liu, Yuhua Li

Huazhong University of Science and Technology, Wuhan, China

Knowledge Editing Model Editing Language Model Editing

Open paper AskPDF arXiv

Null-space-based methods have garnered considerable attention in model editing by constraining updates to the null space of the pre-existing knowledge representation, thereby preserving the model's original behavior. However, in practice these methods rely on an approximate null space--leading to knowledge leakage--and further suffer from severe performance degradation during sequential editing. Recent work shows that history-aware editing strategies can empirically mitigate this decline, yet the underlying reason remains unclear. In this paper, we first expose the knowledge leakage inherent in existing null-space approaches and then analyze why history-aware updates effectively preserve both editing performance and general capabilities during long-horizon editing. Building on these insights, we propose BetaEdit, a refined framework that effectively controls the knowledge leakage and integrates history-aware updates into the null-space paradigm. Extensive experiments on three large language models across two standard benchmarks show that BetaEdit consistently outperforms prior methods in the challenging regime of massive-scale sequential editing. Code is available at: https://github.com/lbq8942/BetaEdit.

Abstract

Explore similar work

BetaEdit: Null-Space Constrained Sequential Model Editing

The Labyrinth and the Thread: Rethinking Regularizations in Sequential Knowledge Editing for Large Language Models

ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspective