Organizations: Xi’an Jiaotong University · Zhejiang University · City University of Hong Kong · Harvard University · Institute of Automation, Chinese Academy of Sciences
Sequential knowledge editing in Large Language Models (LLMs) often leads to a degradation of general capabilities due to cumulative parameter interference. In this paper, we revisit prominent projection-based methods like AlphaEdit and reveal a surprising mismatch: their robustness is primarily driven by update regularization rather than the intended null-space projection. Based on this insight, we propose FisherEdit, a theoretically grounded method built on Fisher geometry, which replaces coarse activation-space projections with importance-aware regularization in parameter space. It penalizes changes to sensitive parameters more strongly while retaining flexibility for new facts, and admits an efficient row-wise closed-form solution. Experiments on Llama-3, Qwen2.5, and Olmo-3 across multiple benchmarks demonstrate that FisherEdit achieves state-of-the-art performance while better preserving model stability and downstream task accuracy.
Figures & tables
Figure 1: Efficacy–locality trade-off on Llama-3-8B-Instruct. Sweeping AlphaEdit’s truncation threshold τ traces a monotone Pareto curve. FisherEdit lands strictly above the curve.
Figure 2: (a) AlphaEdit’s null-space projector Pτ on Llama-3-8B-Instruct is close to the identity matrix, indicating that most edit directions are only weakly affected by the projection. (b) Singular spectrum of K0K0⊤ on the same layer. Only a few dozen directions exceed the truncation threshold τ .
Figure 3: Edit success for AlphaEdit and projector-disabled variant on Llama-3-8B. Across most of the regularization range, the w/ and w/o P curves almost overlap on all three edit-quality metrics.
Figure 4: Within AlphaEdit on Llama-3-8B-Instruct, changing λ from 10−5 to 10 produces a large jump in all three metrics.
Figure 5: (a) Downstream task scores of Llama-3-8B-Instruct on HumanEval, ARC-c, ARC-e, IFEval, WinoGrande and GPQA. (b) Wikitext perplexity of Llama-3-8B-Instruct after applying N∈{200,400,600,800,1000} CounterFact edits.
Eff
Gen
Spe
Avg
Fisher
99.58
91.28
75.13
88.66
Uniform
99.28
91.48
72.22
87.66
Random
95.89
85.74
69.94
83.86
Table 3: Fisher-matrix ablation on CounterFact with Llama-3-8B-Instruct under sequential editing. Fisher diagonal F is replaced with an all-ones matrix ( Uniform ) or with i.i.d. uniform random values ( Random ).
Table 7
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Eff
Gen
Spe
Avg
AlphaEdit
99.69 ± 0.10
90.17 ± 0.75
73.75 ± 0.30
87.87 ± 0.28
FisherEdit
99.79 ± 0.21
92.68 ± 0.70
73.91 ± 0.16
88.80 ± 0.28
Appendix
Table 6: Three-seed comparison on CounterFact with Llama-3-8B-Instruct. Mean ± standard deviation over seeds; each seed reshuffles the sequential edit stream.
CounterFact
Ripple
Method
Eff
Gen
Spe
Avg
Eff
Gen
Spe
Avg
MEMIT
50.26
50.26
49.74
50.09
53.80
53.58
49.49
52.29
RECT
29.85
31.54
75.46
45.61
31.50
32.38
76.15
46.68
PRUNE
32.41
32.36
71.19
45.32
32.50
33.81
73.24
46.52
RLEdit
48.00
48.26
52.91
49.72
49.30
47.19
55.62
50.70
UltraEdit
54.26
54.15
45.83
51.41
67.60
69.38
34.58
57.19
Appendix
Table 7: Sequential editing on Mistral-7B-Instruct-v0.3.
Layer
Pearson (log)
Spearman
Cosine
Top-1% overlap
Top-10% overlap
4
0.878
0.874
0.824
0.61
0.73
5
0.871
0.859
0.947
0.61
0.72
6
0.883
0.864
0.979
0.65
0.72
7
0.911
0.893
0.981
0.71
0.75
8
0.913
0.897
0.919
0.71
0.75
Appendix
Table 9: Agreement between the pre- and post-editing Fisher diagonals after 1,000 sequential edits, per edited layer.
Fisher
Eff
Gen
Spe
Avg
Extra time / batch
Fixed ( γ=1 )
99.90
90.92
75.68
88.83
0
Online ( γ=0.99 )
99.59
90.21
75.14
88.31
∼ 95 s
Online ( γ=0.9 )
99.79
90.56
74.87
88.41
∼ 97 s
Appendix
Table 10: Fixed versus online Fisher on CounterFact with Llama-3-8B-Instruct. Extra time is the additional wall-clock cost per 100-edit batch relative to the fixed Fisher.
Phase
AlphaEdit
FisherEdit
Diff.
Wall-clock time
Preprocessing (min)
648.0
17.7
−97.3%
Editing / 100 edits (s)
195.7
213.0
+8.9%
Evaluation (s)
328.4
318.7
−3.0%
End-to-end (min)
680.4
53.4
−92.1%
Peak memory
Appendix
Table 11: Phase-wise computational cost relative to AlphaEdit.
Figure 6: Hidden-state movement of rewrite and neighborhood prompts at layers 4 and 8.
Figure 7: AlphaEdit update magnitude during sequential editing under different λ .
Figure 8: Full-layer visualization of AlphaEdit’s null-space projector Pτ across edited backbones.
Parameter-based knowledge editing updates the internal knowledge of large language models (LLMs) via localized weight modifications and has attracted significant attention. However, most existing methods overlook fundamental theoretical limitations and are rarely evaluated under realistic, practice-oriented settings. In this paper, we first present a theoretical analysis based on the dimensional Collapse Hypothesis, explaining how localized parameter edits can propagate along fragile directions in the representation space, inducing global interference and ultimately causing reasoning collapse. Building on this insight, we conduct a comprehensive empirical evaluation by systematically varying knowledge complexity, number of edits, evaluation dimensions, and baseline methods. Our results show that parameter-based editing methods consistently damage core LLM capabilities. In contrast, a simple retrieval-based baseline achieves consistently stronger performance than all parameter-editing methods across all evaluated conditions. These findings highlight that preserving the fundamental capabilities of LLMs after knowledge editing should be a central concern for future research.
Wanying Ren, Xin Song, Futing Wang +2
East China Normal University, Shanghai, China · Westlake University, Hangzhou, China · Nanyang Technological University, Singapore
Large Language Models (LLMs) often suffer from catastrophic forgetting and collapse during sequential knowledge editing. This vulnerability stems from the prevailing dense editing paradigm, which treats models as black boxes and relies on coarse-grained parameter interventions that inevitably disrupt preserved knowledge. To address this, we propose SCAN (a sparse editing framework based on Sparse Circuit Anchored Neuron) which transforms editing into a mechanism-aware manipulation by constructing a knowledge circuit via Sparse Transcoders. Experiments on Gemma2, Qwen3, and Llama3.1 across CounterFact, ZsRE and WikiFactDiff demonstrate that SCAN achieves a superior performance, maintaining model integrity on benchmarks like MMLU and GSM8K even after 3,000 sequential edits, whereas other existing methods deteriorate progressively as editing accumulates, eventually resulting in model collapse.
Yuhuan Liu, Haitian Zhong, Xinyuan Xia +3
New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · Cuiying Honors College, Lanzhou University · Zhongguancun Academy +1
Large language models (LLMs) inevitably generate some incorrect or outdated content, necessitating efficient and precise mechanisms for continual knowledge updates. However, existing model editing methods struggle to sequentially edit unstructured long-form knowledge, suffering from severe edit forgetting and degradation of general capabilities. To address these challenges, we reframe knowledge editing from a manifold perspective, viewing it as a localized displacement of an edit sub-manifold within the global knowledge manifold. Under this formulation, the problem can be decomposed into two key questions: (i) how to identify representative edit points that effectively anchor the edit sub-manifold, and (ii) how to preserve the remaining manifold structure during the sub-manifold displacement process. Based on this perspective, we propose ManiEdit, a novel manifold-aware autoregressive editing framework consisting of two core components. Pivot Localization addresses the mediocre-point dilemma by identifying high-leverage pivots to anchor the edit sub-manifold. Manifold-Aware Preservation preserves different knowledge types through an energy-weighted penalty combined with recursive null-space alignment. Experiments on two base LLMs and four unstructured editing benchmarks demonstrate that ManiEdit achieves state-of-the-art performance, outperforming the strongest baseline by up to +27.81 BERTScore and +8.50 ROUGE-L, while maintaining near-original general capabilities across six representative downstream tasks. Our code is available at: https://github.com/Areyliu/ManiEdit
Rui Liu, Chenheng Zhang, Haoxuan Li +1
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University