Organizations: Xi’an Jiaotong University · Zhejiang University · City University of Hong Kong · Harvard University · Institute of Automation, Chinese Academy of Sciences
Sequential knowledge editing in Large Language Models (LLMs) often leads to a degradation of general capabilities due to cumulative parameter interference. In this paper, we revisit prominent projection-based methods like AlphaEdit and reveal a surprising mismatch: their robustness is primarily driven by update regularization rather than the intended null-space projection. Based on this insight, we propose FisherEdit, a theoretically grounded method built on Fisher geometry, which replaces coarse activation-space projections with importance-aware regularization in parameter space. It penalizes changes to sensitive parameters more strongly while retaining flexibility for new facts, and admits an efficient row-wise closed-form solution. Experiments on Llama-3, Qwen2.5, and Olmo-3 across multiple benchmarks demonstrate that FisherEdit achieves state-of-the-art performance while better preserving model stability and downstream task accuracy.
Figures & tables
Figure 1: Efficacy–locality trade-off on Llama-3-8B-Instruct. Sweeping AlphaEdit’s truncation threshold τ traces a monotone Pareto curve. FisherEdit lands strictly above the curve.
Figure 2: (a) AlphaEdit’s null-space projector Pτ on Llama-3-8B-Instruct is close to the identity matrix, indicating that most edit directions are only weakly affected by the projection. (b) Singular spectrum of K0K0⊤ on the same layer. Only a few dozen directions exceed the truncation threshold τ .
Figure 3: Edit success for AlphaEdit and projector-disabled variant on Llama-3-8B. Across most of the regularization range, the w/ and w/o P curves almost overlap on all three edit-quality metrics.
Figure 4: Within AlphaEdit on Llama-3-8B-Instruct, changing λ from 10−5 to 10 produces a large jump in all three metrics.
Figure 5: (a) Downstream task scores of Llama-3-8B-Instruct on HumanEval, ARC-c, ARC-e, IFEval, WinoGrande and GPQA. (b) Wikitext perplexity of Llama-3-8B-Instruct after applying N∈{200,400,600,800,1000} CounterFact edits.
Eff
Gen
Spe
Avg
Fisher
99.58
91.28
75.13
88.66
Uniform
99.28
91.48
72.22
87.66
Random
95.89
85.74
69.94
83.86
Table 3: Fisher-matrix ablation on CounterFact with Llama-3-8B-Instruct under sequential editing. Fisher diagonal F is replaced with an all-ones matrix ( Uniform ) or with i.i.d. uniform random values ( Random ).
Table 7
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Eff
Gen
Spe
Avg
AlphaEdit
99.69 ± 0.10
90.17 ± 0.75
73.75 ± 0.30
87.87 ± 0.28
FisherEdit
99.79 ± 0.21
92.68 ± 0.70
73.91 ± 0.16
88.80 ± 0.28
Appendix
Table 6: Three-seed comparison on CounterFact with Llama-3-8B-Instruct. Mean ± standard deviation over seeds; each seed reshuffles the sequential edit stream.
CounterFact
Ripple
Method
Eff
Gen
Spe
Avg
Eff
Gen
Spe
Avg
MEMIT
50.26
50.26
49.74
50.09
53.80
53.58
49.49
52.29
RECT
29.85
31.54
75.46
45.61
31.50
32.38
76.15
46.68
PRUNE
32.41
32.36
71.19
45.32
32.50
33.81
73.24
46.52
RLEdit
48.00
48.26
52.91
49.72
49.30
47.19
55.62
50.70
UltraEdit
54.26
54.15
45.83
51.41
67.60
69.38
34.58
57.19
Appendix
Table 7: Sequential editing on Mistral-7B-Instruct-v0.3.
Layer
Pearson (log)
Spearman
Cosine
Top-1% overlap
Top-10% overlap
4
0.878
0.874
0.824
0.61
0.73
5
0.871
0.859
0.947
0.61
0.72
6
0.883
0.864
0.979
0.65
0.72
7
0.911
0.893
0.981
0.71
0.75
8
0.913
0.897
0.919
0.71
0.75
Appendix
Table 9: Agreement between the pre- and post-editing Fisher diagonals after 1,000 sequential edits, per edited layer.
Fisher
Eff
Gen
Spe
Avg
Extra time / batch
Fixed ( γ=1 )
99.90
90.92
75.68
88.83
0
Online ( γ=0.99 )
99.59
90.21
75.14
88.31
∼ 95 s
Online ( γ=0.9 )
99.79
90.56
74.87
88.41
∼ 97 s
Appendix
Table 10: Fixed versus online Fisher on CounterFact with Llama-3-8B-Instruct. Extra time is the additional wall-clock cost per 100-edit batch relative to the fixed Fisher.
Phase
AlphaEdit
FisherEdit
Diff.
Wall-clock time
Preprocessing (min)
648.0
17.7
−97.3%
Editing / 100 edits (s)
195.7
213.0
+8.9%
Evaluation (s)
328.4
318.7
−3.0%
End-to-end (min)
680.4
53.4
−92.1%
Peak memory
Appendix
Table 11: Phase-wise computational cost relative to AlphaEdit.
Figure 6: Hidden-state movement of rewrite and neighborhood prompts at layers 4 and 8.
Figure 7: AlphaEdit update magnitude during sequential editing under different λ .
Figure 8: Full-layer visualization of AlphaEdit’s null-space projector Pτ across edited backbones.
New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · Cuiying Honors College, Lanzhou University · Zhongguancun Academy +1
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University