Editing Large Language Models with Geometry-Aware Regularization
Authors: Bingqing Liu, Wei Liu, Jun Wang, Xiaobo Sun, Yuhua Li, Ruixuan Li
Organizations: School of Computer Science and Technology, Huazhong University of Science and Technology · iWudao Tech · Department of Human Genetics, School of Medicine, Emory University
Knowledge editing has become a key technique for updating knowledge in large language models. However, existing methods neglect the geometric structure of the knowledge representation space, leading to two key limitations: they cannot distinguish which pre-trained knowledge should be preferentially preserved (core vs. non-core) nor which new edits are inherently difficult (hard vs. easy). To address this, we propose GEdit, a geometry-aware framework built on a Low-Rank Plus Diagonal (LR+D) factor model that captures the intrinsic low-rank structure of Transformer FFN hidden representations. GEdit features two complementary mechanisms. First, the low-rank regularization applies targeted protection on a low-rank core knowledge subspace. Second, the adaptive regularization dynamically adjusts the regularization strength based on each edit request's editability: relaxing constraints for hard edits to ensure success, and tightening them for easy edits to better preserve pretrained knowledge. Extensive experiments on GPT2-XL, GPT-J, and LLaMA-3-8B under large-scale editing scenarios, including both mini-batch and fully sequential editing settings with up to 10,000 continuous updates, show that GEdit consistently and significantly outperforms state-of-the-art baselines in both editing success and knowledge preservation.
Figures & tables
Figure 1: Top-10 singular values (left axis) and their cumulative ratio of the total singular value sum (right axis).
Method
Weight Update Δ
Editability E(ke)
Output Perturbation
MEMIT
reke⊤(μC0+keke⊤)−1
μ1ke⊤C0−1ke
Δke=1+E(ke)E(ke)re
ROME
re(ke⊤(μC0)−1ke+aI)−1ke⊤(μC0)−1
μa1ke⊤C0−1ke
AlphaEdit
reke⊤P(keke⊤P+μI)−1
μ1ke⊤Pke
Table 1: Unification of output perturbation via editability.
Figure 2: Distribution of editability across 10,000 edits from the CounterFact dataset, along with the editing success rate and locality degradation within each editability bin.
Figure 3: Effect of hyperparameters γ (energy threshold) and c (desired editability) on editing performance. Metrics on the left axis: efficacy and generality; metric on the right axis: locality.
Method
CounterFact
ZsRE
bs = 1
bs = 20
bs = 200
bs = 1
bs = 20
bs = 200
Eff.
Gen.
Loc.
Eff.
Gen.
Loc.
Eff.
Gen.
Loc.
Eff.
Gen.
Loc.
Eff.
Gen.
Loc.
Eff.
Gen.
Loc.
Pre-edited
0.40
0.50
10.6
0.40
0.50
10.6
0.40
0.50
10.6
0.00
0.00
24.3
0.00
0.00
24.3
0.00
0.00
24.3
FT
GPT2-XL-1.5B
0.00
0.00
0.00
0.00
0.00
0.00
7.10
2.50
0.10
0.00
0.00
0.00
0.00
0.00
0.00
10.3
3.60
4.20
RLEdit
0.00
0.00
0.00
33.5
21.8
2.20
45.3
25.5
2.80
0.00
0.00
0.00
48.9
32.3
16.8
37.3
11.1
11.2
RECT
23.6
13.1
4.10
54.5
33.6
6.40
69.5
37.9
6.90
64.0
50.8
24.1
67.9
56.2
24.1
66.6
53.5
24.5
Table 2: Editing performance over 10,000 sequential edits under varying batch sizes (1, 20 and 200). Best and second-best results are bolded and underlined , respectively.
Figure 4: Total editing time for 10,000 sequential edits (batch size = 1) on the CounterFact dataset using three LLMs. From left to right: GPT2-XL, GPT-J-6B, and LLaMA3-8B, with the number of parameters and FFN hidden dimension shown in parentheses.
Figure 7
Method
GPT-J
LLaMA
Eff.
Loc.
Eff.
Loc.
EMMET
54.6->76.4
3.1->4.3
(2k)->(4k)
(2k)->(4k)
HSE
26.7->71.7
2.4->4.7
(7k)->(10k)
(7k)->(10k)
AlphaEdit
83.7->85.4
3.6->4.2
0.0->77.8
0.0->10.1
Table 3: Effect of low-rank regularization on different editing methods.
Method
GPT-J
LLaMA
Eff.
Loc.
Eff.
Loc.
EMMET
54.6->77.5
3.1->4.5
0.0->70.3
0.0->6.8
HSE
26.7->78.1
2.4->4.2
0.0->72.3
0.0->7.4
AlphaEdit
83.7->84.1
3.6->5.7
(2k)->(5k)
(2k)->(5k)
Table 4: Effect of adaptive regularization on different editing methods.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Energy Retention Ratio
0.1
0.3
0.5
0.7
0.8
0.9
0.95
1.0
LLaMA ( d =14336)
21
375
1472
3875
5958
9104
11314
14336
GPT-J ( d =16384)
1
9
639
3326
5911
9773
12453
16384
GPT2-XL ( d =6400)
1
95
629
1852
2820
4154
5047
6400
Appendix
Table 5: Mapping from energy retention ratio to effective rank for each model. Although ranks vary slightly across editing layers, they remain consistent within a narrow range.
Figure 7: Sequential editing performance on CounterFact (top row) and ZsRE (bottom row) as the number of edits scales from 1k to 10k. GEdit variants (denoted with the “-G” suffix) consistently outperform their base editing methods.
Setting
Regularization strength μ
5000
10000
15000
20000
50000
Eff.
Gen.
Loc.
Eff.
Gen.
Loc.
Eff.
Gen.
Loc.
Eff.
Gen.
Loc.
Eff.
Gen.
Loc.
GEdit / GPT-J / CF
76.2
37.3
3.80
80.1
41.8
4.30
85.8
45.4
5.00
87.4
48.1
5.50
42.3
28.9
8.90
GEdit / GPT-J / ZsRE
69.9
52.4
20.1
77.2
69.1
23.8
86.5
78.3
25.4
83.0
74.2
26.3
45.1
34.5
27.0
GEdit / LLaMA / CF
66.1
41.2
3.90
72.8
60.9
5.20
76.3
61.5
7.40
71.1
58.1
9.00
38.6
27.4
16.7
GEdit / LLaMA / ZsRE
80.4
69.1
32.8
89.7
81.5
34.5
85.2
74.4
34.9
81.1
70.9
35.0
39.5
26.2
37.0
Appendix
Table 6: Performance of GEdit and HSE under fixed regularization, which requires per-method, per-dataset, and per-model tuning and is therefore suboptimal. CounterFact is abbreviated as CF.
Figure 13
Figure 10: Adaptive regularization strength across 10,000 edits on the CounterFact dataset, applied to the LLaMA model. The desired editability is set to 1.25 . From left to right: EMMET, MEMIT, AlphaEdit.
Figure 11: SST and MMLU performance across 10,000 sequential edits on the CounterFact dataset, evaluated every 1,000 steps (10 evaluation points total). Results are shown for three base models (GPT-2, GPT-J, and LLaMA) across five editing methods: HSE, AlphaEdit, SPHERE, RECT, and GEdit.
Method
CounterFact
ZsRE
SST
Eff*.
Gen*.
Loc*.
Eff.
Gen.
Loc.
SST
Eff.
Gen.
Loc.
Pre-edited
62.5
22.1
24.4
78.0
0.10
0.40
10.6
62.5
0.00
0.00
25.0
HSE
GPT2-XL
62.0
98.0
88.6
65.7
91.4
58.4
10.6
60.0
94.7
88.4
26.9
EMMET
62.0
92.4
85.6
57.1
71.1
49.9
5.50
62.0
85.0
77.8
24.3
AlphaEdit
62.5
99.6
94.0
65.7
97.3
65.3
6.40
62.5
96.0
88.8
26.8
GEdit
62.5
99.6
94.0
65.6
96.8
65.6
6.40
62.5
95.2
88.0
26.4
Appendix
Table 7: Editing performance under 2,000 sequential edits (batch size=1).
Method
CounterFact
ZsRE
SST
Eff*.
Gen*.
Loc*.
Eff.
Gen.
Loc.
SST
Eff.
Gen.
Loc.
Pre-edited
62.5
21.4
23.8
78.2
0.20
0.40
10.6
62.5
0.00
0.00
24.3
HSE
GPT2-XL
60.0
89.6
76.2
58.0
60.9
34.1
6.30
61.0
92.3
84.6
25.3
EMMET
58.5
87.4
76.3
55.5
54.6
30.6
3.70
61.5
82.1
76.0
21.8
AlphaEdit
62.0
98.0
88.3
60.8
88.7
52.9
4.70
62.0
91.2
84.3
23.6
GEdit
62.0
97.8
87.7
61.0
85.4
50.1
4.90
62.0
90.6
83.3
23.2
Appendix
Table 8: Editing performance under 5,000 sequential edits (batch size=1).
Figure 12: Inference time (in seconds) per input prompt on LLaMA3, GPT-J, and GPT2-XL. Our method consistently achieves the lowest latency across all backbone models.
Method
GPT2-XL
GPT-J
LLaMA3-8B
Eff.
Loc.
CP.
Eff.
Loc.
CP.
Eff.
Loc.
CP.
ROME
99.7
10.4
4.8
99.9
13.8
5.5
99.9
21.2
5.3
MEMIT
89.4
10.6
2.6
99.7
14.0
3.2
96.3
21.6
3.3
AlphaEdit
98.4
10.5
3.4
99.7
14.0
3.0
96.5
21.7
3.0
GEdit
99.2
10.6
1.0
99.9
14.0
1.2
99.9
21.6
1.0
Appendix
Table 9: Performance comparison (in percentage) across 10,000 independent edits.
Method
CounterFact
ZsRE
Eff*.
Gen*.
Loc*.
Eff.
Gen.
Loc.
Eff.
Gen.
Loc.
Pre-edited
22.1
24.4
78.0
0.10
0.40
10.6
0.0
0.0
25.0
SERAC
GPT2-XL
65.2
48.8
66.4
61.8
19.1
10.1
72.7
38.2
26.7
GRACE
98.8
53.4
73.9
96.7
21.8
10.4
94.8
26.9
25.5
MELO
95.2
66.3
63.5
91.9
41.6
9.80
93.5
65.2
27.2
WISE
94.0
72.4
70.0
89.5
49.4
10.2
91.0
72.4
25.9
Appendix
Table 10: Performance comparison with parameter-preserving editing methods across three base models (GPT2-XL, GPT-J, and LLaMA3) on the CounterFact and ZsRE datasets under 2,000 sequential edits (batch size=1).
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University