Organizations: The Hong Kong Polytechnic University · Guangdong Institute of Intelligence Science and Technology · OpenMIA Research · National University of Singapore
Lifelong model editing requires continuously incorporating new facts into deployed large language models (LLMs) without expensive retraining or compromising existing capabilities. Memory-based editors offer an appealing solution by keeping the base parameters untouched, yet their practical use is hindered by linearly growing storage requirements and deteriorating retrieval performance as edits accumulate. To tackle these problems, we introduce BiSpace, a lifelong model editing framework that decouples memory retrieval from knowledge injection across two distinct representation spaces. BiSpace routes queries and identifies relevant edits in an external semantic space, while injecting the corresponding knowledge into the internal representation space of the LLM through sparse and storage-efficient updates. Extensive experiments show that BiSpace shrinks each entry's storage to only about 10.2% of the original and shows strong resilience to long edit sequences of up to 16K edits. On a more challenging benchmark, BiSpace further improves Overall Performance by more than 23% over existing methods. These results advance memory-based editing toward reliable long-term model maintenance at scale.
Figures & tables
Figure 1: Overview of BiSpace. Knowledge retrieval is decoupled from injection by constructing edit keys and queries in an external semantic space. Sparse codebook values reduce storage costs and are reconstructed on the fly for updates in the internal representation space.
Figure 2: Layer-wise analysis of knowledge retrieval and injection in HoReN after 1,000 edits on CounterFact across two models. The primary performance bottleneck lies in memory routing rather than knowledge injection.
Figure 3: Overall Performance over 10K sequential edits on ZsRE, WikiBigEdit, and CounterFact using LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct. BiSpace maintains consistently higher performance as edits accumulate, with larger advantages on the more challenging WikiBigEdit and CounterFact benchmarks.
Figure 4: General capability trajectories over 10K sequential edits on ZsRE using LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct. BiSpace preserves nearly unchanged general performance as edits accumulate.
Method
Edit Latency (s)
Infer. (tok/s)
Vanilla
-
74.40
HoReN
0.98
73.99
BiSpace
1.39
73.62
Table 1: Efficiency comparison under editing and long-form inference. Edit latency is measured on ZsRE after 5K sequential edits with LLaMA-3.1, while throughput is evaluated on UnKEBench with 200-token generation.
Method
Comp.
Data Type
Param.
Storage (KiB)
Total (KiB)
HoReN
Key
BF16
14336
28.00
44.00
Value
FP32
4096
16.00
BiSpace ( ρ =0.125)
Key
BF16
768
1.50
4.50
Value
FP32
512
2.00
Indices
INT16
512
1.00
Table 2: Per-entry storage breakdown of HoReN and BiSpace codebooks with LLaMA-3.1-8B-Instruct, highlighting BiSpace’s substantially lower memory cost.
Figure 5: Storage cost over up to 16K sequential edits on ZsRE using LLaMA-3.1. BiSpace substantially reduces the per-edit memory growth compared with HoReN, while the fixed overhead of the external encoder leads to a storage crossover at approximately 14.8K edits.
Act. Ratio ρ
Train Iter.
Strategy
Rel.
Gen.
Avg.
Edit Latency (s)
0.0625
50
Gradient
98.66 ± 0.00
97.62 ± 0.00
98.14 ± 0.00
1.40
Random
99.72 ± 0.02
98.27 ± 0.18
98.99 ± 0.10
1.46
0.0625
100
Gradient
99.95 ± 0.00
99.38 ± 0.00
99.67 ± 0.00
2.90
Random
99.99 ± 0.02
98.81 ± 0.23
99.40 ± 0.12
2.93
0.125
50
Gradient
99.53 ± 0.00
98.79 ± 0.00
99.16 ± 0.00
1.36
Random
100.00 ± 0.00
99.19 ± 0.06
99.59 ± 0.03
1.45
Table 3: Ablation of activation ratios and dimension selection strategies on WikiBigEdit using LLaMA-3.1-8B-Instruct after 1K edits. High injection effectiveness can be maintained under stronger sparsity, while simple random dimension selection remains competitive with gradient-based selection.
Encoder
Retrieval
Centering
Gen.
Loc.
Avg.
Jina-v5 -Nano
Hopfield
✗
88.60
81.73
85.17
✓
87.35
83.50
85.43
Cos. Sim.
✗
72.60
72.33
72.47
✓
73.60
73.94
73.77
Jina-v5 -Small
Hopfield
✗
87.50
88.04
87.77
✓
87.30
88.33
87.82
Table 4: Ablation of external encoder scale, retrieval strategy, and centering on CounterFact after 1K edits. Modern Hopfield retrieval yields the largest performance gains, while stronger encoders and centering provide further improvements.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Model
LR
Train Iter.
Structured Knowledge Editing
LLaMA-3.1-8B-Instruct
1.025
50
Qwen2.5-7B-Instruct
1.01
50
Mistral-7B-Instruct-v0.3
1.00
65
Unstructured Knowledge Editing
LLaMA-3.1-8B-Instruct
1.00
225
Appendix
Table 5: Knowledge injection hyperparameters for BiSpace across structured and unstructured knowledge editing settings. LR and Train Iter. denote the learning rate and number of training iterations, respectively.
Figure 6: Overall Performance on Mistral-7B-Instruct-v0.3 over up to 10K sequential edits across ZsRE, WikiBigEdit, and CounterFact. BiSpace consistently outperforms UltraEdit across different editing scales and shows a clear advantage over HoReN on the more challenging WikiBigEdit and CounterFact benchmarks.
Benchmark
Method
LLaMA-3.1-8B-Instruct
Qwen2.5-7B-Instruct
Mistral-7B-Instruct-v0.3
Rel.
Gen.
Loc.
O.P.
Rel.
Gen.
Loc.
O.P.
Rel.
Gen.
Loc.
O.P.
ZsRE
UltraEdit
82.01
79.64
6.64
56.10
65.91
62.03
32.99
53.64
58.19
50.67
0.45
36.44
HoReN
99.66
94.09
94.63
96.13
99.89
94.07
98.30
97.42
99.83
95.50
96.89
97.41
BiSpace
99.94
98.82
98.67
99.14
99.74
97.01
98.76
98.51
98.47
95.96
98.65
97.69
WikiBigEdit
UltraEdit
68.22
62.29
21.87
50.79
61.81
54.54
12.09
42.82
52.43
43.15
0.50
32.03
HoReN
98.77
73.76
52.92
75.15
99.91
74.95
48.45
74.44
99.85
64.57
45.58
70.00
Appendix
Table 6: Editing performance at an extended scale of 16K sequential edits on ZsRE and WikiBigEdit across three backbone models. Results demonstrate the scalability of BiSpace under dense lifelong editing, particularly on the more challenging WikiBigEdit benchmark.
Method
N=1000
N=2000
N=3000
ZsRE
HoReN
98.30 ± 0.05
97.94 ± 0.05
97.54 ± 0.09
BiSpace
99.43 ± 0.03
99.51 ± 0.06
99.47 ± 0.05
WikiBigEdit
HoReN
85.47 ± 0.32
83.33 ± 0.57
80.87 ± 1.03
BiSpace
97.82 ± 0.28
97.51 ± 0.50
96.66 ± 0.91
Appendix
Table 7: Robustness evaluation of HoReN and BiSpace using LLaMA-3.1-8B-Instruct across three benchmarks and multiple edit scales. Mean and standard deviation of Overall Performance are reported over three random seeds, showing consistently low variability for BiSpace.
Method
Semantic Similarity
Lexical Similarity
FC.
General Ability
Bert-Score
BLEU
Rouge-1
Rouge-2
Rouge-L
FactScore
MMLU
LLaMA-3.1-8B-Instruct
66.42
UnKE
1.73 / 2.29
0.37 / 0.43
0.30 / 0.32
0.00 / 0.00
0.29 / 0.31
0.00
22.3644.06↓
HoReN
92.72 / 77.16
56.22 / 43.50
99.29 / 64.73
99.04 / 53.63
99.26 / 63.48
21.95
66.300.12↓
BiSpace
93.02 / 86.53
58.98 / 48.85
100.00 / 75.00
99.99 / 64.69
100.00 / 73.75
40.47
66.420.00↓
BiSpace ρ=1.0
93.82 / 90.39
58.49 / 52.80
100.00 / 85.78
99.99 / 79.24
100.00 / 85.06
48.38
66.420.00↓
Appendix
Table 8: Unstructured knowledge editing performance after 1K sequential edits on the full UnKEBench across three backbone models. Values before/after ’/’ denote performance on exact edit prompts/paraphrases, respectively. FC. denotes Factual Correctness. BiSpace maintains strong exact-edit performance, while dense value updates further reveal the capacity limitations of sparse injection for long-form knowledge.
Figure 7: Seven representative cases for each model on ZsRE. BiSpace produces the target output, while all four baselines generate incorrect or incomplete responses.
Figure 8: Seven representative cases for each model on WikiBigEdit. BiSpace produces the target output, while all four baselines generate incorrect or incomplete responses.
Figure 9: Seven representative cases for each model on CounterFact. BiSpace produces the target output, while all four baselines generate incorrect or incomplete responses.
Large language models (LLMs) require frequent knowledge updates to reflect changing facts and mitigate hallucinations. To meet this demand, lifelong knowledge editing has emerged as a continual approach to modify specific pieces of knowledge without retraining the entire model. Existing parameter editing methods struggle with stability during sequential edits due to catastrophic forgetting. While retrieval-based approaches are proposed to alleviate this issue, their applicability remains limited across various datasets because of high training costs. To address these limitations and enhance scalability in lifelong settings, we propose LightEdit. Our framework first selects relevant knowledge from retrieved information to modify the query effectively. It then incorporates a decoding strategy to suppress the model's original knowledge probabilities, thereby enabling efficient edits based on the selected information. Extensive experiments on ZSRE, Counterfact, and RIPE benchmarks demonstrate that LightEdit outperforms existing lifelong knowledge editing methods. Furthermore, by minimizing training costs, LightEdit achieves cost-effective scalability, enabling easy adaptation to various datasets.
Dahyun Jung, Jaewook Lee, Heuiseok Lim
Department of Computer Science and Engineering, Korea University
Lifelong knowledge editing aims to efficiently and sequentially update language models over time, as new knowledge becomes available or when the model makes mistakes, while preserving acceptable performance on past knowledge. One unresolved challenge is that existing methods modify a fixed set of layers for all new knowledge samples, reducing flexibility and increasing catastrophic forgetting. Another is requiring access to previous knowledge and extensive pre-processing to obtain data statistics. To address these challenges, we introduce LOKI, a novel approach that uses dynamic layer selection based on the Hilbert-Schmidt Independence Criterion and projects gradient updates onto the null-space of the model weights, bypassing the requirement for previous knowledge access. We show that LOKI achieves superior performance to existing approaches across a wide variety of experiments, achieving up to a 14% improvement in average accuracy.
Masih Eskandar, Miquel Sirera Perelló, Stratis Ioannidis +1
Department of Electrical and Computer Engineering Northeastern University Boston, MA, 02115
Knowledge editing enables rapid updates of specific factual knowledge in large language models (LLMs) without full retraining. However, more realistic scenarios call for a lifelong framework that handles continual updates rather than one-off modifications. In such settings, existing editing methods often overfit to target prompts, significantly degrading both the generalization of the edited knowledge and the model's general capabilities. To address this issue, we propose GLIME (Generalizable Lifelong Model Editing), which combines knowledge editing with preference optimization over generation behavior. GLIME further incorporates replay-based editing and a gradient constraint to preserve previously edited knowledge. Experimental results show that GLIME significantly improves knowledge generalization in lifelong editing settings while maintaining both editing performance and general capabilities.
Dahyun Jung, Suhyune Son, Heuiseok Lim
Department of Computer Science and Engineering, Korea University