Organizations: The Hong Kong Polytechnic University · Guangdong Institute of Intelligence Science and Technology · OpenMIA Research · National University of Singapore
Lifelong model editing requires continuously incorporating new facts into deployed large language models (LLMs) without expensive retraining or compromising existing capabilities. Memory-based editors offer an appealing solution by keeping the base parameters untouched, yet their practical use is hindered by linearly growing storage requirements and deteriorating retrieval performance as edits accumulate. To tackle these problems, we introduce BiSpace, a lifelong model editing framework that decouples memory retrieval from knowledge injection across two distinct representation spaces. BiSpace routes queries and identifies relevant edits in an external semantic space, while injecting the corresponding knowledge into the internal representation space of the LLM through sparse and storage-efficient updates. Extensive experiments show that BiSpace shrinks each entry's storage to only about 10.2% of the original and shows strong resilience to long edit sequences of up to 16K edits. On a more challenging benchmark, BiSpace further improves Overall Performance by more than 23% over existing methods. These results advance memory-based editing toward reliable long-term model maintenance at scale.
Figures & tables
Figure 1: Overview of BiSpace. Knowledge retrieval is decoupled from injection by constructing edit keys and queries in an external semantic space. Sparse codebook values reduce storage costs and are reconstructed on the fly for updates in the internal representation space.
Figure 2: Layer-wise analysis of knowledge retrieval and injection in HoReN after 1,000 edits on CounterFact across two models. The primary performance bottleneck lies in memory routing rather than knowledge injection.
Figure 3: Overall Performance over 10K sequential edits on ZsRE, WikiBigEdit, and CounterFact using LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct. BiSpace maintains consistently higher performance as edits accumulate, with larger advantages on the more challenging WikiBigEdit and CounterFact benchmarks.
Figure 4: General capability trajectories over 10K sequential edits on ZsRE using LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct. BiSpace preserves nearly unchanged general performance as edits accumulate.
Method
Edit Latency (s)
Infer. (tok/s)
Vanilla
-
74.40
HoReN
0.98
73.99
BiSpace
1.39
73.62
Table 1: Efficiency comparison under editing and long-form inference. Edit latency is measured on ZsRE after 5K sequential edits with LLaMA-3.1, while throughput is evaluated on UnKEBench with 200-token generation.
Method
Comp.
Data Type
Param.
Storage (KiB)
Total (KiB)
HoReN
Key
BF16
14336
28.00
44.00
Value
FP32
4096
16.00
BiSpace ( ρ =0.125)
Key
BF16
768
1.50
4.50
Value
FP32
512
2.00
Indices
INT16
512
1.00
Table 2: Per-entry storage breakdown of HoReN and BiSpace codebooks with LLaMA-3.1-8B-Instruct, highlighting BiSpace’s substantially lower memory cost.
Figure 5: Storage cost over up to 16K sequential edits on ZsRE using LLaMA-3.1. BiSpace substantially reduces the per-edit memory growth compared with HoReN, while the fixed overhead of the external encoder leads to a storage crossover at approximately 14.8K edits.
Act. Ratio ρ
Train Iter.
Strategy
Rel.
Gen.
Avg.
Edit Latency (s)
0.0625
50
Gradient
98.66 ± 0.00
97.62 ± 0.00
98.14 ± 0.00
1.40
Random
99.72 ± 0.02
98.27 ± 0.18
98.99 ± 0.10
1.46
0.0625
100
Gradient
99.95 ± 0.00
99.38 ± 0.00
99.67 ± 0.00
2.90
Random
99.99 ± 0.02
98.81 ± 0.23
99.40 ± 0.12
2.93
0.125
50
Gradient
99.53 ± 0.00
98.79 ± 0.00
99.16 ± 0.00
1.36
Random
100.00 ± 0.00
99.19 ± 0.06
99.59 ± 0.03
1.45
Table 3: Ablation of activation ratios and dimension selection strategies on WikiBigEdit using LLaMA-3.1-8B-Instruct after 1K edits. High injection effectiveness can be maintained under stronger sparsity, while simple random dimension selection remains competitive with gradient-based selection.
Encoder
Retrieval
Centering
Gen.
Loc.
Avg.
Jina-v5 -Nano
Hopfield
✗
88.60
81.73
85.17
✓
87.35
83.50
85.43
Cos. Sim.
✗
72.60
72.33
72.47
✓
73.60
73.94
73.77
Jina-v5 -Small
Hopfield
✗
87.50
88.04
87.77
✓
87.30
88.33
87.82
Table 4: Ablation of external encoder scale, retrieval strategy, and centering on CounterFact after 1K edits. Modern Hopfield retrieval yields the largest performance gains, while stronger encoders and centering provide further improvements.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Model
LR
Train Iter.
Structured Knowledge Editing
LLaMA-3.1-8B-Instruct
1.025
50
Qwen2.5-7B-Instruct
1.01
50
Mistral-7B-Instruct-v0.3
1.00
65
Unstructured Knowledge Editing
LLaMA-3.1-8B-Instruct
1.00
225
Appendix
Table 5: Knowledge injection hyperparameters for BiSpace across structured and unstructured knowledge editing settings. LR and Train Iter. denote the learning rate and number of training iterations, respectively.
Figure 6: Overall Performance on Mistral-7B-Instruct-v0.3 over up to 10K sequential edits across ZsRE, WikiBigEdit, and CounterFact. BiSpace consistently outperforms UltraEdit across different editing scales and shows a clear advantage over HoReN on the more challenging WikiBigEdit and CounterFact benchmarks.
Benchmark
Method
LLaMA-3.1-8B-Instruct
Qwen2.5-7B-Instruct
Mistral-7B-Instruct-v0.3
Rel.
Gen.
Loc.
O.P.
Rel.
Gen.
Loc.
O.P.
Rel.
Gen.
Loc.
O.P.
ZsRE
UltraEdit
82.01
79.64
6.64
56.10
65.91
62.03
32.99
53.64
58.19
50.67
0.45
36.44
HoReN
99.66
94.09
94.63
96.13
99.89
94.07
98.30
97.42
99.83
95.50
96.89
97.41
BiSpace
99.94
98.82
98.67
99.14
99.74
97.01
98.76
98.51
98.47
95.96
98.65
97.69
WikiBigEdit
UltraEdit
68.22
62.29
21.87
50.79
61.81
54.54
12.09
42.82
52.43
43.15
0.50
32.03
HoReN
98.77
73.76
52.92
75.15
99.91
74.95
48.45
74.44
99.85
64.57
45.58
70.00
Appendix
Table 6: Editing performance at an extended scale of 16K sequential edits on ZsRE and WikiBigEdit across three backbone models. Results demonstrate the scalability of BiSpace under dense lifelong editing, particularly on the more challenging WikiBigEdit benchmark.
Method
N=1000
N=2000
N=3000
ZsRE
HoReN
98.30 ± 0.05
97.94 ± 0.05
97.54 ± 0.09
BiSpace
99.43 ± 0.03
99.51 ± 0.06
99.47 ± 0.05
WikiBigEdit
HoReN
85.47 ± 0.32
83.33 ± 0.57
80.87 ± 1.03
BiSpace
97.82 ± 0.28
97.51 ± 0.50
96.66 ± 0.91
Appendix
Table 7: Robustness evaluation of HoReN and BiSpace using LLaMA-3.1-8B-Instruct across three benchmarks and multiple edit scales. Mean and standard deviation of Overall Performance are reported over three random seeds, showing consistently low variability for BiSpace.
Method
Semantic Similarity
Lexical Similarity
FC.
General Ability
Bert-Score
BLEU
Rouge-1
Rouge-2
Rouge-L
FactScore
MMLU
LLaMA-3.1-8B-Instruct
66.42
UnKE
1.73 / 2.29
0.37 / 0.43
0.30 / 0.32
0.00 / 0.00
0.29 / 0.31
0.00
22.3644.06↓
HoReN
92.72 / 77.16
56.22 / 43.50
99.29 / 64.73
99.04 / 53.63
99.26 / 63.48
21.95
66.300.12↓
BiSpace
93.02 / 86.53
58.98 / 48.85
100.00 / 75.00
99.99 / 64.69
100.00 / 73.75
40.47
66.420.00↓
BiSpace ρ=1.0
93.82 / 90.39
58.49 / 52.80
100.00 / 85.78
99.99 / 79.24
100.00 / 85.06
48.38
66.420.00↓
Appendix
Table 8: Unstructured knowledge editing performance after 1K sequential edits on the full UnKEBench across three backbone models. Values before/after ’/’ denote performance on exact edit prompts/paraphrases, respectively. FC. denotes Factual Correctness. BiSpace maintains strong exact-edit performance, while dense value updates further reveal the capacity limitations of sparse injection for long-form knowledge.
Figure 7: Seven representative cases for each model on ZsRE. BiSpace produces the target output, while all four baselines generate incorrect or incomplete responses.
Figure 8: Seven representative cases for each model on WikiBigEdit. BiSpace produces the target output, while all four baselines generate incorrect or incomplete responses.
Figure 9: Seven representative cases for each model on CounterFact. BiSpace produces the target output, while all four baselines generate incorrect or incomplete responses.