Organizations: Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland · National University of Singapore, Singapore · School of Computer Science and Technology, Dalian University of Technology, Dalian, China
The dynamic evolution of real-world necessitates model editing within Large Language Models. While existing methods explore modular isolation or parameter-efficient strategies, they still suffer from semantic drift or knowledge forgetting due to continual updating. To address these challenges, we propose SoLA, a Semantic routing-based LoRA framework for lifelong model editing. In SoLA, each edit is encapsulated as an independent LoRA module, which is frozen after training and mapped to input by semantic routing, allowing dynamic activation of LoRA modules via semantic matching. This mechanism avoids semantic drift caused by cluster updating and mitigates catastrophic forgetting from parameter sharing. More importantly, SoLA supports precise revocation of specific edits by removing key from semantic routing, which restores model's original behavior. To our knowledge, this reversible rollback editing capability is the first to be achieved in existing literature. Furthermore, SoLA integrates decision-making process into edited layer, eliminating the need for auxiliary routing networks and enabling end-to-end decision-making process. Extensive experiments demonstrate that SoLA effectively learns and retains edited knowledge, achieving accurate, efficient, and reversible lifelong model editing.
Figures & tables
Figure 1: Comparison of trainable parameters and ERR accuracy between SoLA (Ours) and other methods. All experiments are conducted on Scotus datasets with the same backbone.
Figure 2: Number of mismatches in LoRA allocation with ERR/TRR accuracy in MELO. All experiments are conducted on Scotus datasets with the same backbone.
Figure 3: Main framework of our method. a) indicates the edited layer in model, where we retain the transformer block of base model frozen and add trainable LoRA module to the edited layer of base model. b) shows the editing process, where every edit will be assigned LoRA module and the query of input will be mapped to assigned LoRA module. c) presents the matching process between query of input and LoRA module in reference. Colour green indicates trainable, while color grey is frozen.
SCOTUS (BERT; Acc ↑)
zsRE (T5; F1 ↑)
Hallucination (GPT2-XL; PPL ↓)
Method
ERR
TRR
Avg.
ERR
TRR
Avg.
ERR
TRR
ARR
EWC
0.51
0.82
0.66
0.67
0.50
0.58
1485.7
29.24
109.59
CMR
0.52
0.52
0.52
0.56
0.82
0.69
1449.3
28.14
107.76
CLEAR
0.67
0.83
0.75
0.27
0.99
0.63
2394.3
35.34
195.82
MEND
0.19
0.27
0.23
0.25
0.27
0.26
1369.8
1754.9
2902.5
SERAC
0.33
0.41
0.37
0.72
0.31
0.52
8183.7
133.3
10.04
Table 1: Comparison of SoLA to existing methods. All the results are obtained after sequential edits. ERR indicates Edit Reliability Rate, TRR presents Task Retention Rate, and ARR is Accuracy Attention Rate. The best performance is shown in bold.
Figure 4: Figure(a)–(c) presents every edit accuracy on SCOTUS datasets with BERT. All experiment settings are the same as Tab. 1 .
Figure 5: Visualization of zsRE dataset encoder output from T5-small with t-SNE and the dots with same color and shape are input and rephrase sentence. All experiment settings are the same as Tab. 1 .
UniEdit (PPL ↓)
WikiBigEdit (PPL ↓)
Model
Method
ERR
TRR
ARR
ERR
TRR
ARR
LLaMA-3-8B
EWC
2.43
1134
22.08
3.58
251117
4.93
CMR
2.79
2239
25.24
2.86
169452
4.49
CLEAR
1.04
982
29.79
1.03
74384
1.52
SERAC
37.10
246
93
60.17
215
59
GRACE
1.00
244
89
1.00
216
58
Table 2: Model Performance Comparison on UniEdit and WikiBigEdit datasets in hallucination correction tasks. All the scores are PPL metric and the best result is in bolded.
Text
Labels
Del
Pred base
Pred edit
Pred del
The date of Tsetserleg earthquake in 1905?
9 July 1905
True
20 February 1905
9 July 1905
20 February 1905
What constellation has Nu Cancri?
Cancer
True
Hydra
Cancer
Hydra
On what date did the Bahrain Grand Prix 2015 occur?
19 April 2015
True
6 April 2017
19 April 2015
6 April 2017
What position does Charles-Joseph Coursol have?
Mayor of Montreal
True
Positioning
Mayor of Montreal
Positioning
Who acted in Blow Dry?
Alan Rickman
False
Jim Carrey
Alan Rickman
Alan Rickman
Table 3: Controllable edit in SoLA on zsRE datasets. ”Text” is the question need to be edited, ”Labels” is the true answer, ”Del” indicates whther delete associated key and Pred base , Pred edit and Pred del is the prediction of base model, edited model and delted model.
Layer
ES
ERR
TRR
Edit Time(min)
0-2
0.77
0.61
0.96
9.21
3-5
1.00
0.68
0.96
8.06
6-8
0.81
0.70
0.97
7.03
9-11
0.94
0.95
0.97
5.97
Table 4: Ablation studies on location of edited layers. Here, ”Layer” indicates the edited layer location. For example, ”0-2” presents edited layer is layers 0, 1 and 2.
LoRA Rank
ES
ERR
TRR
Edit Time(min)
1
0.90
0.84
0.96
5.88
2
0.94
0.91
0.97
5.90
3
1.00
0.91
0.97
5.86
4
1.00
0.95
0.97
5.97
5
1.00
0.92
0.96
5.86
10
1.00
0.71
0.96
5.85
Table 5: Ablation studies on LoRA rank. Here, ”LoRA Rank” indicates the rank value of every LoRA in model.