Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · Alibaba Group · Beihang University · School of Artificial Intelligence, Wuhan University · School of Computer Science, Wuhan University
Large language models (LLMs) require frequent knowledge updates across heterogeneous domains, but existing editing methods fail when applied sequentially to multi-task settings. We trace this collapse to inter-domain null-space misalignment: heterogeneous tasks induce distinct Jacobian geometries, causing their common null space to shrink dramatically and leaving insufficient admissible directions for conflict-free updates. We propose the Conflict Index to quantify this geometric interference, and introduce Mu-Edit, which mitigates multi-task conflicts via (1) conflict-aware ordering to minimize cumulative interference, and (2) dynamic low-rank approximation to expand the null space when ordering alone is insufficient. Experiments across five functional domains and three backbones show that Mu-Edit significantly outperforms existing baselines in multi-task editing performance while preserving general capabilities, and remains effective under incremental task arrival.
Figures & tables
Figure 1: Mechanism and motivation of our Mu-Edit, specifically: (a) Visualization of Mu-Edit’s null space projection mechanism. (b) Empirical validation results of existing methods’ performance decline after editing sequentially across different tasks.
ZsRE
CKnowEdit
SafeEdit
SQuAD
SST-2
Method
Spec ↑
Gen ↑
Loc ↑
Spec ↑
Gen ↑
Loc ↑
Harm ↑
Acc ↑
Acc ↑
AlphaEdit(DO)
0.3009
0.2351
0.3608
0.2873
0.2512
0.3303
0.2778
0.6984
0.8620
AnyEdit(DO)
0.3954
0.3061
0.4021
0.3741
0.3202
0.3495
0.3004
0.7028
0.8643
Mu-Edit(DO)
0.8197
0.7995
0.5684
0.7925
0.7606
0.5642
0.3648
0.7252
0.8694
FT(BO)
0.2861
0.2356
0.2575
0.2559
0.2216
0.2083
0.2210
0.4446
0.6290
ROME(BO)
0.3452
0.3036
0.2418
0.3021
0.2746
0.2327
0.2449
0.4808
0.7073
Table 1: Final performance after multi-task editing on Llama3-8B under Best Order(BO) and Default Order(DO) and ablations. Bold: best. *: significant improvement over AnyEdit ( p<0.05 ).
ZsRE
CKnowEdit
SafeEdit
SQuAD
SST-2
Method
Spec ↑
Gen ↑
Loc ↑
Spec ↑
Gen ↑
Loc ↑
Harm ↑
Acc ↑
Acc ↑
Mu-Edit - (GO)
0.8294
0.7875
0.6168
0.8003
0.7632
0.5913
0.3616
0.6818
0.8752
Mu-Edit(left)(GO)
0.8084
0.7679
0.6112
0.7619
0.5521
0.5619
0.3559
0.6802
0.8695
Mu-Edit(right)(GO)
0.8143
0.7905
0.6146
0.8215
0.7847
0.5896
0.3642
0.6847
0.8774
Mu-Edit(GO)
0.8728 *
0.8432 *
0.6314 *
0.8442 *
0.8130 *
0.6098 *
0.3998 *
0.7442 *
0.8822 *
Table 2: Final performance under Greedy Order (GO) and ablations.
Figure 4
Figure 4: Conflict index visualization under extremely low resource settings
Reference Dataset Size
Size
ZsRE-Specificity
ZsRE-Generalization
ZsRE-Locality
SafeEdit
SQuAD
SST-2
CKnowEdit
50
0.7723
0.7484
0.5385
0.3822
0.7105
0.8293
0.8212
100
0.8176
0.7739
0.5772
0.3897
0.7201
0.8587
0.8317
500
0.8417
0.8083
0.6019
0.3956
0.7348
0.8721
0.8402
10000
0.8845
0.8452
0.6321
0.4043
0.7479
0.8838
0.8505
Table 3: Performance under different reference dataset sizes.
Tasks
Strategy
CI computation
Task ordering
Preprocessing memory
Inference memory
5
BO
168 min
11.4 s
210.6 GB
22.1 GB
5
GO
168 min
5.8 s
216.7 GB
22.5 GB
10
BO
323 min
5502.7 min
1364.2 GB
28.1 GB
10
GO
323 min
22.9 s
224.8 GB
29.5 GB
Table 6: Computational cost breakdown for BO and GO. CI computation and task ordering are timed separately. Memory usage is reported for preprocessing and inference.
Figure 5: Comparing results our method with other multi- task learning methods
Dataset
∣Dtrain∣
∣Dtest∣
Type
ZsRE
28670
9250
English commonsense knowledge
Counterfact
208264
68930
English commonsense knowledge
SafeEdit
4895
1208
Safety knowledge
SQUAD
18704
4352
Reasoning
GSM8k
114095
23610
Math Reasoning
SST-2
8996
1982
Sentiment analysis
Table 7: Statistical information about the training datasets used in the experiments.
Task
Reference source
ZsRE
WikiText-en
CKnowEdit
Chinese Wikipedia
SafeEdit
BeaverTails
SQuAD
WikiText-en
SST-2
IMDb Large Movie Review Dataset
Personality
PersonalityChat
Table 8: External reference sources used to construct task-specific preservation matrices.
ZsRE
CKnowEdit
SafeEdit
SQuAD
SST-2
Method
Reference
Spec
Gen
Loc
Spec
Gen
Loc
Harm
Acc
Acc
AlphaEdit
WikiText
0.4732
0.4274
0.3656
0.6107
0.5853
0.4667
0.2698
0.6205
0.8031
Mu-Edit
WikiText
0.6421
0.6068
0.4996
0.7303
0.7002
0.5032
0.3375
0.6883
0.8442
AlphaEdit
Task-specific
0.5506
0.5101
0.4470
0.6624
0.6313
0.4456
0.3396
0.7016
0.8803
Mu-Edit
Task-specific
0.8845
0.8452
0.6321
0.8505
0.8258
0.6144
0.4043
0.7479
0.8838
Table 9: Effect of reference source on AlphaEdit and Mu-Edit. ZsRE and CKnowEdit report Specificity, Generalization, and Locality; SafeEdit reports Harmful Rate; SQuAD and SST-2 report Accuracy.
Method
Dataset Size
ZsRE
SafeEdit
SQuAD
SST-2
Specificity
Generality
Locality
Harmful rate
Acc
Acc
AlphaEdit
500
0.9891
0.9352
0.6606
0.4356
0.7883
0.9167
1000
0.9787
0.9285
0.6562
0.4310
0.7796
0.9121
2000
0.9665
0.9201
0.6494
0.4217
0.7702
0.9076
3000
0.9505
0.9112
0.6403
0.4125
0.7624
0.9011
AnyEdit
500
0.9899
0.9382
0.6650
0.4601
0.7892
0.9192
Table 10: Editing performance under increasing samples used in each task. x means that we cannot measure due to the memory or time cost is unaffordable.
Counterfact
CKnowEdit
SafeEdit
GSM8k
SST-2
Method
Spec ↑
Gen ↑
Loc ↑
Spec ↑
Gen ↑
Loc ↑
Harm ↑
Acc ↑
Acc ↑
AlphaEdit(DO)
0.3354
0.3034
0.3777
0.2866
0.2507
0.3309
0.2457
0.3959
0.8365
Mu-Edit(DO)
0.7795
0.7664
0.5488
0.3725
0.3208
0.3484
0.3421
0.4498
0.8442
ROME(BO)
0.2841
0.2356
0.1994
0.3013
0.2737
0.2314
0.2015
0.2283
0.5804
MEMIT(BO)
0.3667
0.3424
0.2816
0.2531
0.2223
0.2076
0.2221
0.2499
0.6755
AlphaEdit(BO)
0.5723
0.5218
0.4695
0.6635
0.6304
0.4447
0.3577
0.5225
0.8463
Table 11: Final performance after multi-task editing on Llama3-8B under DO, BO and GO. Counterfact replaces ZsRE; GSM8k replaces SQuAD. Bold: best.
ZsRE
SafeEdit
SQuAD
SST-2
Method
Spec ↑
Gen ↑
Loc ↑
Harm ↑
Acc ↑
Acc ↑
AlphaEdit(DO)
0.2859
0.2160
0.3368
0.2463
0.6702
0.8388
Mu-Edit(DO)
0.7375
0.7504
0.5195
0.3416
0.7071
0.8468
ROME(BO)
0.2544
0.2120
0.1863
0.2024
0.4091
0.6004
MEMIT(BO)
0.3143
0.2860
0.2231
0.2214
0.4461
0.6746
AlphaEdit(BO)
0.5374
0.4886
0.4217
0.3046
0.6775
0.8446
Table 12: Final performance after multi-task editing on GPT2-xl under Default Order (DO) and Best Order (BO). Missing entries are denoted by x. Bold: best.
ZsRE
SafeEdit
SQuAD
SST-2
Method
Spec ↑
Gen ↑
Loc ↑
Harm ↑
Acc ↑
Acc ↑
AlphaEdit(DO)
0.3031
0.2366
0.3615
0.2801
0.7119
0.8702
AnyEdit(DO)
0.4127
0.3298
0.4219
0.3022
0.7264
0.8725
Mu-Edit(DO)
0.8199
0.8006
0.5703
0.3668
0.7539
0.8832
AlphaEdit(BO)
0.5776
0.5145
0.4482
0.3676
0.7203
0.8822
AnyEdit(BO)
0.7142
0.6807
0.5369
0.3878
0.7444
0.8814
Table 13: Final performance after multi-task editing on Qwen2.5-7B under Default Order (DO) and Best Order (BO). Missing entries are denoted by x. Bold: best.
Figure 6: Case Study
Adjacent tasks number
ZsRE-Spec
ZsRE-Gen
ZsRE-Loc
SafeEdit
SQUAD
SST-2
CKnowEdit
Calculating Time
GPU Cost
3
0.8845
0.8452
0.6321
0.4043
0.7479
0.8838
0.8505
168min
210.6 GB
4
0.8903
0.8443
0.6398
0.3996
0.7264
0.8826
0.8513
305min
369.4 GB
5
0.8714
0.8401
0.6452
0.3987
0.7126
0.8830
0.8466
660min
610.3 GB
Table 14: Performance under the number of adjacent tasks during square norms minimization
t parameter influence
t value
ZsRE-Specificity
ZsRE-Generality
ZsRE-Locality
SafeEdit
SQUAD
SST-2
CKnowEdit
0.3
0.5911
0.5385
0.6433
0.3017
0.6852
0.7883
0.6981
0.6
0.8447
0.7926
0.6391
0.3559
0.7226
0.8573
0.8064
0.9
0.8845
0.8452
0.6321
0.4043
0.7479
0.8838
0.8505
1.2
0.8221
0.7806
0.6182
0.3745
0.7158
0.8634
0.8297
1.5
0.7234
0.6695
0.5996
0.3286
0.6660
0.8157
0.7668
Table 15: Performance under different choice of t and α
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University
Renmin University of China · Laboratory for Artificial Intelligence and New Forms of Education, Central China Normal University · School of Computing, National University of Singapore +1