Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · Alibaba Group · Beihang University · School of Artificial Intelligence, Wuhan University · School of Computer Science, Wuhan University
Large language models (LLMs) require frequent knowledge updates across heterogeneous domains, but existing editing methods fail when applied sequentially to multi-task settings. We trace this collapse to inter-domain null-space misalignment: heterogeneous tasks induce distinct Jacobian geometries, causing their common null space to shrink dramatically and leaving insufficient admissible directions for conflict-free updates. We propose the Conflict Index to quantify this geometric interference, and introduce Mu-Edit, which mitigates multi-task conflicts via (1) conflict-aware ordering to minimize cumulative interference, and (2) dynamic low-rank approximation to expand the null space when ordering alone is insufficient. Experiments across five functional domains and three backbones show that Mu-Edit significantly outperforms existing baselines in multi-task editing performance while preserving general capabilities, and remains effective under incremental task arrival.
Figures & tables
Figure 1: Mechanism and motivation of our Mu-Edit, specifically: (a) Visualization of Mu-Edit’s null space projection mechanism. (b) Empirical validation results of existing methods’ performance decline after editing sequentially across different tasks.
ZsRE
CKnowEdit
SafeEdit
SQuAD
SST-2
Method
Spec ↑
Gen ↑
Loc ↑
Spec ↑
Gen ↑
Loc ↑
Harm ↑
Acc ↑
Acc ↑
AlphaEdit(DO)
0.3009
0.2351
0.3608
0.2873
0.2512
0.3303
0.2778
0.6984
0.8620
AnyEdit(DO)
0.3954
0.3061
0.4021
0.3741
0.3202
0.3495
0.3004
0.7028
0.8643
Mu-Edit(DO)
0.8197
0.7995
0.5684
0.7925
0.7606
0.5642
0.3648
0.7252
0.8694
FT(BO)
0.2861
0.2356
0.2575
0.2559
0.2216
0.2083
0.2210
0.4446
0.6290
ROME(BO)
0.3452
0.3036
0.2418
0.3021
0.2746
0.2327
0.2449
0.4808
0.7073
Table 1: Final performance after multi-task editing on Llama3-8B under Best Order(BO) and Default Order(DO) and ablations. Bold: best. *: significant improvement over AnyEdit ( p<0.05 ).
ZsRE
CKnowEdit
SafeEdit
SQuAD
SST-2
Method
Spec ↑
Gen ↑
Loc ↑
Spec ↑
Gen ↑
Loc ↑
Harm ↑
Acc ↑
Acc ↑
Mu-Edit - (GO)
0.8294
0.7875
0.6168
0.8003
0.7632
0.5913
0.3616
0.6818
0.8752
Mu-Edit(left)(GO)
0.8084
0.7679
0.6112
0.7619
0.5521
0.5619
0.3559
0.6802
0.8695
Mu-Edit(right)(GO)
0.8143
0.7905
0.6146
0.8215
0.7847
0.5896
0.3642
0.6847
0.8774
Mu-Edit(GO)
0.8728 *
0.8432 *
0.6314 *
0.8442 *
0.8130 *
0.6098 *
0.3998 *
0.7442 *
0.8822 *
Table 2: Final performance under Greedy Order (GO) and ablations.
Figure 4
Figure 4: Conflict index visualization under extremely low resource settings
Reference Dataset Size
Size
ZsRE-Specificity
ZsRE-Generalization
ZsRE-Locality
SafeEdit
SQuAD
SST-2
CKnowEdit
50
0.7723
0.7484
0.5385
0.3822
0.7105
0.8293
0.8212
100
0.8176
0.7739
0.5772
0.3897
0.7201
0.8587
0.8317
500
0.8417
0.8083
0.6019
0.3956
0.7348
0.8721
0.8402
10000
0.8845
0.8452
0.6321
0.4043
0.7479
0.8838
0.8505
Table 3: Performance under different reference dataset sizes.
Tasks
Strategy
CI computation
Task ordering
Preprocessing memory
Inference memory
5
BO
168 min
11.4 s
210.6 GB
22.1 GB
5
GO
168 min
5.8 s
216.7 GB
22.5 GB
10
BO
323 min
5502.7 min
1364.2 GB
28.1 GB
10
GO
323 min
22.9 s
224.8 GB
29.5 GB
Table 6: Computational cost breakdown for BO and GO. CI computation and task ordering are timed separately. Memory usage is reported for preprocessing and inference.
Figure 5: Comparing results our method with other multi- task learning methods
Dataset
∣Dtrain∣
∣Dtest∣
Type
ZsRE
28670
9250
English commonsense knowledge
Counterfact
208264
68930
English commonsense knowledge
SafeEdit
4895
1208
Safety knowledge
SQUAD
18704
4352
Reasoning
GSM8k
114095
23610
Math Reasoning
SST-2
8996
1982
Sentiment analysis
Table 7: Statistical information about the training datasets used in the experiments.
Task
Reference source
ZsRE
WikiText-en
CKnowEdit
Chinese Wikipedia
SafeEdit
BeaverTails
SQuAD
WikiText-en
SST-2
IMDb Large Movie Review Dataset
Personality
PersonalityChat
Table 8: External reference sources used to construct task-specific preservation matrices.
ZsRE
CKnowEdit
SafeEdit
SQuAD
SST-2
Method
Reference
Spec
Gen
Loc
Spec
Gen
Loc
Harm
Acc
Acc
AlphaEdit
WikiText
0.4732
0.4274
0.3656
0.6107
0.5853
0.4667
0.2698
0.6205
0.8031
Mu-Edit
WikiText
0.6421
0.6068
0.4996
0.7303
0.7002
0.5032
0.3375
0.6883
0.8442
AlphaEdit
Task-specific
0.5506
0.5101
0.4470
0.6624
0.6313
0.4456
0.3396
0.7016
0.8803
Mu-Edit
Task-specific
0.8845
0.8452
0.6321
0.8505
0.8258
0.6144
0.4043
0.7479
0.8838
Table 9: Effect of reference source on AlphaEdit and Mu-Edit. ZsRE and CKnowEdit report Specificity, Generalization, and Locality; SafeEdit reports Harmful Rate; SQuAD and SST-2 report Accuracy.
Method
Dataset Size
ZsRE
SafeEdit
SQuAD
SST-2
Specificity
Generality
Locality
Harmful rate
Acc
Acc
AlphaEdit
500
0.9891
0.9352
0.6606
0.4356
0.7883
0.9167
1000
0.9787
0.9285
0.6562
0.4310
0.7796
0.9121
2000
0.9665
0.9201
0.6494
0.4217
0.7702
0.9076
3000
0.9505
0.9112
0.6403
0.4125
0.7624
0.9011
AnyEdit
500
0.9899
0.9382
0.6650
0.4601
0.7892
0.9192
Table 10: Editing performance under increasing samples used in each task. x means that we cannot measure due to the memory or time cost is unaffordable.
Counterfact
CKnowEdit
SafeEdit
GSM8k
SST-2
Method
Spec ↑
Gen ↑
Loc ↑
Spec ↑
Gen ↑
Loc ↑
Harm ↑
Acc ↑
Acc ↑
AlphaEdit(DO)
0.3354
0.3034
0.3777
0.2866
0.2507
0.3309
0.2457
0.3959
0.8365
Mu-Edit(DO)
0.7795
0.7664
0.5488
0.3725
0.3208
0.3484
0.3421
0.4498
0.8442
ROME(BO)
0.2841
0.2356
0.1994
0.3013
0.2737
0.2314
0.2015
0.2283
0.5804
MEMIT(BO)
0.3667
0.3424
0.2816
0.2531
0.2223
0.2076
0.2221
0.2499
0.6755
AlphaEdit(BO)
0.5723
0.5218
0.4695
0.6635
0.6304
0.4447
0.3577
0.5225
0.8463
Table 11: Final performance after multi-task editing on Llama3-8B under DO, BO and GO. Counterfact replaces ZsRE; GSM8k replaces SQuAD. Bold: best.
ZsRE
SafeEdit
SQuAD
SST-2
Method
Spec ↑
Gen ↑
Loc ↑
Harm ↑
Acc ↑
Acc ↑
AlphaEdit(DO)
0.2859
0.2160
0.3368
0.2463
0.6702
0.8388
Mu-Edit(DO)
0.7375
0.7504
0.5195
0.3416
0.7071
0.8468
ROME(BO)
0.2544
0.2120
0.1863
0.2024
0.4091
0.6004
MEMIT(BO)
0.3143
0.2860
0.2231
0.2214
0.4461
0.6746
AlphaEdit(BO)
0.5374
0.4886
0.4217
0.3046
0.6775
0.8446
Table 12: Final performance after multi-task editing on GPT2-xl under Default Order (DO) and Best Order (BO). Missing entries are denoted by x. Bold: best.
ZsRE
SafeEdit
SQuAD
SST-2
Method
Spec ↑
Gen ↑
Loc ↑
Harm ↑
Acc ↑
Acc ↑
AlphaEdit(DO)
0.3031
0.2366
0.3615
0.2801
0.7119
0.8702
AnyEdit(DO)
0.4127
0.3298
0.4219
0.3022
0.7264
0.8725
Mu-Edit(DO)
0.8199
0.8006
0.5703
0.3668
0.7539
0.8832
AlphaEdit(BO)
0.5776
0.5145
0.4482
0.3676
0.7203
0.8822
AnyEdit(BO)
0.7142
0.6807
0.5369
0.3878
0.7444
0.8814
Table 13: Final performance after multi-task editing on Qwen2.5-7B under Default Order (DO) and Best Order (BO). Missing entries are denoted by x. Bold: best.
Figure 6: Case Study
Adjacent tasks number
ZsRE-Spec
ZsRE-Gen
ZsRE-Loc
SafeEdit
SQUAD
SST-2
CKnowEdit
Calculating Time
GPU Cost
3
0.8845
0.8452
0.6321
0.4043
0.7479
0.8838
0.8505
168min
210.6 GB
4
0.8903
0.8443
0.6398
0.3996
0.7264
0.8826
0.8513
305min
369.4 GB
5
0.8714
0.8401
0.6452
0.3987
0.7126
0.8830
0.8466
660min
610.3 GB
Table 14: Performance under the number of adjacent tasks during square norms minimization
t parameter influence
t value
ZsRE-Specificity
ZsRE-Generality
ZsRE-Locality
SafeEdit
SQUAD
SST-2
CKnowEdit
0.3
0.5911
0.5385
0.6433
0.3017
0.6852
0.7883
0.6981
0.6
0.8447
0.7926
0.6391
0.3559
0.7226
0.8573
0.8064
0.9
0.8845
0.8452
0.6321
0.4043
0.7479
0.8838
0.8505
1.2
0.8221
0.7806
0.6182
0.3745
0.7158
0.8634
0.8297
1.5
0.7234
0.6695
0.5996
0.3286
0.6660
0.8157
0.7668
Table 15: Performance under different choice of t and α
Large language models (LLMs) inevitably generate some incorrect or outdated content, necessitating efficient and precise mechanisms for continual knowledge updates. However, existing model editing methods struggle to sequentially edit unstructured long-form knowledge, suffering from severe edit forgetting and degradation of general capabilities. To address these challenges, we reframe knowledge editing from a manifold perspective, viewing it as a localized displacement of an edit sub-manifold within the global knowledge manifold. Under this formulation, the problem can be decomposed into two key questions: (i) how to identify representative edit points that effectively anchor the edit sub-manifold, and (ii) how to preserve the remaining manifold structure during the sub-manifold displacement process. Based on this perspective, we propose ManiEdit, a novel manifold-aware autoregressive editing framework consisting of two core components. Pivot Localization addresses the mediocre-point dilemma by identifying high-leverage pivots to anchor the edit sub-manifold. Manifold-Aware Preservation preserves different knowledge types through an energy-weighted penalty combined with recursive null-space alignment. Experiments on two base LLMs and four unstructured editing benchmarks demonstrate that ManiEdit achieves state-of-the-art performance, outperforming the strongest baseline by up to +27.81 BERTScore and +8.50 ROUGE-L, while maintaining near-original general capabilities across six representative downstream tasks. Our code is available at: https://github.com/Areyliu/ManiEdit
Rui Liu, Chenheng Zhang, Haoxuan Li +1
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University
Multimodal large language models (MLLMs) need efficient mechanisms to update knowledge without degrading existing capabilities. While intrinsic multimodal knowledge editing achieves strong reliability and locality, it often exhibits limited generality, failing to propagate edits across semantically equivalent visual and linguistic variations. This issue arises from the lack of explicit semantic supervision, rigid editing scopes, and biased anchoring to individual samples in high-dimensional multimodal spaces. We address robust intrinsic multimodal knowledge editing by explicitly targeting generalization. We formalize robustness through knowledge units that group semantically equivalent multimodal inputs and define generality as consistent predictions within each unit. To expose fragile semantic regions, we introduce Latent Adversarial Robustification (LAR), which generates adversarial yet semantically coherent variants in the joint latent space. We further propose Rank-Constrained Subspace Learning (RCSL), enforcing low-rank alignment of adversarial representations at the edit layer via a singular value-based objective. Extensive analysis demonstrates the effectiveness of ASAM empirically.
Haoyuan Wang, Xiaohao Liu, Jiajie Su +2
Zhejiang University · National University of Singapore · Jiangxi Normal University
Knowledge editing has emerged as a promising approach for efficiently updating embedded knowledge in large language models (LLMs). It first computes an ideal target hidden state that steers the LLM toward the new output, and then edits the model parameters so that the original hidden state is mapped to this target. While most existing methods focus on the second stage of parameter editing in preserving other knowledge, we find that the upstream construction of the target hidden state also plays an important role in allocating the model's limited capacity space. We first establish a theoretical connection between interference in the parameter space and interference between target hidden states, showing that the two are positively correlated. Building on this insight, we propose a simple yet effective ℓ1 regularization method that prunes away unnecessary ``tentacles'' of each knowledge vector, retaining only the essential components. With fewer extraneous tentacles, each edit affects less other knowledge, enabling better coexistence. Extensive experiments across multiple LLMs and benchmark datasets verify the effectiveness of our method. Code: https://github.com/wintersdragon-c/STO.
Dongyao Chen, Jiayun Lei, Zhiyuan Ji +3
Renmin University of China · Laboratory for Artificial Intelligence and New Forms of Education, Central China Normal University · School of Computing, National University of Singapore +1