Organizations: School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi, China · Harvard Medical School, Boston, MA 02155, U.S. · School of Mathematics and Statistics, Xidian University, Xi’an, Shaanxi, China · School of Mathematics and Statistics and Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an, Shaanxi, China · Macao Institute of Systems Engineering, Macau University of Science and Technology, Taipa, Macao, China
Multimodal knowledge editing aims to update visually grounded facts in Multimodal Large Language Models (MLLMs) while preserving unrelated text and visual behaviors. However, standard answer-level metrics can hide evidence misgrounding: an edit may produce the target answer on the original image yet remain sensitive to incidental context rather than the intended visual evidence. To examine this issue in a controlled setting, we construct WaterBird-Edit, an analysis set that decouples bird habitat labels from background context and measures whether edited predictions remain stable when the target evidence is preserved but the background changes. Motivated by this diagnosis, we propose VICE (Visual In-Context Editing), a retrieval-based framework that stores edits in external memory and applies them through visual in-context demonstrations designed to expose task-relevant evidence. Experiments on three standard benchmarks show that VICE achieves strong lifelong editing performance, while results on WaterBird-Edit show that it substantially reduces sensitivity to the controlled background shifts. Code and dataset are available in https://github.com/Wang-Z-W/VICE-main.
Figures & tables
Figure 1: An example of background-induced evidence misgrounding. An edit may succeed on the original image but fail when the same bird evidence appears with a different background. Lower performance gap indicates less sensitivity to background context.
Figure 2: Sample-wise pwater defined in Equation 1 for WaterBird-Edit image-rephrase samples after editing. The top two bands show water-bird targets, and the bottom two bands show land-bird targets. Point colors indicate background labels: blue for water backgrounds and orange for land backgrounds. Large markers denote background-group means, and Δbg reports the background-induced mean shift for each target label.
Method
I-Gen. ↑
Mask I-Gen. ↑
Group I-Gen. ↑
PG ↓
WW
WL
LW
LL
FT
85.2
87.8
60.9
80.6
98.5
94.3
24.3
LiveEdit
83.8
92.4
84.3
84.9
67.7
96.2
16.1
VICE
93.4
89.0
88.7
94.6
96.2
93.7
4.7
Table 1: WaterBird-Edit diagnostics on image-rephrase samples. Each group is written as (habitat, background), where “W” and “L” denote water and land. “PG” denotes performance gap. “Mask I-Gen.” is measured on object-only image-rephrase variants that preserve the bird region and mask the background.
Figure 3: Illustration of the VICE pipeline. (a) Evidence-summary demonstrations for Copy, Text-Update (T-U), Image-Update (I-U), Text-Retain (T-R), and Image-Retain (I-R), all using the same template in Equation 3 . (b) Lifelong editing, where each incoming new fact is stored together with its summary and retrieved demonstrations in Sequence Memory. (c) Inference, where a multimodal encoder retrieves the most relevant edit and routes the query augmented by the retrieved context to the MLLM when the retrieval similarity score exceeds the threshold.
Type
Image Summary
New Fact
Prompt
Copy
se=G(ie,xe)
(xe,ye)
(xe,ye)
Text-Update
se=G(ie,xe)
(xe,ye)
(xr,ye)
Image-Update
sr=G(ir,xe)
(xe,ye)
(xe,ye)
Text-Retain
se=G(ie,xe)
(xe,ye)
(xl,yl)
Image-Retain
sl=G(il,xil)
(xe,ye)
(xil,yil)
Table 2: Five evidence-summary demonstration roles instantiated from the template in Equation 3 . (ie,xe,ye) is the anchor example; xr is a text rephrase with answer ye ; ir is an image variant for prompt xe ; (xl,yl) is a text-locality example; and (il,xil,yil) is an image-locality example. Superscript of the k -th demonstration ck is omitted for readability.
EVQA
EIC
VLKEB
Model
# Edits
Method
Rel
T-Gen
I-Gen
T-Loc
I-Loc
Rel
T-Gen
I-Gen
T-Loc
I-Loc
Rel
T-Gen
I-Gen
T-Loc
I-Loc
LLaVA-v1.5
1
FT-VL
100.00
91.03
95.38
95.29
90.53
99.93
94.65
97.34
91.63
82.16
100.00
98.74
99.92
76.13
84.15
TP
33.27
31.33
29.06
96.77
98.83
57.49
57.48
53.91
95.41
96.52
50.34
52.43
50.27
92.59
96.58
MEND
91.23
90.05
91.29
91.02
90.22
92.82
91.81
90.59
96.38
93.69
92.13
91.28
90.22
89.19
90.13
SERAC
89.33
83.72
84.97
82.05
23.78
88.18
81.03
85.61
84.01
28.58
89.77
89.11
87.92
66.68
14.20
LiveEdit
94.28
94.51
88.01
100.00
100.00
82.16
81.01
78.27
100.00
100.00
96.43
95.22
93.72
100.00
100.00
Table 3: Main results of lifelong editing for LLaVA-v1.5 and MiniGPT4 on EVQA, EIC, and VLKEB. “#Edits” represents the number of new facts. MEND, SERAC, and LiveEdit results are from Chen et al. (2025) ; FT-VL, TP, and VICE are from our runs.
# Edits
Ablation
Rel
T-Gen
I-Gen
T-Loc
I-Loc
1
Full VICE
1.00
0.88
0.97
1.00
0.99
w/o Scope Head
1.00
0.86
0.96
1.00
0.85
w/o Image Summary
0.75
0.46
0.80
1.00
1.00
w/o Retrieved Demos
0.32
0.30
0.27
1.00
1.00
1000
Full VICE
1.00
0.88
0.97
1.00
0.98
w/o Scope Head
1.00
0.85
0.95
1.00
0.76
Table 4: Ablation study of VICE on EVQA with LLaVA-v1.5. “w/o Scope Head” uses the raw retrieval route without the learned head hψ , “w/o Image Summary” removes the generated visual evidence summaries from demonstrations, and “w/o Retrieved Demos” removes retrieved visual in-context demonstrations.
Figure 4: Illustration of mean retrieval similarity of raw CLIP-VL features and learned scope-head embeddings on EVQA. For edit, text-rephrase, and image-rephrase queries, we report similarity to the paired edit anchor; for locality, we report nearest-anchor similarity.
Figure 5: Five standard metrics under different similarity thresholds θ on EVQA with LLaVA-v1.5.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Image Set
WW
WL
LW
LL
Edit images
18
190
107
185
Image-rephrase images
115
93
133
159
Appendix
Table 5: WaterBird-Edit test-split group counts. Each group is written as (target habitat, background), where W and L denote water and land. The image-rephrase set is used for the group Image-Generality and PG diagnostics in Section 2 .
Format Consistency
500/500 = 100%
Target Matching Rate
476/500 = 95.2%
Distractor Matching Rate
431/500 = 86.2%
Appendix
Table 6: Summary quality audit.
Query type
Retrieval top-1
Routing acc
Routing error
Rel.
100.0%
100.0%
false reject 0.0%
T-Gen.
98.5%
98.0%
false reject 2.0%
I-Gen.
97.3%
96.9%
false reject 3.1%
T-Loc.
–
100.0%
false accept 0.0%
I-Loc.
–
94.3%
false accept 5.7%
Appendix
Table 7: VICE’s Top-1 retrieval and routing accuracy on EVQA with LLaVA-v1.5 at θ=0.65 .
Class
Class
Class
Class
001 Black footed Albatross (water)
002 Laysan Albatross (water)
003 Sooty Albatross (water)
004 Groove billed Ani (land)
005 Crested Auklet (water)
006 Least Auklet (water)
007 Parakeet Auklet (water)
008 Rhinoceros Auklet (water)
009 Brewer Blackbird (land)
010 Red winged Blackbird (land)
011 Rusty Blackbird (land)
012 Yellow headed Blackbird (land)
013 Bobolink (land)
014 Indigo Bunting (land)
015 Lazuli Bunting (land)
016 Painted Bunting (land)
017 Cardinal (land)
018 Spotted Catbird (land)
019 Gray Catbird (land)
020 Yellow breasted Chat (land)
021 Eastern Towhee (water)
022 Chuck will Widow (land)
023 Brandt Cormorant (water)
024 Red faced Cormorant (water)
Appendix
Table 8: CUB species-level habitat labels used by WaterBird-Edit .
EVQA
EIC
VLKEB
# Edits
Method
Rel
T-Gen
I-Gen
T-Loc
I-Loc
Rel
T-Gen
I-Gen
T-Loc
I-Loc
Rel
T-Gen
I-Gen
T-Loc
I-Loc
1
FT-VL
100.00
91.03
95.38
95.29
90.53
99.93
94.65
97.34
91.63
82.16
100.00
98.74
99.92
76.13
84.15
TP
33.27
31.33
29.06
96.77
98.83
57.49
57.48
53.91
95.41
96.52
50.34
52.43
50.27
92.59
96.58
MEND
91.23
90.05
91.29
91.02
90.22
92.82
91.81
90.59
96.38
93.69
92.13
91.28
90.22
89.19
90.13
SERAC
89.33
83.72
84.97
82.05
23.78
88.18
81.03
85.61
84.01
28.58
89.77
89.11
87.92
66.68
14.20
LiveEdit
94.28
94.51
88.01
100.00
100.00
82.16
81.01
78.27
100.00
100.00
96.43
95.22
93.72
100.00
100.00
Appendix
Table 9: Complete lifelong editing results for LLaVA-v1.5 on EVQA, EIC, and VLKEB. FT-VL, TP, and VICE are from our local runs; MEND, SERAC, and LiveEdit entries are copied from Chen et al. (2025) .
EVQA
EIC
VLKEB
# Edits
Method
Rel
T-Gen
I-Gen
T-Loc
I-Loc
Rel
T-Gen
I-Gen
T-Loc
I-Loc
Rel
T-Gen
I-Gen
T-Loc
I-Loc
1
FT-VL
100.00
99.74
99.86
63.66
35.65
100.00
95.28
96.41
58.21
33.84
100.00
99.75
99.95
56.12
42.03
TP
30.07
28.55
30.40
89.48
71.50
30.79
32.00
34.41
44.72
28.74
37.17
38.31
37.49
74.56
46.15
MEND
93.47
91.32
91.22
81.51
74.97
90.28
83.91
79.05
95.92
93.97
90.28
89.18
89.55
93.63
86.69
SERAC
87.66
69.24
85.96
82.11
24.11
85.09
78.96
81.70
83.36
24.50
89.28
89.06
87.35
64.88
12.72
LiveEdit
94.90
92.63
92.06
100.00
100.00
86.84
83.82
76.95
100.00
100.00
96.41
95.56
92.85
100.00
100.00
Appendix
Table 10: Complete lifelong editing results for MiniGPT4 on EVQA, EIC, and VLKEB. FT-VL, TP, and VICE are from our local runs; MEND, SERAC, and LiveEdit entries are copied from Chen et al. (2025) .
Existing methods in Multimodal Knowledge Editing (MKE) have advanced the ability to correct outdated or inaccurate knowledge in Multimodal Large Language Models (MLLMs). However, they exhibit a critical limitation: while effectively modifying target factual pairs, they fail to generalize edits to logically related queries and often cause unintended alterations to unrelated but visually or semantically linked information. We identify and formalize two underlying failure modes causing this issue: Causal Misalignment, which confines edits to the specific sample, and Feature Entanglement, which causes unintended alterations to coupled but irrelevant information. To address these issues, we propose Localized and Disentangled Knowledge Editing (LDKE), a new framework that achieves precise and generalized editing by localizing fact-specific model layers and disentangling target-relevant inputs from irrelevant ones. Our approach introduces a Fast Localization module to identify and update critical layers efficiently, along with a Disentanglement Classifier that routes inputs appropriately to preserve unrelated knowledge. Extensive experiments across various benchmarks and MLLMs demonstrate that LDKE achieves superior performance in propagating edits to related contexts while maintaining high locality.
Multimodal large language models (MLLMs) need efficient mechanisms to update knowledge without degrading existing capabilities. While intrinsic multimodal knowledge editing achieves strong reliability and locality, it often exhibits limited generality, failing to propagate edits across semantically equivalent visual and linguistic variations. This issue arises from the lack of explicit semantic supervision, rigid editing scopes, and biased anchoring to individual samples in high-dimensional multimodal spaces. We address robust intrinsic multimodal knowledge editing by explicitly targeting generalization. We formalize robustness through knowledge units that group semantically equivalent multimodal inputs and define generality as consistent predictions within each unit. To expose fragile semantic regions, we introduce Latent Adversarial Robustification (LAR), which generates adversarial yet semantically coherent variants in the joint latent space. We further propose Rank-Constrained Subspace Learning (RCSL), enforcing low-rank alignment of adversarial representations at the edit layer via a singular value-based objective. Extensive analysis demonstrates the effectiveness of ASAM empirically.
Haoyuan Wang, Xiaohao Liu, Jiajie Su +2
Zhejiang University · National University of Singapore · Jiangxi Normal University
Online multimodal knowledge editing requires injecting a continual stream of visual-textual corrections into multimodal large language models (MLLMs) with bounded overhead and minimal disruption to unrelated behaviors. Existing editors mainly emphasize edit reliability and long-horizon stability, but rarely control the semantic boundary of each edit. Our pilot analyses of post-edit behaviors and internal neuronal activities reveal a scope gap behind reliable edits: instance-level success neither guarantees transfer to valid cross-modal variants nor prevents leakage to unrelated inputs, while edit-related cross-modal responses concentrate in deeper semantic layers. Therefore, we formulate Edit-Scoped Generalization, reframing online MLLM editing from merely correcting an instance to controlling the propagation boundary of each edit. To this end, we propose ScopeEdit, a scope-aware online editor that decomposes each update into a modality-local absorption branch and an evidence-gated shared generalization branch. The local branch supports stable edit absorption, whereas the shared branch enables cross-modal propagation only when visual and textual evidence are sufficiently aligned. Both branches perform scope-separated write geometries in orthogonal low-rank spaces and maintain branch-wise preconditioners via Sherman--Morrison recursions, yielding constant per-edit overhead. Extensive experiments across diverse benchmarks, long-horizon edit streams, MLLM backbones, real-world VLKEB scenarios, and complex vision-language architectures show that ScopeEdit consistently improves the trade-off between in-scope cross-modal transfer and out-of-scope locality, while preserving edit reliability, stability and online efficiency. Our code is available at https://github.com/lab-klc/ScopeEdit.
Siyuan Li, Youyuan Zhang, Ruitong Liu +2
Harbin Institute of Technology, Shenzhen, China · Peng Cheng Laboratory · Peking University +1