Diagnosing and Mitigating Evidence Misgrounding in Multimodal Knowledge Editing

Zhiwu Wang; Yichen Wu; Jinqian Yang; Yizhou Yang; Renzhen Wang; Qian Zhao; Deyu Meng

cs.CVOct 21, 2026

Diagnosing and Mitigating Evidence Misgrounding in Multimodal Knowledge Editing

Authors: Zhiwu Wang, Yichen Wu, Jinqian Yang, Yizhou Yang, Renzhen Wang, Qian Zhao, Deyu Meng

Organizations: School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi, China · Harvard Medical School, Boston, MA 02155, U.S. · School of Mathematics and Statistics, Xidian University, Xi’an, Shaanxi, China · School of Mathematics and Statistics and Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an, Shaanxi, China ·

Abstract

Multimodal knowledge editing aims to update visually grounded facts in Multimodal Large Language Models (MLLMs) while preserving unrelated text and visual behaviors. However, standard answer-level metrics can hide evidence misgrounding: an edit may produce the target answer on the original image yet remain sensitive to incidental context rather than the intended visual evidence. To examine this issue in a controlled setting, we construct WaterBird-Edit, an analysis set that decouples bird habitat labels from background context and measures whether edited predictions remain stable when the target evidence is preserved but the background changes. Motivated by this diagnosis, we propose VICE (Visual In-Context Editing), a retrieval-based framework that stores edits in external memory and applies them through visual in-context demonstrations designed to expose task-relevant evidence. Experiments on three standard benchmarks show that VICE achieves strong lifelong editing performance, while results on WaterBird-Edit show that it substantially reduces sensitivity to the controlled background shifts. Code and dataset are available in https://github.com/Wang-Z-W/VICE-main.

Figures & tables

Figure 1: An example of background-induced evidence misgrounding. An edit may succeed on the original image but fail when the same bird evidence appears with a different background. Lower performance gap indicates less sensitivity to background context.

Figure 2: Sample-wise

p_{\text{water}}

defined in Equation 1 for WaterBird-Edit image-rephrase samples after editing. The top two bands show water-bird targets, and the bottom two bands show land-bird targets. Point colors indicate background labels: blue for water backgrounds and orange for land backgrounds. Large markers denote background-group means, and

\Delta_{bg}

reports the background-induced mean shift for each target label.

Method	I-Gen. $\uparrow$	Mask I-Gen. $\uparrow$	Group I-Gen. $\uparrow$				PG $\downarrow$
Method	I-Gen. $\uparrow$	Mask I-Gen. $\uparrow$	WW	WL	LW	LL	PG $\downarrow$
FT	85.2	87.8	60.9	80.6	98.5	94.3	24.3
LiveEdit	83.8	92.4	84.3	84.9	67.7	96.2	16.1
VICE	93.4	89.0	88.7	94.6	96.2	93.7	4.7

Table 1: WaterBird-Edit diagnostics on image-rephrase samples. Each group is written as (habitat, background), where “W” and “L” denote water and land. “PG” denotes performance gap. “Mask I-Gen.” is measured on object-only image-rephrase variants that preserve the bird region and mask the background.

Figure 3: Illustration of the VICE pipeline. (a) Evidence-summary demonstrations for Copy, Text-Update (T-U), Image-Update (I-U), Text-Retain (T-R), and Image-Retain (I-R), all using the same template in Equation 3 . (b) Lifelong editing, where each incoming new fact is stored together with its summary and retrieved demonstrations in Sequence Memory. (c) Inference, where a multimodal encoder retrieves the most relevant edit and routes the query augmented by the retrieved context to the MLLM when the retrieval similarity score exceeds the threshold.

Type	Image Summary	New Fact	Prompt
Copy	$s_{e}=G(i_{e},x_{e})$	$(x_{e},y_{e})$	$(x_{e},y_{e})$
Text-Update	$s_{e}=G(i_{e},x_{e})$	$(x_{e},y_{e})$	$(x_{r},y_{e})$
Image-Update	$s_{r}=G(i_{r},x_{e})$	$(x_{e},y_{e})$	$(x_{e},y_{e})$
Text-Retain	$s_{e}=G(i_{e},x_{e})$	$(x_{e},y_{e})$	$(x_{l},y_{l})$
Image-Retain	$s_{l}=G(i_{l},x_{il})$	$(x_{e},y_{e})$	$(x_{il},y_{il})$

Table 2: Five evidence-summary demonstration roles instantiated from the template in Equation 3 .

(i_{e},x_{e},y_{e})

is the anchor example;

x_{r}

is a text rephrase with answer

y_{e}

;

i_{r}

is an image variant for prompt

x_{e}

;

(x_{l},y_{l})

is a text-locality example; and

(i_{l},x_{il},y_{il})

is an image-locality example. Superscript of the

k

-th demonstration

c_{k}

is omitted for readability.

			EVQA					EIC					VLKEB
Model	# Edits	Method	Rel	T-Gen	I-Gen	T-Loc	I-Loc	Rel	T-Gen	I-Gen	T-Loc	I-Loc	Rel	T-Gen	I-Gen	T-Loc	I-Loc
LLaVA-v1.5	1	FT-VL	100.00	91.03	95.38	95.29	90.53	99.93	94.65	97.34	91.63	82.16	100.00	98.74	99.92	76.13	84.15
		TP	33.27	31.33	29.06	96.77	98.83	57.49	57.48	53.91	95.41	96.52	50.34	52.43	50.27	92.59	96.58
		MEND	91.23	90.05	91.29	91.02	90.22	92.82	91.81	90.59	96.38	93.69	92.13	91.28	90.22	89.19	90.13
		SERAC	89.33	83.72	84.97	82.05	23.78	88.18	81.03	85.61	84.01	28.58	89.77	89.11	87.92	66.68	14.20
		LiveEdit	94.28	94.51	88.01	100.00	100.00	82.16	81.01	78.27	100.00	100.00	96.43	95.22	93.72	100.00	100.00

Table 3: Main results of lifelong editing for LLaVA-v1.5 and MiniGPT4 on EVQA, EIC, and VLKEB. “#Edits” represents the number of new facts. MEND, SERAC, and LiveEdit results are from Chen et al. (2025) ; FT-VL, TP, and VICE are from our runs.

# Edits	Ablation	Rel	T-Gen	I-Gen	T-Loc	I-Loc
1	Full VICE	1.00	0.88	0.97	1.00	0.99
	w/o Scope Head	1.00	0.86	0.96	1.00	0.85
	w/o Image Summary	0.75	0.46	0.80	1.00	1.00
	w/o Retrieved Demos	0.32	0.30	0.27	1.00	1.00
1000	Full VICE	1.00	0.88	0.97	1.00	0.98
	w/o Scope Head	1.00	0.85	0.95	1.00	0.76

Table 4: Ablation study of VICE on EVQA with LLaVA-v1.5. “w/o Scope Head” uses the raw retrieval route without the learned head

h_{\psi}

, “w/o Image Summary” removes the generated visual evidence summaries from demonstrations, and “w/o Retrieved Demos” removes retrieved visual in-context demonstrations.

Figure 4: Illustration of mean retrieval similarity of raw CLIP-VL features and learned scope-head embeddings on EVQA. For edit, text-rephrase, and image-rephrase queries, we report similarity to the paired edit anchor; for locality, we report nearest-anchor similarity.

Figure 5: Five standard metrics under different similarity thresholds

\theta

on EVQA with LLaVA-v1.5.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Image Set	WW	WL	LW	LL
Edit images	18	190	107	185
Image-rephrase images	115	93	133	159

Appendix

Table 5: WaterBird-Edit test-split group counts. Each group is written as (target habitat, background), where W and L denote water and land. The image-rephrase set is used for the group Image-Generality and PG diagnostics in Section 2 .

Format Consistency	500/500 = 100%
Target Matching Rate	476/500 = 95.2%
Distractor Matching Rate	431/500 = 86.2%

Appendix

Table 6: Summary quality audit.

Query type	Retrieval top-1	Routing acc	Routing error
Rel.	100.0%	100.0%	false reject 0.0%
T-Gen.	98.5%	98.0%	false reject 2.0%
I-Gen.	97.3%	96.9%	false reject 3.1%
T-Loc.	–	100.0%	false accept 0.0%
I-Loc.	–	94.3%	false accept 5.7%

Appendix

Table 7: VICE’s Top-1 retrieval and routing accuracy on EVQA with LLaVA-v1.5 at

\theta=0.65

Class	Class	Class	Class
001 Black footed Albatross (water)	002 Laysan Albatross (water)	003 Sooty Albatross (water)	004 Groove billed Ani (land)
005 Crested Auklet (water)	006 Least Auklet (water)	007 Parakeet Auklet (water)	008 Rhinoceros Auklet (water)
009 Brewer Blackbird (land)	010 Red winged Blackbird (land)	011 Rusty Blackbird (land)	012 Yellow headed Blackbird (land)
013 Bobolink (land)	014 Indigo Bunting (land)	015 Lazuli Bunting (land)	016 Painted Bunting (land)
017 Cardinal (land)	018 Spotted Catbird (land)	019 Gray Catbird (land)	020 Yellow breasted Chat (land)
021 Eastern Towhee (water)	022 Chuck will Widow (land)	023 Brandt Cormorant (water)	024 Red faced Cormorant (water)

Appendix

Table 8: CUB species-level habitat labels used by WaterBird-Edit .

		EVQA					EIC					VLKEB
# Edits	Method	Rel	T-Gen	I-Gen	T-Loc	I-Loc	Rel	T-Gen	I-Gen	T-Loc	I-Loc	Rel	T-Gen	I-Gen	T-Loc	I-Loc
1	FT-VL	100.00	91.03	95.38	95.29	90.53	99.93	94.65	97.34	91.63	82.16	100.00	98.74	99.92	76.13	84.15
	TP	33.27	31.33	29.06	96.77	98.83	57.49	57.48	53.91	95.41	96.52	50.34	52.43	50.27	92.59	96.58
	MEND	91.23	90.05	91.29	91.02	90.22	92.82	91.81	90.59	96.38	93.69	92.13	91.28	90.22	89.19	90.13
	SERAC	89.33	83.72	84.97	82.05	23.78	88.18	81.03	85.61	84.01	28.58	89.77	89.11	87.92	66.68	14.20
	LiveEdit	94.28	94.51	88.01	100.00	100.00	82.16	81.01	78.27	100.00	100.00	96.43	95.22	93.72	100.00	100.00

Appendix

Table 9: Complete lifelong editing results for LLaVA-v1.5 on EVQA, EIC, and VLKEB. FT-VL, TP, and VICE are from our local runs; MEND, SERAC, and LiveEdit entries are copied from Chen et al. (2025) .

		EVQA					EIC					VLKEB
# Edits	Method	Rel	T-Gen	I-Gen	T-Loc	I-Loc	Rel	T-Gen	I-Gen	T-Loc	I-Loc	Rel	T-Gen	I-Gen	T-Loc	I-Loc
1	FT-VL	100.00	99.74	99.86	63.66	35.65	100.00	95.28	96.41	58.21	33.84	100.00	99.75	99.95	56.12	42.03
	TP	30.07	28.55	30.40	89.48	71.50	30.79	32.00	34.41	44.72	28.74	37.17	38.31	37.49	74.56	46.15
	MEND	93.47	91.32	91.22	81.51	74.97	90.28	83.91	79.05	95.92	93.97	90.28	89.18	89.55	93.63	86.69
	SERAC	87.66	69.24	85.96	82.11	24.11	85.09	78.96	81.70	83.36	24.50	89.28	89.06	87.35	64.88	12.72
	LiveEdit	94.90	92.63	92.06	100.00	100.00	86.84	83.82	76.95	100.00	100.00	96.41	95.56	92.85	100.00	100.00

Appendix

Table 10: Complete lifelong editing results for MiniGPT4 on EVQA, EIC, and VLKEB. FT-VL, TP, and VICE are from our local runs; MEND, SERAC, and LiveEdit entries are copied from Chen et al. (2025) .

Multimodal Knowledge Editing

See all 11 papers

Towards Localized and Disentangled Knowledge Editing for Multimodal Large Language Models
May 28, 2026Leijiang Gu, Zhen Zeng, Feng Li +2Knowledge Editing Multimodal Large Language Models
1Hefei University of Technology, Hefei, China
Beyond Binary Edits Robust Multimodal Knowledge Editing with Adversarial Subspace Alignment
May 22, 2026Haoyuan Wang, Xiaohao Liu, Jiajie Su +2Multimodal Robustness Adversarial Robustness
Zhejiang University · National University of Singapore · Jiangxi Normal University
Multimodal Knowledge Edit-Scoped Generalization for Online Recursive MLLM Editing
Jul 2, 2026Siyuan Li, Youyuan Zhang, Ruitong Liu +2Knowledge Editing Multimodal Large Language Models
Harbin Institute of Technology, Shenzhen, China · Peng Cheng Laboratory · Peking University +1

May 28, 2026J/K move · Enter open · S save

Towards Localized and Disentangled Knowledge Editing for Multimodal Large Language Models

Leijiang Gu, Zhen Zeng, Feng Li, Xinjian Gao+1

1Hefei University of Technology, Hefei, China

Knowledge Editing Multimodal Large Language Models Multimodal Knowledge Editing

Open paper AskPDF arXiv

Existing methods in Multimodal Knowledge Editing (MKE) have advanced the ability to correct outdated or inaccurate knowledge in Multimodal Large Language Models (MLLMs). However, they exhibit a critical limitation: while effectively modifying target factual pairs, they fail to generalize edits to logically related queries and often cause unintended alterations to unrelated but visually or semantically linked information. We identify and formalize two underlying failure modes causing this issue: Causal Misalignment, which confines edits to the specific sample, and Feature Entanglement, which causes unintended alterations to coupled but irrelevant information. To address these issues, we propose Localized and Disentangled Knowledge Editing (LDKE), a new framework that achieves precise and generalized editing by localizing fact-specific model layers and disentangling target-relevant inputs from irrelevant ones. Our approach introduces a Fast Localization module to identify and update critical layers efficiently, along with a Disentanglement Classifier that routes inputs appropriately to preserve unrelated knowledge. Extensive experiments across various benchmarks and MLLMs demonstrate that LDKE achieves superior performance in propagating edits to related contexts while maintaining high locality.

Abstract

Multimodal Knowledge Editing

Explore similar work

Towards Localized and Disentangled Knowledge Editing for Multimodal Large Language Models

Beyond Binary Edits Robust Multimodal Knowledge Editing with Adversarial Subspace Alignment

Multimodal Knowledge Edit-Scoped Generalization for Online Recursive MLLM Editing