Evaluating Cultural Knowledge Editing in Vision-Language Models

Malvina Nikandrou; Georgios Pantazopoulos; Ioannis Konstas; Alessandro Suglia

cs.CVOct 29, 2026

Evaluating Cultural Knowledge Editing in Vision-Language Models

Authors: Malvina Nikandrou, Georgios Pantazopoulos, Ioannis Konstas, Alessandro Suglia

Organizations: University of Edinburgh · Heriot-Watt University

Abstract

Current Vision-Language Models (VLMs) frequently fail to recognize culture-specific concepts, especially from underrepresented regions. Effectively injecting this long-tail knowledge into existing models remains a challenge. In this work, we investigate knowledge editing as a targeted alternative by introducing CulturalBind, a benchmark comprising 3.2K images of 1K concepts from 6 domains, designed to evaluate culture-aware cross-modal knowledge as well as concept-level knowledge editing in VLMs. Pre-edit evaluation reveals that textual and visual knowledge is decoupled: models often demonstrate text-based knowledge of cultural concepts but fail to identify them from images. We then evaluate editing methods in single and sequential scenarios. Although current methods successfully correct individual visual recognition errors, they exhibit low cross-modal portability, failing to integrate edits into the model's broader parametric knowledge. Furthermore, performance degrades rapidly under sequential editing, highlighting the need for comprehensive adaptation approaches that better consolidate multimodal knowledge over time.

Figures & tables

Figure 1: Structure of the CulturalBind editing dataset. Each concept is paired with an edit sample and six auxiliary probes targeting generalization (Text and Image Generality), knowledge preservation (Text and Image Locality), and cross-modal knowledge integration (Text and Image Portability).

Dimension	Visual Input	Text Input	Tests
Reliability	Edit image	Recognition question	Edit success
Text-Generality	Edit image	Paraphrased question	Text generalization
Image-Generality	Held-out image	Recognition question	Image generalization
Text-Locality	—	Unrelated text question	Preservation of text knowledge
Image-Locality	Unrelated image	Unrelated visual question	Preservation of VL knowledge
Text-Portability	—	Visual attribute question	Textual attribute recall

Table 1: Evaluation dimensions for knowledge editing. Generality targets the generalization of the edit, Locality focuses preservation of unrelated knowledge, and Portability tests cross-modal knowledge integration.

Region	Concepts	Samples
Eastern Asia	199	571
South-eastern Asia	164	533
Southern Asia	168	519
Western & Central Asia	118	366
Southern Europe	149	356
Northern Europe	81	166

Table 2: CulturalBind distribution by region.

	Text-Based Metrics			Image-Based Metrics
Model	OE Description	Encyclopedic	Attribute	Recognition	VG Encyclopedic
InternVL3.5 (4B)	35.7	24.8	31.6	18.6	15.5
InternVL3.5 (8B)	46.2	30.0	37.2	18.7	16.4
InternVL3.5 (14B)	54.0	33.1	42.3	19.8	17.3
InternVL3.5 (38B)	60.9	38.9	47.3	25.7	20.1
Qwen3-VL (4B)	56.2	32.4	39.5	34.6	23.9

Table 3: Cross-modal knowledge evaluation. Text-based measures (Open-Ended Description, Encyclopedic QA, Attribute QA) probe parametric cultural knowledge via text, while image-based measures (Recognition, Visually-Grounded Encyclopedic QA) test whether knowledge about the same concepts is accessible through visual input.

Model	Known	Unknown	$\Delta$	$r_{pb}^{*}$
InternVL3.5 (4B)	25.8	10.8	15.0	0.267
InternVL3.5 (8B)	22.9	10.5	12.4	0.221
InternVL3.5 (14B)	22.3	10.6	11.7	0.215
InternVL3.5 (38B)	29.5	11.9	17.6	0.265
Qwen3-VL (4B)	42.4	15.9	26.5	0.374
Qwen3-VL (8B)	37.2	16.2	21.1	0.286

Table 4: Recognition accuracy for correctly versus incorrectly recalled concepts. Known : model accurately described the concept; Unknown : model failed to describe it. Correlation (

r_{pb}

) indicates relationship strength.

{}^{*}p<.01

Model	Method	Reliability	T-Gen	I-Gen	T-Loc	I-Loc	T-Port	I-Port	Avg
	FT-V	98.5	79.7	25.1	93.0	77.7	15.7	16.0	58.0
	FT-T	100.0	79.8	63.4	75.7	84.2	14.0	14.1	61.6
	DPO	99.8	78.9	62.8	75.4	84.3	14.0	14.2	61.3
Qwen3-VL (4B)	IKE	99.7	98.9	99.8	23.4	31.7	15.8	17.4	55.2
N=871	MEND	92.3	88.1	85.9	86.1	82.3	12.7	12.5	65.7
	GRACE	99.5	2.4	4.6	93.0	97.4	15.7	15.0	46.8

Table 5: Method performance on single edits to correct N model errors on cultural concept recognition.

Panel of Figure 2 — Figure 2: Average accuracy of editing metrics as a function of the number of sequential edits.

Model	Portability	Corrected	Damaged	Unchanged
Qwen3-VL (4B)	Text	48	61	367
Qwen3-VL (4B)	Image	23	29	396
Qwen3-VL (8B)	Text	47	28	398
Qwen3-VL (8B)	Image	35	21	366
Gemma 3 (12B)	Text	43	34	393
Gemma 3 (12B)	Image	40	25	370

Table 6: Portability outcomes before and after editing.

Method	Reliability	T-Gen	I-Gen
FT-V	8.8 $\rightarrow$ 53.6	6.7 $\rightarrow$ 43.2	2.7 $\rightarrow$ 5.6
FT-T	19.2 $\rightarrow$ 54.4	16.0 $\rightarrow$ 39.2	10.4 $\rightarrow$ 35.2
DPO	16.8 $\rightarrow$ 56.8	16.8 $\rightarrow$ 44.0	15.3 $\rightarrow$ 34.7
WISE-T	29.6 $\rightarrow$ 24.8	28.0 $\rightarrow$ 26.0	20.8 $\rightarrow$ 25.6
WISE-V	29.6 $\rightarrow$ 31.2	32.0 $\rightarrow$ 28.7	26.4 $\rightarrow$ 28.0

Table 7: Accuracy of Qwen3-VL (4B) after sequential editing, on the earliest

\rightarrow

last 25% of edits.

Panel of Figure 3 — Figure 3: Distribution of the 15 largest Wikidata instance of (P31) groups among CulturalGround entities with English Wikipedia pages and multiple images. Before filtering, the majority of items belong to named-entity categories, which do not satisfy the concept criterion and are excluded at the group level.

Figure 4: Geographic distribution of concepts in CulturalGround (after filtering, left) and CROPE (after retrieving images for distractor concepts, right).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	Question Clarity	Answer Faithfulness
Encyclopedic QA	89.3	94.6
Visual Attribute QA	90.5	96.5

Appendix

Table 8: Manual audit of 200 generated QA pairs (%).

Panel of Figure 5 — Figure 5: Prompt for filtering the CulturalGround entities.

Panel of Figure 6 — Figure 6: Prompt for filtering the dataset images.

Panel of Figure 7 — Figure 7: Prompt for extracting visual descriptions from the Wikipedia page.

Panel of Figure 8 — Figure 8: LLM prompt to generate encyclopedic questions.

Domain	Concepts	Samples
Foods & Beverages	432	1,235
Animals & Plants	177	508
Music & Performing Arts	152	543
Artefacts	135	462
Traditions	126	300
Sports	69	168

Appendix

Table 9: CulturalBind distribution by domain.

Question	Description	Count
Historical	Dates, events, historical figures	1,322
Geographic	Regional origins, specific locations	1,163
Linguistic	Etymology, meaning, language of origin	868
Cultural	Ethnic groups, practices, traditions	633
Function	Purpose, use case, role	449
Comparison	Similarities, equivalents, related items	441

Appendix

Table 10: Distribution of 5,298 encyclopedic (top) and 5,321 visual attribute (bottom) knowledge questions.

Figure 9: CulturalBind editing examples.

	Generality	Locality
CulturalBind	0.756 $\pm$ 0.124	0.548 $\pm$ 0.095
VLKEB	0.853 $\pm$ 0.073	0.688 $\pm$ 0.113
MMKE-Bench	0.845 $\pm$ 0.077	0.425 $\pm$ 0.084

Appendix

Table 11: SigLIP2 cosine similarity (mean

\pm

sd) between edit images and their paired generality and locality images.

Panel of Figure 10 — Figure 10: Evaluation rubric for Open-Ended Description.

Panel of Figure 11 — Figure 11: Evaluation rubric for factual QA tasks.

Panel of Figure 12 — Figure 12: Evaluation rubric for visual recognition QA tasks, adapted from LAVE ( Mañas et al., 2024 ) .

Method	Edited Component	Modified Params	Added Params	Auxiliary Component
Qwen3-VL (4B), 36 LM layers
FT-V	VL connector	27M	0	—
FT-T	$W_{down}$ , layer 36	25M	0	—
DPO	$W_{down}$ , layer 36	25M	0	—
IKE	—	0	0	SentenceBERT
MEND	FFN, layer 36	75M	0	Hypernet. (189M)

Appendix

Table 12: Configuration of editing methods per model. Modified Params count model parameters updated in place; Added Params count new parameters. IKE retrieves

k=32

in-context examples per edit, embedded with all-MiniLM-L6-v2 SentenceBERT ( Reimers and Gurevych, 2019 ) . MEND’s hypernetwork is trained before editing. GRACE stores a key-value entry per edit : when the key (the FFN intermediate state) is matched, the learned value replaces the

W_{down}

output. WISE adds a copy of

W_{down}

, updated sparsely per edit (20% at random).

Panel of Figure 13 — Figure 13: Text-based cross-modal evaluation results per domain and per region.

Panel of Figure 14 — Figure 14: Image-based cross-modal evaluation results per domain and per region.

Model	Name (no image)	Name + image	Ref. expr. + image
Qwen3-VL (4B)	32.4	34.5	23.9
Qwen3-VL (8B)	39.3	39.7	24.3
Gemma 3 (12B)	43.1	45.2	26.8

Appendix

Table 13: Encyclopedic QA accuracy when the concept is referenced by name without the image, by name with the image, or by a referring expression with the image (Visually-Grounded Encyclopedic QA).

Panel of Figure 15 — Figure 15: Results per editing metric for sequential edits.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 738 papers

Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images
Apr 8, 2026Yuechen Jiang, Enze Zhang, Md Mohsinul Kabir +4Cultural Knowledge in Language Models VLM Evaluation
University of Manchester Manchester, UK · School of Artificial Intelligence, Wuhan University Wuhan, Hubei, China · Getty Conservation Institute Los Angeles, CA, USA
DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editing
Apr 9, 2026Gyanendra Das, Sai Satyam JenaContinual Learning for VLMs Vision-Language Alignment
Zynix AI, FL, USA
Evaluating and Understanding Model Editing for Medical Vision Language Models
Jul 6, 2026Guli Zhu, Chenwei Wu, Liyue ShenKnowledge Editing Medical VLMs
EECS, University of Michigan, Ann Arbor, MI 48105, USA

Apr 8, 2026J/K move · Enter open · S save

Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images

Yuechen Jiang, Enze Zhang, Md Mohsinul Kabir, Qianqian Xie+3

University of Manchester Manchester, UK · School of Artificial Intelligence, Wuhan University Wuhan, Hubei, China · Getty Conservation Institute Los Angeles, CA, USA

Cultural Knowledge in Language Models VLM Evaluation Cross-Cultural Language Model Evaluation

Open paper AskPDF arXiv

Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage. However, inferring structured cultural metadata (e.g., creator, origin, period) from visual input remains underexplored. We introduce a multi-category, cross-cultural benchmark for this task and evaluate VLMs using an LLM-as-Judge framework that measures semantic alignment with reference annotations. To assess cultural reasoning, we report exact-match, partial-match, and attribute-level accuracy across cultural regions. Results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types, leading to inconsistent and weakly grounded predictions. These findings highlight the limitations of current VLMs in structured cultural metadata inference beyond visual perception.

Abstract

Vision-Language Models

Explore similar work

Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images

DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editing

Evaluating and Understanding Model Editing for Medical Vision Language Models