This study presents a benchmark for evaluating the Visual Word Sense Disambiguation (Visual-WSD) task in Ukrainian. The main goal of the Visual-WSD task is to identify, with minimal contextual information, the most appropriate representation of a given ambiguous word from a set of ten images. To construct this benchmark, we followed a methodology similar to that proposed by (CITATION), who previously introduced benchmarks for the Visual-WSD task in English, Italian, and Farsi. This approach allows us to incorporate the Ukrainian benchmark into a broader framework for cross-language model performance comparisons. We collected the benchmark data semi-automatically and refined it with input from domain experts. We then assessed eight multilingual and multimodal large language models using this benchmark. All tested models performed worse than the zero-shot CLIP-based baseline model (CITATION) used by (CITATION) for the English Visual-WSD task. Our analysis revealed a significant performance gap in the Visual-WSD task between Ukrainian and English.
Figures & tables
Figure 1: An illustration of GPT4-Vision visual hallucination caused by ambiguous target word.
Figure 2: Example of the benchmark entry. The word Коса (en: braid, transl: kosa) is ambiguous. It corresponds to the meaning Заплетене волосся; довге волосся (en: braided hair; long hair, transl: zapletene volossya; dovhe volossya). The word Волосся (en: hair, transl: volossya) is the trigger word. The image that corresponds to the intended meaning is b (underlined). The other three images are examples of negative samples. Note: While the task involves nine negative images, we only display three negative images for simplicity.
Model
HIT@1
MRR
XLM-Roberta-Large-
Vit-B-16Plus
42.78
60.30
GPT-4o
42.52
43.75
XLM-Roberta-Large-Vit-L-14
40.21
58.65
XLM-Roberta-Large-Vit-B-32
39.69
57.69
GPT4-Vision
38.50
45.29
Table 1: The HIT@1 and MRR metrics for multiple multimodal models evaluated on the assembled benchmark and sorted by HIT@1. Baseline results for Visual-WSD in the English language are also included for comparison Raganato et al. (2023) .
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.