Multilingual VLM Evaluation

VLM: Vision-Language Model

Overview

This field tests whether vision-language models can combine visual and linguistic evidence across languages, scripts, and cultural settings. Benchmarks cover tasks such as OCR and document parsing, visual question answering, reasoning, captioning, translation, and safety, using translated or parallel examples as well as native, human-verified data. Evaluations increasingly compare performance across languages and scripts, use text-only or image-only controls to check modality reliance, and measure consistency, grounding, and hallucinations rather than relying on aggregate accuracy alone. A central challenge is that strong English scores or gains from multilingual training do not guarantee reliable performance in low-resource languages, unfamiliar scripts, or culturally specific contexts.

Latest papers

61 papers5 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Aug 19, 2025

VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark

We introduce VMMU, a Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark designed to evaluate how vision-language models (VLMs) interpret and reason over visual and textual information beyond English. VMMU consists of 2.5k multimodal questions across 7 tasks, covering a diverse range of problem contexts, including STEM problem solving, data interpretation, rule-governed visual reasoning, and abstract visual reasoning. All questions require genuine multimodal integration, rather than reliance on text-only cues or OCR-based shortcuts. We evaluate a diverse set of state-of-the-art proprietary and open-source VLMs on VMMU. Despite strong Vietnamese OCR performance, proprietary models achieve only 66% mean accuracy. Further analysis shows that the primary source of failure is not OCR, but instead multimodal grounding and reasoning over text and visual evidence. Code and data are available at https://vmmu-bench.github.io/