Multilingual VLM Evaluation

VLM: Vision-Language Model

This field tests whether vision-language models can combine visual and linguistic evidence across languages, scripts, and cultural settings. Benchmarks cover tasks such as OCR and document parsing, visual question answering, reasoning, captioning, translation, and safety, using translated or parallel examples as well as native, human-verified data. Evaluations increasingly compare performance across languages and scripts, use text-only or image-only controls to check modality reliance, and measure consistency, grounding, and hallucinations rather than relying on aggregate accuracy alone. A central challenge is that strong English scores or gains from multilingual training do not guarantee reliable performance in low-resource languages, unfamiliar scripts, or culturally specific contexts.

Latest papers

56 papers1 paper in the last four weeks, down 90% on the four weeks before. 0.0% of all new papers.

VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics
Sep 29, 2026Bo Lv, Mao Zheng, Zheng Li +3Multilingual VLM Evaluation Rubric-Based Evaluation
Foundation Model Department, Tencent
Vroom-Vroom at SHROOM-Visions: A Multi-Judge Committee for Detecting Hallucinated Spans in Vision-Language Outputs
Sep 15, 2026Toqeer Ehsan, Nico Penttilä, Richard Schmidt +2VLM Evaluation Hallucination Detection in VLMs
Reliable Intelligence Team, Physical AI, VTT Technical Research Centre of Finland
ChitraMiti: Benchmarking Visual Grounding and Modality Reliance in Bengali Geometric Reasoning
Sep 11, 2026Khan Raiyan Ibne Reza, Sanjana Aktar Maria, Sumaiya Tabassum Nimi +1Mathematical Reasoning Benchmarks Multilingual VLM Evaluation
North South University
BanglaMemeX: Advancing Cultural Metaphoric Image Interpretation in Bangla with a Multimodal Explainable Dataset
Sep 7, 2026Md. Sadman Sakib, Zisan Mahmud, Md. Fahim Arefin +1Figurative Language Understanding Multilingual VLM Evaluation
University of Dhaka · University of Texas at Dallas
MM-IFEval-Pro: A Multilingual and Attack-Resistant Benchmark for Instruction-Following in Vision-Language Models
Sep 4, 2026Changming Xiao, Zhenliang Ni, Jinhui He +2VLM Robustness Adversarial Attacks on VLMs
Huawei Technologies Ltd. · Beijing, China · Shenzhen, China +1
IchthyoNoma: Nomenclature and Context Sensitivity of Zero-Shot Biological Vision--Language Models for Bangladeshi Freshwater Fish Recognition
Sep 3, 2026Nazim-E-Alam, Tarek Rahman, Md Kishor MorolVLM Robustness Prompt Sensitivity
American International University Bangladesh · United International University · ELITE Research Lab, Queens, New York, USA
CulturalMenuBench: Probing the Knowledge-Application Gap in Multimodal Culinary Reasoning
Sep 3, 2026Bo Zeng, Linfeng Gao, Peiqin Lin +9Multimodal Reasoning Multilingual VLM Evaluation
Alibaba Group · Xiamen University · University of Hamburg
MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models
Sep 1, 2026Tawsif Tashwar Dipto, Mehedi Ahamed, Radib Bin Kabir +5VLM Evaluation Multilingual VLM Evaluation
1Islamic University of Technology · 2South East University · 3Vector Institute +3
Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?
Aug 31, 2026Donghoon Han, SungHyun Moon, Aidyn Zhakatayev +2Cross-Lingual Representation Alignment Multilingual VLM Evaluation
Dnotitia Inc.
ImageEval 2026: Culturally Grounded Arabic Multimodal Evaluation
Aug 31, 2026Samir Abdaljalil, Hunzalah Hassan Bhatti, Ahlam Bashiti +11Hallucination Detection T2I Generation
Texas A&M University · Qatar Computing Research Institute, Qatar · Birzeit University, Palestine +2
Read the Room, Read the Image: Understanding Indirect Speech Acts in Multimodal Visual Contexts
Aug 31, 2026Jaehee Kim, Ji Hoon Chung, Seoyoon Park +5Multimodal QA Multilingual VLM Evaluation
Department of Korean Language and Literature, Yonsei University · Interdisciplinary Graduate Program of Linguistics and Informatics, Yonsei University · LG AI Research +1
What Can Low Resource Languages Learn From Each Other?
Aug 27, 2026Achyuth P, Kahaan Shah, Chetan AroraOptical Character Recognition VLM Adaptation
IIT Delhi, New Delhi, India
Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models
Aug 27, 2026Basel Mousi, Fahim Dalvi, Shammur Chowdhury +2Multimodal Model Evaluation Multimodal Robustness
Qatar Computing Research Institute, HBKU, Qatar
Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Models
Aug 26, 2026Raúl Vázquez, Aman Sinha, Chuyuan Li +12Hallucination Detection in VLMs Hallucination Detection
University of Helsinki · Politecnico di Torino · Universiteit Utrecht +5
Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA
Aug 23, 2026Jingbo Wang, Sendong Zhao, Haochun Wang +2Medical VQA Medical VLMs
Research Center for Social Computing and Interactive Robotics Harbin Institute of Technology, China
BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models
Aug 18, 2026Liubov Chubarova, Alexandra Kuleshova, Daniil Volkov +2Multimodal Reasoning Multilingual VLM Evaluation
Yandex · Applied AI Institute
Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs
Aug 15, 2026Yihang Du, Juhao Liang, Zhengzhao Lai +2Multimodal Large Language Models Vision-Language Alignment
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · National Health Data Institute (Shenzhen)
NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video
Aug 13, 2026Yuheng Huang, Jianlang Chen, Jiayang Song +6Video Understanding Video-Language Model Evaluation
The University of Tokyo, Tokyo, Japan · Kyushu University, Fukuoka, Japan · Macau University of Science and Technology, Macau, China +2
Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence
Aug 13, 2026Jakub Pokrywka, Łukasz Grzybowski, Antoni Lasik +3Medical VQA Multilingual VLM Evaluation
Adam Mickiewicz University · ARAAI Poland · NASK National Research Institute +2
Jako Tako or Fluent? Presenting PoVisLE: A Polish Vision-Language Evaluation
Aug 7, 2026Anna Kołos, Grzegorz Statkiewicz, Karolina Seweryn +3VLM Evaluation Visual Question Answering
NASK National Research Institute, Warsaw, Poland
Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarking
Aug 2, 2026Timothee Mickus, Claudio Savelli, Eduardo Calò +10VLM Evaluation Hallucination Detection in VLMs
University of Helsinki
Khondo: A Multimodal Benchmark for Document Packet Splitting of Bangla Forms
Jul 23, 2026Abu Tyeb Azad, Fahim Ahmed, Ishita Sur Apan +7Document Chunking Multilingual VLM Evaluation
Wichita State University, USA · CCDS, Independent University, Bangladesh · Amazon GenAI, USA
Test-Time Scaling for Small VLMs on Multilingual Visual MCQ
Jul 10, 2026Spiros Baxevanakis, Peng-Jian YangVisual Question Answering Test-Time Scaling for VLMs
University of Amsterdam, Science Park 904, 1098 XH Amsterdam, The Netherlands
Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models
Jul 8, 2026Kaito Watanabe, Taisei Yamamoto, Tomoki Doi +1Visual Spatial Reasoning Multilingual VLM Evaluation
The University of Tokyo · Riken · Tohoku University
Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability
Jul 7, 2026Alicia Parrish, Rajat Shinde, Sanket Badhe +57VLM Evaluation Cultural Safety Evaluation of VLMs
MORE: A Multilingual Document Parsing Benchmark and Evaluation
Jul 3, 2026Long Xu, Binghong Wu, Tinghao Yu +6Document Parsing Benchmarks Document Parsing
Tencent, Shenzhen, China
Resolution Thresholds in VLM Detection of Harmful ASCII Art Across Construction Modes and Languages
Jun 28, 2026Yikai Hua, Peter WestVLM Evaluation VLM Robustness
Department of Computer Science The University of British Columbia
Beyond 'One Language, One Script': Quantifying Orthographic Bias in Multilingual VLMs with PuMVR
Jun 18, 2026Prabhjot Singh, Bhushan Pawar, Madhu ReddiboinaVLM Evaluation Vision-Language Models
RediMinds Inc., USA
Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation
Jun 15, 2026Prabhjot Singh, Bhushan Pawar, Madhu Reddiboina +1VLM Evaluation Multilingual Language Models
RediMinds Inc., USA · The University of Texas at Austin, USA · Indian AI Research Organization (IAIRO), India
When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Models
Jun 10, 2026Xuan Dong, Zhe Han, Tianhao Niu +2VLM Robustness Multilingual VLM Evaluation
Harbin Institute of Technology

Sep 29, 2026J/K move · Enter open · S save

VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics

Bo Lv, Mao Zheng, Zheng Li, Fangxu Liu+2

Foundation Model Department, Tencent

Multilingual VLM Evaluation Rubric-Based Evaluation MT Evaluation

Open paper AskPDF arXiv

Image translation is a fundamental capability of multimodal models for multilingual applications, requiring visual understanding and meaning preservation across languages. However, existing benchmarks have limited language coverage and often lack explicit image-specific evaluation criteria, making it difficult to comprehensively assess this capability. To systematically evaluate this capability, we introduce VISTA-Bench, covering 22 languages and 10 domains, and develop an image-specific rubric evaluation protocol. The benchmark combines sampling for language and scenario coverage with model-assisted, human-verified annotations that group related text into coherent semantic units and provide multilingual reference translations. The rubrics specify essential content, semantic relations, and acceptable translation variants, yielding separate output-based scores for translation quality and the preservation of visual and knowledge-dependent information. We conduct extensive evaluations of 16 mainstream models, including 12 multimodal models and four text-input models, and provide systematic analyses across languages, domains, and evaluation dimensions.