M-MiniGPT4: Multilingual VLLM Alignment via Translated Data

Seung Hun Han; Youssef Mohamed; Mohamed Elhoseiny

cs.CLMar 31, 2026

M-MiniGPT4: Multilingual VLLM Alignment via Translated Data

Authors: Seung Hun Han, Youssef Mohamed, Mohamed Elhoseiny

Organizations: MBZUAI · KAUST

Abstract

This paper presents a Multilingual Vision Large Language Model, named M-MiniGPT4. Our model exhibits strong vision-language understanding (VLU) capabilities across 11 languages. We utilize a mixture of native multilingual and translated data to push the multilingual VLU performance of the MiniGPT4 architecture. In addition, we propose a multilingual alignment training stage that uses parallel text corpora to further enhance the multilingual capabilities of our model. M-MiniGPT4 achieves 36% accuracy on the multilingual MMMU benchmark, outperforming state-of-the-art models in the same weight class, including foundation models released after the majority of this work was completed. We open-source our models, code, and translated datasets to facilitate future research in low-resource and multilingual settings.

Figures & tables

Model	E-MMMU	BT-MMMU
Pretrained	34.61	34.45
Finetuned	34.14	33.02

Table 1: Performance validation via back-translation. We report the average accuracy after back-translation from all target languages. E-MMMU : English MMMU; BT-MMMU : Back-Translated MMMU.

Stage 2	Stage 3	MMMU	MMMU Multi
(ccSBU, LAION, LAVAM, PALO)	-	31.02	29.83
+ CI	-	34.14	31.21
+ CI M	-	36.69	33.57
+ CI	CI M + CT M + MText	37.07	32.90
+ CI M	CI M + CT M + MText	37.27	33.45
+ CI	CI + CT	35.19	32.93

Table 2: Ablation Studies on Training Data Combinations. CI : Cambrian Image; CT : Cambrian Text; M : Translated/Multilingual version.

Model	MMMU	MMMU Multi
PALO	28.36	13.12
Qwen-VL 2.5	52.89	25.46
Our Model	37.27	33.45

Table 3: Comparison to SOTA Vision-Language Models.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

Multilingual Training and Evaluation Resources for Vision-Language Models
Apr 20, 2026Daniela Baiamonte, Elena Fano, Matteo Gabburo +3VLM Evaluation Vision-Language Models
Aithlas
Lost in Translation: Do LVLM Judges Generalize Across Languages?
Apr 21, 2026Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5Multimodal Model Evaluation Multilingual VLM Evaluation
York University · University of Alberta · Nanyang Technological University +1
Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models
Jun 2, 2026Hashmat Shadab Malik, Muzammal Naseer, Salman KhanMLLM Safety Multilingual Language Model Evaluation
Mohamed Bin Zayed University of AI, UAE · Khalifa University, UAE · Australian National University, Australia

Apr 20, 2026J/K move · Enter open · S save

Multilingual Training and Evaluation Resources for Vision-Language Models

Daniela Baiamonte, Elena Fano, Matteo Gabburo, Stefano Simonazzi+2

Aithlas

VLM Evaluation Vision-Language Models Multilingual VLM Evaluation Vision-Language Pretraining

Open paper AskPDF arXiv

Vision Language Models (VLMs) achieved rapid progress in the recent years. However, despite their growth, VLMs development is heavily grounded on English, leading to two main limitations: (i) the lack of multilingual and multimodal datasets for training, and (ii) the scarcity of comprehensive evaluation benchmarks across languages. In this work, we address these gaps by introducing a new comprehensive suite of resources for VLMs training and evaluation spanning five European languages (English, French, German, Italian, and Spanish). We adopt a regeneration-translation paradigm that produces high-quality cross-lingual resources by combining curated synthetic generation and manual annotation. Specifically, we build Multi-PixMo, a training corpus obtained regenerating examples from Pixmo pre-existing datasets with permissively licensed models: PixMo-Cap, PixMo-AskModelAnything, and CoSyn-400k. On the evaluation side, we construct a set of multilingual benchmarks derived translating widely used English datasets (MMbench, ScienceQA, MME, POPE, AI2D). We assess the quality of these resources through qualitative and quantitative human analyses, measuring inter-annotator agreement. Additionally, we perform ablation studies to demonstrate the impact of multilingual data, with respect to English only, in VLMs training. Experiments, comprising 3 different models show that using multilingual, multimodal examples for training VLMs aids is consistently beneficial on non-English benchmarks, with positive transfer to English as well.

Abstract

Vision-Language Models

Explore similar work

Multilingual Training and Evaluation Resources for Vision-Language Models

Lost in Translation: Do LVLM Judges Generalize Across Languages?

Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models