Benchmarking Multilingual Document Understanding in Vision-Language Models
Organizations: Aithlas · Villanova
Abstract
Recent Vision-Language Models (VLMs) perform strongly on multimodal tasks, Document Understanding (DU) among them, but the benchmarks that measure them mostly cover English documents and narrow problems. How well current VLMs generalize across languages, tasks, styles and layouts therefore remains unclear. We introduce MDU, a multilingual DU benchmark of 23146 examples spanning 13 tasks and five languages: English, French, German, Italian, and Spanish. The tasks isolate complementary capabilities, from surface-level text and Markdown extraction through summarization, classification and named-entity recognition to layout-sensitive passage and layout extraction. We evaluate 45 state-of-the-art open-weight VLMs from 1.7B to 235B parameters, plus one commercial system under the same protocol. The benchmark is far from saturated: the best average is 69.0, only eight of the 45 evaluated systems reach 65, and the two best results come from 27B models rather than from the largest systems we evaluate. Performance is uneven in ways that scale does not fix. The gap between the strongest and weakest language does not close as models grow, and it survives controlling for the domain composition of each language. Capabilities also do not improve together: once we partial out overall model quality, per-task scores are largely uncorrelated, and transcription ability in particular trades off against semantic extraction.
Figures & tables
| Task | Category | Metric | A + B | Examples per language | Total | ||||
| de | en | es | fr | it | |||||
| Text Extraction | Parsing | 97.6 | 400 | 481 | 546 | 511 | 327 | 2,265 | |
| Markdown Extraction | Parsing | (norm.) | 97.2 | 400 | 365 | 537 | 553 | 389 | 2,244 |
| Layout Extraction | Layout | over covered words | 96.0 | 400 | 639 | 380 | 322 | 589 | 2,330 |
| NE Extraction | Inf. Extr. | value-set micro- | 94.0 | 400 | 400 | 400 | 400 | 1,368 | 2,968 |
| Passage Extraction | Inf. Extr. | chrF | 98.4 | 96 | 98 | 103 | 108 | 95 | 500 |
| Model (params) | AVG | Parsing | Layout | Inf. Extr. | Classif. | Question Answering | Summarization | Trans | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text | MD | NER | Pass | Type | Dom | QA | TabQA | FigQA | Sum | CSum | ||||
| Small ( 10B) | ||||||||||||||
| EuroVLM (9B) | 40.6 | 24.6 | 41.5 | 11.6 | 20.5 | 39.5 | 51.1 | 58.7 | 53.9 | 44.8 | 42.5 | 40.2 | 77.0 | 21.4 |
| GLM-4.1V (9B) | 46.1 | 56.9 | 50.8 | 37.2 | 32.6 | 45.7 | 52.5 | 46.3 | 36.7 | 53.4 | 44.6 | 42.8 | 59.4 | 39.9 |
| Granite-Vision-3.3 (2B) | 35.4 | 52.9 | 10.6 | 41.9 | 19.1 | 35.9 | 23.5 | 39.7 | 51.1 | 31.5 | 38.0 | 24.9 | 68.1 | 23.4 |
| Idefics3 (8B) | 35.5 | 31.6 | 41.2 | 8.2 | 19.4 | 32.2 | 42.8 | 54.1 | 47.9 | 32.3 | 34.3 | 29.2 | 71.6 | 17.2 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Type | Description |
|---|---|
| single_sentence | summarize in a single short sentence |
| bulleted_list | summarize as a bulleted list |
| numbered_list | summarize as a numbered list |
| actions_intentions | summarize with a focus on actions and intentions |
| key_info | summarize focusing only on key information |
| n_key_concepts | summarize in n key concepts |
| Task | Lang | Instruction | Reference |
|---|---|---|---|
| Text Extraction | fr | Peux-tu lire et recopier tout le texte ? | digilor FICHE TECHNIQUE BORNE IPAD COLUMN Cette borne iPad COLUMN au design tout en courbe s’intègre parfaitement sur vos points de vente ou lors de vos salons. Elle accueille votre […] |
| Markdown Extraction | fr | Convertissez cette page de document en Markdown. | Et après en avoir délibéré, > CONSTATE effectivement un allongement de la durée du chantier qui devrait s’achever en juillet 2024. > DÉCIDE la signature d’un marché complémentaire avec le […] |
| Layout Extraction | it | Individua le didascalie presenti in questa pagina e forniscimi le loro posizioni. | In questa pagina è presente una didascalia con il testo "Sala del Rifrattore". La sua posizione corrisponde alla regione [x1=0.406, y1=0.316, x2=0.593, y2=0.333] in coordinate normalizzate, […] |
| NE Extraction | it | Analizza l’immagine allegata per identificare le entità presenti. Restituisci il risultato come un oggetto JSON in cui ogni chiave è il testo esatto dell’entità trovata e ogni valore è la […] | {"Repubblica Italiana": "country", "COMUNE DI CASTELPIZZUTO": "city", "Provincia di Isernia": "state", "n. 2 del 23-02-2016": "DATE", "S2": "ID", "Folgara": "LOCATION" } |
| Passage Extraction | en | What specific modification does the guide recommend when ending the Butterfly stitch to avoid creating an unstable loop? | After numerous additional stitches, I’m ready to end the Butterfly stitch, so I tighten up this last loop, but this time, instead of making a loop to stick through this last loop, I want to […] |
| Type Classification | en | Operate as a document classifier. You receive ONE page of a document as an image. TASK: assign ONE label - the document TYPE (structure/nature). Pick exactly one of the 10 type classes […] | justification,has_main_body_text,type_confidence,type "The document is a formal letter addressed to Dr Brent Layton (Chair of the Electricity Authority) from a group of stakeholders. It […] |
| Model (params) | P | R | Ref. | |||
|---|---|---|---|---|---|---|
| EuroVLM (1.7B) | 0.002 | 0.00 | 0.00 | 0.000 | 0.028 | 0.36 |
| EuroVLM (9B) | 0.126 | 0.14 | 0.18 | 0.013 | 0.091 | 0.41 |
| GLM-4.1V (9B) | 0.367 | 0.47 | 0.37 | 0.101 | 0.186 | 1.53 |
| Gemma3 (4B) | 0.260 | 0.34 | 0.30 | 0.019 | 0.159 | 1.38 |
| Gemma4 (E2B) | 0.275 | 0.42 | 0.26 | 0.061 | 0.075 | 1.74 |
| Gemma4 (E4B) | 0.403 | 0.53 | 0.39 | 0.122 | 0.142 | 1.24 |
| Model (params) | AVG | Parsing | Layout | Inf. Extr. | Classif. | Question Answering | Summarization | Trans | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text | MD | NER | Pass | Type | Dom | QA | TabQA | FigQA | Sum | CSum | ||||
| Small ( 10B) | ||||||||||||||
| EuroVLM (1.7B) | 25.5 | 15.0 | 9.1 | 0.2 | 6.0 | 34.2 | 10.4 | 14.6 | 45.9 | 40.0 | 37.9 | 34.6 | 68.0 | 15.3 |
| EuroVLM (9B) | 40.6 | 24.6 | 41.5 | 11.6 | 20.5 | 39.5 | 51.1 | 58.7 | 53.9 | 44.8 | 42.5 | 40.2 | 77.0 | 21.4 |
| GLM-4.1V (9B) | 46.1 | 56.9 | 50.8 | 37.2 | 32.6 | 45.7 | 52.5 | 46.3 | 36.7 | 53.4 | 44.6 | 42.8 | 59.4 | 39.9 |
| Gemma3 (4B) | 48.5 | 51.3 | 37.1 | 26.0 | 30.5 | 43.0 | 57.1 | 67.8 | 48.8 | 53.7 | 47.4 | 39.1 | 88.0 | 41.3 |
| Model (params) | AVG ∗ | Language | |||||
|---|---|---|---|---|---|---|---|
| de | en | es | fr | it | |||
| Small ( 10B) | |||||||
| EuroVLM (1.7B) | 27.8 | 25.8 | 31.3 | 28.4 | 26.7 | 27.0 | 5.5 |
| EuroVLM (9B) | 38.0 | 34.9 | 44.0 | 36.7 | 36.9 | 37.2 | 9.0 |
| GLM-4.1V (9B) | 45.5 | 42.4 | 46.8 | 46.7 | 44.7 | 46.6 | 4.4 |
| Gemma3 (4B) | 46.0 | 44.3 | 49.1 | 46.8 | 43.8 | 46.0 | 5.3 |
Multilingual VLM Evaluation
Multilingual vision-language model evaluation measures how well models interpret images, documents, video, and other visual inputs across languages and cultures.
See all 61 papers