cs.CLAug 22, 2026
SavePUMA: A Polish Benchmark for Culturally Grounded Multimodal Understanding
Organizations: National Information Processing Institute al. Niepodległości 188b, 00-608 Warsaw, Poland
Abstract
Large language models are increasingly moving beyond text processing, adding support for other modalities such as images and audio. While text understanding and generation have been extensively studied, multimodal data processing capabilities, particularly in the context of cultures and languages other than English, have not yet been evaluated comprehensively. In this paper, we propose PUMA (Polish Unified Multimodal Assessment), a novel benchmark of 900 hand-crafted tasks designed to probe the limits of multimodal models in the Polish cultural and linguistic context. The dataset evaluates both cultural understanding and practical skill in processing text, images, audio, and visually rich documents. Our extensive evaluation of frontier commercial models, open-weights models, and specialized smaller systems highlights a significant performance gap. While top commercial models achieve high scores in visual question answering, most models struggle with complex audio or document understanding. We open-source our evaluation framework to advance localized multimodal AI research.
Figures & tables
Figure 1: A diagram showing the breakdown of categories in our benchmark. The nine categories, representing tasks types or subject areas, span three modalities: images, audio, and documents.
| Images | Audio | Documents | ||||||||
| Model | History & culture | Contemp. life | Geography & env. | ASR | Speech QA | Sound & music QA | OCR | Document QA | Structured extraction | Mean score |
| Multimodal models | ||||||||||
| Phi-4-Multimodal-Instruct | 2 (6.2) | 2 (3.4) | 5 (11.8) | 0 (2.2) | 0 (2.2) | 2 (4.0) | 0 (0.6) | 6 (11.8) | 0 (14.8) | 1.9 (6.3) |
| MiMo-V2-Omni | 21 (30.6) | 27 (33.5) | 28 (42.9) | 3 (48.6) | 14 (28.5) | 23 (28.0) | 57 (82.2) | 53 (68.1) | 16 (39.5) | 26.9 (44.6) |
| MiMo-V2.5 | 22 (30.6) | 23 (32.5) | 31 (42.0) | 1 (49.1) | 15 (29.4) | 22 (28.4) | 57 (82.8) | 62 (72.1) | 39 (88.7) | 30.2 (50.6) |
| Gemini-2.5-Flash | 50 (58.2) | 40 (51.0) | 53 (66.2) | 69 (89.7) | 41 (52.2) | 23 (29.8) | 73 (91.0) | 52 (64.4) | 53 (91.9) | 50.4 (66.0) |
Table 1: Model results on our benchmark comprising 900 tasks (100 per category). We report strict scores and soft scores (the smaller gray numbers in parentheses). The overall best score in each column is highlighted with a gray background and boldface, while the best score within each model group is underlined. The evaluation includes multimodal models, pairs of models from the same model family, and selected significant VLMs for comparison.
| Scores | |||
| Model | Strict | Soft | Soft BN |
| ASR models | |||
| Whisper-Tiny | 0 | 49.42 | 49.42 |
| Qwen3-ASR-0.6B | 4 | 65.97 | 61.92 |
| Whisper-Small | 22 | 75.82 | 75.82 |
| Qwen3-ASR-1.7B | 24 | 74.79 | 72.26 |
Table 2: ASR evaluation results.
| Scores | |||
| Model | Strict | Soft | Soft BN |
| OCR models | |||
| PaddleOCR-VL-1.5 | 8 | 47.05 | 44.87 |
| GLM-OCR | 8 | 58.64 | 53.73 |
| Qianfan-OCR | 12 | 57.84 | 55.57 |
| Tesseract-OCR | 13 | 35.13 | 32.07 |
Table 3: OCR evaluation results.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: The number of examples in audio categories (ASR, Speech QA, Sound and music QA) by recording length, number of speakers and recording quality.
Figure 3: The types of documents included in our benchmark with examples.
Figure 4: The number of examples in document categories (OCR, Document QA, Structured extraction), by document type, creation date, and quality.
| Model name | Model | Refusal | Rules |
| Gemini-3.1-Pro | 115 | 2 | 3 (0.5%) |
| GPT-5.5 + GPT-Audio | 181 | 36 | 6 (1.0%) |
| Gemini-3-Flash | 211 | 4 | 4 (0.7%) |
| MiMo-V2.5 | 421 | 0 | 4 (0.7%) |
Table 4: Manual classification of evaluation errors. The Rules rate is calculated relative to all 600 QA tasks in each run.
| Images | Documents | |||||||
| Model | Reference | History & culture | Contemp. life | Geography & env. | OCR | Document QA | Structured extraction | Mean score |
| Gemini-2.5-Flash | 12 | 50 (58.2) | 40 (51.0) | 53 (66.2) | 73 (91.0) | 52 (64.4) | 53 (91.9) | 53.5 (70.4) |
| Gemini-3.1-Flash-Lite | 56 | 58 (63.2) | 49 (58.2) | 62 (76.2) | 76 (91.1) | 44 (60.0) | 45 (91.0) | 55.7 (73.3) |
| Gemini-3-Flash | 56 | 70 (76.2) | 74 (80.2) | 69 (79.7) | 84 (92.5) | 64 (76.8) | 68 (96.7) | 71.5 (83.7) |
| Gemini-2.5-Pro | 12 | 70 (76.0) | 72 (80.2) | 72 (80.8) | 80 (92.3) | 72 (83.6) | 68 (96.4) | 72.3 (84.9) |
| Gemini-3.5-Flash | 56 | 82 (85.8) | 79 (85.3) | 86 (92.6) | 86 (94.3) | 88 (94.7) | 66 (94.9) | 81.2 (91.3) |
Table 5: Results of the evaluation of VLMs and multimodal models on visual categories. We report strict scores and soft scores (the smaller gray numbers in parentheses). The best score in each column is highlighted with light gray shading and boldface, while the highest score within each model group is underlined.
Figure 5: A chart showing the distribution of files according to the most common sources we used. The private source refers to files created by the annotators themselves.
Figure 6: A sample screenshot of the task edit view in our annotator app, along with a description of the available features.
| Category: History and culture Instruction: Na litografii ukazano jedną z największych średniowiecznych warowni Europy. Na podstawie przedstawienia rozpoznaj ją i podaj jej budowniczych oraz rok przejęcia przez Królestwo Polskie. Instruction (english translation): The lithograph depicts one of the largest medieval fortresses in Europe. Based on the depiction, identify it and provide its builders, as well as the year it was taken over by the Kingdom of Poland. Verification rules: Include ([Krzyżacy, Zakon Krzyżacki], 1457) | |
| Category: History and culture Instruction: W mennicy którego polskiego władcy została wybita moneta widoczna na fotografii? Instruction (english translation): In which Polish ruler’s mint was the coin shown in the photograph struck? Verification rules: Include ([Bolesław Chrobry, Bolesław I Chrobry]) | |
| Category: Contemporary life Instruction: Obrazy oznaczone literami A, B, C, D przedstawiają kadry z polskich gier komputerowych. Czy jesteś w stanie rozpoznać te gry? Podaj tytuły w kolejności od A do D, bez dodatkowych komentarzy. Instruction (english translation): The images marked with letters A, B, C, D show screenshots from Polish video games. Can you recognize these games? Provide the titles in order from A to D, without any additional comments. Verification rules: Order (Cyberpunk, Polanie, Wiedźmin, This War of Mine) | |
| Category: Contemporary life Instruction: Podaj nazwy przedstawionych na zdjęciach zup. Zwróć tylko listę w formacie [numer zdjęcia] - [nazwa zupy], bez dodatkowych komentarzy. Instruction (english translation): Provide the names of the soups shown in the photos. Return only a list in the format [photo number] - [soup name], without any additional comments. Verification rules: Order (pieczarkowa, krupnik, rosół, ogórkowa, pomidorowa, fasolowa, [barszcz czerwony, barszcz z uszkami], szczawiowa) | |
| Category: Geography and environment Instruction: Wyodrębnij i uszereguj widoczne na powyższym zdjęciu, charakterystyczne polskie szczyty górskie w kolejności od najniższego do najwyższego. Każda z gór została oznaczona literą A, B, C lub D. Odpowiedz tylko ciągiem liter A, B, C, D w odpowiedniej kolejności. Instruction (english translation): Identify and sort the characteristic Polish mountain peaks visible in the photo above from lowest to highest. Each mountain has been marked with the letter A, B, C, or D. Answer only with a sequence of the letters A, B, C, D in the correct order. Verification rules: Order (A, B, D, C) | |
| Category: Geography and environment Instruction: Podaj nazwy miast zaznaczonych na powyższej mapie. Posortuj ich nazwy według kierunku północ-południe zaczynając od miasta wysuniętego najbardziej na północ. Podaj tylko posortowną listę miast i nie uwzględniaj dodatkowego komentarza. Instruction (english translation): Provide the names of the cities marked on the map above. Sort their names from north to south, starting with the northernmost city. Provide only the sorted list of cities and do not include any additional comments. Verification rules: Order (Gdańsk, Szczecin, Białystok, Warszawa, Wrocław, Katowice, Kraków) | |
| File description: An explanatory recording about how a moka pot works and how to use it. | Category: ASR Instruction: [General prompt for ASR] Verification rules: WAcc ( Jak zaparzyć kawę w kawiarce? Przyznam się, że gdy kupiłem swoją pierwszą kawiarkę, to kompletnie nie wiedziałem co mam z nią zrobić. Jest to natomiast fantastyczne, a przede wszystkim tanie naczynie, które pozwala przygotować bardzo dobrą kawę. Zasada działania jest prosta. Pod wpływem temperatury woda w dolnym zbiorniku przelewa się pod ciśnieniem przez kawę w pojemniku powyżej i ostatecznie trafia w postaci docelowego napoju do górnego zbiornika. (…)) |