KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context

Nahyun Lee; Guijin Son; Hyunwoo Ko; Chanyoung Kim; JunYoung An; Kyubeen Han; Il-Youp Kwak

cs.CLMar 18, 2026

KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context

Authors: Nahyun Lee, Guijin Son, Hyunwoo Ko, Chanyoung Kim, JunYoung An, Kyubeen Han, Il-Youp Kwak

Organizations: Chung-Ang University · HAE-RAE · Seoul National University · OnelineAI ·

Abstract

Multimodal evaluation is strengthened when benchmarks reflect the languages, institutions, and disciplinary conventions that shape expert visual reasoning. We introduce KMMMU, a native Korean benchmark built from Korean-native examination materials. KMMMU contains 3,466 questions spanning nine disciplines and nine visual modality categories. For targeted evaluation on challenging questions and Korean-specific knowledge, we additionally provide KMMMU-Adv and KMMMU-Local, respectively. Experiments show that the strongest evaluated open-weight model reaches 42.05% on the KMMMU, while the best evaluated proprietary model achieves 52.42% on the KMMMU-Adv. Performance varies substantially across disciplines and answer formats, with Korean-specific questions showing gaps of up to 13.43%. Error analysis suggests that inspected failures often involve convention-to-label mapping, few-shot symbolic induction, localized knowledge recall, and domain-specific standards, not reasoning depth alone. KMMMU provides a testbed for native Korean expert-level multimodal understanding beyond English-centric or translation-based evaluation.

Figures & tables

Figure 1: Comparison of English (MMMU, MMMU-Pro), Japanese (JMMMU, JMMMU-Pro), and Korean (others) multimodal benchmarks. Each point is positioned by benchmark size (x-axis, log scale) and difficulty proxy (100 $-$ peak public score), with lighter colors indicating more recent releases. Shaded regions mark two common limitations: small size (left) and high peak model performance (bottom).

Figure 2: Examples of KMMMU questions. Examples include the original questions, associated images, English translations, and metadata such as visual modality, question format, and Korean-specific labels.

Benchmark	Size	Native	Expert	Multi-disc.	Diverse vis.	Local subset
K-MMBench	4.3k	✗	✗	✗	✗	✗
K-SEED	3.0k	✗	✗	✗	✗	✗
K-MMStar	1.5k	✗	✗	✗	✗	✗
K-DTCBench	0.2k	✓	✗	✗	$\triangle$	✗
K-Viscuit	0.7k	✓	✗	✗	✗	$\triangle$
KoFFVQA	0.3k	✓	✗	✗	✗	$\triangle$

Table 1: Comparison with Korean multimodal benchmarks. Native denotes items sourced without translation, and Local subset denotes a dedicated subset for Korean-localized knowledge (KMMMU-Local in our case).

\triangle

denotes partial coverage.

Benchmark	Year	Size	Filtering Method
HellaSwag Commonsense NLI	2019	70K	Adversarially selected hard generated distractors
MMLU-Pro Text-only exams	2024	12K	Remove items solved by $>4$ of 8 LLMs
SimpleQA Short-form factuality	2024	4.3K	Collect questions adversarially against GPT-4 responses
MMMU-Pro Multimodal exams	2025	3.5K	Remove items solved by $\geq 3$ of 4 text-only LLMs
KMMLU-Pro Text-only Korean exams	2025	2.8K	Filter items solved by $\geq 4$ of 7 small LLMs
HLE Expert-level knowledge	2025	2.5K	Frontier-LLM difficulty check before expert review

Table 2: Comparison of benchmark construction strategies that preserve evaluation difficulty through adversarial selection, model-aware filtering, or frontier-model difficulty checks.

Figure 3: Embedding-based distributional comparison after selection. Question text embeddings from the original 68k corpus, the KMMMU , and the KMMMU-Adv are projected using PCA followed by 3D UMAP.

Figure 4: Discipline-wise visual modality composition of KMMMU. Stacked bars show the number of questions for each visual modality in each discipline, with total counts shown beneath the labels. Scatter points indicate Korean-specific items overlaid on the corresponding discipline–modality segments, and jittered randomly.

Open-Weight Multilingual Non-Reasoning Models
Model	Arts & Design	Business & Public	CS & IT	Engineering	General	Law & Ethics	Math & Stats	Natural Sciences	Social Sciences	Overall Acc.
Gemma-3-4B-IT	29.96 1.58	26.51 1.58	18.83 2.40	23.19 0.66	19.24 1.67	22.76 2.27	17.03 1.81	15.25 1.13	19.29 2.32	20.49 0.41
Gemma-3-12B-IT	28.27 1.58	25.85 2.45	20.99 1.57	22.95 0.53	14.09 2.03	31.09 1.98	16.74 2.94	18.49 1.29	22.83 2.25	21.59 0.55
Gemma-3-27B-IT	23.63 6.72	31.02 2.85	25.41 1.91	22.48 0.33	17.62 0.38	30.77 3.93	23.35 0.95	21.74 1.35	26.64 3.04	23.78 0.88
Qwen3-VL-2B-IT	24.05 1.03	22.88 1.33	19.55 0.52	13.75 0.65	8.67 2.03	23.08 2.08	8.52 1.36	9.61 0.60	15.75 0.85	14.24 0.34
Qwen3-VL-4B-IT	25.32 1.03	34.43 1.38	26.75 1.62	16.73 0.79	13.28 1.38	24.36 2.52	13.22 0.95	17.68 0.70	21.78 0.67	19.88 0.42

Table 3: Accuracy (%) on the KMMMU by disciplines. Overall accuracy is averaged across disciplines. Mean accuracy is reported in percentage, with standard deviation shown as a subscript. Best in each model group is shown in bold .

Figure 5: Human alignment of the LLM scorer. Each x-axis label indicates the source model of the sampled responses. We independently sample and re-annotate 100 outputs per model. H-H Agr. denotes inter-annotator agreement between human annotators, and LLM-H Agr. denotes agreement between human annotations and the LLM scorer. The LLM scorer is Gemini-2.5-Flash , the same judge used throughout our evaluation pipeline.

Model	Arts & Design	Business & Public	CS & IT	Engineering	General	Law & Ethics	Math & Stats	Natural Sciences	Social Sciences	Overall Acc.
Claude-Opus-4.5	28.00 6.53	32.38 3.56	25.73 2.19	23.12 2.31	21.93 3.28	24.07 6.93	19.54 3.25	22.94 1.22	35.19 4.72	24.51 0.54
Claude-Sonnet-4.5	16.00 0.00	24.76 5.39	14.62 4.60	18.58 2.04	12.28 5.41	37.04 5.24	17.24 5.63	18.40 0.81	19.44 2.27	18.55 1.31
Gemini-3-Flash	48.00 0.00	42.86 7.00	48.54 1.65	46.52 1.32	21.93 3.28	46.30 9.44	37.93 0.00	46.10 1.84	56.48 3.46	45.14 0.98
Gemini-3-Pro	50.67 4.99	61.90 1.35	57.31 0.83	54.04 2.28	27.19 1.24	46.30 5.24	48.28 2.82	52.60 3.23	58.33 2.27	52.42 0.94
Mistral-Large-3-675B-IT	20.00 3.27	16.19 1.35	15.79 3.79	15.32 2.28	4.39 1.24	16.67 4.54	6.90 2.82	16.23 0.53	21.30 3.46	15.15 1.17
GPT-5	20.00 3.27	26.67 3.56	26.90 2.19	30.35 1.40	24.56 3.28	18.52 2.62	34.48 2.82	33.55 2.51	45.37 3.46	30.57 0.40

Table 4: Accuracy (%) on the KMMMU-Adv by disciplines. Overall accuracy is averaged across disciplines. Mean accuracy is reported in percentage, with standard deviation shown as a subscript. The best result is shown in bold .

Open-Weight Multilingual Models
Model	Non Korean–spec	Korean –spec	Raw gap	Controlled gap
Qwen3-VL-8B-IT	24.33	27.11	+2.78	+4.01
Qwen3-VL-30B-A3B-IT	30.16	22.00	-8.16	-5.97
Qwen3-VL-30B-A3B-Thinking	36.32	26.67	-9.65	-7.99
Qwen3-VL-32B-IT	36.56	26.22	-10.33	-8.09
Qwen3-VL-32B-Thinking	38.70	28.33	-10.37	-10.00

Table 5: Performance on KMMMU-Local subset. Raw gap is the accuracy difference between Korean-specific and non-Korean-specific questions; controlled gap is the discipline-controlled difference. Negative values indicate worse performance on Korean-specific questions. The largest positive gap is green shaded, and the largest negative gap is red shaded.

Model	Ko Acc.	En Acc.	$\Delta$	Ko/En-only	$p$
GPT-5	42.5	29.0	+13.5	33/6	$1.43{\times}10^{-5}$
Gemini-3-Flash	58.0	40.0	+18.0	38/2	$1.49{\times}10^{-9}$

Table 6: Cross-lingual ablation on 200 Korean-specific paired items. We translate the textual question and answer options into English. Ko/En-only denotes paired discordant cases answered correctly only in Korean or only in English.

p

is from an exact McNemar test.

Figure 6: Example of a Korean-specific regulatory category mismatch. Qwen3-VL-235B-A22B-IT reads the table correctly, but maps small vehicle to the wrong category and applies the wrong standard. This is a failure of institutional knowledge recall and lexical category matching, not OCR.

Figure 7: Reasoning gain by discipline and question type in Qwen3-VL-32B (IT vs. Thinking).

Figure 8: Annotation tool interface used for OCR verification. The tool displays the original PDF page on the left and the parsed text and images on the right, allowing annotators to correct OCR errors and validate image cropping in real time.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Filtering model	Input	Removed	Remaining
Initial	–	–	–	68,214
1	Phi-3.5-Vision-Instruct	68,214	18,309	49,905
2	InternVL-3.5-38B	49,905	8,390	41,515
3	Gemini-2.5-Flash-Lite	41,515	18,368	23,147
4	Gemini-2.5-Flash	23,147	19,681	3,466

Appendix

Table 7: Stage-wise retention during adversarial filtering. At each stage, questions answered correctly by the corresponding filtering model are removed from the candidate pool.

Figure 9: Data Card for a Korean-Specific Question. The figure aggregates the raw inputs and their translations. [Original Image] The original visual input containing a text-rich regulation box. [Original Question] The original question text in Korean. [Translation] English translations for both the visual context and the question. Correctly answering this question requires retrieving specific legal provisions regarding slope limits for soil extraction permits in South Korea, demonstrating the benchmark’s focus on localized expert knowledge.

Statistic	Count
Total Questions	3,466
KMMMU-Adv	627
Discipline Categories	9
Sub-discipline Categories	45
Visual Modality Types	9
Question Types	5

Appendix

Table 8: Dataset distribution outline. We report counts and percentages for key attributes such as in-image text and Korean-specific content.

Subject	Cnt	Subject	Cnt	Subject	Cnt
Physics	474	General Knowledge & Interdisciplinary	67	Economics	33
Civil & Structural Engineering	332	Industrial & Systems Engineering	64	Earth & Geological Sciences	33
Mechanical Engineering	223	Architecture & Urban Studies	57	Library, Archival & Information Science	30
Electrical & Electronics Engineering	221	Human Resources & Organizational Studies	56	Transportation & Logistics	29
Computer Science	164	Linguistics & Language Studies	54	Agriculture & Life Sciences	27
Statistics & Probability	149	Biology	43	Software & Programming	27

Appendix

Table 9: Distribution of sub-discipline categories in KMMMU .

Question Type	Count	%
Multiple Choice (Single Answer)	2,831	81.68
Multiple Choice (Multiple Answers)	346	9.98
Open Format (Numerical / Calculation)	207	5.98
Open Format (Text)	67	1.93
Essay (Descriptive)	15	0.43
Total	3466	100

Appendix

Table 10: Question type distribution.

Figure 10: Representative examples of fine-grained visual types in KMMMU. Before consolidation into the final macro-level visual modality categories, the dataset included diverse fine-grained visual types, such as specialized engineering diagrams, document-style text images, and South Korean geographic maps.

Discipline	Multiple Choice (Single Answer)	Multiple Choice (Multiple Answers)	Open Format (Numerical)	Open Format (Text)	Essay (Descriptive)
Engineering	1182	25	20	2	0
Natural Sciences	450	180	154	27	11
CS & IT	311	11	2	0	0
Business & Public	263	37	2	2	0
Social Sciences	193	55	2	4	0
Math & Stats	161	36	27	3	0

Appendix

Table 11: Dataset attributes by disciplines and answer type.

Figure 11: Per-dimension density comparison after adversarial filtering. Kernel density estimates over the three UMAP dimensions for the original 68k corpus, the KMMMU , and the KMMMU-Adv . The filtered subsets broadly retain the major density peaks and multimodal trends of the original distribution, although the KMMMU-Adv shows a somewhat larger deviation in Dimension 3.

Model family	Temperature	Top-p	Top-k
Gemma-3	1.0	0.95	64
Qwen3-VL-Instruct	0.7	0.8	20
Qwen3-VL-Thinking	1.0	0.95	20
Llama-4	0.6	0.9	–
HyperCLOVAX-SEED-Vision-3B	1.0	1.0	50
VARCO-VISION-2.0	1.0	1.0	–

Appendix

Table 12: Decoding settings used for model evaluation. Dashes indicate parameters that were not explicitly specified or not exposed by the corresponding provider interface.

Model	API identifier	Access	Released	Accessed
GPT-5	gpt-5	OpenAI API	2025-08-07	2025-12-29
GPT-5-mini	gpt-5-mini	OpenAI API	2025-08-07	2025-12-31
Claude-Opus-4.5	anthropic/claude-opus-4.5	OpenRouter	2025-11-24	2026-01-01
Claude-Sonnet-4.5	anthropic/claude-sonnet-4.5	OpenRouter	2025-09-29	2025-12-30
Gemini-3-Pro	gemini-3-pro-preview	Gemini API	2025-11-18	2025-12-31
Gemini-3-Flash	gemini-3-flash-preview	Gemini API	2025-12-17	2025-12-31

Appendix

Table 13: API identifiers and access information for proprietary models.

Figure 12: Discipline-wise visual modality composition of KMMMU-Adv subset. Stacked bars show the number of ques- tions for each visual modality in each discipline, with total counts shown beneath the labels. Scatter points indicate Korean-specific ( KMMMU-Local ) items overlaid on the corresponding discipline–modality segments. The KMMMU-Adv is concentrated in Engineering and Natural Sciences , similar to Full set.

Split	Korean Specific	Not Korean Specific	Ratio (%)
KMMMU	300	3166	8.65
KMMMU-Adv	76	551	12.12

Appendix

Table 14: Number of Korean-specific questions each in Full set and KMMMU-Adv .

Open-weight Multilingual Non-Reasoning Models
Model	Arts & Design	Business & Public	CS & IT	Engineering	General	Law & Ethics	Math & Stats	Natural Sciences	Social Sciences	Overall Acc.
Gemma-3-4B-IT	22.67 1.89	21.90 1.35	9.94 4.14	19.01 2.95	14.04 1.24	18.52 6.93	10.34 2.82	10.82 1.33	24.07 5.24	16.06 1.43
Gemma-3-12B-IT	14.67 3.77	8.57 4.04	11.70 4.60	15.32 1.84	6.14 3.28	22.22 0.00	5.75 4.30	8.87 1.70	12.04 7.29	12.01 0.72
Gemma-3-27B-IT	16.44 8.46	11.63 10.45	15.11 7.09	14.04 4.78	14.04 12.59	16.20 15.06	9.51 8.57	12.61 6.48	16.24 0.60	13.72 5.88
Llama-4-Scout-17B-16E-IT	13.33 3.77	9.52 1.35	9.36 2.19	17.73 0.53	10.53 2.15	20.37 2.62	6.90 2.82	11.26 2.72	19.44 4.54	13.98 0.53
Llama-4-Maverick-17B-128E-IT	17.33 3.77	13.33 1.35	20.47 1.65	16.74 1.06	10.53 2.15	22.22 4.54	17.24 4.88	14.29 1.91	21.30 3.46	16.37 1.08

Appendix

Table 15: Accuracy (%) on the KMMMU-Adv subset by disciplines in Open-Weight models. This table reports results recomputed by restricting full-set evaluation outputs to the adversarially filtered Adv subset. Overall accuracy is averaged across all disciplines. Mean accuracy is reported in percentage, with standard deviation shown as a subscript. The best result for each discipline and overall accuracy is shown in bold.

Model	$n$	Ko Acc.	En Acc.	$\Delta$	Both Correct	Ko-only	En-only	Both Wrong
GPT-5	200	42.5	29.0	+13.5	52	33	6	109
Gemini-3-Flash	200	58.0	40.0	+18.0	78	38	2	82

Appendix

Table 16: Detailed paired outcomes for the cross-lingual ablation on Korean-specific items. We evaluate 200 Korean-specific paired items in Korean and English. The English condition translates the textual question and answer options while keeping the other conditions are fixed. Accuracies are percentages.

\Delta

denotes Korean accuracy minus English accuracy. Both Correct and Both Wrong indicate paired cases where the model is correct or incorrect in both language conditions. Ko-only and En-only indicate paired discordant cases where the model answers correctly only in Korean or only in English, respectively. Exact McNemar tests over the paired predictions show significant paired differences for both GPT-5 (

p=1.43{\times}10^{-5}

) and Gemini-3-Flash (

p=1.49{\times}10^{-9}

Discipline	$n_{\text{KS}}$	$n_{\text{Non}}$	Acc. ${}_{\text{KS}}$	Acc. ${}_{\text{Non}}$	Gap	95% CI
Business & Public	76	227	23.2	69.5	$-46.2$	$[-55.4,-36.6]$
Law & Ethics	82	22	27.6	65.2	$-37.5$	$[-46.7,-28.3]$
Arts & Design †	1	78	0.0	25.6	$-25.6$	–
Natural Sciences	31	792	17.2	42.4	$-25.1$	$[-34.6,-14.1]$
CS & IT	16	308	14.6	38.6	$-24.1$	$[-38.4,-6.6]$
General	28	95	21.4	25.6	$-4.2$	$[-17.9,10.5]$

Appendix

Table 17: Discipline-level Korean-specific gaps of Qwen3-VL-235B-A22B-IT. KS and Non denote Korean-specific and non-Korean-specific items. Accuracy and gaps are in percentage points. † denotes disciplines with fewer than ten Korean-specific items.

Figure 13: Annotation interface for manual validation of LLM scorer outputs. For each sample, annotators review the question, image, gold answer, model response, and parsed answer, and record parsing consistency, correctness judgments, metadata consistency, and optional comments.

Model	Human agr.	No-answer rate	Overall judge–human agr.		No-answer acc.		Answered acc.
Model	Human agr.	No-answer rate	Parsed	Response	Parsed	Response	Parsed	Response
Qwen3-2B-IT	0.99	0.51	0.980	0.980	0.96	0.96	1.000	1.000
Qwen3-30B-IT	0.97	0.18	0.959	0.928	0.94	0.72	0.962	0.975
Qwen3-32B-Thinking	0.93	0.31	0.925	0.495	1.00	0.20	0.889	0.734
Qwen3-30B-Thinking	0.91	0.22	0.880	0.326	0.95	0.15	0.729	0.347
VARCO2-14B	0.97	0.04	0.948	0.929	1.00	1.00	0.946	0.926

Appendix

Table 18: Human agreement and judge–human alignment across six model runs. “Human agr.” denotes pairwise human agreement. “Parsed” and “Response” report judge–human alignment under parsed-answer-based and full-response-based evaluation, respectively. The no-answer rate denotes the proportion of outputs without a complete final answer. The no-answer and answered columns further break down judge–human accuracy by response completeness.

Figure 14: Accuracy by visual modality for Qwen3-VL-32B-IT and Qwen3-VL-32B-Thinking. Performance remains broadly similar across visual modality categories, suggesting that explicit reasoning does not systematically change raw visual evidence extraction. The main differences appear to arise after evidence extraction, such as in task framing, constraint tracking, and answer finalization.

Figure 15: Rigid conceptual framing in a Natural Sciences reversal. Qwen3-VL-32B-IT correctly applies the relevant conductivity criterion, whereas Qwen3-VL-32B-Thinking overcommits to an overly rigid band-gap-based schema and rejects the crucial statement about the Fermi level in the conduction band.

Figure 16: Structural misinterpretation in an Engineering reversal (Fault Tree analysis). Qwen3-VL-32B-IT partially corrects an early gate-level misinterpretation, whereas Qwen3-VL-32B-Thinking persists with an incorrect top-level gate reading and derives the wrong recovery point.

Figure 17: Exact architectural category misclassification in Arts & Design . The model recognizes the overall structure of the plan, but fails to map it to the correct standardized architectural category. Instead, it overcommits to an orthogonal-plan interpretation and supports it with a plausible but incorrect villa association.

Figure 18: Few-shot symbolic induction failure in General . A Shanghainese-language item requiring the model to infer a latent mapping from a small set of diagram–expression pairs and apply it to new cases. Although Qwen3-VL-235B-A22B-IT produces a detailed step-by-step analysis, it fails to recover the full correspondence system and instead relies on partial surface analogies, leading to a plausible but incorrect answer.

Figure 19: Exact standards and rule-criterion misapplication in General . An item testing the official Romanization rule for hyphen use. The model gives a broadly reasonable explanation of the rule, but selects the wrong option because it applies an approximate plausibility-based criterion rather than the exact condition required by the formal standard.

Model	Original	Text-only
Gemini-3-Flash	45.15 0.98	20.43 0.48
GPT-5-Mini	21.32 0.33	9.75 0.85

Appendix

Table 19: Ablation results for image dependency. We compare the average accuracy and standard deviation (

Acc\textsubscript{std}

) of models on the original multimodal dataset versus the text-only baseline.

Source	Model	Exactness NH	Exactness H	Refusal NH	Refusal H	Hallucination NH	Hallucination H
PSAT	Gemini-3-Flash	14.20	13.14	17.53	24.70	76.10	68.13
	Gemini-3-Pro	0.34	0.46	95.60	94.78	3.20	3.21
	GPT-5-Mini	0.00	0.00	99.60	100.00	0.00	0.00
NCS	Gemini-3-Flash	10.05	6.54	29.08	50.20	50.20	35.57
	Gemini-3-Pro	0.18	0.04	99.21	100.00	0.40	0.00
	GPT-5-Mini	0.65	0.45	95.65	100.00	1.19	1.19

Appendix

Table 20: Prefix-completion analysis for potential data contamination. Models are given the first 35% of each question together with the associated image, and asked to generate the remaining continuation. Exactness denotes a judge-assigned 0–100 faithfulness rating with respect to the reference continuation. Refusal and Hallucination denote judge-labeled failure modes, reported as percentages, under the No-hint (

NH

) and Hinted (

H

) settings.

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 770 papers

Oct 15, 2026

KBS: A Korean Benchmark Suite for Foundation Model Evaluation

We present a Korean Benchmark Suite (KBS) comprising Ko-ARC, Ko-GSM8K, Ko-EQ-Bench, Ko-WinoGrande, Ko-LAMBADA, and Ko-IFEval, covering six capabilities across 9,396 items. The suite combines review, localization, direct Korean construction, and redesign for Korean, avoiding a translation-only approach to benchmark adaptation. We evaluate Korean-developed models, multilingual open models, and closed API models under protocols that account for model access, prompt format, and scoring differences. Results show that Korean benchmark scores depend on task construction, scoring rules, prompting, and model post-training. These findings suggest that Korean evaluation should report both final scores and the settings used to obtain them.

Yeonkyoung So, Jongmin Kim, Sungmok Jung +13

Graduate School of Data Science, Seoul National University · Dept. of Computer Science and Engineering, Seoul National University

LLM Evaluation Multilingual Language Model Evaluation

Ask PDF

Apr 27, 2026

K-MetBench: A Multi-Dimensional Benchmark for Fine-Grained Evaluation of Expert Reasoning, Locality, and Multimodality in Meteorology

The development of practical (multimodal) large language model assistants for Korean weather forecasters is hindered by the absence of a multidimensional, expert-level evaluation framework grounded in authoritative sources. To address this, we introduce K-MetBench, a diagnostic benchmark grounded in national qualification exams. It exposes critical gaps across four dimensions: expert visual reasoning of charts, logical validity via expert-verified rationales, Korean-specific geo-cultural comprehension, and fine-grained domain analysis. Our evaluation of 55 models reveals a profound modality gap in interpreting specialized diagrams and a reasoning gap where models hallucinate logic despite correct predictions. Crucially, Korean models outperform significantly larger global models in local contexts, demonstrating that parameter scaling alone cannot resolve cultural dependencies. K-MetBench serves as a roadmap for developing reliable, culturally aware expert AI agents. The dataset is available at https://huggingface.co/datasets/soyeonbot/K-MetBench .

Soyeon Kim, Cheongwoong Kang, Myeongjin Lee +3

KAIST, INEEJI · Seongnam, Korea · KAIST +2

LLM Evaluation Cultural Bias in Language Models Multimodal Model Evaluation

Ask PDF

Aug 19, 2025

VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark

We introduce VMMU, a Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark designed to evaluate how vision-language models (VLMs) interpret and reason over visual and textual information beyond English. VMMU consists of 2.5k multimodal questions across 7 tasks, covering a diverse range of problem contexts, including STEM problem solving, data interpretation, rule-governed visual reasoning, and abstract visual reasoning. All questions require genuine multimodal integration, rather than reliance on text-only cues or OCR-based shortcuts. We evaluate a diverse set of state-of-the-art proprietary and open-source VLMs on VMMU. Despite strong Vietnamese OCR performance, proprietary models achieve only 66% mean accuracy. Further analysis shows that the primary source of failure is not OCR, but instead multimodal grounding and reasoning over text and visual evidence. Code and data are available at https://vmmu-bench.github.io/

Vy Tuong Dang, An Vo, Emilio Villa-Cueva +4

KAIST · MBZUAI

VLM Evaluation Multimodal Reasoning Multilingual VLM Evaluation Vision-Language Grounding Visually Grounded Reasoning Multimodal Foundation Models+2

Ask PDF

Abstract

Multimodal Model Evaluation

Explore similar work

KBS: A Korean Benchmark Suite for Foundation Model Evaluation

K-MetBench: A Multi-Dimensional Benchmark for Fine-Grained Evaluation of Expert Reasoning, Locality, and Multimodality in Meteorology

VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark