cs.CLMar 18, 2026

KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context

Authors: Nahyun Lee, Guijin Son, Hyunwoo Ko, Chanyoung Kim, JunYoung An, Kyubeen Han, Il-Youp Kwak

Organizations: Chung-Ang University · HAE-RAE · Seoul National University · OnelineAI · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multimodal evaluation is strengthened when benchmarks reflect the languages, institutions, and disciplinary conventions that shape expert visual reasoning. We introduce KMMMU, a native Korean benchmark built from Korean-native examination materials. KMMMU contains 3,466 questions spanning nine disciplines and nine visual modality categories. For targeted evaluation on challenging questions and Korean-specific knowledge, we additionally provide KMMMU-Adv and KMMMU-Local, respectively. Experiments show that the strongest evaluated open-weight model reaches 42.05% on the KMMMU, while the best evaluated proprietary model achieves 52.42% on the KMMMU-Adv. Performance varies substantially across disciplines and answer formats, with Korean-specific questions showing gaps of up to 13.43%. Error analysis suggests that inspected failures often involve convention-to-label mapping, few-shot symbolic induction, localized knowledge recall, and domain-specific standards, not reasoning depth alone. KMMMU provides a testbed for native Korean expert-level multimodal understanding beyond English-centric or translation-based evaluation.

Figures & tables

Figure 1: Comparison of English (MMMU, MMMU-Pro), Japanese (JMMMU, JMMMU-Pro), and Korean (others) multimodal benchmarks. Each point is positioned by benchmark size (x-axis, log scale) and difficulty proxy (100 −- peak public score), with lighter colors indicating more recent releases. Shaded regions mark two common limitations: small size (left) and high peak model performance (bottom).
Figure 2: Examples of KMMMU questions. Examples include the original questions, associated images, English translations, and metadata such as visual modality, question format, and Korean-specific labels.
BenchmarkSizeNativeExpertMulti-disc.Diverse vis.Local subset
K-MMBench4.3k✗✗✗✗✗
K-SEED3.0k✗✗✗✗✗
K-MMStar1.5k✗✗✗✗✗
K-DTCBench0.2k✓✗✗△\triangle✗
K-Viscuit0.7k✓✗✗✗△\triangle
KoFFVQA0.3k✓✗✗✗△\triangle
Table 1: Comparison with Korean multimodal benchmarks. Native denotes items sourced without translation, and Local subset denotes a dedicated subset for Korean-localized knowledge (KMMMU-Local in our case). △\triangle denotes partial coverage.
BenchmarkYearSizeFiltering Method
HellaSwag Commonsense NLI201970KAdversarially selected hard generated distractors
MMLU-Pro Text-only exams202412KRemove items solved by >4>4 of 8 LLMs
SimpleQA Short-form factuality20244.3KCollect questions adversarially against GPT-4 responses
MMMU-Pro Multimodal exams20253.5KRemove items solved by ≥3\geq 3 of 4 text-only LLMs
KMMLU-Pro Text-only Korean exams20252.8KFilter items solved by ≥4\geq 4 of 7 small LLMs
HLE Expert-level knowledge20252.5KFrontier-LLM difficulty check before expert review
Table 2: Comparison of benchmark construction strategies that preserve evaluation difficulty through adversarial selection, model-aware filtering, or frontier-model difficulty checks.
Figure 3: Embedding-based distributional comparison after selection. Question text embeddings from the original 68k corpus, the KMMMU , and the KMMMU-Adv are projected using PCA followed by 3D UMAP.
Figure 4: Discipline-wise visual modality composition of KMMMU. Stacked bars show the number of questions for each visual modality in each discipline, with total counts shown beneath the labels. Scatter points indicate Korean-specific items overlaid on the corresponding discipline–modality segments, and jittered randomly.
ModelArts & DesignBusiness & PublicCS & ITEngineeringGeneralLaw & EthicsMath & StatsNatural SciencesSocial SciencesOverall Acc.
Open-Weight Multilingual Non-Reasoning Models
Gemma-3-4B-IT29.96 1.5826.51 1.5818.83 2.4023.19 0.6619.24 1.6722.76 2.2717.03 1.8115.25 1.1319.29 2.3220.49 0.41
Gemma-3-12B-IT28.27 1.5825.85 2.4520.99 1.5722.95 0.5314.09 2.0331.09 1.9816.74 2.9418.49 1.2922.83 2.2521.59 0.55
Gemma-3-27B-IT23.63 6.7231.02 2.8525.41 1.9122.48 0.3317.62 0.3830.77 3.9323.35 0.9521.74 1.3526.64 3.0423.78 0.88
Qwen3-VL-2B-IT24.05 1.0322.88 1.3319.55 0.5213.75 0.658.67 2.0323.08 2.088.52 1.369.61 0.6015.75 0.8514.24 0.34
Qwen3-VL-4B-IT25.32 1.0334.43 1.3826.75 1.6216.73 0.7913.28 1.3824.36 2.5213.22 0.9517.68 0.7021.78 0.6719.88 0.42
Table 3: Accuracy (%) on the KMMMU by disciplines. Overall accuracy is averaged across disciplines. Mean accuracy is reported in percentage, with standard deviation shown as a subscript. Best in each model group is shown in bold .
Figure 5: Human alignment of the LLM scorer. Each x-axis label indicates the source model of the sampled responses. We independently sample and re-annotate 100 outputs per model. H-H Agr. denotes inter-annotator agreement between human annotators, and LLM-H Agr. denotes agreement between human annotations and the LLM scorer. The LLM scorer is Gemini-2.5-Flash , the same judge used throughout our evaluation pipeline.
ModelArts & DesignBusiness & PublicCS & ITEngineeringGeneralLaw & EthicsMath & StatsNatural SciencesSocial SciencesOverall Acc.
Claude-Opus-4.528.00 6.5332.38 3.5625.73 2.1923.12 2.3121.93 3.2824.07 6.9319.54 3.2522.94 1.2235.19 4.7224.51 0.54
Claude-Sonnet-4.516.00 0.0024.76 5.3914.62 4.6018.58 2.0412.28 5.4137.04 5.2417.24 5.6318.40 0.8119.44 2.2718.55 1.31
Gemini-3-Flash48.00 0.0042.86 7.0048.54 1.6546.52 1.3221.93 3.2846.30 9.4437.93 0.0046.10 1.8456.48 3.4645.14 0.98
Gemini-3-Pro50.67 4.9961.90 1.3557.31 0.8354.04 2.2827.19 1.2446.30 5.2448.28 2.8252.60 3.2358.33 2.2752.42 0.94
Mistral-Large-3-675B-IT20.00 3.2716.19 1.3515.79 3.7915.32 2.284.39 1.2416.67 4.546.90 2.8216.23 0.5321.30 3.4615.15 1.17
GPT-520.00 3.2726.67 3.5626.90 2.1930.35 1.4024.56 3.2818.52 2.6234.48 2.8233.55 2.5145.37 3.4630.57 0.40
Table 4: Accuracy (%) on the KMMMU-Adv by disciplines. Overall accuracy is averaged across disciplines. Mean accuracy is reported in percentage, with standard deviation shown as a subscript. The best result is shown in bold .
ModelNon Korean–specKorean –specRaw gapControlled gap
Open-Weight Multilingual Models
Qwen3-VL-8B-IT24.3327.11+2.78+4.01
Qwen3-VL-30B-A3B-IT30.1622.00-8.16-5.97
Qwen3-VL-30B-A3B-Thinking36.3226.67-9.65-7.99
Qwen3-VL-32B-IT36.5626.22-10.33-8.09
Qwen3-VL-32B-Thinking38.7028.33-10.37-10.00
Table 5: Performance on KMMMU-Local subset. Raw gap is the accuracy difference between Korean-specific and non-Korean-specific questions; controlled gap is the discipline-controlled difference. Negative values indicate worse performance on Korean-specific questions. The largest positive gap is green shaded, and the largest negative gap is red shaded.
ModelKo Acc.En Acc.Δ\DeltaKo/En-onlypp
GPT-542.529.0+13.533/61.43×10−51.43{\times}10^{-5}
Gemini-3-Flash58.040.0+18.038/21.49×10−91.49{\times}10^{-9}
Table 6: Cross-lingual ablation on 200 Korean-specific paired items. We translate the textual question and answer options into English. Ko/En-only denotes paired discordant cases answered correctly only in Korean or only in English. pp is from an exact McNemar test.
Figure 6: Example of a Korean-specific regulatory category mismatch. Qwen3-VL-235B-A22B-IT reads the table correctly, but maps small vehicle to the wrong category and applies the wrong standard. This is a failure of institutional knowledge recall and lexical category matching, not OCR.
Figure 7: Reasoning gain by discipline and question type in Qwen3-VL-32B (IT vs. Thinking).
Figure 8: Annotation tool interface used for OCR verification. The tool displays the original PDF page on the left and the parsed text and images on the right, allowing annotators to correct OCR errors and validate image cropping in real time.
Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

StageFiltering modelInputRemovedRemaining
Initial–––68,214
1Phi-3.5-Vision-Instruct68,21418,30949,905
2InternVL-3.5-38B49,9058,39041,515
3Gemini-2.5-Flash-Lite41,51518,36823,147
4Gemini-2.5-Flash23,14719,6813,466
Appendix
Table 7: Stage-wise retention during adversarial filtering. At each stage, questions answered correctly by the corresponding filtering model are removed from the candidate pool.
Figure 9: Data Card for a Korean-Specific Question. The figure aggregates the raw inputs and their translations. [Original Image] The original visual input containing a text-rich regulation box. [Original Question] The original question text in Korean. [Translation] English translations for both the visual context and the question. Correctly answering this question requires retrieving specific legal provisions regarding slope limits for soil extraction permits in South Korea, demonstrating the benchmark’s focus on localized expert knowledge.
StatisticCount
Total Questions3,466
KMMMU-Adv627
Discipline Categories9
Sub-discipline Categories45
Visual Modality Types9
Question Types5
Appendix
Table 8: Dataset distribution outline. We report counts and percentages for key attributes such as in-image text and Korean-specific content.
SubjectCntSubjectCntSubjectCnt
Physics474General Knowledge & Interdisciplinary67Economics33
Civil & Structural Engineering332Industrial & Systems Engineering64Earth & Geological Sciences33
Mechanical Engineering223Architecture & Urban Studies57Library, Archival & Information Science30
Electrical & Electronics Engineering221Human Resources & Organizational Studies56Transportation & Logistics29
Computer Science164Linguistics & Language Studies54Agriculture & Life Sciences27
Statistics & Probability149Biology43Software & Programming27
Appendix
Table 9: Distribution of sub-discipline categories in KMMMU .
Question TypeCount%
Multiple Choice (Single Answer)2,83181.68
Multiple Choice (Multiple Answers)3469.98
Open Format (Numerical / Calculation)2075.98
Open Format (Text)671.93
Essay (Descriptive)150.43
Total3466100
Appendix
Table 10: Question type distribution.
Figure 10: Representative examples of fine-grained visual types in KMMMU. Before consolidation into the final macro-level visual modality categories, the dataset included diverse fine-grained visual types, such as specialized engineering diagrams, document-style text images, and South Korean geographic maps.
DisciplineMultiple Choice (Single Answer)Multiple Choice (Multiple Answers)Open Format (Numerical)Open Format (Text)Essay (Descriptive)
Engineering1182252020
Natural Sciences4501801542711
CS & IT31111200
Business & Public26337220
Social Sciences19355240
Math & Stats161362730
Appendix
Table 11: Dataset attributes by disciplines and answer type.
Figure 11: Per-dimension density comparison after adversarial filtering. Kernel density estimates over the three UMAP dimensions for the original 68k corpus, the KMMMU , and the KMMMU-Adv . The filtered subsets broadly retain the major density peaks and multimodal trends of the original distribution, although the KMMMU-Adv shows a somewhat larger deviation in Dimension 3.
Model familyTemperatureTop-pTop-k
Gemma-31.00.9564
Qwen3-VL-Instruct0.70.820
Qwen3-VL-Thinking1.00.9520
Llama-40.60.9–
HyperCLOVAX-SEED-Vision-3B1.01.050
VARCO-VISION-2.01.01.0–
Appendix
Table 12: Decoding settings used for model evaluation. Dashes indicate parameters that were not explicitly specified or not exposed by the corresponding provider interface.
ModelAPI identifierAccessReleasedAccessed
GPT-5gpt-5OpenAI API2025-08-072025-12-29
GPT-5-minigpt-5-miniOpenAI API2025-08-072025-12-31
Claude-Opus-4.5anthropic/claude-opus-4.5OpenRouter2025-11-242026-01-01
Claude-Sonnet-4.5anthropic/claude-sonnet-4.5OpenRouter2025-09-292025-12-30
Gemini-3-Progemini-3-pro-previewGemini API2025-11-182025-12-31
Gemini-3-Flashgemini-3-flash-previewGemini API2025-12-172025-12-31
Appendix
Table 13: API identifiers and access information for proprietary models.
Figure 12: Discipline-wise visual modality composition of KMMMU-Adv subset. Stacked bars show the number of ques- tions for each visual modality in each discipline, with total counts shown beneath the labels. Scatter points indicate Korean-specific ( KMMMU-Local ) items overlaid on the corresponding discipline–modality segments. The KMMMU-Adv is concentrated in Engineering and Natural Sciences , similar to Full set.
SplitKorean SpecificNot Korean SpecificRatio (%)
KMMMU30031668.65
KMMMU-Adv7655112.12
Appendix
Table 14: Number of Korean-specific questions each in Full set and KMMMU-Adv .
ModelArts & DesignBusiness & PublicCS & ITEngineeringGeneralLaw & EthicsMath & StatsNatural SciencesSocial SciencesOverall Acc.
Open-weight Multilingual Non-Reasoning Models
Gemma-3-4B-IT22.67 1.8921.90 1.359.94 4.1419.01 2.9514.04 1.2418.52 6.9310.34 2.8210.82 1.3324.07 5.2416.06 1.43
Gemma-3-12B-IT14.67 3.778.57 4.0411.70 4.6015.32 1.846.14 3.2822.22 0.005.75 4.308.87 1.7012.04 7.2912.01 0.72
Gemma-3-27B-IT16.44 8.4611.63 10.4515.11 7.0914.04 4.7814.04 12.5916.20 15.069.51 8.5712.61 6.4816.24 0.6013.72 5.88
Llama-4-Scout-17B-16E-IT13.33 3.779.52 1.359.36 2.1917.73 0.5310.53 2.1520.37 2.626.90 2.8211.26 2.7219.44 4.5413.98 0.53
Llama-4-Maverick-17B-128E-IT17.33 3.7713.33 1.3520.47 1.6516.74 1.0610.53 2.1522.22 4.5417.24 4.8814.29 1.9121.30 3.4616.37 1.08
Appendix
Table 15: Accuracy (%) on the KMMMU-Adv subset by disciplines in Open-Weight models. This table reports results recomputed by restricting full-set evaluation outputs to the adversarially filtered Adv subset. Overall accuracy is averaged across all disciplines. Mean accuracy is reported in percentage, with standard deviation shown as a subscript. The best result for each discipline and overall accuracy is shown in bold.
ModelnnKo Acc.En Acc.Δ\DeltaBoth CorrectKo-onlyEn-onlyBoth Wrong
GPT-520042.529.0+13.552336109
Gemini-3-Flash20058.040.0+18.07838282
Appendix
Table 16: Detailed paired outcomes for the cross-lingual ablation on Korean-specific items. We evaluate 200 Korean-specific paired items in Korean and English. The English condition translates the textual question and answer options while keeping the other conditions are fixed. Accuracies are percentages. Δ\Delta denotes Korean accuracy minus English accuracy. Both Correct and Both Wrong indicate paired cases where the model is correct or incorrect in both language conditions. Ko-only and En-only indicate paired discordant cases where the model answers correctly only in Korean or only in English, respectively. Exact McNemar tests over the paired predictions show significant paired differences for both GPT-5 ( p=1.43×10−5p=1.43{\times}10^{-5} ) and Gemini-3-Flash ( p=1.49×10−9p=1.49{\times}10^{-9} ).
DisciplinenKSn_{\text{KS}}nNonn_{\text{Non}}Acc. KS{}_{\text{KS}}Acc. Non{}_{\text{Non}}Gap95% CI
Business & Public7622723.269.5−46.2-46.2[−55.4,−36.6][-55.4,-36.6]
Law & Ethics822227.665.2−37.5-37.5[−46.7,−28.3][-46.7,-28.3]
Arts & Design †1780.025.6−25.6-25.6–
Natural Sciences3179217.242.4−25.1-25.1[−34.6,−14.1][-34.6,-14.1]
CS & IT1630814.638.6−24.1-24.1[−38.4,−6.6][-38.4,-6.6]
General289521.425.6−4.2-4.2[−17.9,10.5][-17.9,10.5]
Appendix
Table 17: Discipline-level Korean-specific gaps of Qwen3-VL-235B-A22B-IT. KS and Non denote Korean-specific and non-Korean-specific items. Accuracy and gaps are in percentage points. † denotes disciplines with fewer than ten Korean-specific items.
Figure 13: Annotation interface for manual validation of LLM scorer outputs. For each sample, annotators review the question, image, gold answer, model response, and parsed answer, and record parsing consistency, correctness judgments, metadata consistency, and optional comments.
ModelHuman agr.No-answer rateOverall judge–human agr.No-answer acc.Answered acc.
ParsedResponseParsedResponseParsedResponse
Qwen3-2B-IT0.990.510.9800.9800.960.961.0001.000
Qwen3-30B-IT0.970.180.9590.9280.940.720.9620.975
Qwen3-32B-Thinking0.930.310.9250.4951.000.200.8890.734
Qwen3-30B-Thinking0.910.220.8800.3260.950.150.7290.347
VARCO2-14B0.970.040.9480.9291.001.000.9460.926
Appendix
Table 18: Human agreement and judge–human alignment across six model runs. “Human agr.” denotes pairwise human agreement. “Parsed” and “Response” report judge–human alignment under parsed-answer-based and full-response-based evaluation, respectively. The no-answer rate denotes the proportion of outputs without a complete final answer. The no-answer and answered columns further break down judge–human accuracy by response completeness.
Figure 14: Accuracy by visual modality for Qwen3-VL-32B-IT and Qwen3-VL-32B-Thinking. Performance remains broadly similar across visual modality categories, suggesting that explicit reasoning does not systematically change raw visual evidence extraction. The main differences appear to arise after evidence extraction, such as in task framing, constraint tracking, and answer finalization.
Figure 15: Rigid conceptual framing in a Natural Sciences reversal. Qwen3-VL-32B-IT correctly applies the relevant conductivity criterion, whereas Qwen3-VL-32B-Thinking overcommits to an overly rigid band-gap-based schema and rejects the crucial statement about the Fermi level in the conduction band.
Figure 16: Structural misinterpretation in an Engineering reversal (Fault Tree analysis). Qwen3-VL-32B-IT partially corrects an early gate-level misinterpretation, whereas Qwen3-VL-32B-Thinking persists with an incorrect top-level gate reading and derives the wrong recovery point.
Figure 17: Exact architectural category misclassification in Arts & Design . The model recognizes the overall structure of the plan, but fails to map it to the correct standardized architectural category. Instead, it overcommits to an orthogonal-plan interpretation and supports it with a plausible but incorrect villa association.
Figure 18: Few-shot symbolic induction failure in General . A Shanghainese-language item requiring the model to infer a latent mapping from a small set of diagram–expression pairs and apply it to new cases. Although Qwen3-VL-235B-A22B-IT produces a detailed step-by-step analysis, it fails to recover the full correspondence system and instead relies on partial surface analogies, leading to a plausible but incorrect answer.
Figure 19: Exact standards and rule-criterion misapplication in General . An item testing the official Romanization rule for hyphen use. The model gives a broadly reasonable explanation of the rule, but selects the wrong option because it applies an approximate plausibility-based criterion rather than the exact condition required by the formal standard.
ModelOriginalText-only
Gemini-3-Flash45.15 0.9820.43 0.48
GPT-5-Mini21.32 0.339.75 0.85
Appendix
Table 19: Ablation results for image dependency. We compare the average accuracy and standard deviation ( Acc\textsubscriptstdAcc\textsubscript{std} ) of models on the original multimodal dataset versus the text-only baseline.
SourceModelExactness NHExactness HRefusal NHRefusal HHallucination NHHallucination H
PSATGemini-3-Flash14.2013.1417.5324.7076.1068.13
 Gemini-3-Pro0.340.4695.6094.783.203.21
 GPT-5-Mini0.000.0099.60100.000.000.00
NCSGemini-3-Flash10.056.5429.0850.2050.2035.57
 Gemini-3-Pro0.180.0499.21100.000.400.00
 GPT-5-Mini0.650.4595.65100.001.191.19
Appendix
Table 20: Prefix-completion analysis for potential data contamination. Models are given the first 35% of each question together with the associated image, and asked to generate the remaining continuation. Exactness denotes a judge-assigned 0–100 faithfulness rating with respect to the reference continuation. Refusal and Hallucination denote judge-labeled failure modes, reported as percentages, under the No-hint ( NHNH ) and Hinted ( HH ) settings.
More in this topic

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 770 papers

Explore similar work

CardsList

  1. KBS: A Korean Benchmark Suite for Foundation Model Evaluation

    Oct 15, 2026Yeonkyoung So, Jongmin Kim, Sungmok Jung +13LLM EvaluationMultilingual Language Model Evaluation

    Graduate School of Data Science, Seoul National University · Dept. of Computer Science and Engineering, Seoul National University

  2. K-MetBench: A Multi-Dimensional Benchmark for Fine-Grained Evaluation of Expert Reasoning, Locality, and Multimodality in Meteorology

    Apr 27, 2026Soyeon Kim, Cheongwoong Kang, Myeongjin Lee +3LLM EvaluationCultural Bias in Language Models

    KAIST, INEEJI · Seongnam, Korea · KAIST +2

  3. VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark

    Aug 19, 2025Vy Tuong Dang, An Vo, Emilio Villa-Cueva +4VLM EvaluationMultimodal Reasoning

    KAIST · MBZUAI