EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams

Jaeseong Kim; Chaehwan Lim; Sang Hyun Gil; Suan Lee

cs.CVMar 28, 2026

EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams

Authors: Jaeseong Kim, Chaehwan Lim, Sang Hyun Gil, Suan Lee

Organizations: School of Computer Science / Data Intelligence Lab, Semyung University, Jecheon, Republic of Korea

Abstract

We present EuraGovExam, a multilingual and multimodal benchmark sourced from real-world civil service examinations across five representative Eurasian regions: South Korea, Japan, Taiwan, India, and the European Union. Designed to reflect the authentic complexity of public-sector assessments, the dataset contains over 8,000 high-resolution scanned multiple-choice questions covering 17 diverse academic and administrative domains. Unlike existing benchmarks, EuraGovExam embeds all question content--including problem statements, answer choices, and visual elements--within a single image, providing only a minimal standardized instruction for answer formatting. This design demands that models perform layout-aware, cross-lingual reasoning directly from visual input. All items are drawn from real exam documents, preserving rich visual structures such as tables, multilingual typography, and form-like layouts. Evaluation results show that even state-of-the-art vision-language models (VLMs) achieve only 86% accuracy, underscoring the benchmark's difficulty and its power to diagnose the limitations of current models. By emphasizing cultural realism, visual complexity, and linguistic diversity, EuraGovExam establishes a new standard for evaluating VLMs in high-stakes, multilingual, image-grounded settings. It also supports practical applications in e-governance, public-sector document analysis, and equitable exam preparation.

Figures & tables

Benchmark	#Q	#Lang.	#Scripts	#Domains	Multi-modal	Image-only	Orig. script	Real exam	N $\times$ D
MMLU ( Hendrycks et al., 2021 )	15,908	1	1	57	✗	✗	–	✓	✗
MMMU ( Yue et al., 2024 )	11,550	1	1	30	✓	✗	–	✓	✗
MATHVISTA ( Lu et al., 2024a )	6,141	1	1	5	✓	✗	–	Partial	✗
AGIEval ( Zhong et al., 2024 )	8,062	2	2	20	✗	✗	✓	✓	✗
EXAMS ( Hardalov et al., 2020 )	24,000	16	2	24	✗	✗	✓	✓	✗
M3Exam ( Zhang et al., 2023 )	12,317	9	3	3	$\dagger$	✗	✓	✓	✗

Table 1. Comparison of EuraGovExam with representative multimodal and exam-based benchmarks. Image-only : the model receives only an image with no separate text input. Orig. script : questions preserved in their source language without translation. N

\times

D : benchmark structure supports factorial analysis of nation/region and domain effects.

\dagger

subset of items are multimodal.

Figure 1. EuraGovExam Dataset Construction Pipeline

Panel of Figure 2 — Figure 2. Distribution of the EuraGovExam dataset.

Models	Overall	Nation					Task
Models	Overall
Closed-source Models
o3 ( OpenAI, 2025 )	84.26	68.64	84.49	93.72	82.37	90.06	85.51	90.55	73.83	92.44	87.20	89.23	74.03	81.85	79.90	86.33	81.53	91.49	91.18	84.16	90.93	78.08	78.98
o4-mini ( OpenAI, 2025 )	79.40	63.38	76.95	92.29	82.52	82.49	85.02	85.83	75.70	85.71	82.86	83.02	64.09	89.63	73.21	67.97	71.21	84.19	88.24	81.98	92.52	63.92	80.61
GPT-4o ( Hurst et al., 2024 )	42.04	40.99	63.73	66.66	25.97	33.25	50.72	36.22	17.76	29.41	31.89	42.73	37.02	33.52	34.93	35.55	47.48	49.73	50.00	33.72	58.96	42.00	43.38
GPT-4.1 ( OpenAI, 2025 )	54.73	48.10	66.44	72.58	48.10	54.23	68.12	55.12	33.64	46.22	46.42	53.50	48.07	54.63	45.93	48.83	54.57	60.14	69.75	50.00	73.47	47.54	58.06

Table 2. EuraGovExam performance. Icons denote regions ( India, EU, Taiwan, Japan, S. Korea) and domains ( Chem, Phil, Earth, Psych, Econ, Bio, Geo, Phys, Pol, Hist, Admin, Lang, Med, Eng, CS, Law, Math). All models evaluated under the image-only protocol (§ 4 ) with greedy decoding and single inference.

Model	Ovr.	JP	IN	KR	EU	TW
Gemini-2.5-Pro	87.0	87.6 $\pm$ 1.3	69.2 $\pm$ 2.2	91.1 $\pm$ 1.3	88.1 $\pm$ 1.8	95.5 $\pm$ 1.7
GPT-5	85.8	87.5 $\pm$ 1.3	68.4 $\pm$ 2.2	91.2 $\pm$ 1.3	83.9 $\pm$ 2.0	94.8 $\pm$ 1.9
o3	84.3	82.4 $\pm$ 1.5	68.6 $\pm$ 2.2	90.1 $\pm$ 1.4	84.5 $\pm$ 2.0	93.7 $\pm$ 2.0
o4-mini	79.4	82.5 $\pm$ 1.5	63.4 $\pm$ 2.3	82.5 $\pm$ 1.7	77.0 $\pm$ 2.3	92.3 $\pm$ 2.2
Gem.-3-flash	75.3	56.7 $\pm$ 1.9	51.7 $\pm$ 2.4	84.5 $\pm$ 1.6	89.5 $\pm$ 1.7	97.3 $\pm$ 1.4
Gem.-3-pro †	68.5	31.3 $\pm$ 2.0	66.5 $\pm$ 2.9	74.4 $\pm$ 1.7	89.0 $\pm$ 1.4	96.1 $\pm$ 1.6

Table 3. Nation-specific accuracy (%) for selected models.

\pm

: Wilson 95% CI. Bold : highest region; underline : lowest region. Gemini-3-Pro † : largest cross-regional gap (64.8%p).

Figure 3. Cross-regional performance analysis. (a) Model $\times$ Region interaction heatmap: each cell shows $\Delta=\text{Region}-\text{Overall}$ accuracy; magenta = underperformance, blue = overperformance (values shown for $|\Delta|\geq 14$ %p). Model names are color-coded by source type (blue: closed, orange: open). (b) Average accuracy by region with 95% Wilson CIs. (c) Ten most difficult domains.

Nation	Cnt	%	Domain	Cnt
India	30	2.9	Mathematics	9
S. Korea	8	0.3	Economics	7
Japan	6	0.3	Engineering	6
EU	6	0.3	Administration	5
Taiwan	0	0.0	History	4
Total	50	0.6	Other (11)	15

Table 4. Universal failure questions (all 28 models incorrect). India accounts for 60% despite 12.7% of the dataset.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4. Scaling analysis for open-source models. We analyze the relationship between parameter scale (in billions, $S$ ) and overall benchmark accuracy ( $A$ ) for open-source VLMs using a linear regression in log scale. The dashed line shows the fitted trend, and the shaded region indicates a $\pm 1$ standard-deviation band of the regression residuals (an empirical $\pm 1$ SD band). We obtain $R^{2}=0.41$ with $p<0.01$ , and observe an empirical trend of the form $A\approx 16.8\cdot\log(S)$ .

Figure 5. Task difficulty by model tier (mean accuracy). We partition models into Top, Middle, and Bottom tiers by overall accuracy, and visualize per-task mean accuracy as a heatmap. This reveals whether the relative ordering of task difficulty is preserved as models improve, and which tasks remain persistent bottlenecks.

Figure 6. Cross-task correlation with hierarchical clustering. For each task, we form a model-wise accuracy vector and compute Pearson correlations $r(t,t^{\prime})$ between tasks. We then apply hierarchical clustering (average linkage) using Euclidean distance between correlation profiles (rows/columns of the correlation matrix). High-correlation blocks suggest that subsets of tasks co-vary due to shared latent capability axes .

#	Region	Domain	Type	Model	Pred (GT)
1	S. Korea	Economics	KF	Gemini-2.5-Pro	D (A)
2	S. Korea	Administration	RF	Claude-Sonnet-4	B (D)
3	Japan	Physics	RF	o4-mini	B (C)
4	Japan	Engineering	KF + VP	Qwen2-VL-72B	B (C)
5	India	Mathematics	RF + HC	Llama-3.2-11B-Vision	A (B)
6	India	Economics	VP	LLaVA-1.5-13B	– (C)

Appendix

Table 5. Summary of qualitative failure cases. VP : Visual Parsing, RF : Reasoning, KF : Knowledge, HC : Hallucination. “–” in the Pred column indicates that the model failed to produce a valid answer choice.

Multilingual VLM Evaluation

Multilingual vision-language model evaluation measures how well models interpret images, documents, video, and other visual inputs across languages and cultures.

See all 56 papers

May 12, 2026

Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions from Japan's National Assessment of Academic Ability

Authentic school examinations provide a high-validity test bed for evaluating multimodal large language models (MLLMs), yet benchmarks grounded in Japanese K-12 assessments remain scarce. We present a multimodal dataset constructed from Japan's National Assessment of Academic Ability, comprising officially released middle-school items in Science, Mathematics, and Japanese Language. Unlike existing benchmarks based on synthetic or curated data, our dataset preserves real exam layouts, diagrams, and Japanese educational text, together with nationwide aggregated student response distributions (N

\approx

900{,}000). These features enable direct comparison between human and model performance under a unified evaluation framework. We benchmark recent multimodal LLMs using exact-match accuracy and character-level F1 for open-ended responses, observing substantial variation across subjects and strong sensitivity to visual reasoning demands. Human evaluation and LLM-as-judge analyses further assess the reliability of automatic scoring. Our dataset establishes a reproducible, human-grounded benchmark for multimodal educational reasoning and supports future research on evaluation, feedback generation, and explainable AI in authentic assessment contexts. Our dataset is available at: https://github.com/KyosukeTakami/gakucho-benchmark

Kyosuke Takami, Yuka Tateisi, Satoshi Sekine +1

Osaka Kyoiku University · University of Tokyo · NII LLMC

Educational Assessment Multimodal Reasoning Multimodal Model Evaluation Multimodal Understanding MLLM Evaluation Multimodal Dataset

Ask PDF

Apr 15, 2026

EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation

While Large Language Models (LLMs) have demonstrated high proficiency on English-centric medical examinations, their performance often declines when faced with non-English languages and multimodal diagnostic tasks. This study protocol describes the development of EuropeMedQA, the first comprehensive, multilingual, and multimodal medical examination dataset sourced from official regulatory exams in Italy, France, Spain, and Portugal. Following FAIR data principles and SPIRIT-AI guidelines, we describe a rigorous curation process and an automated translation pipeline for comparative analysis. We evaluate contemporary multimodal LLMs using a zero-shot, strictly constrained prompting strategy to assess cross-lingual transfer and visual reasoning. EuropeMedQA aims to provide a contamination-resistant benchmark that reflects the complexity of European clinical practices and fosters the development of more generalizable medical AI.

Francesco Andrea Causio, Vittorio De Vita, Olivia Riccomi +17

Section of Hygiene, University Department of Life Sciences and Public Health, Università Cattolica del Sacro Cuore, Rome, Italy · Società Italiana Intelligenza Artificiale in Medicina (SIIAM), Rome, Italy · NSBproject, Mantova, Italy +8

Multilingual Language Model Evaluation Multilingual VLM Evaluation Biomedical QA Multimodal QA Cross-Lingual Transfer

Ask PDF

Mar 18, 2026

KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context

Multimodal evaluation is strengthened when benchmarks reflect the languages, institutions, and disciplinary conventions that shape expert visual reasoning. We introduce KMMMU, a native Korean benchmark built from Korean-native examination materials. KMMMU contains 3,466 questions spanning nine disciplines and nine visual modality categories. For targeted evaluation on challenging questions and Korean-specific knowledge, we additionally provide KMMMU-Adv and KMMMU-Local, respectively. Experiments show that the strongest evaluated open-weight model reaches 42.05% on the KMMMU, while the best evaluated proprietary model achieves 52.42% on the KMMMU-Adv. Performance varies substantially across disciplines and answer formats, with Korean-specific questions showing gaps of up to 13.43%. Error analysis suggests that inspected failures often involve convention-to-label mapping, few-shot symbolic induction, localized knowledge recall, and domain-specific standards, not reasoning depth alone. KMMMU provides a testbed for native Korean expert-level multimodal understanding beyond English-centric or translation-based evaluation.

Nahyun Lee, Guijin Son, Hyunwoo Ko +4

Chung-Ang University · HAE-RAE · Seoul National University +2

Multimodal Model Evaluation Multimodal Understanding Multilingual VLM Evaluation Cross-Cultural Language Model Evaluation Cultural Knowledge in Language Models

Ask PDF

Abstract

Multilingual VLM Evaluation

Explore similar work

Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions from Japan's National Assessment of Academic Ability

EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation

KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context