EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams

Jaeseong Kim; Chaehwan Lim; Sang Hyun Gil; Suan Lee

cs.CVMar 28, 2026

EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams

Authors: Jaeseong Kim, Chaehwan Lim, Sang Hyun Gil, Suan Lee

Organizations: School of Computer Science / Data Intelligence Lab, Semyung University, Jecheon, Republic of Korea

Abstract

We present EuraGovExam, a multilingual and multimodal benchmark sourced from real-world civil service examinations across five representative Eurasian regions: South Korea, Japan, Taiwan, India, and the European Union. Designed to reflect the authentic complexity of public-sector assessments, the dataset contains over 8,000 high-resolution scanned multiple-choice questions covering 17 diverse academic and administrative domains. Unlike existing benchmarks, EuraGovExam embeds all question content--including problem statements, answer choices, and visual elements--within a single image, providing only a minimal standardized instruction for answer formatting. This design demands that models perform layout-aware, cross-lingual reasoning directly from visual input. All items are drawn from real exam documents, preserving rich visual structures such as tables, multilingual typography, and form-like layouts. Evaluation results show that even state-of-the-art vision-language models (VLMs) achieve only 86% accuracy, underscoring the benchmark's difficulty and its power to diagnose the limitations of current models. By emphasizing cultural realism, visual complexity, and linguistic diversity, EuraGovExam establishes a new standard for evaluating VLMs in high-stakes, multilingual, image-grounded settings. It also supports practical applications in e-governance, public-sector document analysis, and equitable exam preparation.

Figures & tables

Benchmark	#Q	#Lang.	#Scripts	#Domains	Multi-modal	Image-only	Orig. script	Real exam	N $\times$ D
MMLU ( Hendrycks et al., 2021 )	15,908	1	1	57	✗	✗	–	✓	✗
MMMU ( Yue et al., 2024 )	11,550	1	1	30	✓	✗	–	✓	✗
MATHVISTA ( Lu et al., 2024a )	6,141	1	1	5	✓	✗	–	Partial	✗
AGIEval ( Zhong et al., 2024 )	8,062	2	2	20	✗	✗	✓	✓	✗
EXAMS ( Hardalov et al., 2020 )	24,000	16	2	24	✗	✗	✓	✓	✗
M3Exam ( Zhang et al., 2023 )	12,317	9	3	3	$\dagger$	✗	✓	✓	✗

Table 1. Comparison of EuraGovExam with representative multimodal and exam-based benchmarks. Image-only : the model receives only an image with no separate text input. Orig. script : questions preserved in their source language without translation. N

\times

D : benchmark structure supports factorial analysis of nation/region and domain effects.

\dagger

subset of items are multimodal.

Figure 1. EuraGovExam Dataset Construction Pipeline

Panel of Figure 2 — Figure 2. Distribution of the EuraGovExam dataset.

Models	Overall	Nation					Task
Models	Overall
Closed-source Models
o3 ( OpenAI, 2025 )	84.26	68.64	84.49	93.72	82.37	90.06	85.51	90.55	73.83	92.44	87.20	89.23	74.03	81.85	79.90	86.33	81.53	91.49	91.18	84.16	90.93	78.08	78.98
o4-mini ( OpenAI, 2025 )	79.40	63.38	76.95	92.29	82.52	82.49	85.02	85.83	75.70	85.71	82.86	83.02	64.09	89.63	73.21	67.97	71.21	84.19	88.24	81.98	92.52	63.92	80.61
GPT-4o ( Hurst et al., 2024 )	42.04	40.99	63.73	66.66	25.97	33.25	50.72	36.22	17.76	29.41	31.89	42.73	37.02	33.52	34.93	35.55	47.48	49.73	50.00	33.72	58.96	42.00	43.38
GPT-4.1 ( OpenAI, 2025 )	54.73	48.10	66.44	72.58	48.10	54.23	68.12	55.12	33.64	46.22	46.42	53.50	48.07	54.63	45.93	48.83	54.57	60.14	69.75	50.00	73.47	47.54	58.06

Table 2. EuraGovExam performance. Icons denote regions ( India, EU, Taiwan, Japan, S. Korea) and domains ( Chem, Phil, Earth, Psych, Econ, Bio, Geo, Phys, Pol, Hist, Admin, Lang, Med, Eng, CS, Law, Math). All models evaluated under the image-only protocol (§ 4 ) with greedy decoding and single inference.

Model	Ovr.	JP	IN	KR	EU	TW
Gemini-2.5-Pro	87.0	87.6 $\pm$ 1.3	69.2 $\pm$ 2.2	91.1 $\pm$ 1.3	88.1 $\pm$ 1.8	95.5 $\pm$ 1.7
GPT-5	85.8	87.5 $\pm$ 1.3	68.4 $\pm$ 2.2	91.2 $\pm$ 1.3	83.9 $\pm$ 2.0	94.8 $\pm$ 1.9
o3	84.3	82.4 $\pm$ 1.5	68.6 $\pm$ 2.2	90.1 $\pm$ 1.4	84.5 $\pm$ 2.0	93.7 $\pm$ 2.0
o4-mini	79.4	82.5 $\pm$ 1.5	63.4 $\pm$ 2.3	82.5 $\pm$ 1.7	77.0 $\pm$ 2.3	92.3 $\pm$ 2.2
Gem.-3-flash	75.3	56.7 $\pm$ 1.9	51.7 $\pm$ 2.4	84.5 $\pm$ 1.6	89.5 $\pm$ 1.7	97.3 $\pm$ 1.4
Gem.-3-pro †	68.5	31.3 $\pm$ 2.0	66.5 $\pm$ 2.9	74.4 $\pm$ 1.7	89.0 $\pm$ 1.4	96.1 $\pm$ 1.6

Table 3. Nation-specific accuracy (%) for selected models.

\pm

: Wilson 95% CI. Bold : highest region; underline : lowest region. Gemini-3-Pro † : largest cross-regional gap (64.8%p).

Figure 3. Cross-regional performance analysis. (a) Model $\times$ Region interaction heatmap: each cell shows $\Delta=\text{Region}-\text{Overall}$ accuracy; magenta = underperformance, blue = overperformance (values shown for $|\Delta|\geq 14$ %p). Model names are color-coded by source type (blue: closed, orange: open). (b) Average accuracy by region with 95% Wilson CIs. (c) Ten most difficult domains.

Nation	Cnt	%	Domain	Cnt
India	30	2.9	Mathematics	9
S. Korea	8	0.3	Economics	7
Japan	6	0.3	Engineering	6
EU	6	0.3	Administration	5
Taiwan	0	0.0	History	4
Total	50	0.6	Other (11)	15

Table 4. Universal failure questions (all 28 models incorrect). India accounts for 60% despite 12.7% of the dataset.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4. Scaling analysis for open-source models. We analyze the relationship between parameter scale (in billions, $S$ ) and overall benchmark accuracy ( $A$ ) for open-source VLMs using a linear regression in log scale. The dashed line shows the fitted trend, and the shaded region indicates a $\pm 1$ standard-deviation band of the regression residuals (an empirical $\pm 1$ SD band). We obtain $R^{2}=0.41$ with $p<0.01$ , and observe an empirical trend of the form $A\approx 16.8\cdot\log(S)$ .

Figure 5. Task difficulty by model tier (mean accuracy). We partition models into Top, Middle, and Bottom tiers by overall accuracy, and visualize per-task mean accuracy as a heatmap. This reveals whether the relative ordering of task difficulty is preserved as models improve, and which tasks remain persistent bottlenecks.

Figure 6. Cross-task correlation with hierarchical clustering. For each task, we form a model-wise accuracy vector and compute Pearson correlations $r(t,t^{\prime})$ between tasks. We then apply hierarchical clustering (average linkage) using Euclidean distance between correlation profiles (rows/columns of the correlation matrix). High-correlation blocks suggest that subsets of tasks co-vary due to shared latent capability axes .

#	Region	Domain	Type	Model	Pred (GT)
1	S. Korea	Economics	KF	Gemini-2.5-Pro	D (A)
2	S. Korea	Administration	RF	Claude-Sonnet-4	B (D)
3	Japan	Physics	RF	o4-mini	B (C)
4	Japan	Engineering	KF + VP	Qwen2-VL-72B	B (C)
5	India	Mathematics	RF + HC	Llama-3.2-11B-Vision	A (B)
6	India	Economics	VP	LLaVA-1.5-13B	– (C)

Appendix

Table 5. Summary of qualitative failure cases. VP : Visual Parsing, RF : Reasoning, KF : Knowledge, HC : Hallucination. “–” in the Pred column indicates that the model failed to produce a valid answer choice.

Multilingual VLM Evaluation

Multilingual vision-language model evaluation measures how well models interpret images, documents, video, and other visual inputs across languages and cultures.

See all 56 papers

Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions from Japan's National Assessment of Academic Ability
May 12, 2026Kyosuke Takami, Yuka Tateisi, Satoshi Sekine +1Educational Assessment Multimodal Reasoning
Osaka Kyoiku University · University of Tokyo · NII LLMC
EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation
Apr 15, 2026Francesco Andrea Causio, Vittorio De Vita, Olivia Riccomi +17Multilingual Language Model Evaluation Multilingual VLM Evaluation
Section of Hygiene, University Department of Life Sciences and Public Health, Università Cattolica del Sacro Cuore, Rome, Italy · Società Italiana Intelligenza Artificiale in Medicina (SIIAM), Rome, Italy · NSBproject, Mantova, Italy +8
KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context
Mar 18, 2026Nahyun Lee, Guijin Son, Hyunwoo Ko +4Multimodal Model Evaluation Multimodal Understanding
Chung-Ang University · HAE-RAE · Seoul National University +2

May 12, 2026J/K move · Enter open · S save

Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions from Japan's National Assessment of Academic Ability

Kyosuke Takami, Yuka Tateisi, Satoshi Sekine, Yusuke Miyao

Osaka Kyoiku University · University of Tokyo · NII LLMC

Educational Assessment Multimodal Reasoning Multimodal Model Evaluation Multimodal Understanding MLLM Evaluation Multimodal Dataset

Open paper AskPDF arXiv

Authentic school examinations provide a high-validity test bed for evaluating multimodal large language models (MLLMs), yet benchmarks grounded in Japanese K-12 assessments remain scarce. We present a multimodal dataset constructed from Japan's National Assessment of Academic Ability, comprising officially released middle-school items in Science, Mathematics, and Japanese Language. Unlike existing benchmarks based on synthetic or curated data, our dataset preserves real exam layouts, diagrams, and Japanese educational text, together with nationwide aggregated student response distributions (N

\approx

900{,}000). These features enable direct comparison between human and model performance under a unified evaluation framework. We benchmark recent multimodal LLMs using exact-match accuracy and character-level F1 for open-ended responses, observing substantial variation across subjects and strong sensitivity to visual reasoning demands. Human evaluation and LLM-as-judge analyses further assess the reliability of automatic scoring. Our dataset establishes a reproducible, human-grounded benchmark for multimodal educational reasoning and supports future research on evaluation, feedback generation, and explainable AI in authentic assessment contexts. Our dataset is available at: https://github.com/KyosukeTakami/gakucho-benchmark

Abstract

Multilingual VLM Evaluation

Explore similar work

Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions from Japan's National Assessment of Academic Ability

EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation

KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context