JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation

Issa Sugiura; Koki Maeda; Shuhei Kurita; Yusuke Oda; Daisuke Kawahara; Naoaki Okazaki

cs.CVApr 1, 2026

JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation

Authors: Issa Sugiura, Koki Maeda, Shuhei Kurita, Yusuke Oda, Daisuke Kawahara, Naoaki Okazaki

Organizations: Kyoto University · NII LLMC · Institute of Science Tokyo · NII · Waseda University

Abstract

Reliable evaluation is essential for the development of vision-language models (VLMs). However, Japanese VQA benchmarks have undergone far less iterative refinement than their English counterparts. As a result, many existing benchmarks contain issues such as ambiguous questions, incorrect answers, and instances that can be solved without visual grounding, undermining evaluation reliability and leading to misleading conclusions in model comparisons. To address these limitations, we introduce JAMMEval, a refined collection of Japanese benchmarks for reliable VLM evaluation. It is constructed by systematically refining seven existing Japanese benchmark datasets through two rounds of human annotation, improving both data quality and evaluation reliability. In our experiments, we evaluate open-weight and proprietary VLMs on JAMMEval and analyze the capabilities of recent models on Japanese VQA. We further demonstrate the effectiveness of our refinement by showing that the resulting benchmarks yield evaluation scores that better reflect model capability, exhibit lower run-to-run variance, and improve the ability to distinguish between models of different capability levels. We release our dataset and code to advance reliable evaluation of VLMs.

Figures & tables

Figure 1: Examples of inappropriate instances in existing Japanese VQA evaluation datasets. (a) Open-ended questions with inherent ambiguity that do not admit a unique correct answer. (b) Questions that can be answered with high confidence using only the text, without referring to the image. (c) Instances with incorrect ground-truth answers. (d) Subjective questions for which answers vary across annotators.

Figure 2: Construction pipeline of JAMMEval. Starting from seven seed datasets, all instances undergo two rounds of manual review and re-annotation to produce a refined benchmark collection.

Figure 3: An example of re-annotation. An ambiguous open-ended question is replaced with a specific, objectively answerable question targeting information visible in the image.

Figure 4: Breakdown of refinement operations per dataset. Identical instances required no modification; other categories indicate the type of correction applied.

Dataset	Seed Dataset	Samples	Images	Answer Format	Category
CC-OCR-JA-Refined	Yang et al. (2024)	145	145	Short answer	OCR
JA-VLM-Bench-Refined	Akiba et al. (2025)	49	42	Short answer	Japanese culture
Heron-Bench-Refined	Inoue et al. (2024)	88	21	Short answer	Japanese culture
CVQA-JA-Refined	Mogrovejo et al. (2024)	200	94	Multiple-choice	Japanese culture
JA-Multi-Image-VQA-Refined	Yuichi et al. (2024)	53	39	Short answer	Multi-image
JDocQA-Refined	Onami et al. (2024)	861	793	Short answer	Document

Table 1: Statistics of JAMMEval. The collection spans seven tasks across diverse domains, totaling 1,592 instances after refinement.

Figure 5: Model performance on JAMMEval across seven tasks. Note that Gemini 3 Pro is evaluated with reasoning enabled, while all other models are evaluated without reasoning. Gemini 3 Pro achieves the highest scores overall.

Figure 6: Distribution of Gemini 3 Pro errors by category across datasets. Knowledge errors are prominent in CVQA-JA-Refined, reflecting limited Japanese cultural knowledge, while perception errors appear broadly across datasets. Judge errors in short-answer datasets reflect limitations of LLM-based grading rather than model capability.

	Accuracy (%)			Run Std			Performance Gap		Rank Corr.
Dataset	Before	After	$\Delta$	Before	After	$\Delta$	Before	After
CC-OCR-Ja	22.4	65.8	+43.4	0.9	0.6	-0.3	26.0	44.6	0.78
JA-VLM-Bench	64.1	64.9	+0.8	1.6	0.8	-0.9	28.7	53.1	0.88
Heron-Bench	46.9	62.2	+15.2	2.0	0.6	-1.4	43.0	48.9	0.95
CVQA-JA	62.6	65.3	+2.7	0.9	0.8	-0.1	58.0	56.7	0.96
JA-Multi-Image-VQA	66.5	71.1	+4.6	1.0	0.5	-0.5	61.2	52.8	0.99

Table 2: Evaluation statistics before and after refinement. Accuracy and Run Std denote the mean accuracy and the run-to-run standard deviation averaged across models per dataset. Performance Gap measures the score difference between the best and worst models. Rank Corr. denotes the Spearman rank correlation between model rankings before and after refinement. Refinement consistently raises accuracy and reduces run-to-run variance while preserving model rankings (Rank Corr.

\approx

1).

Figure 7: Effect of refinement on Heron-Bench. After refinement, accuracy increases across all models and run-to-run variance decreases, likely because problematic instances are removed and ambiguous questions are replaced with objectively answerable ones, resulting in more stable and reliable evaluation scores.

Figure 8: Error distribution of Gemini 3 Pro on Heron-Bench and Heron-Bench-Refined. In Heron-Bench, a large portion of errors are attributed to judge errors, primarily due to the prevalence of ambiguous QA pairs.

Figure 9: Interface of the annotation tool. For each example, the (Image, Original Question, Original Answer) is displayed. Annotators provide a (New Question, New Answer) for the image and then click the Submit button. If they cannot create a valid VQA, they can click the Skip button to move on to the next example.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Examples of errors made by Gemini 3 Pro.

Figure 11: Model performance on each dataset: original (left) and refined (right).

Explore similar work

Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models
Apr 2, 2026Issa Sugiura, Keito Sasagawa, Keisuke Nakao +8Vision-Language Models Visual Question Answering
Institute of Science Tokyo · NII LLMC · Waseda University +4
Lost in Translation: Do LVLM Judges Generalize Across Languages?
Apr 21, 2026Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5VLM Evaluation Multilingual VLM Evaluation
York University · University of Alberta · Nanyang Technological University +1
Can Argus Judge Them All? Comparing VLMs Across Domains
Jun 23, 2025Harsh Joshi, Gautam Siddharth Kashyap, Rafiq Ali +5VLM Evaluation VLM Robustness
Bharati Vidyapeeth, New Delhi, India · Macquarie University, Sydney, Australia · DSEU-Okhla, New Delhi, India +3

Apr 2, 2026J/K move · Enter open · S save

Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models

Issa Sugiura, Keito Sasagawa, Keisuke Nakao, Koki Maeda+7

Institute of Science Tokyo · NII LLMC · Waseda University+4

Vision-Language Models Visual Question Answering Multimodal Instruction Tuning

Open paper AskPDF arXiv

Developing vision-language models (VLMs) that generalize across diverse tasks requires large-scale training datasets with diverse content. In English, such datasets are typically constructed by aggregating and curating numerous existing visual question answering (VQA) resources. However, this strategy does not readily extend to other languages, where VQA datasets remain limited in both scale and domain coverage. In this work, we introduce Jagle, the largest Japanese multimodal post-training dataset to date, comprising approximately 9.2 million instances across diverse tasks. Rather than relying on existing VQA datasets, we collect heterogeneous source data, including images, image-text pairs, and PDF documents, and generate VQA pairs through multiple strategies such as VLM-based QA generation, translation, and text rendering. Experiments demonstrate that a 2.2B model trained with Jagle surpasses InternVL3.5-2B in average score across ten Japanese evaluation tasks and comes within five points of Qwen3-VL-2B-Instruct. Moreover, combining Jagle with FineVision improves English performance over training with FineVision alone. These benefits generalize across model backbones and scales: LLM-jp-4-VL 9B beta, a 9B-scale model built on the Japanese-centric LLM-jp-4-8B-instruct, achieves Japanese performance comparable to Qwen3-VL-8B-Instruct. To facilitate reproducibility and future research, we release the dataset, models, and code.

Abstract

Explore similar work

Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models

Lost in Translation: Do LVLM Judges Generalize Across Languages?

Can Argus Judge Them All? Comparing VLMs Across Domains