JaWildText: A Benchmark for Vision-Language Models on Japanese Scene Text Understanding

Koki Maeda; Naoaki Okazaki

cs.CVMar 30, 2026

JaWildText: A Benchmark for Vision-Language Models on Japanese Scene Text Understanding

Authors: Koki Maeda, Naoaki Okazaki

Organizations: Institute of Science Tokyo, Tokyo, Japan · Research and Development Center for Large Language Models, National Institute of Informatics, Tokyo, Japan

Abstract

Japanese scene text poses challenges that multilingual benchmarks often fail to capture, including mixed scripts, frequent vertical writing, and a character inventory far larger than the Latin alphabet. Although Japanese is included in several multilingual benchmarks, these resources do not adequately capture the language-specific complexities. Meanwhile, existing Japanese visual text datasets have primarily focused on scanned documents, leaving in-the-wild scene text underexplored. To fill this gap, we introduce JaWildText, a diagnostic benchmark for evaluating vision-language models (VLMs) on Japanese scene text understanding. JaWildText contains 3,241 instances from 2,961 images newly captured in Japan, with 1.12 million annotated characters spanning 3,643 unique character types. It comprises three complementary tasks that vary in visual organization, output format, and writing style: (i) Dense Scene Text Visual Question Answering (STVQA), which requires reasoning over multiple pieces of visual text evidence; (ii) Receipt Key Information Extraction (KIE), which tests layout-aware structured extraction from mobile-captured receipts; and (iii) Handwriting OCR, which evaluates page-level transcription across various media and writing directions. We evaluate 14 open-weight VLMs and find that the best model achieves an average score of 0.64 across the three tasks. Error analyses show recognition remains the dominant bottleneck, especially for kanji. JaWildText enables fine-grained, script-aware diagnosis of Japanese scene text capabilities, and will be released with evaluation code.

Figures & tables

Figure 1: Overview of the JaWildText benchmark: (i) Dense STVQA, (ii) Receipt KIE, and (iii) Handwriting OCR. We added English translations for readability.

Subset	#Instances	#Images	#Regions	#Chars	#Unique Chars
Dense STVQA	1,025	745	33,608	571,979	3,313
Receipt KIE	1,151	1,151	56,095	433,558	2,001
Handwriting OCR	1,065	1,065	6,002	111,977	1,830
Total	3,241	2,961	95,705	1,117,514	3,643

Table 1: Summary statistics of JaWildText . #Instances denotes the evaluation unit: an instance is a question–answer pair in Dense STVQA, and a single image in Receipt KIE and Handwriting OCR. #Regions denotes the number of annotated quadrilateral text regions.

Figure 2: Representative images from each task, illustrating the diversity of JaWildText . Dense STVQA covers signboards, posters, and product packages under varying conditions. Receipt KIE includes receipts with creases, folds, and diverse perspectives. Handwriting OCR spans multiple writing media and directions.

Panel of Figure 3 — Figure 3: Image-level text properties in JaWildText .

Figure 4: Stacked character-type composition by task. Japanese scripts dominate Dense STVQA and Handwriting OCR, while Receipt KIE allocates a large fraction to ASCII digits.

Table 6

Model	Params	Overall	Dense STVQA	Receipt KIE	Handwriting OCR
			Accuracy	F1	1-CER
Qwen3-VL-8B	8B	0.64	0.62	0.53	0.79
Qwen3-VL-4B	4B	0.60	0.52	0.50	0.77
Qwen3-VL-2B	2B	0.52	0.31	0.48	0.76
InternVL3.5-38B	38B	0.55	0.44	0.50	0.71
InternVL3.5-14B	14B	0.49	0.30	0.47	0.71

Table 5: Results on the JaWildText benchmark.

Figure 5: Performance scaling trends with model size across benchmark tasks. Larger models improve overall performance, but gains differ by task family.

Model	Params	Store Name	Address	Receipt ID	Date	Time	Total	Tax
Qwen3-VL-8B	8B	0.14	0.52	0.13	0.81	0.91	0.81	0.77
Qwen3-VL-4B	4B	0.16	0.55	0.11	0.28	0.92	0.84	0.80
Qwen3-VL-2B	2B	0.11	0.43	0.08	0.52	0.88	0.82	0.82
InternVL3.5-38B	38B	0.15	0.34	0.18	0.52	0.92	0.83	0.82
InternVL3.5-14B	14B	0.15	0.35	0.13	0.26	0.89	0.83	0.79
InternVL3.5-8B	8B	0.11	0.31	0.13	0.50	0.89	0.79	0.74

Table 6: Receipt KIE field-level accuracy on JaWildText.

Figure 8: Representative failure cases on Handwriting OCR. (Left) Gemma3-12B misrecognizes kanji characters, substituting visually dissimilar characters. (Center) InternVL3.5-14B confuses visually similar katakana characters. (Right) Gemma3-4B produces hallucinated output entirely unrelated to the input image. Red bold text indicates erroneous characters in the model predictions.

Model	Params	Handwriting OCR (1-CER)
olmOCR-2-7B	7B	0.74
PaddleOCR-VL	0.9B	0.61
DeepSeek-OCR	3B	0.53

Table 7: Handwriting OCR results for OCR-specialized models.

Explore similar work

Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models
Apr 2, 2026Issa Sugiura, Keito Sasagawa, Keisuke Nakao +8Vision-Language Models Visual Question Answering
Institute of Science Tokyo · NII LLMC · Waseda University +4
BanglaWild: An In-the-Wild Bengali Scene Text Recognition Benchmark for OCR and Vision-Language Models
Aug 4, 2026Sadab Shiper, Tawsif Tashwar Dipto, Mir Md Inzamam +1VLM Evaluation Scene Text Recognition
Islamic University of Technology, Bangladesh · BRAC University, Bangladesh
JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
Apr 1, 2026Issa Sugiura, Koki Maeda, Shuhei Kurita +3VLM Evaluation Benchmark Construction
Kyoto University · NII LLMC · Institute of Science Tokyo +2

Apr 2, 2026J/K move · Enter open · S save

Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models

Issa Sugiura, Keito Sasagawa, Keisuke Nakao, Koki Maeda+7

Institute of Science Tokyo · NII LLMC · Waseda University+4

Vision-Language Models Visual Question Answering Multimodal Instruction Tuning

Open paper AskPDF arXiv

Developing vision-language models (VLMs) that generalize across diverse tasks requires large-scale training datasets with diverse content. In English, such datasets are typically constructed by aggregating and curating numerous existing visual question answering (VQA) resources. However, this strategy does not readily extend to other languages, where VQA datasets remain limited in both scale and domain coverage. In this work, we introduce Jagle, the largest Japanese multimodal post-training dataset to date, comprising approximately 9.2 million instances across diverse tasks. Rather than relying on existing VQA datasets, we collect heterogeneous source data, including images, image-text pairs, and PDF documents, and generate VQA pairs through multiple strategies such as VLM-based QA generation, translation, and text rendering. Experiments demonstrate that a 2.2B model trained with Jagle surpasses InternVL3.5-2B in average score across ten Japanese evaluation tasks and comes within five points of Qwen3-VL-2B-Instruct. Moreover, combining Jagle with FineVision improves English performance over training with FineVision alone. These benefits generalize across model backbones and scales: LLM-jp-4-VL 9B beta, a 9B-scale model built on the Japanese-centric LLM-jp-4-8B-instruct, achieves Japanese performance comparable to Qwen3-VL-8B-Instruct. To facilitate reproducibility and future research, we release the dataset, models, and code.

Abstract

Explore similar work

Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models

BanglaWild: An In-the-Wild Bengali Scene Text Recognition Benchmark for OCR and Vision-Language Models

JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation