JaWildText: A Benchmark for Vision-Language Models on Japanese Scene Text Understanding
Authors: Koki Maeda, Naoaki Okazaki
Organizations: Institute of Science Tokyo, Tokyo, Japan · Research and Development Center for Large Language Models, National Institute of Informatics, Tokyo, Japan
Japanese scene text poses challenges that multilingual benchmarks often fail to capture, including mixed scripts, frequent vertical writing, and a character inventory far larger than the Latin alphabet. Although Japanese is included in several multilingual benchmarks, these resources do not adequately capture the language-specific complexities. Meanwhile, existing Japanese visual text datasets have primarily focused on scanned documents, leaving in-the-wild scene text underexplored. To fill this gap, we introduce JaWildText, a diagnostic benchmark for evaluating vision-language models (VLMs) on Japanese scene text understanding. JaWildText contains 3,241 instances from 2,961 images newly captured in Japan, with 1.12 million annotated characters spanning 3,643 unique character types. It comprises three complementary tasks that vary in visual organization, output format, and writing style: (i) Dense Scene Text Visual Question Answering (STVQA), which requires reasoning over multiple pieces of visual text evidence; (ii) Receipt Key Information Extraction (KIE), which tests layout-aware structured extraction from mobile-captured receipts; and (iii) Handwriting OCR, which evaluates page-level transcription across various media and writing directions. We evaluate 14 open-weight VLMs and find that the best model achieves an average score of 0.64 across the three tasks. Error analyses show recognition remains the dominant bottleneck, especially for kanji. JaWildText enables fine-grained, script-aware diagnosis of Japanese scene text capabilities, and will be released with evaluation code.
Figures & tables
Figure 1: Overview of the JaWildText benchmark: (i) Dense STVQA, (ii) Receipt KIE, and (iii) Handwriting OCR. We added English translations for readability.
Subset
#Instances
#Images
#Regions
#Chars
#Unique Chars
Dense STVQA
1,025
745
33,608
571,979
3,313
Receipt KIE
1,151
1,151
56,095
433,558
2,001
Handwriting OCR
1,065
1,065
6,002
111,977
1,830
Total
3,241
2,961
95,705
1,117,514
3,643
Table 1: Summary statistics of JaWildText . #Instances denotes the evaluation unit: an instance is a question–answer pair in Dense STVQA, and a single image in Receipt KIE and Handwriting OCR. #Regions denotes the number of annotated quadrilateral text regions.
Figure 2: Representative images from each task, illustrating the diversity of JaWildText . Dense STVQA covers signboards, posters, and product packages under varying conditions. Receipt KIE includes receipts with creases, folds, and diverse perspectives. Handwriting OCR spans multiple writing media and directions.
Figure 3: Image-level text properties in JaWildText .
Figure 4: Stacked character-type composition by task. Japanese scripts dominate Dense STVQA and Handwriting OCR, while Receipt KIE allocates a large fraction to ASCII digits.
Table 6
Model
Params
Overall
Dense STVQA
Receipt KIE
Handwriting OCR
Accuracy
F1
1-CER
Qwen3-VL-8B
8B
0.64
0.62
0.53
0.79
Qwen3-VL-4B
4B
0.60
0.52
0.50
0.77
Qwen3-VL-2B
2B
0.52
0.31
0.48
0.76
InternVL3.5-38B
38B
0.55
0.44
0.50
0.71
InternVL3.5-14B
14B
0.49
0.30
0.47
0.71
Table 5: Results on the JaWildText benchmark.
Figure 5: Performance scaling trends with model size across benchmark tasks. Larger models improve overall performance, but gains differ by task family.
Model
Params
Store Name
Address
Receipt ID
Date
Time
Total
Tax
Qwen3-VL-8B
8B
0.14
0.52
0.13
0.81
0.91
0.81
0.77
Qwen3-VL-4B
4B
0.16
0.55
0.11
0.28
0.92
0.84
0.80
Qwen3-VL-2B
2B
0.11
0.43
0.08
0.52
0.88
0.82
0.82
InternVL3.5-38B
38B
0.15
0.34
0.18
0.52
0.92
0.83
0.82
InternVL3.5-14B
14B
0.15
0.35
0.13
0.26
0.89
0.83
0.79
InternVL3.5-8B
8B
0.11
0.31
0.13
0.50
0.89
0.79
0.74
Table 6: Receipt KIE field-level accuracy on JaWildText.
Figure 10
Figure 8: Representative failure cases on Handwriting OCR. (Left) Gemma3-12B misrecognizes kanji characters, substituting visually dissimilar characters. (Center) InternVL3.5-14B confuses visually similar katakana characters. (Right) Gemma3-4B produces hallucinated output entirely unrelated to the input image. Red bold text indicates erroneous characters in the model predictions.
Model
Params
Handwriting OCR (1-CER)
olmOCR-2-7B
7B
0.74
PaddleOCR-VL
0.9B
0.61
DeepSeek-OCR
3B
0.53
Table 7: Handwriting OCR results for OCR-specialized models.
Developing vision-language models (VLMs) that generalize across diverse tasks requires large-scale training datasets with diverse content. In English, such datasets are typically constructed by aggregating and curating numerous existing visual question answering (VQA) resources. However, this strategy does not readily extend to other languages, where VQA datasets remain limited in both scale and domain coverage. In this work, we introduce Jagle, the largest Japanese multimodal post-training dataset to date, comprising approximately 9.2 million instances across diverse tasks. Rather than relying on existing VQA datasets, we collect heterogeneous source data, including images, image-text pairs, and PDF documents, and generate VQA pairs through multiple strategies such as VLM-based QA generation, translation, and text rendering. Experiments demonstrate that a 2.2B model trained with Jagle surpasses InternVL3.5-2B in average score across ten Japanese evaluation tasks and comes within five points of Qwen3-VL-2B-Instruct. Moreover, combining Jagle with FineVision improves English performance over training with FineVision alone. These benefits generalize across model backbones and scales: LLM-jp-4-VL 9B beta, a 9B-scale model built on the Japanese-centric LLM-jp-4-8B-instruct, achieves Japanese performance comparable to Qwen3-VL-8B-Instruct. To facilitate reproducibility and future research, we release the dataset, models, and code.
Issa Sugiura, Keito Sasagawa, Keisuke Nakao +8
Institute of Science Tokyo · NII LLMC · Waseda University +4
In-the-wild Bengali scene text recognition is largely unmeasured: existing resources target handwritten documents or constrained sign-board parsing, report only aggregate edit-distance metrics, and evaluate either conventional OCR or VLMs, never both on the same in-the-wild data. To address this gap, we introduce BANGLAWILD, a benchmark of 2,535 Bengali scene text images, each paired with a verbatim gold transcription, two categorical axes, four diagnostic attributes, and an orthographically standard form where the in-image text deviates from canonical spelling. We evaluate fifteen VLMs and three conventional OCR systems under three prompting strategies, fine-tune 6 open-source models with LoRA, and complement edit-distance metrics with an LLM-as-a-Judge evaluation. Our results reveal a persistent gap in which larger models within the same family do not outperform smaller ones. Our fifteen-class error taxonomy shows that visual mis-recognition accounts for ~60% of errors in the strongest systems, while conjunct-related errors contribute under 2%, challenging a long-standing assumption in Bengali OCR research; the same visual dominant profile also holds across architectures, including the one conventional baseline that reads Bengali reliably. Prompt language mainly affects cross-script drift and LoRA reduces catastrophic failures in weak models without lifting the ceiling on already competent ones. Code and data will be publicly released.
Sadab Shiper, Tawsif Tashwar Dipto, Mir Md Inzamam +1
Islamic University of Technology, Bangladesh · BRAC University, Bangladesh
Reliable evaluation is essential for the development of vision-language models (VLMs). However, Japanese VQA benchmarks have undergone far less iterative refinement than their English counterparts. As a result, many existing benchmarks contain issues such as ambiguous questions, incorrect answers, and instances that can be solved without visual grounding, undermining evaluation reliability and leading to misleading conclusions in model comparisons. To address these limitations, we introduce JAMMEval, a refined collection of Japanese benchmarks for reliable VLM evaluation. It is constructed by systematically refining seven existing Japanese benchmark datasets through two rounds of human annotation, improving both data quality and evaluation reliability. In our experiments, we evaluate open-weight and proprietary VLMs on JAMMEval and analyze the capabilities of recent models on Japanese VQA. We further demonstrate the effectiveness of our refinement by showing that the resulting benchmarks yield evaluation scores that better reflect model capability, exhibit lower run-to-run variance, and improve the ability to distinguish between models of different capability levels. We release our dataset and code to advance reliable evaluation of VLMs.
Issa Sugiura, Koki Maeda, Shuhei Kurita +3
Kyoto University · NII LLMC · Institute of Science Tokyo +2