Shape vs. Context: Examining Human--AI Gaps in Ambiguous Japanese Character Recognition

Daichi Haraguchi

cs.HCFeb 27, 2026

Shape vs. Context: Examining Human--AI Gaps in Ambiguous Japanese Character Recognition

Authors: Daichi Haraguchi

Organizations: CyberAgent, Tokyo, Japan

Abstract

High text recognition performance does not guarantee that Vision-Language Models (VLMs) share human-like decision patterns when resolving ambiguity. We investigate this behavioral gap by directly comparing humans and VLMs using continuously interpolated Japanese character shapes generated via a $β$ -VAE. We estimate decision boundaries in a single-character recognition (shape-only task) and evaluate whether VLM responses align with human judgments under shape in context (i.e., embedding an ambiguous character near the human decision boundary in word-level context). We find that human and VLM decision boundaries differ in the shape-only task, and that shape in context can improve human alignment in some conditions. These results highlight qualitative behavioral differences, offering foundational insights toward human--VLM alignment benchmarking.

Figures & tables

Figure 1. Probing Human–VLM alignment. Using interpolated images of the visually similar Japanese characters ‘ソ’ ( so ) and ‘ン’ ( n ), we visualize the behavioral gap in two scenarios: RQ1 compares decision boundaries in a single character recognition (shape-only task), while RQ2 tests whether VLM responses align with human judgments in shape-in-context conditions. teaser

Figure 2. Construction of contextual word images. Examples of sole-occurrence and co-occurrence contexts, where a single character is replaced with an ambiguous glyph (highlighted in red box) selected to yield approximately 50% recognition by human participants in the shape-only task (RQ1). In the co-occurrence context, additional so or n characters appear elsewhere in the word.

Figure 3. Single character recognition by each interpolation parameter (RQ1), aggregated across 10 fonts. Representative interpolated characters from one example font are shown along the x-axis for visual reference.

Figure 4. Shape-in-context word recognition (RQ2). The target character was replaced with an ambiguous glyph X (an interpolation between so and n ) in either so-biased or n-biased word contexts, under (a) sole-occurrence and (b) co-occurrence conditions. Asterisks denote significant differences from humans after Bonferroni correction ( * $p_{\mathrm{adj}}<.05$ , *** $p_{\mathrm{adj}}<.001$ ).

Explore similar work

Sorry Robot, Happy Human: Vision-Language Models Read Only One of Two Legible Typographic Layers
Sep 25, 2026Mert İncidelen, Yamen Kashkash, Asya Berker +1Vision-Language Models VLM Robustness
Department of Artificial Intelligence and Data Engineering, Fırat University, Elazığ, Türkiye · Department of Software Engineering, Fırat University, Elazığ, Türkiye
When VLMs Trust Context: Evaluating Scene Text Recognition under Misleading Context
Sep 28, 2026Yuxing Cheng, Yuan Wu, Yi ChangVLM Evaluation Counterfactual Evaluation of VLMs
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs
Mar 19, 2026Rui Hong, Shuxue QuanVision-Language Models Vision-Language Grounding
George Mason University, Fairfax, VA, USA · Independent Researcher

Sep 25, 2026J/K move · Enter open · S save

Sorry Robot, Happy Human: Vision-Language Models Read Only One of Two Legible Typographic Layers

Mert İncidelen, Yamen Kashkash, Asya Berker, Murat Aydoğan

Department of Artificial Intelligence and Data Engineering, Fırat University, Elazığ, Türkiye · Department of Software Engineering, Fırat University, Elazığ, Türkiye

Vision-Language Models VLM Robustness VLM Evaluation

Open paper AskPDF arXiv

Vision-language models (VLMs), despite their success in optical character recognition (OCR) tasks, are vulnerable to typographic attacks and have a fragile structure for images with multiple text layers. In this study, the DecoyBench dataset was created using the Decoy Font method. The dataset consists of 300 images, each containing text with sharp contour lines superimposed on another text with soft shading. Six recent closed-source models from three different model families were evaluated using this dataset under two different prompting conditions (naive and guided) and at two different resolutions (

512\times512

and

64\times64

). A validation study showed that human participants could read both text layers with high accuracy. In contrast, the models, with most variants and both prompting methods, read the contour text with near-human accuracy at high resolution, but almost never fully extracted the shading text. At low resolution, the contour text could not be read by either the models or humans, while the shading text could be extracted with high accuracy. The findings indicate that the evaluated VLMs exhibit a consistent behavioral limitation when processing typographic structures containing multiple spatial frequency layers.

Abstract

Explore similar work

Sorry Robot, Happy Human: Vision-Language Models Read Only One of Two Legible Typographic Layers

When VLMs Trust Context: Evaluating Scene Text Recognition under Misleading Context

To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs