Evaluating the Robustness of Japanese LLMs to IME-Related and Typographical Errors

Ryota Mibayashi; Hiroaki Ohshima

cs.CLOct 1, 2026

Evaluating the Robustness of Japanese LLMs to IME-Related and Typographical Errors

Authors: Ryota Mibayashi, Hiroaki Ohshima

Organizations: Kobe University, Hyogo, Japan · University of Hyogo, Hyogo, Japan

Abstract

Large language models (LLMs) have achieved strong performance across various natural language processing tasks. However, their robustness to typographical errors remains underexplored, particularly in Japanese, where text input involves multiple writing systems and IME-based conversion. In this study, we evaluate the robustness of Japanese LLMs against realistic Japanese-specific typos. We introduce five typo categories: Character Transposition, Character Replacement, Homophone Conversion, Japanese IME Conversion, and Full-Width Conversion. These perturbations are applied to three Japanese benchmark datasets (JMMLU, JCommonsenseQA, and JamC-QA), and eleven Japanese and multilingual LLMs are evaluated. The results show that Character Transposition and Character Replacement typos consistently reduce accuracy across benchmarks, whereas IME Conversion, Full-Width Conversion, and Homophone Conversion have relatively limited impact. These findings reveal that current Japanese LLMs remain vulnerable to realistic Japanese typing errors, particularly those that substantially distort the original input, highlighting the importance of robustness evaluation in practical input environments.

Figures & tables

Figure 1: Overview of the proposed evaluation framework. Original and typo-injected prompts are input to Japanese LLMs, and the resulting outputs are compared to measure the impact of different typo patterns on model performance.

Typo Type	Example
Character Transposition	Original: san ka $\rightarrow$ 参加 (participation) Typo: san ak $\rightarrow$ 三亜k (nonsensical output)
Character Replacement	Original: nose r u $\rightarrow$ 乗せる (place on) Typo: nose t u $\rightarrow$ の節 (phrase)
Homophone Conversion	Original: sanka $\rightarrow$ 参加 (participation) Typo: sanka $\rightarrow$ 酸化 (oxidation)
Japanese IME Conversion	Original: minecraft $\rightarrow$ minecraft Typo: minecraft $\rightarrow$ みねｃらｆｔ (nonsensical output)
Full-Width Conversion	Original: AI $\rightarrow$ AI Typo: AI $\rightarrow$ ＡＩ

Table 1: Examples of the proposed typo transformations

Model	Orig.	Trans.	Repl.	Homo.	IME	FullW.
Ministral-8B	0.583	0.565	0.503	0.578	0.576	0.573
Swallow-8B	0.677	0.607	0.615	0.672	0.677	0.672
ELYZA-8B	0.568	0.529	0.516	0.560	0.565	0.565
llm-jp-13B	0.615	0.578	0.531	0.602	0.607	0.609
llm-jp-7.2B	0.578	0.513	0.477	0.578	0.578	0.581
Sarashina-3B	0.698	0.620	0.638	0.688	0.695	0.701

Table 2: Accuracy of each model under different Japanese typo conditions. Orig. denotes the original prompts without typos. Values lower than the original accuracy are underlined. Average represents the mean accuracy across all evaluated models.

Figure 2: Accuracy of each model under different typo conditions averaged over the three benchmarks. The black line indicates the average accuracy across all evaluated models. Arrows indicate changes relative to the original prompts.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Orig.	Trans.	Repl.	Homo.	IME	FullW.
Ministral-8B	0.516	0.508	0.406	0.508	0.508	0.500
Swallow-8B	0.633	0.586	0.602	0.625	0.625	0.609
ELYZA-8B	0.461	0.477	0.430	0.445	0.461	0.453
llm-jp-13B	0.438	0.430	0.398	0.438	0.422	0.422
llm-jp-7.2B	0.406	0.383	0.305	0.398	0.398	0.406
Sarashina-3B	0.570	0.500	0.539	0.539	0.555	0.570

Appendix

Table 3: Accuracy under different typo conditions for each benchmark.

Figure 3: Accuracy of each model under different typo conditions for JMMLU, JCommonsenseQA, and JamC-QA. The black line denotes the average accuracy across all evaluated models, and arrows indicate changes from the original condition.

Explore similar work

Quantifying and Mitigating Korean Jamo-Level Typographical Vulnerabilities in Large Language Models
Aug 31, 2026Seojin Lee, Hwanhee LeeGrammatical Error Correction Large Language Models Fail
Chung-Ang University, Seoul, Korea
YOMI-Bench: A Benchmark for Evaluating Kanji Reading and Phonological Understanding of LLMs for Japanese
Jul 1, 2026Ryota Mibayashi, Hiroya Takamura, Hitomi YanakaGrapheme-To-Phoneme
Kobe University · National Institute of Advanced Industrial Science and Technology (AIST) · The University of Tokyo +2
Benchmarking Large Language Models for Grapheme-to-Phoneme Conversion: A Japanese Case Study
Jun 20, 2026Tomoki KoriyamaGrapheme-To-Phoneme Large Language Model Benchmarks
CyberAgent, Japan

Aug 31, 2026, cs.CLJ/K move · Enter open · S save

Quantifying and Mitigating Korean Jamo-Level Typographical Vulnerabilities in Large Language Models

Seojin Lee, Hwanhee Lee

Chung-Ang University, Seoul, Korea

Grammatical Error Correction Large Language Models Fail Corrupted Inputs Few-Shot Font Generation Chain-of-Thought Reasoning Quantifying

Open paper AskPDF arXiv

Korean introduces an additional typographical perturbation level not captured by ordinary character-level edit models: because syllable blocks are internally composed of sub-character units called jamo, keyboard-level errors can occur within a syllable, either producing a valid but semantically altered character or exposing raw jamo on the surface. Both outcomes disrupt sub-word tokenization and are not reliably corrected by existing grammatical error correction pipelines, leaving LLMs directly exposed to corrupted inputs. To quantify this vulnerability, we apply five jamo-level perturbation types to the KMMLU benchmark and evaluate four language models, finding that accuracy declines monotonically with perturbation intensity and that parameter scaling does not confer robustness against intra-syllabic noise. We further show that typo-corrupted inputs induce a distinct shift in internal representations that is not reducible to ordinary answer incorrectness, and that a simple linear probe trained on these representations detects unseen perturbation types with high AUROC. Motivated by this signal, we propose Typo-Aware Chain-of-Thought (TACoT), which routes inputs to chain-of-thought inference only when the probe detects a likely typo, recovering a substantial portion of the CoT accuracy gain at a fraction of the inference cost.

Abstract

Explore similar work

Quantifying and Mitigating Korean Jamo-Level Typographical Vulnerabilities in Large Language Models

YOMI-Bench: A Benchmark for Evaluating Kanji Reading and Phonological Understanding of LLMs for Japanese

Benchmarking Large Language Models for Grapheme-to-Phoneme Conversion: A Japanese Case Study