Large language models (LLMs) have achieved strong performance across various natural language processing tasks. However, their robustness to typographical errors remains underexplored, particularly in Japanese, where text input involves multiple writing systems and IME-based conversion. In this study, we evaluate the robustness of Japanese LLMs against realistic Japanese-specific typos. We introduce five typo categories: Character Transposition, Character Replacement, Homophone Conversion, Japanese IME Conversion, and Full-Width Conversion. These perturbations are applied to three Japanese benchmark datasets (JMMLU, JCommonsenseQA, and JamC-QA), and eleven Japanese and multilingual LLMs are evaluated. The results show that Character Transposition and Character Replacement typos consistently reduce accuracy across benchmarks, whereas IME Conversion, Full-Width Conversion, and Homophone Conversion have relatively limited impact. These findings reveal that current Japanese LLMs remain vulnerable to realistic Japanese typing errors, particularly those that substantially distort the original input, highlighting the importance of robustness evaluation in practical input environments.
Figures & tables
Figure 1: Overview of the proposed evaluation framework. Original and typo-injected prompts are input to Japanese LLMs, and the resulting outputs are compared to measure the impact of different typo patterns on model performance.
Typo Type
Example
Character Transposition
Original: san ka → 参加 (participation) Typo: san ak → 三亜k (nonsensical output)
Character Replacement
Original: nose r u → 乗せる (place on) Typo: nose t u → の節 (phrase)
Table 1: Examples of the proposed typo transformations
Model
Orig.
Trans.
Repl.
Homo.
IME
FullW.
Ministral-8B
0.583
0.565
0.503
0.578
0.576
0.573
Swallow-8B
0.677
0.607
0.615
0.672
0.677
0.672
ELYZA-8B
0.568
0.529
0.516
0.560
0.565
0.565
llm-jp-13B
0.615
0.578
0.531
0.602
0.607
0.609
llm-jp-7.2B
0.578
0.513
0.477
0.578
0.578
0.581
Sarashina-3B
0.698
0.620
0.638
0.688
0.695
0.701
Table 2: Accuracy of each model under different Japanese typo conditions. Orig. denotes the original prompts without typos. Values lower than the original accuracy are underlined. Average represents the mean accuracy across all evaluated models.
Figure 2: Accuracy of each model under different typo conditions averaged over the three benchmarks. The black line indicates the average accuracy across all evaluated models. Arrows indicate changes relative to the original prompts.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Orig.
Trans.
Repl.
Homo.
IME
FullW.
Ministral-8B
0.516
0.508
0.406
0.508
0.508
0.500
Swallow-8B
0.633
0.586
0.602
0.625
0.625
0.609
ELYZA-8B
0.461
0.477
0.430
0.445
0.461
0.453
llm-jp-13B
0.438
0.430
0.398
0.438
0.422
0.422
llm-jp-7.2B
0.406
0.383
0.305
0.398
0.398
0.406
Sarashina-3B
0.570
0.500
0.539
0.539
0.555
0.570
Appendix
Table 3: Accuracy under different typo conditions for each benchmark.
Figure 3: Accuracy of each model under different typo conditions for JMMLU, JCommonsenseQA, and JamC-QA. The black line denotes the average accuracy across all evaluated models, and arrows indicate changes from the original condition.
Korean introduces an additional typographical perturbation level not captured by ordinary character-level edit models: because syllable blocks are internally composed of sub-character units called jamo, keyboard-level errors can occur within a syllable, either producing a valid but semantically altered character or exposing raw jamo on the surface. Both outcomes disrupt sub-word tokenization and are not reliably corrected by existing grammatical error correction pipelines, leaving LLMs directly exposed to corrupted inputs. To quantify this vulnerability, we apply five jamo-level perturbation types to the KMMLU benchmark and evaluate four language models, finding that accuracy declines monotonically with perturbation intensity and that parameter scaling does not confer robustness against intra-syllabic noise. We further show that typo-corrupted inputs induce a distinct shift in internal representations that is not reducible to ordinary answer incorrectness, and that a simple linear probe trained on these representations detects unseen perturbation types with high AUROC. Motivated by this signal, we propose Typo-Aware Chain-of-Thought (TACoT), which routes inputs to chain-of-thought inference only when the probe detects a likely typo, recovering a substantial portion of the CoT accuracy gain at a fraction of the inference cost.
We propose YOMI-Bench, a benchmark for evaluating kanji reading and phonological understanding of large language models (LLMs) for Japanese. In Japanese, a single kanji character often has multiple possible readings, making it difficult to infer the correct reading from surface-level text alone. Due to these linguistic characteristics, it is empirically known that LLMs exhibit low performance in kanji reading for Japanese. The proposed YOMI-Bench consists of four tasks specifically designed to evaluate kanji reading performance in Japanese. In our evaluation using YOMI-Bench, we assessed one multilingual open LLM, four Japanese-specific open LLMs, and five commercial LLMs. As a result, we found that even Japanese-specific models show low performance, and that commercial models also perform poorly on generation tasks that require consideration of kanji readings.
Ryota Mibayashi, Hiroya Takamura, Hitomi Yanaka
Kobe University · National Institute of Advanced Industrial Science and Technology (AIST) · The University of Tokyo +2
Grapheme-to-phoneme (G2P) conversion is essential for controllable and robust text-to-speech, and large language models (LLMs), with broad linguistic knowledge, offer a promising approach. We benchmarked over 30 LLMs on Japanese G2P, comparing them with conventional morphological analyzers on 3000 manually annotated sentences. We evaluated two prompting strategies: a parse mode, where the LLM performs morphological analysis followed by rule-based kana conversion, and a direct mode, where the LLM directly predicts kana readings. The results show that model size, version, and Japanese-specialized training are key factors, with the best LLMs achieving kana character error rate below 0.52% vs. the best conventional tool (1.03%). Parse mode outperforms direct mode for most models, as rule-based post-processing relieves the LLM of handling complex pronunciation rules. We also show that feeding LLM-predicted kana into a kana-input TTS yields better pronunciation than end-to-end TTS.