Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness

Zhipeng Yang; Shu Yang; Lijie Hu; Di Wang

cs.CLMar 11, 2026

Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness

Authors: Zhipeng Yang, Shu Yang, Lijie Hu, Di Wang

Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Provable Responsible AI and Data Analytics (PRADA) Lab · King Abdullah University of Science and Technology

Abstract

Large language models (LLMs) trained with canonical tokenization exhibit surprising robustness to non-canonical inputs such as character-level tokenization, yet the mechanisms underlying this robustness remain unclear. We study this phenomenon through mechanistic interpretability and identify a core process we term word recovery. We first introduce a decoding-based method to detect word recovery, showing that hidden states reconstruct canonical word-level token identities from character-level inputs. We then provide causal evidence by removing the corresponding subspace from hidden states, which consistently degrades downstream task performance. Finally, we conduct a fine-grained attention analysis and show that in-group attention among characters belonging to the same canonical token is critical for word recovery: masking such attention in early layers substantially reduces both recovery scores and task performance. Together, our findings provide a mechanistic explanation for tokenization robustness and identify word recovery as a key mechanism enabling LLMs to process character-level inputs.

Figures & tables

Figure 1: Overview of word recovery in large language models under character-level tokenization. A character-level input (e.g., “What is natural gas?”) is processed by the transformer, where early-layer in-group attention aggregates information among characters belonging to the same canonical token. This enables the model to reconstruct word-level representations in hidden states, which are then used for downstream contextual understanding.

Benchmark	Gemma-2-9B-It			Qwen2.5-7B-Instruct			Llama-3.2-3B-Instruct
Benchmark	Canon	Char $\Delta$	Recovery	Canon	Char $\Delta$	Recovery	Canon	Char $\Delta$	Recovery
ARC-E	$96.8$	$-3.50$	96.8	$95.6$	$-2.30$	53.8	$88.6$	$-35.80$	69.9
ARC-C	$90.6$	$-4.00$	96.6	$91.6$	$-4.00$	55.4	$76.6$	$-33.80$	71.2
CSQA	$80.7$	$-5.90$	96.4	$85.3$	$-9.10$	51.1	$73.9$	$-33.40$	55.9
OpenbookQA	$87.8$	$-7.80$	95.7	$86.6$	$-8.20$	51.6	$77.0$	$-30.80$	58.6

Table 1: Performance and word recovery under character-level tokenization. For each model and benchmark, we report accuracy under canonical tokenization ( Canon ), the change in accuracy under character-level tokenization ( Char

\Delta

), and the corresponding word recovery score measured from character-level inputs. Since recovery is defined layerwise, we report the maximum recovery score across layers in this table.

Panel of Figure 2 — Figure 2: Layerwise word recovery under character-level tokenization. We show the word recovery score across four datasets and three models. Recovery patterns are consistent across datasets within each model, while the layerwise behavior of recovery varies across models.

Panel of Figure 3 — Figure 3: Layerwise effects of targeted word-recovery intervention for Gemma-2-9B-It. The line plot shows task performance under targeted intervention applied starting from each transformer layer, while the shaded area shows the corresponding word recovery score at the intervention starting layer under character-level tokenization.

Panel of Figure 4 — Figure 4: Layerwise effects of targeted word-recovery intervention for Qwen2.5-7B-Instruct. The line plot shows task performance under targeted intervention applied starting from each transformer layer, while the shaded area shows the corresponding word recovery score at the intervention starting layer under character-level tokenization.

Panel of Figure 5 — Figure 5: Layerwise effects of targeted word-recovery intervention for Llama-3.2-3B-Instruct. The line plot shows task performance under targeted intervention applied starting from each transformer layer, while the shaded area shows the corresponding word recovery score at the intervention starting layer under character-level tokenization.

Panel of Figure 6 — Figure 6: Layerwise effects of in-group attention masking. For each model, we plot task accuracy under character-level tokenization when in-group attention (attention among characters belonging to the same canonical token) is masked starting from different transformer layers. Results are shown across four datasets.

Panel of Figure 7 — Figure 7: Early-layer in-group attention supports word recovery. We plot word recovery scores across layers when in-group attention is masked in the first five layers, compared to the baseline without masking (dashed).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Layerwise word recovery under random segmentation tokenization. Recovery dynamics remain qualitatively similar to the character-level setting across all models, suggesting that word recovery generalizes beyond character-level inputs.

Length	2	3	4	5+
Gemma	0.83	0.90	0.91	0.82
Qwen	0.86	0.76	0.63	0.39
Llama	0.89	0.83	0.75	0.72

Appendix

Table 2: Maximum recovery scores across canonical token lengths. Longer tokens are generally harder to recover, but recovery remains broadly distributed across token categories.

Panel of Figure 9 — Figure 9: Layerwise recovery curves grouped by canonical token length. Longer tokens generally exhibit weaker recovery, though the qualitative recovery dynamics remain consistent across token categories.

Model	Canon	Char $\Delta$
Qwen2.5-1.5B-Instruct	89.6	-24.5
Qwen2.5-3B-Instruct	92.3	-6.70
Qwen2.5-7B-Instruct	95.6	-2.30
Gemma-2-2B-It	87.0	-33.7
Gemma-2-9B-It	96.8	-3.50
Mistral-7B-Instruct-v0.1	83.5	-58.4

Appendix

Table 3: Performance degradation under character-level tokenization on ARC-E.

Panel of Figure 10 — Figure 10: Layerwise word recovery across additional models under character-level tokenization. (a) Within the Qwen family, recovery strength increases consistently with model size despite differences in embedding tying. (b) Within the Gemma family, the larger model exhibits substantially stronger early-layer recovery and robustness. (c) Mistral-7B-Instruct-v0.1 shows recovery dynamics comparable to other 7B-scale models despite using sliding-window attention, suggesting that this architectural feature alone does not explain robustness.

Clean run	Masking (France)
“Paris”: 0.9997710586	“Paris”: 0.1813361049
“paris”: 9.157e-05	“Rome”: 0.1692125201
“Par”: 5.018e-05	“Berlin”: 0.1167379990
“París”: 3.708e-05	“Madrid”: 0.0662391782
“パリ” : 1.445e-05	“Toronto”: 0.0468508899
“Pari”: 9.783e-06	“Re”: 0.0389084555

Appendix

Table 4: Top-10 next-token predictions for the query “What is the capital of France?” under character-level tokenization. In-group attention masking is applied only to the characters corresponding to “France”.

Panel of Figure 11 — Figure 11: Effect of top- $K$ choice on word recovery for Gemma-2-9B-It. We report layerwise word recovery scores under character-level tokenization for different values of $K\in\{1,2,3,5,10,20\}$ across four datasets. While larger $K$ yields uniformly higher absolute recovery scores, the overall layerwise trends and relative recovery dynamics remain consistent across choices of $K$ , indicating that our findings are not sensitive to the specific value of $K$ .

Panel of Figure 12 — Figure 12: In-group word recovery decoding across layers. We report word recovery scores when decoding is restricted to the character span corresponding to each canonical token (in-group decoding). Dashed lines show the original set-based decoding over the full character sequence. While in-group decoding yields lower absolute recovery scores, the layerwise recovery dynamics and relative trends remain consistent, indicating that word recovery does not arise from spurious matches elsewhere in the sequence.

Language Model Robustness

Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.

See all 471 papers

When Is Noise Response Universal? Tokenization as the Hidden Variable in Language Models
Aug 26, 2026Yefan Tao, Gerald Friedland, Luyang KongLanguage Model Robustness Tokenization in Language Models
Amazon
From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks
Apr 1, 2026Ayan Datta, Mounika Marreddy, Alexander Mehler +2Symbolic Reasoning in Language Models LLM Interpretability
IIIT Hyderabad · Goethe University, Frankfurt am Main, Germany · University of Sheffield, UK
Language Models are not Equally Robust to Non-Canonical Tokenization across Languages
Jul 29, 2026Poulami Ghosh, Preethi JyothiMultilingual Language Model Evaluation Language Model Robustness
*IIT Bombay, India

Aug 26, 2026J/K move · Enter open · S save

When Is Noise Response Universal? Tokenization as the Hidden Variable in Language Models

Yefan Tao, Gerald Friedland, Luyang Kong

Amazon

Language Model Robustness Tokenization in Language Models Subword Tokenization

Open paper AskPDF arXiv

The performance of textual neural models often degrades when their inputs are corrupted by noise such as typos, OCR errors, or dropped words. We study the degradation rate across neural models, both sentence embeddings and decoder-only LLMs, and find that how consistent it is depends on the scale of the noise: under word-level noise, models with very different architectures decline along nearly the same curve, while under character-level noise they separate. We further identify the determining factor to be the training objective, not the architecture: eight encoders spanning six pretraining paradigms are scattered initially, and collapse onto a common curve after a short contrastive training recipe. We trace the word/character split to tokenization: a single character edit forces the tokenizer to re-segment the surrounding word, disturbing the token sequence far more than dropping a whole word does. This finding and its underlying mechanism provide a practical means to predict a model's robustness to noise without any noisy evaluation, and to install robustness at a chosen noise scale through noise-augmented training.

Abstract

Language Model Robustness

Explore similar work

When Is Noise Response Universal? Tokenization as the Hidden Variable in Language Models

From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks

Language Models are not Equally Robust to Non-Canonical Tokenization across Languages