The performance of textual neural models often degrades when their inputs are corrupted by noise such as typos, OCR errors, or dropped words. We study the degradation rate across neural models, both sentence embeddings and decoder-only LLMs, and find that how consistent it is depends on the scale of the noise: under word-level noise, models with very different architectures decline along nearly the same curve, while under character-level noise they separate. We further identify the determining factor to be the training objective, not the architecture: eight encoders spanning six pretraining paradigms are scattered initially, and collapse onto a common curve after a short contrastive training recipe. We trace the word/character split to tokenization: a single character edit forces the tokenizer to re-segment the surrounding word, disturbing the token sequence far more than dropping a whole word does. This finding and its underlying mechanism provide a practical means to predict a model's robustness to noise without any noisy evaluation, and to install robustness at a chosen noise scale through noise-augmented training.
Figures & tables
Figure 1: The embedding word/character split (word left, character right). Top: the nine models’ normalized curves; the amber ±1σ across-model band is thin under word noise (CV 2.2%), six times wider under character noise (14.2%). Bottom: per-model decay rate km with bootstrap 95% CI. Under word noise every interval crosses a common k≈1.5 (shaded): one universality class; under character noise the rates spread with no common overlap ( Am≈1 , so km is unambiguous).
Figure 2: Eight encoders with different pretraining paradigms, before contrastive training (dashed) and after an identical recipe (solid). Under word dropout (left) the scattered rates converge to k≈1.5 (CV 14.3% → 2.8%); under character noise (right) training tightens them only partly (29.7% → 13.0%).
Figure 3: Decoder-only LLMs show the same word/character split as embeddings. Each curve is the retained agreement 1−JS(ε) ; the amber shading is the ±1σ spread across models (centered on the cross-model mean, black), whose width is the cross-model CV. It is thin under word dropout (left, CV 2.0%) and widens under character noise (right, CV 4.8%). The two largest models (32B, 72B), run on a coarser grid, appear in Appendix D only.
Figure 4: A masked-LM BERT-base checkpoint (red) falls off the word-dropout universal curve (black dashed, mean of the nine sentence-trained models); after contrastive fine-tuning (green) it lies on it.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Noise
Across-model retention range
Spread
Word, ε=0.1
87–89%
2 pts
Word, ε=0.3
63–65%
2 pts
Character, ε=0.1
51–63%
12 pts
Appendix
Table 1: The word-noise agreement is present before normalization does any work. For each noise level we list the range, across the nine models, of STS-B quality retained relative to clean ( ρ(ε)/ρ(0) ); since the clean baselines ρ(0) themselves span only 0.80 – 0.88 , the absolute scores cluster just as tightly. Under word dropout the models stay within a two-point band at every level; under character noise they already fan out roughly six times more.
Architecture
Scale
Exponent CV
95% CI
Embedding
word
4.3%
[2.2, 5.0]
Embedding
char
13.1%
[5.3, 17.7]
LLM
word
5.3%
[4.8, 6.4]
LLM
char
15.2%
[12.6, 20.4]
Appendix
Table 2: Across-model dispersion of the decay exponent (CV of km , %), with bootstrap 95% CIs. Embedding uses the raw exponent (with a bootstrap-over-models CI); LLM uses the amplitude-controlled (saturation-normalized) exponent (joint-bootstrap CI, refitting Am and km on each resample). In every cell the word exponent is far more shared than the character exponent, and the word/character CIs are disjoint.
Model
Backbone
Params
Dim
Pooling
Training
kchar
Word-universal
all-MiniLM-L6-v2
MiniLM
23M
384
mean
contrastive
8.38
yes
all-MiniLM-L12-v2
MiniLM
33M
384
mean
contrastive
8.34
yes
all-mpnet-base-v2
MPNet
110M
768
mean
contrastive
6.54
yes
bge-base-en-v1.5
BERT
110M
768
CLS
contrastive
8.70
yes
bge-large-en-v1.5
BERT
335M
1024
CLS
contrastive
7.97
yes
e5-base-v2
BERT
110M
768
mean
contrastive
7.41
yes
Appendix
Table 3: Embedding model inventory. The nine sentence-trained models differ in backbone, dimension, and pooling but join the common word-dropout curve (“word-universal”); the four baselines do not. kchar is the fitted character-noise decay rate on STS-B and varies widely even among the word-universal models, reflecting the tokenizer dependence of character-level response.
Model
Family
Params
GPT2-XL
GPT-2
1.5B
Qwen2.5-1.5B
Qwen2.5
1.5B
Qwen2.5-3B-Instruct
Qwen2.5
3B
Qwen2.5-7B
Qwen2.5
7B
Qwen2.5-7B-Instruct
Qwen2.5
7B
Qwen2.5-14B-Instruct
Qwen2.5
14B
Appendix
Table 4: The 18 decoder-only LLMs, spanning nine families, two Qwen generations, base/instruct/chat variants, and a 48 × range in scale (1.5B–72B). All use byte-pair-encoding (BPE) tokenizers. Per-model character-noise JS is in Table 5 .
Model
Params
JS@ ε =0.1
Llama-2-7B
7B
0.241
Mistral-7B-Instruct
7B
0.249
Qwen3-1.7B
1.7B
0.253
Llama-2-7B-Chat
7B
0.262
Phi-3-mini
3.8B
0.265
Qwen2.5-7B
7B
0.266
Appendix
Table 5: Raw JS divergence between clean and noisy next-token distributions at ε=0.1 under character noise, for the 18 LLMs. Both the absolute values and the normalized curve shapes spread across models (normalized-curve CV 4.8%, versus 2.0% under word noise).
Architecture
Pretraining paradigm
Word ret. @ 0.2
BERT-base
masked LM
0.73
RoBERTa-base
masked LM (byte-BPE)
0.75
DistilBERT
masked LM (distilled)
0.74
ALBERT
param. sharing
0.75
ELECTRA
replaced-token det.
0.72
DeBERTa
disentangled attn.
0.76
Appendix
Table 6: Per-model results for the cross-architecture experiment. The bottom rows give the before/after inter-model CV; the per-model column is word retention (fraction of clean STS performance kept at ε=0.2 under word dropout) after training.
Figure 5: Normalized STS-B degradation curves for the nine sentence-trained embedding models under five noise types (each thin line is one model; black is the mean). Curves collapse under word dropout (CV 2.2%) and diverge under all character-level noise (CV 10–20%).
Model / task
SFT
accuracy at ε=
0.0
0.05
0.1
0.15
0.2
0.3
Qwen2.5-1.5B, full SFT
AG-News
standard
0.67
0.68
0.67
0.63
0.62
0.46
AG-News
noise-aug
0.68
0.65
0.69
0.69
0.67
0.59
SST-2
standard
0.90
0.85
0.79
0.81
0.71
0.57
SST-2
noise-aug
0.87
0.83
0.77
0.76
0.68
0.62
Appendix
Table 7: Top-1 accuracy under character noise, standard SFT vs. noise-augmented SFT, at each noise level. Clean accuracy ( ε=0 ) is preserved while accuracy under noise improves, the gain growing with ε . Full-parameter SFT on Qwen2.5-1.5B (top) and LoRA SFT on Qwen2.5-7B-Instruct (bottom).
Architecture
Scale
Cross-model CV
Reading
Embedding
word
2.2%
universal
Embedding
char
14.2%
non-universal
LLM
word
2.0%
universal
LLM
char
4.8%
partly broken
untrained encoders
14.3%
(before obj.)
Appendix
Table 8: The word/character split across both architectures. For each architecture and noise scale we report the cross-model CV of the normalized degradation curve. Word noise collapses to near 2% for both, two model classes that share nothing but the training signal; character noise is far larger for embeddings and milder for LLMs. The last row gives, for comparison, the same architecturally diverse encoders measured before the shared objective: 14.3%, far more scattered.
Noise type
Scale
CV
Word dropout
word
2.2%
Character substitution
character
14.2%
Keyboard typo (QWERTY)
character
11.4%
OCR confusion
character
10.2%
Adjacent swap
character
20.1%
Appendix
Table 9: Word-level noise response is universal across the nine sentence-trained models; character-level noise spreads them apart. Pointwise CV of the normalized STS-B degradation curve (averaged over ε>0 ).
Table 10: Noise-augmented training in LLMs (mean JS@ ε=0.1 , ↓ ; lower is more robust) raises robustness mostly on the character scale: either augmentation cuts character-noise JS from 0.30 to 0.17, while word-noise JS is already low and barely moves. Mean over the augmented panel (9 models for character augmentation, 6 for word).
Tokenizer
Word ED
Char ED
Char/Word
Embedding models
WordPiece (MiniLM, BGE)
0.09
0.72
7.8 ×
SentencePiece (GTR-T5)
0.09
0.86
9.8 ×
LLMs
BPE (GPT-2)
0.11
0.71
6.8 ×
BPE (Qwen2.5)
0.11
0.66
6.2 ×
Appendix
Table 11: Normalized token-edit distance (Levenshtein on token-id sequences ÷ clean length) at ε=0.1 and the character/word ratio, for five tokenizer families spanning both model sets (WordPiece and SentencePiece for the embedding models, BPE for the LLMs). A character edit perturbs the token sequence several-fold more than a word edit for every tokenizer.
Noise type
Scale
CV
Random character
character
4.8%
Keyboard typo (QWERTY)
character
5.5%
OCR confusion
character
4.6%
Adjacent swap
character
4.9%
Word dropout
word
2.0%
Appendix
Table 12: Normalized-curve CV across LLMs: word dropout stays tight (2.0%) while every character-level noise type spreads (4.6–5.5%).
Figure 6: The tokenizer drives character-level divergence. The character-noise CV across embedding models drops from 14.2% (all nine) to 5.6–8.6% within a shared-tokenizer group.
Figure 7: Cross-model spread tracks token perturbation. A single block-corruption family with tunable block length g (characters) sweeps the induced token-edit distance: as g grows, the same corruption rate touches fewer words and shatters less (token-edit distance per token falls from 0.73 at g=1 to 0.36 at g=12 ), and the inter-model CV falls monotonically in lockstep, from 14.3% at g=1 to 5.6% at g=12 .
Figure 8: Noise-augmented contrastive training slows the character-noise decay. Under character noise, the standard model ( k=8.4 ) degrades fastest; char- and mixed-augmented training ( k=6.2 , 5.8 ) decay more slowly. Curves are normalized STS ρ across the embedding models.
Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.
Large language models (LLMs) trained with canonical tokenization exhibit surprising robustness to non-canonical inputs such as character-level tokenization, yet the mechanisms underlying this robustness remain unclear. We study this phenomenon through mechanistic interpretability and identify a core process we term word recovery. We first introduce a decoding-based method to detect word recovery, showing that hidden states reconstruct canonical word-level token identities from character-level inputs. We then provide causal evidence by removing the corresponding subspace from hidden states, which consistently degrades downstream task performance. Finally, we conduct a fine-grained attention analysis and show that in-group attention among characters belonging to the same canonical token is critical for word recovery: masking such attention in early layers substantially reduces both recovery scores and task performance. Together, our findings provide a mechanistic explanation for tokenization robustness and identify word recovery as a key mechanism enabling LLMs to process character-level inputs.
Zhipeng Yang, Shu Yang, Lijie Hu +1
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Provable Responsible AI and Data Analytics (PRADA) Lab · King Abdullah University of Science and Technology
Despite the existence of exponentially many valid tokenizations for a given string, language models operate on a single canonical sequence deterministically produced by the tokenizer, leaving the broader tokenization space largely uncharacterized. In this paper, we investigate this overlooked space by studying the behavior of language models under non-canonical tokenizations across diverse languages. For English, prior work shows that models are largely invariant to alternative tokenizations that represent the same underlying string. We ask whether this invariance generalizes to other languages beyond English. We conduct a multilingual study across 27 languages spanning diverse scripts and evaluate LLM behavior under alternative tokenizations across six downstream tasks. We find that tokenization invariance does not generalize: model behavior varies substantially across languages with instruction-tuned models exhibiting an average relative performance drop of 23.7% for Llama-3.1-8B, 11.4% for Qwen3-8B, and 9.9% for Gemma-3-12B. The variation of tokenization invariance is systematic across languages. Languages that exhibit higher token fragmentation show significantly greater sensitivity to non-canonical tokenizations. Our study of tokenization robustness serves as a diagnostic of how tightly a model is coupled to its tokenizer. These results demonstrate that tokenization robustness is not a universal property of language models, but depends strongly on the language and its interaction with the tokenizer. We also show that LoRA fine-tuning with multi-tokenization training data provides an effective mitigation for tokenization sensitivity. Fine-tuning on English alone improves tokenization robustness across languages, while systematically sampling diverse non-canonical tokenizations achieves the strongest overall performance.
Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs). Despite the importance of tokenization, its role in LM performance and behavior is poorly understood due to the challenge of measuring the impact of tokenization in isolation. To address this need, we present TokSuite, a collection of models and a benchmark that supports research into tokenization's influence on LMs. Specifically, we release fourteen pre-trained models that use different off-the-shelf tokenizers but are otherwise identical, using the same architecture, dataset, training budget, and initialization. We also release a multilingual robustness benchmark that measures model performance under real-world perturbations in English, Chinese, Farsi, Italian, and Turkish, curated by native annotators. Together, TokSuite allows robust decoupling of the influence of a model's tokenizer, supporting a series of novel findings that elucidate the respective benefits and shortcomings of a wide range of popular tokenizers.
Gül Sena Altıntaş, Malikeh Ehghaghi, Brian Lester +4
University of Toronto · Vector Institute · Google DeepMind +4