When Is Noise Response Universal? Tokenization as the Hidden Variable in Language Models

Yefan Tao; Gerald Friedland; Luyang Kong

cs.CLAug 26, 2026

When Is Noise Response Universal? Tokenization as the Hidden Variable in Language Models

Authors: Yefan Tao, Gerald Friedland, Luyang Kong

Organizations: Amazon

Abstract

The performance of textual neural models often degrades when their inputs are corrupted by noise such as typos, OCR errors, or dropped words. We study the degradation rate across neural models, both sentence embeddings and decoder-only LLMs, and find that how consistent it is depends on the scale of the noise: under word-level noise, models with very different architectures decline along nearly the same curve, while under character-level noise they separate. We further identify the determining factor to be the training objective, not the architecture: eight encoders spanning six pretraining paradigms are scattered initially, and collapse onto a common curve after a short contrastive training recipe. We trace the word/character split to tokenization: a single character edit forces the tokenizer to re-segment the surrounding word, disturbing the token sequence far more than dropping a whole word does. This finding and its underlying mechanism provide a practical means to predict a model's robustness to noise without any noisy evaluation, and to install robustness at a chosen noise scale through noise-augmented training.

Figures & tables

Figure 1: The embedding word/character split (word left, character right). Top: the nine models’ normalized curves; the amber

\pm 1\sigma

across-model band is thin under word noise (CV 2.2%), six times wider under character noise (14.2%). Bottom: per-model decay rate

k_{m}

with bootstrap 95% CI. Under word noise every interval crosses a common

k\approx 1.5

(shaded): one universality class; under character noise the rates spread with no common overlap (

A_{m}\approx 1

, so

k_{m}

is unambiguous).

Figure 2: Eight encoders with different pretraining paradigms, before contrastive training (dashed) and after an identical recipe (solid). Under word dropout (left) the scattered rates converge to

k\approx 1.5

(CV 14.3%

\,\to\,

2.8%); under character noise (right) training tightens them only partly (29.7%

\,\to\,

13.0%).

Figure 3: Decoder-only LLMs show the same word/character split as embeddings. Each curve is the retained agreement

1-\text{JS}(\varepsilon)

; the amber shading is the

\pm 1\sigma

spread across models (centered on the cross-model mean, black), whose width is the cross-model CV. It is thin under word dropout (left, CV 2.0%) and widens under character noise (right, CV 4.8%). The two largest models (32B, 72B), run on a coarser grid, appear in Appendix D only.

Figure 4: A masked-LM BERT-base checkpoint (red) falls off the word-dropout universal curve (black dashed, mean of the nine sentence-trained models); after contrastive fine-tuning (green) it lies on it.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Noise	Across-model retention range	Spread
Word, $\varepsilon{=}0.1$	87–89%	2 pts
Word, $\varepsilon{=}0.3$	63–65%	2 pts
Character, $\varepsilon{=}0.1$	51–63%	12 pts

Appendix

Table 1: The word-noise agreement is present before normalization does any work. For each noise level we list the range, across the nine models, of STS-B quality retained relative to clean (

\rho(\varepsilon)/\rho(0)

); since the clean baselines

\rho(0)

themselves span only

0.80

–

0.88

, the absolute scores cluster just as tightly. Under word dropout the models stay within a two-point band at every level; under character noise they already fan out roughly six times more.

Architecture	Scale	Exponent CV	95% CI
Embedding	word	4.3%	[2.2, 5.0]
Embedding	char	13.1%	[5.3, 17.7]
LLM	word	5.3%	[4.8, 6.4]
LLM	char	15.2%	[12.6, 20.4]

Appendix

Table 2: Across-model dispersion of the decay exponent (CV of

k_{m}

, %), with bootstrap 95% CIs. Embedding uses the raw exponent (with a bootstrap-over-models CI); LLM uses the amplitude-controlled (saturation-normalized) exponent (joint-bootstrap CI, refitting

A_{m}

and

k_{m}

on each resample). In every cell the word exponent is far more shared than the character exponent, and the word/character CIs are disjoint.

Model	Backbone	Params	Dim	Pooling	Training	$k_{\text{char}}$	Word-universal
all-MiniLM-L6-v2	MiniLM	23M	384	mean	contrastive	8.38	yes
all-MiniLM-L12-v2	MiniLM	33M	384	mean	contrastive	8.34	yes
all-mpnet-base-v2	MPNet	110M	768	mean	contrastive	6.54	yes
bge-base-en-v1.5	BERT	110M	768	CLS	contrastive	8.70	yes
bge-large-en-v1.5	BERT	335M	1024	CLS	contrastive	7.97	yes
e5-base-v2	BERT	110M	768	mean	contrastive	7.41	yes

Appendix

Table 3: Embedding model inventory. The nine sentence-trained models differ in backbone, dimension, and pooling but join the common word-dropout curve (“word-universal”); the four baselines do not.

k_{\text{char}}

is the fitted character-noise decay rate on STS-B and varies widely even among the word-universal models, reflecting the tokenizer dependence of character-level response.

Model	Family	Params
GPT2-XL	GPT-2	1.5B
Qwen2.5-1.5B	Qwen2.5	1.5B
Qwen2.5-3B-Instruct	Qwen2.5	3B
Qwen2.5-7B	Qwen2.5	7B
Qwen2.5-7B-Instruct	Qwen2.5	7B
Qwen2.5-14B-Instruct	Qwen2.5	14B

Appendix

Table 4: The 18 decoder-only LLMs, spanning nine families, two Qwen generations, base/instruct/chat variants, and a 48

\times

range in scale (1.5B–72B). All use byte-pair-encoding (BPE) tokenizers. Per-model character-noise JS is in Table 5 .

Model	Params	JS@ $\varepsilon$ =0.1
Llama-2-7B	7B	0.241
Mistral-7B-Instruct	7B	0.249
Qwen3-1.7B	1.7B	0.253
Llama-2-7B-Chat	7B	0.262
Phi-3-mini	3.8B	0.265
Qwen2.5-7B	7B	0.266

Appendix

Table 5: Raw JS divergence between clean and noisy next-token distributions at

\varepsilon=0.1

under character noise, for the 18 LLMs. Both the absolute values and the normalized curve shapes spread across models (normalized-curve CV 4.8%, versus 2.0% under word noise).

Architecture	Pretraining paradigm	Word ret. @ 0.2
BERT-base	masked LM	0.73
RoBERTa-base	masked LM (byte-BPE)	0.75
DistilBERT	masked LM (distilled)	0.74
ALBERT	param. sharing	0.75
ELECTRA	replaced-token det.	0.72
DeBERTa	disentangled attn.	0.76

Appendix

Table 6: Per-model results for the cross-architecture experiment. The bottom rows give the before/after inter-model CV; the per-model column is word retention (fraction of clean STS performance kept at

\varepsilon{=}0.2

under word dropout) after training.

Figure 5: Normalized STS-B degradation curves for the nine sentence-trained embedding models under five noise types (each thin line is one model; black is the mean). Curves collapse under word dropout (CV 2.2%) and diverge under all character-level noise (CV 10–20%).

Qwen2.5-1.5B, full SFT
Model / task	SFT	accuracy at $\varepsilon=$
		0.0	0.05	0.1	0.15	0.2	0.3
AG-News	standard	0.67	0.68	0.67	0.63	0.62	0.46
AG-News	noise-aug	0.68	0.65	0.69	0.69	0.67	0.59
SST-2	standard	0.90	0.85	0.79	0.81	0.71	0.57
SST-2	noise-aug	0.87	0.83	0.77	0.76	0.68	0.62

Appendix

Table 7: Top-1 accuracy under character noise, standard SFT vs. noise-augmented SFT, at each noise level. Clean accuracy (

\varepsilon{=}0

) is preserved while accuracy under noise improves, the gain growing with

\varepsilon

. Full-parameter SFT on Qwen2.5-1.5B (top) and LoRA SFT on Qwen2.5-7B-Instruct (bottom).

Architecture	Scale	Cross-model CV	Reading
Embedding	word	2.2%	universal
Embedding	char	14.2%	non-universal
LLM	word	2.0%	universal
LLM	char	4.8%	partly broken
untrained encoders		14.3%	(before obj.)

Appendix

Table 8: The word/character split across both architectures. For each architecture and noise scale we report the cross-model CV of the normalized degradation curve. Word noise collapses to near 2% for both, two model classes that share nothing but the training signal; character noise is far larger for embeddings and milder for LLMs. The last row gives, for comparison, the same architecturally diverse encoders measured before the shared objective: 14.3%, far more scattered.

Noise type	Scale	CV
Word dropout	word	2.2%
Character substitution	character	14.2%
Keyboard typo (QWERTY)	character	11.4%
OCR confusion	character	10.2%
Adjacent swap	character	20.1%

Appendix

Table 9: Word-level noise response is universal across the nine sentence-trained models; character-level noise spreads them apart. Pointwise CV of the normalized STS-B degradation curve (averaged over

\varepsilon>0

	robustness ([email protected], $\downarrow$ )
Augmentation	char	word
None (baseline)	0.30	0.17
Character-aug	0.17	0.15
Word-aug	0.17	0.15

Appendix

Table 10: Noise-augmented training in LLMs (mean JS@

\varepsilon{=}0.1

\downarrow

; lower is more robust) raises robustness mostly on the character scale: either augmentation cuts character-noise JS from 0.30 to 0.17, while word-noise JS is already low and barely moves. Mean over the augmented panel (9 models for character augmentation, 6 for word).

Embedding models
Tokenizer	Word ED	Char ED	Char/Word
WordPiece (MiniLM, BGE)	0.09	0.72	7.8 $\times$
SentencePiece (GTR-T5)	0.09	0.86	9.8 $\times$
LLMs
BPE (GPT-2)	0.11	0.71	6.8 $\times$
BPE (Qwen2.5)	0.11	0.66	6.2 $\times$

Appendix

Table 11: Normalized token-edit distance (Levenshtein on token-id sequences

\div

clean length) at

\varepsilon{=}0.1

and the character/word ratio, for five tokenizer families spanning both model sets (WordPiece and SentencePiece for the embedding models, BPE for the LLMs). A character edit perturbs the token sequence several-fold more than a word edit for every tokenizer.

Noise type	Scale	CV
Random character	character	4.8%
Keyboard typo (QWERTY)	character	5.5%
OCR confusion	character	4.6%
Adjacent swap	character	4.9%
Word dropout	word	2.0%

Appendix

Table 12: Normalized-curve CV across LLMs: word dropout stays tight (2.0%) while every character-level noise type spreads (4.6–5.5%).

Figure 6: The tokenizer drives character-level divergence. The character-noise CV across embedding models drops from 14.2% (all nine) to 5.6–8.6% within a shared-tokenizer group.

Figure 7: Cross-model spread tracks token perturbation. A single block-corruption family with tunable block length

g

(characters) sweeps the induced token-edit distance: as

g

grows, the same corruption rate touches fewer words and shatters less (token-edit distance per token falls from 0.73 at

g{=}1

to 0.36 at

g{=}12

), and the inter-model CV falls monotonically in lockstep, from 14.3% at

g{=}1

to 5.6% at

g{=}12

Figure 8: Noise-augmented contrastive training slows the character-noise decay. Under character noise, the standard model (

k{=}8.4

) degrades fastest; char- and mixed-augmented training (

k{=}6.2

5.8

) decay more slowly. Curves are normalized STS

\rho

across the embedding models.

Language Model Robustness

Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.

See all 471 papers

Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness
Mar 11, 2026Zhipeng Yang, Shu Yang, Lijie Hu +1Language Model Robustness Tokenization in Language Models
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Provable Responsible AI and Data Analytics (PRADA) Lab · King Abdullah University of Science and Technology
Language Models are not Equally Robust to Non-Canonical Tokenization across Languages
Jul 29, 2026Poulami Ghosh, Preethi JyothiMultilingual Language Model Evaluation Language Model Robustness
*IIT Bombay, India
TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
Dec 23, 2025Gül Sena Altıntaş, Malikeh Ehghaghi, Brian Lester +4Subword Tokenization Tokenization in Language Models
University of Toronto · Vector Institute · Google DeepMind +4

Mar 11, 2026J/K move · Enter open · S save

Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness

Zhipeng Yang, Shu Yang, Lijie Hu, Di Wang

Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Provable Responsible AI and Data Analytics (PRADA) Lab · King Abdullah University of Science and Technology

Language Model Robustness Tokenization in Language Models LLM Interpretability

Open paper AskPDF arXiv

Large language models (LLMs) trained with canonical tokenization exhibit surprising robustness to non-canonical inputs such as character-level tokenization, yet the mechanisms underlying this robustness remain unclear. We study this phenomenon through mechanistic interpretability and identify a core process we term word recovery. We first introduce a decoding-based method to detect word recovery, showing that hidden states reconstruct canonical word-level token identities from character-level inputs. We then provide causal evidence by removing the corresponding subspace from hidden states, which consistently degrades downstream task performance. Finally, we conduct a fine-grained attention analysis and show that in-group attention among characters belonging to the same canonical token is critical for word recovery: masking such attention in early layers substantially reduces both recovery scores and task performance. Together, our findings provide a mechanistic explanation for tokenization robustness and identify word recovery as a key mechanism enabling LLMs to process character-level inputs.

Abstract

Language Model Robustness

Explore similar work

Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness

Language Models are not Equally Robust to Non-Canonical Tokenization across Languages

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior