GlitchPatch: Repairing Glitch Tokens in Frozen Language Models via Local Retokenization

Kunsheng Tang; Peigui Qi; Yide Song; Peijun Huang; Weiming Zhang; Nenghai Yu

cs.CLOct 3, 2026

GlitchPatch: Repairing Glitch Tokens in Frozen Language Models via Local Retokenization

Authors: Kunsheng Tang, Peigui Qi, Yide Song, Peijun Huang, Weiming Zhang, Nenghai Yu

Organizations: University of Science and Technology of China · University of Washington · Wuhan University

Abstract

Glitch tokens are anomalous vocabulary entries that can cause large language models (LLMs) to produce outputs inconsistent with their inputs. Existing repair methods require access to model internals, making them impractical for frozen checkpoints. We investigate whether glitch tokens can be repaired outside the model by optimizing the input tokenization. An empirical study on BPE merge-rule deletion reveals that (1)deleting a glitch token's merge rule can fix a substantial fraction of failures, yet disrupting normal tokens sharing intermediate merge nodes causes the overall glitch rate to rise, and (2)different decomposition granularities yield non-monotonic fix rates while collateral damage on normal tokens grows monotonically. Motivated by these findings, we propose GlitchPatch, a repair framework for frozen language models based on local retokenization, consisting of two stages: the offline stage uses Behavioral Path Optimization (BPO) to find the behaviorally optimal replacement token sequence for each glitch token and compiles validated replacements into a rule table; the online stage substitutes only the IDs of matched glitch tokens in the canonical token sequence, with no modification to model parameters or internal states. Experiments on ten models spanning six tokenizer families show that GlitchPatch achieves an 85.10% mean fix rate, outperforming the strongest baseline by 14.37 percentage points, and reduces the average glitch rate from 14.88% to 2.27%. GlitchPatch achieves a 0.00% RR in full-vocabulary evaluation and leaves rule-unmatched inputs unchanged by design. We further evaluate the practical impact of repair from the perspectives of time cost, language understanding, and capability, supporting its deployment feasibility. We hope this work provides a practical option for improving tokenizer reliability.

Figures & tables

Figure 1: A glitch-token example and our GlitchPatch repair. The strings “fortawesome” (glitch token) and “fort awesome” (normal tokens) differ by one space. The model describes an icon library for the latter but a fort-building game for the former. GlitchPatch restores the relevant answer without changing the input text.

Figure 2: BPE tokenization of “Hello!” with Qwen3.8-27B.

Figure 3: Repair gains and new errors from global merge deletion. (a) FR, RR, and GR after depth-1 backtracking on two models. (b) The three metrics at increasing backtracking depths.

Figure 4: Collateral effects of shared merges and path variation across depths on Qwen3.8-27B. (a) Deleting the rule for the glitch token “rastructure” blocks the construction path of the normal token “Infrastructure”, changing its encoding from one token to three and causing detection failure. (b) Different backtracking depths produce different sub-token paths for both tokens.

Figure 5: The two-stage GlitchPatch framework. The offline stage uses BPO to search for the behaviorally optimal replacement sequence for each glitch token and compiles validated results into a rule table. The online stage performs budget-constrained local ID substitution on the canonical tokenization output.

Model	Glitch	Fixed	Reg.	FR% $\,\uparrow$	RR% $\,\downarrow$	GR% $\,\downarrow$
Llama-3.1-8B	22,026	18,722	0	85.00	0.00	2.58 $-14.60$
Llama-3.1-70B	15,878	14,455	0	91.04	0.00	1.11 $-11.27$
Gemma-4-31B	36,045	32,080	0	89.00	0.00	1.51 $-12.24$
Gemma-4-E4B	48,287	38,873	0	80.50	0.00	3.59 $-14.83$
Qwen3.8-27B	29,451	25,770	0	87.50	0.00	1.48 $-10.38$
Qwen3.6-27B	31,040	26,546	0	85.52	0.00	1.81 $-10.69$

Table 1: GlitchPatch repair results across models. Full-vocabulary, token-level evaluation. Red subscripts show GR changes from the original model in percentage points.

Model	GlitchProber	GlitchEdit	GlitchCleaner	GlitchPatch
Llama-3.1-8B	21.29 $+63.71$	59.16 $+25.84$	68.07 $+16.93$	85.00
Llama-3.1-70B	30.97 $+60.07$	65.03 $+26.01$	78.48 $+12.56$	91.04
Gemma-4-31B	36.19 $+52.81$	66.37 $+22.63$	81.28 $+7.72$	89.00
Gemma-4-E4B	12.35 $+68.15$	51.77 $+28.73$	59.12 $+21.38$	80.50
Qwen3.8-27B	27.01 $+60.49$	60.48 $+27.02$	74.02 $+13.48$	87.50
Qwen3.6-27B	30.09 $+55.43$	61.67 $+23.85$	76.06 $+9.46$	85.52

Table 2: Fix Rate comparison across four methods (%

\uparrow

). Green subscripts show GlitchPatch ’s gains over each baseline in percentage points.

Figure 6: Practicality comparison across three representative models. (a) Offline build time and online latency increase. (b) Relative reduction in token-level loss. (c) Change in General Score.

Paths	Count	Proportion%	FR%
0	1,473	5.00	0.0
1–3	8,847	30.04	85.4
4–10	11,742	39.87	93.8
11–50	5,849	19.86	97.1
$>$ 50	1,540	5.23	98.7

Table 3: Repairability analysis on Qwen3.8-27B.

$K$	FR (%)	Est. build cost
1	71.83	$\sim$ 47,000
2	77.46	$\sim$ 76,000
4	82.91	$\sim$ 135,000
8	87.50	$\sim$ 253,000
16	87.93	$\sim$ 488,000

Table 4: Candidate count

K

on Qwen3.8-27B.

Setting	FR (%)	Note
Full GlitchPatch	87.50
BPE merge history only	73.82	No out-of-tree paths
Remove pair term	81.73	Proxy ignores adjacency
Remove full-path evaluation	85.34	Proxy score only
Lower utility threshold ( $\gamma=0$ )	90.23	Lower behavioral stability

Table 5: Component ablation on Qwen3.8-27B.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	Instruction	Expected output	Scoring criterion
Repetition	Repeat the string: {target}	The target itself	Exact byte match
Spelling	Spell character by character: {target}	Character sequence	Unicode character order match
Length	Count the characters: {target}	Correct count	Extracted number matches Unicode char count

Appendix

Table A1: The three GlitchQuiz tasks and their scoring rules.

Model	Vocab	Glitch	Glitch %	Normal
Llama-3.1-8B	128,256	22,026	17.17	106,230
Llama-3.1-70B	128,256	15,878	12.38	112,378
Gemma-4-31B	262,144	36,045	13.75	226,099
Gemma-4-E4B	262,144	48,287	18.42	213,857
Qwen3.8-27B	248,320	29,451	11.86	218,869
Qwen3.6-27B	248,320	31,040	12.50	217,280

Appendix

Table A2: Vocabulary sizes and initial glitch counts per model.

Method	Fixed	Avg. FR(%)	Regressed	Avg. RR(%)	Avg. GR(%)
GlitchProber	63,614	24.63	382	0.024	11.37
GlitchEdit	154,382	58.39	182	0.012	6.30
GlitchCleaner	185,940	70.73	713	0.046	4.53
GlitchPatch	225,197	85.10	0	0.00	2.27

Appendix

Table B1: Aggregate repair results across all four methods.

Method	Fixed	Unfixed	FR%	Reg.	Maint.	RR%	GR%
Llama-8B $\|G_{0}\|=22,026$ ; $\|N_{0}\|=106,230$
GlitchProber	4,689	17,337	21.29	32	106,198	0.03	13.54
GlitchEdit	13,031	8,995	59.16	11	106,219	0.01	7.02
GlitchCleaner	14,993	7,033	68.07	53	106,177	0.05	5.52
GlitchPatch	18,722	3,304	85.00	0	106,230	0.00	2.58
Llama-70B $\|G_{0}\|=15,878$ ; $\|N_{0}\|=112,378$

Appendix

Table B2: Per-model complete counts for all four methods. Glitch and normal counts are in model heading.

Figure 7: Candidate paths and rule selection. (a) Normal vocabulary entries define different paths over the same byte sequence. The original glitch ID is excluded from replacement edges. (b) Proxy ranking, full-sequence evaluation, and independent calibration select a rule.

Model	Type	Vocab	Adaptation
Llama-3.1-8B	Byte-level BPE	128,256	Raw byte graph
Llama-3.1-70B	Byte-level BPE	128,256	Shared with 8B
Gemma-4-31B	SentencePiece	262,144	Normalizer + state
Gemma-4-E4B	SentencePiece	262,144	Shared with 31B
Qwen3.8-27B	Byte-level BPE	248,320	Raw byte graph
Qwen3.6-27B	Byte-level BPE	248,320	Same as above

Appendix

Table C1: Tokenizer adapter targets per model.

Case	Required handling	Invalid shortcut
Byte-level BPE	Build edges from exact raw bytes	Substitute visible Unicode for incomplete bytes
SentencePiece BPE	Preserve normalizer, leading space, byte fallback, state	Infer algorithm from file format alone
Partial UTF-8	Expand to complete verifiable span; compare joint decoding	Decode to replacement character and build from it
Added/special tokens	Preserve IDs, order, control structure; exclude as edges	Reconstruct control markers via string matching
Normalization	Use $T(x)$ ’s actual byte content as reference	Claim preservation without checking
Incompatible/over-budget	Retain canonical IDs	Delete chars, insert spaces, or rewrite text

Appendix

Table C2: Tokenizer adapter contract.

Dim.	Variant	Prompt structure	Criterion
Rep.	Direct	Repeat $g$ exactly	Exact match
	Embedded	Extract $g$ from sentence	Exact extraction
	Label bind	Bind $g$ to label A; query	Exact match
	Selection	Select $g$ from list	Correct selection
Spell	Forward	List chars of $g$ in order	Correct sequence
	Reverse	List chars in reverse	Correct reverse

Appendix

Table D1: Twelve selection probes: four variants per dimension.

g

denotes the target string.

Parameter	Value	Meaning
$K$	8	Max candidate paths per target
$B$	8	Max added tokens per span
Span content limit	64 bytes	Retain if exceeded
$\eta$	0.5	Adjacent-pair proxy weight
$\lambda_{r}$	0.02	Proxy length cost
$\kappa$	2	Regression penalty coefficient

Appendix

Table D2: Default GlitchPatch parameters.

Method	Mechanism & config	Parameter selection
GlitchProber	Activation statistics + neuron correction; 2,000 ref. tokens; threshold 0.99; later MLP layers	Relative layer interval grid; ${\sim}[0.60L,0.90L]$
GlitchEdit	Target embedding mixed with normal neighbors; $\alpha\in[0,1]$ , step 0.1; 1–10 neighbors	Behavioral acceptance; retain failure fallback
GlitchCleaner	Target-triggered MLP gated LoRA; rank 4, scaling 4, lr $3{\times}10^{-4}$	Freeze base weights; independent calib. from $\leq$ 20 epochs

Appendix

Table D3: Baseline construction and adaptation settings.

$\gamma$	0.00	0.05	0.10
FR (%)	90.23	87.50	80.86
Coverage (%)	94.8	89.2	81.8

Appendix

Table D4: Utility threshold

\gamma

$\eta$	0.00	0.25	0.50	1.00
FR (%)	81.73	85.03	87.50	86.47

Appendix

Table D5: Adjacent-pair weight

\eta

$B$	4	8	16
FR (%)	82.48	87.50	88.52
Mean growth (%)	1.8	2.6	4.1

Appendix

Table D6: Span token budget

B

Setting	Llama-8B	Qwen3.8	Gemma-31B
Full GlitchPatch	85.00	87.50	89.00
BPE history only	72.41	73.82	76.53
Remove pair term	79.27	81.73	84.18
Remove path eval.	82.86	85.34	87.12
Threshold $\gamma=0$	88.16	90.23	91.74

Appendix

Table D7: Three-model component ablation (FR%).

Model	Mean $\rho(-J,U)$
Llama-3.1-8B	0.72
Qwen3.8-27B	0.68
Gemma-4-31B	0.71

Appendix

Table D8: Proxy score vs. true utility.

Model	TP	Base (s)	Incr. (%)	Host Prep. (ms)	Build (h)
Llama-3.1-8B	1	1.280	0.071	0.231	0.61
Llama-3.1-70B	4	4.600	0.058	0.247	2.08
Gemma-4-31B	2	2.950	0.069	0.258	1.27
Gemma-4-E4B	1	1.100	0.061	0.219	0.49
Qwen3.8-27B	2	2.750	0.079	0.243	1.02
Qwen3.6-27B	2	2.800	0.068	0.238	0.96

Appendix

Table E1: GlitchPatch per-model serving latency and offline build time.

Model	Prober	Edit	Cleaner	GlitchPatch
Llama-3.1-8B	0.80	2.80	7.20	0.61
Llama-3.1-70B	4.80	9.80	22.50	2.08
Gemma-4-31B	2.80	5.20	12.80	1.27
Gemma-4-E4B	2.00	4.00	9.80	0.49
Qwen3.8-27B	2.50	4.50	11.50	1.02
Qwen3.6-27B	2.40	4.30	11.00	0.96

Appendix

Table E2: Offline build time comparison across methods (hours).

Method	Build (h)	Latency incr.	Extra params	Extra calls
GlitchProber	2.19	$\sim$ 9.20%	0	0
GlitchEdit	4.28	0.00%	0	0
GlitchCleaner	10.41	$\sim$ 1.00%	Gated LoRA	0
GlitchPatch	0.89	$<$ 0.1%	0	0

Appendix

Table E3: Combined cost summary (ten-model average).

Model	Method	MMLU-Pro	ARC	GSM8K	HumanEval
Llama-8B	Original	540	790	805	86
Llama-8B	GlitchPatch	540	791	805	86
Llama-70B	Original	660	875	922	121
Llama-70B	GlitchPatch	660	875	922	121
Gemma-31B	Original	640	860	910	118
Gemma-31B	GlitchPatch	641	860	910	118

Appendix

Table F1: Original vs. GlitchPatch correct counts on general benchmarks.

Model	Original	GlitchPatch	$\Delta$ (pp)
Llama-3.1-8B	66.48	66.51	$+$ 0.025
Llama-3.1-70B	79.87	79.87	$+$ 0.000
Gemma-4-31B	78.24	78.26	$+$ 0.025
Gemma-4-E4B	60.48	60.45	$-$ 0.025
Qwen3.8-27B	77.03	77.06	$+$ 0.025
Qwen3.6-27B	76.23	76.25	$+$ 0.025

Appendix

Table F2: General Score and change (

\Delta

in pp).

Case	Observable evidence	System action
No budget-feasible path	Graph search: path count $=0$	Retain original
All probes fail on original ( $m_{+}=0$ )	Selection probes: $m_{+}=0$	Retain original
Unreliable composition	Full-path probe failure	Reject candidate
Insufficient utility	$U(g,p)\leq\gamma$	Retain original
Calibration failure	Independent probe failure	Reject rule
Per-span limit	$\Delta(p)>B$ or content exceeds 64 bytes	Retain original

Appendix

Table F3: Failure and retention cases for GlitchPatch .

Model	Prober	Edit	Cleaner	GlitchPatch
Llama-3.1-8B	2.2	6.4	8.3	15.5
Llama-3.1-70B	1.8	4.8	7.4	11.9
Gemma-4-31B	3.2	5.6	8.8	12.5
Gemma-4-E4B	1.5	5.8	6.4	14.4
Qwen3.8-27B	2.3	5.1	8.8	13.9
Qwen3.6-27B	2.7	5.4	8.5	12.8

Appendix

Table G1: Relative token-level loss reduction across methods (%).

	GlitchProber		GlitchEdit		GlitchCleaner		GlitchPatch
Model	Acc.	$\Delta$	Acc.	$\Delta$	Acc.	$\Delta$	Acc.	$\Delta$
Mistral-7B (orig. 87.4)	86.8	$-$ 0.6	87.2	$-$ 0.2	87.0	$-$ 0.4	87.2	$-$ 0.2
Llama-3.1-70B (orig. 91.6)	90.8	$-$ 0.8	91.2	$-$ 0.4	91.0	$-$ 0.6	91.8	$+$ 0.2
Qwen3.8-27B (orig. 89.2)	88.4	$-$ 0.8	88.8	$-$ 0.4	88.8	$-$ 0.4	89.0	$-$ 0.2
Mean (orig. 89.4)	88.7	$-$ 0.7	89.1	$-$ 0.3	88.9	$-$ 0.5	89.3	$-$ 0.1

Appendix

Table H1: Contextual regression: short-answer QA accuracy (%) on normal tokens adjacent to repaired glitch tokens. Each per-model accuracy is computed from 500 binary exact-match outcomes.

\Delta

is the change in percentage points relative to the original (no-repair) baseline.

Task	Model	Orig.	Prober	Edit	Cleaner	Ours
Entity QA	Mistral-7B	41.8	43.2	49.6	56.4	67.8
	Llama-70B	48.2	50.8	57.0	63.8	75.0
	Qwen3.8-27B	44.6	46.6	52.8	59.2	71.6
Verb. Copy	Mistral-7B	38.2	39.8	45.4	52.8	64.6
	Llama-70B	43.6	45.6	52.0	59.6	72.4
	Qwen3.8-27B	40.8	42.4	48.2	55.0	68.2

Appendix

Table H2: Out-of-distribution downstream task accuracy (%) on inputs containing glitch tokens. 500 glitch tokens uniformly sampled from

G_{0}

per model. Best result per row in bold.

Language Model Robustness

Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.

See all 493 papers

Sep 14, 2026

Safe Error Correction for Language Models: Frozen-Base Adjustment with Capability Preservation

We study a practical question: can a small correction module fix errors in a frozen language model's outputs without degrading its base capabilities? We propose CRN v2, a lightweight logit-level correction module (~34M trainable parameters, 0.73% of the 4.65B text module) that sits atop a fully frozen Gemma 4 E2B model. The base model is never updated; only the correction module learns, via supervised fine-tuning followed by reference-free DPO on 83,400 error-correction pairs. On a 60-question domain exam (CEHRI: Certified Human-Robot Intelligence, covering facts, arithmetic, and implicit-goal reasoning), CRN v2 corrects 53.3% of base-model errors (reworded variant: 43.3%) while showing no degradation on tested capability benchmarks (MMLU/BoolQ N=200; car-wash N=8). A LoRA baseline at the matched CRN v1 budget (6.6M params, rank 19) achieves 83.3% correction but suffers 30-75% capability loss on the same benchmarks -- the correction-capability tradeoff. An ablation shows that the KL preservation term (lambda=0.1) is critical: lowering it to 0.01 degrades correction to 35.0%. A hidden-state injection variant at earlier layers (1.6M params, SFT-only) reaches 50.0%/55.8% but does not exceed logit correction; shallower injection (layer 4) drops to 30.0%/28.3%; multi-depth logit correction (~35M) reaches only 40%; and longer training (5,000 SFT + 2,000 DPO) stays at 53.3% -- none of the alternative configurations we tested exceeded the rank-128 logit result, consistent with a best-achieved result of ~53% rather than a floor. This is a study of a design principle (frozen base + logit correction + KL anchoring), not a claim of architectural novelty. All code, main-result weights, and evaluation scripts are released (deep variant as code only -- no trained deep checkpoints).

Gautam Kishore

Eulogik

Logit Adjustment Parameter-Efficient Fine-Tuning

Ask PDF

Sep 28, 2026

Spontaneous Context Restoration: How Language Models Recover from Corrupted Inputs

Language models sometimes produce correct outputs even when their inputs are corrupted by deletion, replacement, or misspelling. We study the internal processes accompanying this behavior, which we call context restoration, in controlled attention-only transformers and five pretrained LLMs (1B-32B parameters) across arithmetic, reading comprehension, and multiple-choice reasoning tasks. In the attention-only transformers, restoration emerges spontaneously despite training exclusively on clean sequences, without corruption training or an explicit denoising objective. We find that context restoration follows a two-phase process: early layers localize effects associated with repair at corrupted positions, while later layers accumulate these effects at uncorrupted positions through the residual stream and ultimately concentrate them at the output position. Repair outcome is predictable from hidden states: cosine alignment with the clean state is highly predictive in attention-only models, while linear probes recover additional information in pretrained LLMs. A linear probe using only the corrupted prompt's first-block hidden state predicts failure with mean ROC-AUC 0.78. This enables failure triage under matched or even partially shifted deployment conditions and may reduce unnecessary verification or computation. Failed examples also show substantially greater nonlinearity along corruption directions. Moderate-corruption finetuning increases corruption tolerance while simultaneously reducing displacement-normalized linearization error, associating improved robustness with a more nearly linear response to corruption.

Pranjal Garg, Jacob Beck

Independent Researcher

Language Model Error Detection Mechanistic Interpretability Language Model Robustness Representation Geometry in Language Models

Ask PDF

Aug 22, 2026

Lexical Perturbations Disrupt LLM Reasoning: An Empirical Study of Attention Diversion

Large Language Models (LLMs) achieve strong reasoning performance, but their robustness to realistic lexical corruption remains poorly understood. We evaluate four open-weight instruction-tuned models and frontier models across four reasoning benchmarks under keyboard noise, character swaps, and filler insertion. Character-level perturbations substantially degrade accuracy, especially on multi-step reasoning tasks, while filler insertion has little effect. We trace this asymmetry to Attention Diversion: lexical corruption fragments subword tokenization, and the resulting fragments attract disproportionate attention mass, concentrated in middle and final transformer layers. Length-matched controls confirm that fragmentation, not prompt length, drives the loss. A factorial intervention then shows why the damage is hard to undo: fragmentation corrupts token content and attention allocation together, and the two are coupled. Restoring clean attention while the content remains corrupted is actively harmful, restoring content alone is insufficient, and only restoring both recovers a substantial share of the gap. This coupling explains why inference-time strategies, including chain-of-thought prompting, spell-checking, self-repair, and stronger repair models, fail to consistently recover performance: each addresses one channel at a time. Code and data are available at https://github.com/Jiaqian-Janelle/Attention-Diversion

Jiaqian Zhu, Yang Zhang, Junhua Ding +1

Missouri University of Science and Technology, Rolla, MO, USA · University of North Texas, Denton, TX, USA

Language Model Robustness Causal Interventions in Language Models

Ask PDF

Abstract

Language Model Robustness

Explore similar work

Safe Error Correction for Language Models: Frozen-Base Adjustment with Capability Preservation

Spontaneous Context Restoration: How Language Models Recover from Corrupted Inputs

Lexical Perturbations Disrupt LLM Reasoning: An Empirical Study of Attention Diversion