The Vocabulary Tax: Safely Trimming Unused Tokens from LLMs for On-Device Use
Organizations: Independent Researcher, Bangkok, Thailand
Abstract
Open large language models (LLMs) and speech models ship vocabularies of up to 262k tokens, and every user downloads the tables those tokens index (the input embedding, an untied output head and, in recent on-device models, per-layer embeddings), whatever languages they need. We call the bytes in rows a deployment never uses the vocabulary tax and measure it, without downloading weights, for 1,244 generative and speech-recognition models on the Hugging Face Hub. For a Thai and English deployment it is a median 4.4% of model bytes below 0.5B parameters and 2.1% above 8B, and 1.93 GB in Gemma 4 E2B: it falls hardest on the small models that edge devices run. Keeping only the rows of the deployment's writing systems is exact. Keeping the tokens seen in the user's own text removes much more, a median 29.0% below 0.5B, but the usual way of doing it breaks BPE tokenizers: it drops the intermediate tokens that merges build on, so new text is split differently 1.9 times as often as the standard check reports. We repair this with merge closure, at a cost of 5.9% more rows, and add a trim certificate: a Good-Turing estimate, from the trimming text alone, of how much new text the trimmed tokenizer will split differently (correlation 0.996 over 300 settings in 22 languages) and of how much more text a target needs. Without test data our method never exceeds a 2% target, while prior keep rules checked on test data exceed it in 9-13 of 44 settings. On six models of 270M to 3B parameters in 11 languages, trimmed models keep their predictions: where the certificate passes, a median 89.7% of rows is removed and bits per character rise by at most 2.0%. The census, the trimming and the certificate are released as an open Python package, and the per-model tax for 41 languages as an open dataset.
Figures & tables
| Family | Vocabulary | Rows unused (%) | Token ratio TH/EN | Median tax (%) | |
|---|---|---|---|---|---|
| Baichuan | 64,000 | 55.8 | 3.86 | 0 | – |
| Aquila | 100,008 | 53.2 | 4.66 | 0 | – |
| XLM-R | 250,048 | 51.9 | 1.06 | 0 | – |
| Gemma 3/3n/4 | 262,144 | 35.0 | 1.53 | 170 | 5.3 |
| Qwen 2–3.5 | 151,936 | 28.8 | 2.36 | 496 | 4.1 |
| Command R | 256,000 | 28.1 | 4.31 | 1 | 1.7 |
| Family | Kept / vocabulary | Rebuild | Same TH | Same Latin | Round trip | Other scripts |
|---|---|---|---|---|---|---|
| Qwen 2–3.5 | 108,232 / 151,936 | 100.0 | 100 | 100 | 100 | 1.30–4.97 |
| Llama 3 | 108,101 / 128,256 | 97.9 | 100 | 100 | 100 | 2.04–5.43 |
| GPT-2 | 50,021 / 50,257 | 100.0 | 100 | 100 | 100 | 1.00–1.63 |
| Gemma 3/3n/4 | 170,274 / 262,144 | 100.0 | 100 | 100 | 100 | 3.62–7.23 |
| Mistral v3 | 28,726 / 32,768 | – | 100 | 100 | 100 | 1.91–4.65 |
| Closure | Real / nominal | Extra tokens (%) | ||||
|---|---|---|---|---|---|---|
| Family | Orphans | cost (%) | usual | closed | usual | closed |
| Gemma 3/3n/4 | 1,146 | 8.7 | 2.05 | 0.95 | 6.31 / 62 | 1.80 / 12 |
| Qwen 2–3.5 | 989 | 4.7 | 2.10 | 0.95 | 2.86 / 52 | 0.72 / 9 |
| Llama 3 | 638 | 11.6 | 1.81 | 0.91 | 2.07 / 49 | 0.88 / 10 |
| GPT-2 | 567 | 8.1 | 1.67 | 0.94 | 0.76 / 51 | 0.36 / 9 |
| Oracle | Deployed, =1% | Deployed, =2% | |||||
|---|---|---|---|---|---|---|---|
| Keep rule | Matched | Orphans | =1% | Removed | Over | Removed | Over |
| Script rule [ 4 , 22 ] | – | 87 | 17.3 | 17.3 | 0 | 17.3 | 0 |
| Rank top- (first pieces) | 8.4 | 87 | 22.4 | 22.4 | 0 | 24.0 | 0 |
| Frequency top- [ 4 , 10 , 14 ] | 1.32 | 424 | 46.5 | 51.1 | 3 | 68.1 | 9 |
| Seen in corpus [ 25 ] | 1.30 | 720 | 46.6 | 51.2 | 6 | 68.5 | 13 |
| Ours (merge-closed, certificate) | 1 | 0 | 52.6 | 52.6 | 3 | 68.2 | 0 |
| Family | Script | 1,000 | 2,000 | 4,000 | 15,000 |
|---|---|---|---|---|---|
| Gemma 3/3n/4 | 35.0 | 92.1 / 3.33 | 91.4 / 2.35 | 90.7 / 1.59 | 88.5 / 0.67 |
| Qwen 2–3.5 | 28.8 | 90.9 / 2.00 | 90.0 / 1.46 | 89.3 / 0.92 | 86.5 / 0.40 |
| Llama 3 | 15.7 | 89.5 / 2.28 | 88.3 / 1.65 | 87.3 / 1.05 | 83.5 / 0.44 |
| GPT-2 | 0.5 | 90.5 / 0.58 | 88.4 / 0.41 | 86.3 / 0.28 | 79.1 / 0.10 |
| Script rule | Merge-closed corpus rule | |||||
|---|---|---|---|---|---|---|
| Languages | Tax 0.5B | Gemma 4 E2B | Removed | Real (pooled) | Worst language | Cert./real |
| th, zh, en | 3.3 | 1.52 | 85.7 | 1.93 | 7.08 | 0.98 |
| ar, fr, en | 4.4 | 1.80 | 83.2 | 1.86 | 8.08 | 1.04 |
| ja, ko, zh, en | 2.7 | 1.37 | 85.3 | 1.37 | 6.66 | 0.96 |
| es, pt, fr, en | 4.5 | 1.97 | 79.9 | 1.99 | 6.26 | 1.09 |
| ru, uk, en | 2.4 | 1.69 | 89.7 | 0.47 | 2.91 | 1.11 |
| Language | Measured at 4k (%) | Lines needed (median) | Lines needed (worst) |
|---|---|---|---|
| Arabic | 0.04 | 4k | 4k |
| Persian | 0.14 | 4k | 4k |
| Hebrew | 0.00 | 4k | 4k |
| Hindi | 0.00 | 4k | 8k |
| Korean | 0.41 | 4k | 4k |
| Russian | 0.39 | 4k | 6k |
| Language | Certificate | Real, in-domain | Real, PUD | PUD / in-domain | Nominal 200 / real |
|---|---|---|---|---|---|
| Arabic | 0.06 | 0.04 | 0.15 | – | 1.22 |
| Czech | 1.87 | 1.76 | 3.14 | 1.75 | 0.77 |
| English | 8.95 | 7.69 | 13.65 | 1.78 | 0.82 |
| French | 3.10 | 3.04 | 2.96 | 0.98 | 0.96 |
| German | 2.35 | 2.26 | 4.74 | 2.10 | 0.78 |
| Hindi | 0.00 | 0.00 | 0.10 | 2.13 | 0.85 |
| Model | Rule | n | Removed | Re-split | BPC | Range | Greedy | Belebele |
|---|---|---|---|---|---|---|---|---|
| (%) | (%) | (%) | (%) | (%) | (pp) | |||
| Gemma 3 270M | Script | 11 | 34.1 | 0.00 | 0.01 | 0.1, 0.4 | 100 | 0.0 |
| Corpus 4k | 11 | 88.5 | 1.58 | 2.93 | 0.2, 23.3 | 63 | 0.2 | |
| Corpus 15k | 10 | 87.9 | 0.80 | 1.13 | 0.3, 8.1 | 80 | 0.2 | |
| Qwen3 0.6B | Script | 11 | 28.9 | 0.00 | 0.20 | 0.4, 0.1 | 100 | 0.0 |
| Corpus 4k | 10 | 89.1 | 0.95 | 0.54 | 1.9, 14.0 | 76 | 0.5 |
| Budget | Over budget | Fit after script rule | Fit after corpus rule |
|---|---|---|---|
| Stored precision, 1 GB | 182 | 3 | 25 |
| Stored precision, 2 GB | 131 | 5 | 16 |
| Stored precision, 4 GB | 64 | 9 | 19 |
| INT8, 1 GB | 124 | 3 | 6 |
| INT8, 2 GB | 54 | 10 | 21 |
| INT4, 1 GB | 54 | 10 | 21 |
| Model | Size (GB) | Script TH+EN (%) | Script TH+EN+ZH (%) | Corpus TH (%) |
|---|---|---|---|---|
| Gemma 3 270M | 0.54 | 21.9 (0.12) | 17.3 | 55.4 (0.30) |
| Qwen3 0.6B | 1.50 | 11.9 (0.18) | 5.0 | 35.8 (0.54) |
| Gemma 3 1B | 2.00 | 10.6 (0.21) | 8.4 | 26.7 (0.53) |
| Qwen3 1.7B | 4.06 | 8.8 (0.36) | 3.7 | 26.5 (1.08) |
| Whisper large-v3-turbo | 1.62 | 1.2 (0.02) | 0.9 | – |
| Gemma 4 E2B | 10.25 | 18.8 (1.93) | 14.9 | 47.5 (4.87) |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Languages | 0.5B | 0.5–2B | 2–8B | 8B | Edge, weighted | Gemma 4 E2B (GB) |
|---|---|---|---|---|---|---|
| Dutch, English, French, German, Indonesian, Italian, Lao, Malay, Polish, Portuguese, Punjabi, Spanish, Swahili, Tagalog, Turkish, Vietnamese | 4.5 | 4.6 | 2.9 | 2.2 | 5.1 | 1.97 |
| Bengali | 4.5 | 4.6 | 2.9 | 2.2 | 5.1 | 1.77 |
| Telugu | 4.5 | 4.6 | 2.9 | 2.2 | 5.1 | 1.94 |
| Gujarati, Kannada | 4.5 | 4.6 | 2.9 | 2.2 | 5.1 | 1.95 |
| Amharic, Burmese, Khmer | 4.5 | 4.6 | 2.9 | 2.2 | 5.1 | 1.96 |
| Thai | 4.4 | 4.3 | 2.8 | 2.1 | 4.8 | 1.93 |
| Family | ar | cs | de | en | es | fi | fr | gl | hi | id | is | it | ja | ko | pl | pt | ru | sv | th | tr | zh |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 3/3n/4 | 33 | 36 | 36 | 36 | 36 | 36 | 36 | 36 | 31 | 36 | 36 | 36 | 27 | 34 | 36 | 36 | 31 | 36 | 35 | 36 | 29 |
| Qwen 2–3.5 | 28 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 12 | 28 | 30 | 30 | 28 | 30 | 29 | 30 | 14 |
| Llama 3 | 14 | 17 | 17 | 17 | 17 | 17 | 17 | 17 | 16 | 17 | 17 | 17 | 13 | 15 | 17 | 17 | 12 | 17 | 16 | 17 | 13 |
| Llama 2 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 11 | 13 | 13 | 13 | 4 | 13 | 13 | 13 | 11 |
| Mistral Tekken | 20 | 28 | 28 | 28 | 28 | 28 | 28 | 28 | 26 | 28 | 28 | 28 | 24 | 24 | 28 | 28 | 22 | 28 | 27 | 28 | 25 |
| Whisper | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 14 | 11 | 11 | 14 | 14 | 9 | 14 | 14 | 14 | 12 |
| Language | Gemma | Qwen | Llama 3 | GPT-2 |
|---|---|---|---|---|
| Arabic | 92/0.14/0.23 | 91/0.05/1.09 | 90/0.03/0.05 | 94/0.00/0.00 |
| Chinese (2k) | 88/3.25/3.69 | 86/7.24/10.19 | 89/1.01/1.43 | 92/0.25/0.29 |
| Czech | 90/2.72/4.76 | 88/1.38/2.25 | 84/2.14/3.96 | 83/1.06/1.61 |
| Dutch | 89/4.10/7.77 | 88/2.74/4.88 | 85/2.81/4.51 | 81/1.86/2.99 |
| English (web) | 86/8.71/26.93 | 83/7.64/26.18 | 78/7.74/23.54 | 66/7.21/25.37 |
| French | 86/5.06/10.57 | 85/3.08/5.71 | 81/2.99/5.00 | 74/2.14/3.82 |