Efficiency Measured in English: Tokeniser Coverage Bounds Small Language Models on Indic Scripts
Organizations: School of Computer Science, University of Leeds, United Kingdom
Abstract
Small Language Models with quantisation and parameter-efficient adaptation are widely offered as the route to multilingual AI under compute constraints, and their efficiency is almost always reported as a single memory figure. We show that this figure is measured in English and does not transfer. Across a 131k-sentence corpus spanning eleven Indic scripts, TinyLlama-1.1B's tokeniser emits 1.34 subword tokens per Latin-script word and 22.60 per Kannada word. This is not inefficiency; it is absence. Its vocabulary contains 0% of Odia's codepoints, 1% of Telugu's and 2% of Kannada's, and for five of eleven scripts the measured fertility sits at the UTF-8 byte-fallback ceiling, on Odia at 99.6% of it, with 93.6% of emitted tokens being raw bytes. Because SentencePiece byte fallback never raises an unknown-token marker, this failure is entirely silent: TinyLlama's UNK rate is 0.0% on every script, including the one it reads as bytes. We further show that low fertility can indicate the opposite of efficiency: mBERT posts its best score anywhere (1.10 tokens/word, on Odia) while discarding 87.9% of the text as UNK. MuRIL, an Indic-vocabulary encoder with a fifth of TinyLlama's parameters, reads every script at 1.23-2.26 tokens/word with zero UNK and zero byte fallback. The binding constraint is the vocabulary, not the parameter count. The consequence is concrete: on a 2048-token context window, the same model gives an English user 1,528 words of usable context and a Kannada user 91.
Figures & tables
| TinyLlama-1.1B (32k vocab) | mBERT (120k vocab) | MuRIL (197k Indic vocab) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Script | % corp. | cov. | fert. | fall. | cov. | fert. | fall. | cov. | fert. | fall. |
| Latin | 7.4 | 50% | 1.34 | 0.0 | 47% | 1.32 | 0.0 | 20% | 1.23 | 0.0 |
| Perso-Arabic | 0.9 | 20% | 4.84 | 9.8 | 38% | 1.77 | 0.0 | 47% | 1.25 | 0.0 |
| Devanagari | 17.4 | 30% | 6.67 | 9.6 | 67% | 2.70 | 0.1 | 72% | 1.40 | 0.0 |
| Bengali | 16.5 | 24% | 9.59 | 40.3 | 75% | 3.74 | 0.2 | 78% | 1.45 | 0.0 |
| Tamil | 8.3 | 26% | 11.32 | 19.7 | 65% | 3.87 | 0.0 | 64% | 1.69 | 0.0 |