The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India
Organizations: Havon AI Mumbai, India
Abstract
Commercial language model APIs bill per token, so a tokenizer that needs more tokens for the same content in one language makes that language more expensive, slower, and able to fit less text in a fixed context window. We measure this disparity for nine public tokenizers, including those of GPT-4, GPT-4o, Qwen2.5, DeepSeek-V3, Mistral-Nemo and four Sarvam models, on all 25 languages of India available in FLORES-200 (2,009 professionally translated sentences aligned with English) and on 13,732 human-translated Hinglish-English social media pairs from PHINC. On GPT-4o's tokenizer the same content needs between 1.57 times (Hindi) and 5.01 times (Odia) the tokens of English, and 13.7 times for Santali in Ol Chiki script; GPT-4's tokenizer needs up to 12.8 times. Sarvam-30B/105B is the only third-party tokenizer that stays below 2.6 times on every language (median 1.53). In an illustrative scenario using official list prices accessed on 5 November 2026 and stated assumptions, a support bot handling 100,000 chats a month on GPT-4o would cost about INR 56,640 in English and INR 102,829 in Marathi; these are estimates, not measured production costs. Against an openly licensed Hindi and Marathi morphological segmentation set, no tokenizer places more than 35% of its word-internal boundaries on morpheme boundaries (boundary F1 at most 0.48). In a companion paper we present a tokenizer that reduces this tax. Code, pinned artifacts and raw token counts are available at https://github.com/HavonAI/indic-token-tax.
Figures & tables
| Tokenizer | Vocab | Marathi | Hindi | Hinglish |
|---|---|---|---|---|
| GPT-4 (cl100k_base) | 100,277 | 5.04 | 4.78 | 1.36 |
| Qwen2.5 | 151,665 | 4.61 | 4.43 | 1.36 |
| DeepSeek-V3 | 128,815 | 3.00 | 2.81 | 1.34 |
| Mistral-Nemo | 131,072 | 2.14 | 1.79 | 1.33 |
| Sarvam-M | 131,072 | 2.14 | 1.79 | 1.33 |
| GPT-4o (o200k_base) | 200,019 | 1.82 | 1.57 | 1.28 |
| Tokenizer | English | Marathi | Hindi |
|---|---|---|---|
| GPT-4o (o200k_base) | 1.00 | 1.82 | 1.57 |
| GPT-4 (cl100k_base) | 1.01 | 5.10 | 4.84 |
| Qwen2.5 | 1.03 | 4.74 | 4.55 |
| DeepSeek-V3 | 1.01 | 3.02 | 2.82 |
| Mistral-Nemo / Sarvam-M | 1.04 | 2.22 | 1.86 |
| Sarvam-30B / 105B | 1.01 | 1.41 | 1.33 |
| Language | FLORES code | GPT-4 | Qwen2.5 | DeepSeek-V3 | Mistral-Nemo | GPT-4o | Sarvam-1 | Sarvam-30B |
|---|---|---|---|---|---|---|---|---|
| cl100k | / Sarvam-M | o200k | / 105B | |||||
| Hindi | hin_Deva | 4.78 | 4.43 | 2.81 | 1.79 | 1.57 | 1.14 | 1.31 |
| Nepali | npi_Deva | 4.77 | 4.42 | 2.98 | 2.14 | 1.62 | 1.54 | 1.53 |
| Urdu | urd_Arab | 4.36 | 3.16 | 2.37 | 1.76 | 1.66 | 7.41 | 1.53 |
| Awadhi | awa_Deva | 4.77 | 4.32 | 2.91 | 2.03 | 1.71 | 1.28 | 1.47 |
| Sindhi (Arabic) | snd_Arab | 3.98 | 2.84 | 2.89 | 2.50 | 1.71 | 6.74 | 2.02 |
| PHINC | Hing- | Me- | ||
|---|---|---|---|---|
| Tokenizer | English | Hinglish | Corpus | Corpus |
| GPT-4o | 1.36 | 1.61 | 1.45 | 2.14 |
| GPT-4 (cl100k) | 1.40 | 1.77 | 1.60 | 2.35 |
| Qwen2.5 | 1.41 | 1.77 | 1.59 | 2.37 |
| DeepSeek-V3 | 1.41 | 1.75 | 1.58 | 2.32 |
| Mistral-Nemo / Sarvam-M | 1.44 | 1.76 | 1.64 | 2.31 |
| Tokenizer | Hin | Mar | Ben | Guj | Pan | Tam | Tel | Kan | Mal | Urd | Snd | Sin |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-4o (o200k) | 1.08 | 0.97 | 0.96 | 1.03 | 0.70 | 1.04 | 0.90 | 0.95 | 1.00 | 1.10 | 1.07 | 0.70 |
| GPT-4 (cl100k) | 0.40 | 0.39 | 0.32 | 0.28 | 0.26 | 0.31 | 0.25 | 0.25 | 0.25 | 0.48 | 0.53 | 0.25 |
| Qwen2.5 | 0.43 | 0.43 | 0.37 | 0.32 | 0.28 | 0.39 | 0.29 | 0.32 | 0.31 | 0.67 | 0.75 | 0.31 |
| DeepSeek-V3 | 0.68 | 0.64 | 0.88 | 0.55 | 0.45 | 0.77 | 0.54 | 0.58 | 0.50 | 0.88 | 0.75 | 0.46 |
| Mistral-Nemo | 1.00 | 0.87 | 0.89 | 0.76 | 0.66 | 1.01 | 0.84 | 0.92 | 0.83 | 1.11 | 0.83 | 0.17 |
| Sarvam-30B/105B | 1.23 | 1.20 | 1.37 | 1.22 | 1.09 | 1.36 | 1.12 | 1.24 | 1.15 | 1.16 | 0.92 | 0.15 |
| Model (API) | Tokenizer | Input | Output |
|---|---|---|---|
| GPT-4o | o200k_base | USD 2.50 | USD 10.00 |
| GPT-4.1 mini | o200k_base | USD 0.40 | USD 1.60 |
| GPT-5 | o200k_base | USD 1.25 | USD 10.00 |
| GPT-5 mini | o200k_base | USD 0.25 | USD 2.00 |
| deepseek-flash a | DeepSeek-V3 | USD 0.30 | USD 1.20 |
| deepseek-v4-pro a | DeepSeek-V3 | USD 1.32 | USD 3.96 |
| Language | GPT-4o | GPT-5 mini | deepseek-flash | Sarvam 105B |
|---|---|---|---|---|
| English | 56.6 | 9.4 | 6.8 | 3.6 |
| Hinglish (PHINC) | 72.6 | 12.1 | 9.1 | 4.3 |
| Hindi | 89.1 | 14.9 | 19.1 | 4.7 |
| Marathi | 102.8 | 17.1 | 20.4 | 5.0 |
| Bengali | 97.3 | 16.2 | 13.8 | 4.4 |
| Tamil | 112.0 | 18.7 | 20.9 | 5.3 |
| Hindi | Marathi | |||||||
| Tokenizer | P | R | F1 | T/w | P | R | F1 | T/w |
| GPT-4 (cl100k) a | .13 | .72 | .22 | 7.96 | .19 | .98 | .32 | 9.08 |
| DeepSeek-V3 | .14 | .70 | .24 | 4.99 | .20 | .95 | .33 | 5.65 |
| Mistral-Nemo b | .18 | .65 | .29 | 3.83 | .24 | .88 | .37 | 4.51 |
| GPT-4o (o200k) | .20 | .62 | .31 | 3.37 | .28 | .85 | .43 | 3.79 |
| Sarvam-1 | .29 | .79 | .43 | 3.25 | .34 | .73 | .46 | 3.01 |