Subword Tokenization

Subword tokenization studies how to divide text into reusable pieces that let language models handle unfamiliar words without treating every character as a separate unit. Common methods include frequency-based Byte Pair Encoding (BPE), probabilistic Unigram models, and approaches that incorporate morphology, syllables, scripts, or learned boundaries. Researchers compare tokenizers using compression and fertility measures, bits per byte, morphological alignment, multilingual benchmarks, and downstream language-model performance, since token choices affect context length, computation, and language coverage. A central tension is that compression-efficient segments do not always match linguistic structure or improve task performance, and token costs remain uneven across languages and writing systems.