Subword Tokenization
Overview
Subword tokenization studies how to divide text into reusable pieces that let language models handle unfamiliar words without treating every character as a separate unit. Common methods include frequency-based Byte Pair Encoding (BPE), probabilistic Unigram models, and approaches that incorporate morphology, syllables, scripts, or learned boundaries. Researchers compare tokenizers using compression and fertility measures, bits per byte, morphological alignment, multilingual benchmarks, and downstream language-model performance, since token choices affect context length, computation, and language coverage. A central tension is that compression-efficient segments do not always match linguistic structure or improve task performance, and token costs remain uneven across languages and writing systems.
Latest papers
83 papers10 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.