Token Efficiency
Overview
Token efficiency asks how language models and agent workflows can accomplish the same work with fewer tokens, lowering costs and extending usable context. Approaches include tokenizer redesign, compact structured-data formats, prompt and context management, and pruning or routing agents and harness components. Evaluations compare token counts with language-model loss, task accuracy, latency, semantic fidelity, and performance across languages and benchmarks. A central tension is that shorter representations do not always preserve meaning or predictive quality, and savings can shift costs across languages, models, and workflow stages.
Latest papers
31 papers1 paper in the last four weeks, down 86% on the four weeks before. 0.0% of all new papers.
Tokenization directly affects the inference efficiency of large language models, since fragmented tokenization increases sequence length and generation cost. Although longer, multi-word tokens can reduce fertility, naively adding them often degrades language model performance. We propose Thunder-Tok, a subword tokenizer that reduces fertility while preserving downstream performance. Thunder-Tok first constructs a large seed vocabulary from corpus substrings and filters structurally incomplete candidates, including invalid Unicode byte fragments and word-boundary violations. It then prunes the seed vocabulary using a likelihood-based token score derived from a uniform Jensen lower bound of the training-data probability. Experiments show that Thunder-Tok reduces fertility by approximately 25% in English and 9% in Korean compared with the standard BPE tokenizer while maintaining competitive performance.