cs.CLJul 25, 2026

Joint Optimization for Greedy Longest-match Tokenization

Authors: Adhiraj SinghDeepanshu ModyGhina Al ShdaifatHamza AlshamyAdam WiemerslageVarshini ReddyCraig W. Schmidt

Organizations: Center for Data Science, New York University New York, NY, USA · Kensho Technologies Cambridge, MA, USA

Abstract

Recent work has shown that subword vocabularies can be trained to optimize compression for a specific inference rule rather than relying on greedy heuristics such as Byte Pair Encoding (BPE). We extend this approach to greedy left-to-right longest-match decoding, the fast and widely used inference rule underlying WordPiece. We introduce Joint Optimization for Greedy Longest-Match Tokenization (JOLT), which formulates vocabulary learning as an integer program over vocabulary-selection and segmentation-choice variables. Greedy-consistency constraints ensure that each optimized segmentation exactly matches the segmentation produced by longest-match decoding under the selected vocabulary, aligning the training objective with deployment-time tokenization. To scale the optimization, we solve a linear programming relaxation and selectively introduce higher-order segmentations only for unresolved pretokens. The resulting relaxation is nearly integral: rounded solutions fall within 0.008 - 0.176 % of the LP lower bound on the training scope. The bound also shows that BPE is already within 1 - 2 % of the best achievable compression under greedy longest-match decoding, while JOLT closes 89.6 - 99.4 % of the remaining gap. On held-out validation data across four training scopes and vocabulary sizes of 32,000 and 64,000, JOLT produces up to 0.78 % fewer tokens than BPE, with improvements generally increasing as the training scope grows. These results demonstrate that inference-aligned vocabulary optimization can recover most of the limited compression headroom left by BPE while providing a certificate of near-optimality.

Explore similar work

CardsList
  1. Tokenization with Split Trees

    May 21, 2026Craig W. Schmidt, Michael Krumdick, Adam Wiemerslage +4TokenizationSubword Tokenization

  2. Tokenisation via Convex Relaxations

    May 21, 2026Jan Tempus, Philip Whittington, Craig W. Schmidt +2TokenizationToken-Level Uncertainty