GPUTOK: GPU Accelerated Byte Level BPE Tokenization

Venu Gopal Kadamba; Kanishkha Jaisankar

cs.CLMar 3, 2026

GPUTOK: GPU Accelerated Byte Level BPE Tokenization

Authors: Venu Gopal Kadamba, Kanishkha Jaisankar

Organizations: M.S. in Data Science, New York University

Abstract

As large language models move toward million-token context windows, CPU tokenizers become a major slowdown because they process text one step at a time while powerful GPUs sit unused. We built a GPU-based byte-level BPE tokenizer that follows GPT-2's merge rules. It includes a basic BlockBPE-style kernel and a faster, optimized version that uses cuCollections static map, CUB reductions, and a pybind11 interface for Python. On WikiText103 sequences up to 131k tokens, the optimized GPU tokenizer produces the same tokens as a CPU version and, for the longest inputs, is about 1.7x faster than tiktoken and about 7.6x faster than the HuggingFace GPT-2 tokenizer. Nsight profiling shows that 70-80% of CUDA API time goes to memory allocation, so adding memory pooling should give the biggest speed boost next. Tests on generation tasks using WikiText103 prompts show that our GPU tokenizer's outputs stay within about one percentage point of tiktoken and HuggingFace GPT-2 on similarity and overlap metrics, meaning it keeps output quality while making long-context inference more practical.

Figures & tables

Figure 1: CUDA API breakdown from Nsight Systems on the long-document workload. Memory allocations dominate API time, indicating that device-side memory pooling could substantially reduce overhead.

Figure 2: Tokenization latency versus sequence length on WikiText103 in latency mode. The optimized GPU kernel overtakes tiktoken at around two thousand tokens and reaches a clear advantage at the longest context lengths.

Figure 3: Speedup of the optimized GPU tokenizer relative to CPU-based tokenizers. Values above one indicate that the GPU implementation is faster. The crossover point where the GPU becomes advantageous is near two thousand tokens.

Seq. Len	HF GPT2	tiktoken	CPU-BPE	GPU-Base	GPU-Opt
256	4.21	1.92	12.84	5.77	4.98
1K	12.7	6.32	48.5	10.9	9.42
4K	49.4	22.1	189.2	18.3	14.5
16K	201	91.3	749.9	29.5	24.1
131K	1673	598.1	6143	89.3	53.4

Table 1: Latency comparison across tokenizers on WikiText-103 (ms).

Seq. Len	GPU-O / CPU	GPU-O / tiktoken	GPU-O / HF
256	2.58	0.38	0.91
1K	5.15	0.67	1.35
4K	13.0	1.53	3.41
16K	31.1	3.79	8.34
131K	115	1.67	31.3

Table 2: Speedup ratios of GPU-Optimized relative to CPU and CPU-based tokenizers. Values

>1

indicate GPU advantage.

Figure 4: Relative throughput comparison at the longest sequence length. The GPU tokenizer surpasses both tiktoken and HuggingFace GPT2.

Subword Tokenization

Subword tokenization divides text into reusable units smaller than words to support efficient and flexible language modeling.

See all 81 papers

Incremental BPE Tokenization
May 29, 2026Shenghu Jiang, Ruihao GongSubword Tokenization Byte-Pair Encoding
1Beihang University · 2SenseTime Research.
Taking Tokenization off the Host for Agentic LLM Serving
Oct 9, 2026Mohammad Siavashi, Pascal Schwinte, Ali Banaei +3Tokenization LLM Serving
KTH Royal Institute of Technology, Stockholm, Sweden
Online Tokenization with Optimal Delay in Linear Time
Oct 13, 2026Mina Dalirrooyfard, Christopher YeTokenization Subword Tokenization
Morgan Stanley · University of California San Diego

May 29, 2026J/K move · Enter open · S save

Incremental BPE Tokenization

Shenghu Jiang, Ruihao Gong

1Beihang University · 2SenseTime Research.

Subword Tokenization Byte-Pair Encoding Tokenization in Language Models Tokenization

Open paper AskPDF arXiv

We propose a novel algorithm for incremental Byte Pair Encoding (BPE) tokenization. The algorithm processes each input byte in worst-case

\mathcal{O}(\log^2 t)

time, leading to an overall complexity of

\mathcal{O}(n \log^2 t)

, where

n

is the input length and

t

is the maximum token length. The algorithm incrementally maintains BPE tokenization results for every prefix of the input text, implementing the standard BPE merge procedure defined by a fixed set of merge rules. This enables efficient partial tokenization in streaming settings. Functioning as a drop-in replacement for standard BPE, our approach achieves a speedup of up to

{\sim}3\times

over Hugging Face's tokenizers, and demonstrates significant latency reductions over OpenAI's tiktoken on pathological inputs. We further introduce an eager output algorithm that enables streaming output, emitting tokens as soon as token boundaries are determined during incremental tokenization. Overall, our results demonstrate that BPE tokenization can be performed incrementally with strong worst-case guarantees, while providing practical latency benefits in modern large language model pipelines. Code: https://github.com/ModelTC/mtc-inc-bpe

Abstract

Subword Tokenization

Explore similar work

Incremental BPE Tokenization

Taking Tokenization off the Host for Agentic LLM Serving

Online Tokenization with Optimal Delay in Linear Time