cs.CLOct 14, 2026
SaveDelta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUs
Organizations: Arizona State University
Abstract
An agent that drives an LLM sends the model the same conversation again and again, and each turn adds only a little text at the end. Inference engines reuse the KV cache of the shared prefix, so only the new tokens pass through the model. However, tokenization has no such shortcut. The CPU front end of the serving engine converts every request back into token IDs from its first character, which lengthens the time to first token (TTFT) and consumes CPU on every turn. We present Delta Tokenizer, a CPU tokenization front end that tokenizes only the delta between a request and the previous request of the same session. It retains the earlier token sequence, encodes again only the appended text and the short tail whose tokens the append could affect, and joins the two only at a position the tokenizer's splitting rules themselves cannot move, which keeps the result exact. For long requests with no state to reuse, it cuts the text where practical tokenizers already keep neighboring pieces independent, encodes the pieces on several cores at once, and concatenates the outputs into the reference encoding, exactly. A shared CPU pool schedules these chunks together with the short incremental updates, so an update runs between two chunks rather than behind an entire long job. The incremental path shortens the tokenization of a session turn by roughly 40x relative to the Hugging Face reference tokenizer, and on 16 cores the chunk-parallel path encodes a four-million-character request 8.2x faster than a serial encode and 1.6-2.1x faster than LoPT. Inside SGLang, Delta Tokenizer reduces the median TTFT through the chat interface by 59% at 100K tokens and brings the P99 TTFT of agent sessions down 9.5x under load and 23x when long requests tie up the front end. Every output we checked matches the reference, among them 41.8 billion documents encoded by its fast encoder.
Figures & tables
Figure 1: The four steps of the design: reuse across turns, parallel full encodes, scheduling at chunk granularity, and an adapter that plugs into an unmodified serving engine. C stands for a chunk and U for an update.
Figure 2: Agent requests and what a stateless front end pays for them. (a) Append (characters, both agents pooled) and context (tokens, Claude Code and Codex) per call in the recorded traces, dots at the medians. (b) Mean TTFT in a vLLM sweep (Qwen3-1.7B, 10K-token prompts), reference encode timed separately. (c) Reference encode of Qwen3-8B’s tokenizer on one core, clean and accented transcripts, two runs each, and tiktoken on gpt-oss-120b’s tokenizer, where it is native, on the same texts. (d, e) Median TTFT of Qwen3-8B in SGLang with a cached prefix, raw text by cores, and at 100K tokens by request form against the fast encoder without session state.
Figure 3: Runtime architecture of Delta Tokenizer, with the request path numbered as in § 3.1 . Unsupported configurations take the reference path. The three labels mark where the designs of Figure 1 live and the sections that describe them.
Figure 4: An append can change tokens that were already emitted. Amber marks the tokens that change, and ␣ marks a space. Both examples hold for Qwen3 and Llama-3.1.
Figure 5: Incremental repair. (a) Only the old tail and the append are re-encoded. (b) A matching run that contains a stable transition lets the result keep the cached IDs before index and take the window IDs from index on. A failed check widens the window or hands the unsealed tail and the append to the reference.
Figure 6: Splitting a request for parallel encoding. (a) Earlier systems cut blindly, encode overlapping chunks, and check the seams, retrying when a check fails. (b) Delta Tokenizer moves each target cut back to a transition that ends a piece in every context, so the chunks are disjoint and nothing is retried.
Figure 7: A cut must preserve every stage that precedes BPE. Special-token strings stay intact, normalization gives the same text on both sides, and the pre-tokenizer the same pieces, so the outputs concatenate to the full encode.
Figure 8: Chunk boundaries are scheduling points. A queued update cannot interrupt a whole encode, but with chunks it can start once the current chunk completes. Each row shows one worker, and the P99 values are from the standard mix of § 4.4 with chunks in FIFO order.
| Baseline | Wrong IDs | Reuse | Result |
|---|---|---|---|
| Reference | none | no | 93–99 ms per encode (Qwen3) |
| SGLang workers | none | no | median TTFT 6–10% below reference |
| tiktoken | none | no | 14.0 ms vs. 85.8 for Hugging Face |
| Gigatoken 0.9.0 | 558 of 41.8B | no | compared in corrected form |
| Fast stateless | none | no | no faster than reference on chat |
| fastokens 0.1.2 | 78 of 279 | unchecked | as fast as Delta Tokenizer |
Table 1: Baselines. Encode times are at 100K tokens. SGLang workers means four tokenizer workers or dynamic batching, measured under a burst. tiktoken runs on gpt-oss-120b’s tokenizer. Fast stateless is our corrected Gigatoken without session state, and LoPT is our reimplementation. Wrong IDs counts encodes that differ from the reference, on the web corpus of § 4.8 for Gigatoken, on replayed SGLang requests for fastokens, and on CLongEval for LoPT (supplement, §S5). Reuse means carrying a session’s tokens from one call to the next.
| Session | Added by Delta Tokenizer | Model | |||
|---|---|---|---|---|---|
| Context | Text | IDs | Blocks | Added | KV cache |
| 8K | 31 KB | 37 KB | 117 KB | 155 KB | 1.18 GB |
| 32K | 125 KB | 133 KB | 562 KB | 696 KB | 4.72 GB |
| 100K | 391 KB | 526 KB | 562 KB | 1,089 KB | 14.75 GB |
| 250K | 977 KB | 1,051 KB | 562 KB | 1,614 KB | 36.86 GB |
| 500K | 1,954 KB | 2,099 KB | 562 KB | 2,662 KB | 73.73 GB |
Table 2: State per session by context in tokens (1 KB = 1,000 B), against the KV cache of Qwen3-8B for the same tokens. Text is the session’s transcript. IDs include the tail’s positions, blocks are prefix blocks shared through the store, counted as if no session shared them, and Added is IDs plus blocks plus digests under 1 KB. The rows from 500K on follow the same accounting.
Figure 9: Warm-turn median TTFT in SGLang. (a) Qwen3-8B by context, where fastokens returns token IDs that differ from the reference. (b) Qwen3-8B at 100K tokens by request form, extending Figure 2 e with Delta Tokenizer serial, with 95% bootstrap intervals. (c) gpt-oss-120b by context.
Figure 10: Full-encode speedup from chunking. (a) Qwen3 on 16 cores, relative to a one-worker pool. (b) Speedup over serial encoding with the same Hugging Face encoder, Qwen3 with eight chunks on eight cores, for LoPT as published, whose seam matching is not exact, and for Delta Tokenizer’s safe cuts. The 1.6–2.1 range in the text spans eight and sixteen chunks and two tokenizers (supplement, §S5).
Figure 11: Unit of work and queue order. (a) One four-worker FIFO queue with the same arrival sequence, where only the unit of work changes, for million-character stateless requests. (b) Every request of (a) at 26 requests/s. (c, d) P99 latency against achieved load for five pool designs on four workers, with stateless requests of 100K and 1M characters. Whole-request designs are gray, chunked designs blue, the shortest-first chunked design is Delta Tokenizer pool, and open markers denote overloaded runs.
Figure 12: SGLang’s tokenizer stage replayed in one process, with session turns and 2.3% stateless requests, for sessions of (a) 100K and (b) 250K characters.
Figure 13: Front-end interference on served agent turns in SGLang, while four stateless requests of one million characters arrive every eight seconds and are tokenized before SGLang declines them for their length. (a) Warm-turn P99 TTFT by warm-turn rate. (b) Share of warm turns slower than a given TTFT at eight turns per second.
Figure 14: Warm-turn P99 TTFT in SGLang next to served documents of about 100K tokens. Bars pool a configuration’s arrival seeds, two or three of 60 s each, error bars give each seed’s P99, and the labels give the reference’s P99 over that of Delta Tokenizer serial and Delta Tokenizer pool. Qwen3-8B unless noted, twelve sessions, SGLang 0.5.19. (a) A repeated document, resent every 8 s and cached by the model, and two new documents every 40 s, under first-come-first-served (FCFS) and longest-prefix-match (LPM) order. (b) One change at a time on the repeated document with FCFS order: prefill chunks of 2,048 tokens, the server pinned to three, two, or one cores, and Qwen3-1.7B in place of Qwen3-8B, alone and together with one core.
| Check | Compared | Differ |
|---|---|---|
| Fast encoder, whole encodes, 11 tokenizers | 41.8B | 0 |
| Web corpus split at Delta Tokenizer’s cuts, 4 tokenizers | 409.6M | 0 |
| Special-token cuts, chat prompts | 610 | 0 |
| Turn-by-turn replay, 610 chat transcripts | 323,680 | 0 |
| Adversarial battery, 6 tokenizers | 220,000 | 0 |
| control, expected to differ , without the normalization check, Qwen3 | 100,000 | 2,115 |
Table 3: Exactness checks against the reference. Each comparison covers a complete token-ID sequence, except the prompt-token counts that SGLang reports and the greedy completions, which compare the model’s output. The shaded row is a control, not a check of Delta Tokenizer. It removes the normalization check, and its nonzero count shows that the battery detects what that check prevents.
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
| Tokenizer | Repository; revision and file digest | Normalizer | Special strings | Margin | Bound | Also in |
|---|---|---|---|---|---|---|
| Qwen3-8B | Qwen/Qwen3-8B b968826d9c46 aeb13307a71a | NFC | 26 | 20 | 128 | 79 |
| Llama-3.1-8B | meta-llama/Llama-3.1-8B d04e592bb4f6 76e48799b099 | none | 256 | 30 | 128 | 3 |
| gpt-oss-120b | openai/gpt-oss-120b b5c939de8f75 0614fe83cada | none | 21 | 19 | 128 | 3 |
| DeepSeek-V3 | deepseek-ai/DeepSeek-V3 e815299b0bcb 621ac2e32d0d | empty list | 818 | 23 | 128 | 2 |
| Qwen3.5-0.8B | Qwen/Qwen3.5-0.8B 2fc06364715b 5f9e4d4901a9 | NFC | 33 | 21 | 128 | 19 |
| MiniMax-M3 | MiniMaxAI/MiniMax-M3 50942730318c bb1f1626cf01 | NFC | 61 | 31 | 128 | 1 |
Table S1: The eleven admitted tokenizers. Each tokenizer file was frozen from the named repository at the revision given by the first hexadecimal string. The second string begins the SHA-256 digest of the file, which admission compares. Special strings are counted in the loaded configuration. Margin and Bound are the repair parameters in characters. Also in counts the other public repositories whose tokenizer file has the same digest (Table S2 ).
| Tokenizer | Count | Public repositories with the same tokenizer file |
|---|---|---|
| Qwen3-8B | 79 | Qwen/Qwen3-0.6B (+FP8, GPTQ-Int8, MLX-4bit, MLX-6bit, MLX-8bit, MLX-bf16); Qwen/Qwen3-1.7B (+FP8, GPTQ-Int8, MLX-4bit, MLX-6bit, MLX-8bit, MLX-bf16); Qwen/Qwen3-4B (+FP8, AWQ, MLX-4bit, MLX-6bit, MLX-8bit, MLX-bf16); Qwen/Qwen3-4B-Instruct-2507 (+FP8); Qwen/Qwen3-4B-SafeRL; Qwen/Qwen3-4B-Thinking-2507 (+FP8); Qwen/Qwen3-8B (FP8, AWQ, MLX-4bit, MLX-6bit, MLX-8bit, MLX-bf16); Qwen/Qwen3-14B (+FP8, AWQ, MLX-4bit, MLX-6bit, MLX-8bit, MLX-bf16); Qwen/Qwen3-30B-A3B (+FP8, GPTQ-Int4, MLX-4bit, MLX-6bit, MLX-8bit, MLX-bf16); Qwen/Qwen3-30B-A3B-Instruct-2507 (+FP8); Qwen/Qwen3-30B-A3B-Thinking-2507-FP8; Qwen/Qwen3-32B (+FP8, AWQ, MLX-4bit, MLX-6bit, MLX-8bit, MLX-bf16); Qwen/Qwen3-235B-A22B (+FP8, GPTQ-Int4, MLX-6bit, MLX-8bit, MLX-bf16); Qwen/Qwen3-235B-A22B-Instruct-2507 (+FP8); Qwen/Qwen3-235B-A22B-Thinking-2507-FP8; Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8; Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8; Qwen/Qwen3-Next-80B-A3B-Instruct (+FP8); Qwen/Qwen3-Next-80B-A3B-Thinking (+FP8); Qwen/Qwen3-Reranker-0.6B; Qwen/Qwen3-Reranker-4B; Qwen/Qwen3-Reranker-8B; Qwen/Qwen3-VL-Reranker-2B; Qwen/Qwen3-VL-Reranker-8B; nvidia/Nemotron-Terminal-8B; nvidia/Nemotron-Terminal-14B; nvidia/Nemotron-Terminal-32B |
| Llama-3.1-8B | 3 | meta-llama/Llama-3.1-70B; meta-llama/Llama-3.1-405B (+FP8) |
| gpt-oss-120b | 3 | openai/gpt-oss-20b; openai/gpt-oss-safeguard-120b; openai/gpt-oss-safeguard-20b |
| DeepSeek-V3 | 2 | deepseek-ai/DeepSeek-V3-0324; deepseek-ai/DeepSeek-V3-Base |
| Qwen3.5-0.8B | 19 | Qwen/Qwen3.5-2B; Qwen/Qwen3.5-4B; Qwen/Qwen3.5-9B; Qwen/Qwen3.5-27B (+FP8, GPTQ-Int4); Qwen/Qwen3.5-35B-A3B (+FP8, GPTQ-Int4); Qwen/Qwen3.5-122B-A10B (+FP8, GPTQ-Int4); Qwen/Qwen3.5-397B-A17B (+FP8, GPTQ-Int4); Qwen/Qwen3.6-27B (+FP8); Qwen/Qwen3.6-35B-A3B (+FP8) |
| MiniMax-M3 | 1 | MiniMaxAI/MiniMax-M3-MXFP8 |
Table S2: Public model repositories whose tokenizer file has the same SHA-256 digest as the frozen file of a tokenizer. A parenthesized list names suffixes that further repositories append to the name before it, and a leading plus sign means that the name itself is included. Qwen/Qwen3-4B-Instruct-2507 (+FP8) thus stands for two repositories, the model and its FP8 release, and nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B (BF16, NVFP4) for the two suffixed ones.
| Check | Qwen3-8B | Llama- 3.1-8B | gpt-oss- 120b | DeepSeek- V3 | Qwen3.5- 0.8B | MiniMax- M3 |
| Fast encoder, whole encodes, documents | 3.8B | 3.8B | 3.8B | 3.8B | 3.8B | 3.8B |
| Web corpus split at Delta Tokenizer’s cuts, documents | 102.4M | 102.4M | 102.4M | 102.4M | – | – |
| Chunking sweep, Zen 5 server | 16,050 | 6,560 | 6,560 | – | – | – |
| Chunking sweep, two desktops | – | – | – | – | ||
| Serial encoder, two desktops | 2,645 | 2,645 | 2,645 | 2,645 | – | – |
| Special-token cuts, chat prompts | 610 | – | – | – | – | – |
Table S3: Checks against the reference, by tokenizer. Each comparison covers a complete token-ID sequence, except the prompt-token counts that SGLang reports and the greedy completions, which compare the model’s output. Apart from the control, none finds a difference. The control row counts the strings that differ once the normalization check is removed. The sweep rows count every output assembled at Delta Tokenizer’s cuts, whichever encoder encoded the chunks (Delta Tokenizer’s fast encoder, the reference, or fastokens), and the whole encodes of Delta Tokenizer’s fast encoder. The serial-encoder row counts Delta Tokenizer’s fast encoder called in three ways, through the native runtime with and without building the index for content lookup (Section S6.3 ) and through the Python tokenizer interface. The gpt-oss-120b shadow re-encodes are not part of the main paper’s count of 2,195. The row for served long documents includes runs with Qwen3-1.7B, which uses the same tokenizer files as Qwen3-8B, and twelve runs on a GH200 node with an ARM build of the fast encoder (Section S7 ).
| Check | Compared | Differ |
|---|---|---|
| Chunking sweep, Zen 5 server | 29,170 | 0 |
| Chunking sweep and serial encoder, two Zen 5 desktops | 31,572 | 0 |
| Rust prototype, cuts and IDs | 167,912 | 0 |
| Scheduling testbed, server and desktop | 448,676 | 0 |
| Capacity study, 128-core Zen 3 node, 215 runs | 429,457 | 0 |
| Total | 1,106,787 | 0 |
Table S4: The rows that Table 3 of the main paper combines into its row of platform and runtime repeats. Each comparison covers a complete token-ID sequence, and none differs.
| Transition probe, unsplit | Seam test | ||||||
|---|---|---|---|---|---|---|---|
| Tokenizer | Pattern | Normalizer | at exclusions | elsewhere | documents | wrong | whole |
| Qwen3-8B | A | NFC | 1,760 | 0 | – | – | – |
| Llama-3.1-8B | B | none | 1,760 | 0 | – | – | – |
| gpt-oss-120b | C | none | 2,208 | 0 | – | – | – |
| DeepSeek-V3 | D | empty list | 1,760 | 0 | – | – | – |
| Qwen3.5-0.8B | E | NFC | 1,760 | 0 | 3,000 | 0 | 94 |
Table S5: Configuration checks for all eleven tokenizers. Tokenizers with the same pattern letter split text with the same regular expressions. The probe has 57,728 strings per tokenizer, and a string is unsplit when the reference does not start a new piece at the class change it probes. The seam test was run for the seven tokenizers other than the main paper’s four; whole counts documents encoded whole after the retries.
| Desktop A | Desktop B | ||||||
|---|---|---|---|---|---|---|---|
| Tokenizer | Normalizer | Reference | Fast encoder | Speedup | Reference | Fast encoder | Speedup |
| Qwen3-8B | NFC | 3.3 | 54.4 | 16.6 | 3.1 | 51.2 | 16.4 |
| Llama-3.1-8B | none | 4.1 | 126.3 | 30.7 | 3.9 | 118.6 | 30.3 |
| gpt-oss-120b | none | 4.1 | 131.6 | 32.3 | 3.8 | 122.6 | 32.4 |
| DeepSeek-V3 | empty list | 3.5 | 113.6 | 32.9 | 3.2 | 109.9 | 33.8 |
Table S6: Single-core throughput in MB/s of UTF-8 text on English web documents of one million characters, measured on two identical Zen 5 desktops. Each value is the median over nine of ten documents, leaving out the first document that the process encodes. Speedups divide the unrounded medians.
| One core | Sixteen cores | Best speedup over | |||||
| Input | Reference | Whole | 1 chunk | 16 chunks | 32 chunks | whole | 1 chunk |
| Qwen3-8B | |||||||
| web, 1M | 374.37 | 27.21 | 30.81 | 7.37 | 5.48 | 5.0 | 5.6 |
| web, 4M | 1,617.90 | 150.06 | 170.28 | 23.62 | 18.31 | 8.2 | 9.3 |
| code, 1M | 435.35 | 30.96 | 33.10 | 7.56 | 5.56 | 5.6 | 6.0 |
| code, 4M | 1,866.23 | 149.47 | 154.59 | 23.22 | 19.30 | 7.7 | 8.0 |
Table S7: Median encode time in milliseconds on sixteen cores of the Zen 5 server, over 20 documents per setting, each timed at its first encode in the process. Reference and Whole encode the document on one core with the reference and with the fast encoder. The 1-chunk column uses the worker pool with a single chunk, which adds the cost of handing the document to a worker. The speedups use the best chunk count, 32 for Qwen3-8B and 16 for the others. Inputs are English web text, source code, text from public coding-agent transcripts, and multilingual text; sizes are in characters.
| Passed to the reference | Median time | ||||||
|---|---|---|---|---|---|---|---|
| Tokenizer | Input | Documents | all | normalization | special string | passed | others |
| Qwen3-8B | web, 64K | 20 | 1 | 1 | 0 | 17.39 | 1.64 |
| Qwen3-8B | web, 256K | 20 | 1 | 1 | 0 | 77.36 | 6.60 |
| Qwen3-8B | web, 1M | 20 | 4 | 4 | 0 | 286.32 | 25.41 |
| Qwen3-8B | web, 4M | 20 | 7 | 7 | 0 | 1202.29 | 129.26 |
| Qwen3-8B | web, 10M | 10 | 7 | 7 | 0 | 2881.65 | 397.84 |
Table S8: Documents that the fast encoder, encoding whole on one core of the Zen 5 server, passed to the reference, by the guard that triggered, with median times in milliseconds. Settings without such documents are omitted, including every gpt-oss-120b setting.
| Median | 99th percentile | |||||
|---|---|---|---|---|---|---|
| Model and host | Context | Reference | Delta Tokenizer | change | Reference | Delta Tokenizer |
| Qwen3-8B, Zen 5 server (RTX PRO 6000) | 8K tokens | 29.13 | 29.18 | % | 33.55 | 33.49 |
| 16K tokens | 35.48 | 30.84 | % | 38.85 | 33.53 | |
| 32K tokens | 55.67 | 37.52 | % | 63.58 | 42.98 | |
| 64K tokens | 94.85 | 48.02 | % | 104.25 | 55.69 | |
| 100K tokens | 136.83 | 53.72 | % | 158.49 | 74.97 | |
Table S9: Time to first token in milliseconds in SGLang for one session that grows by 400 characters per turn, with the reference tokenizer and with Delta Tokenizer. Qwen3-8B ran with three server processes per tokenizer on the Zen 5 server and two on the Zen 5 desktop, and gpt-oss-120b with two on the Zen 5 server. The gpt-oss-120b contexts are 100K, 250K, and 350K characters.
| Tokenizer | Small | Medium | Large | With ID check |
|---|---|---|---|---|
| Qwen3 | 8 | 1 | 0 | 0 |
| Llama-3.1 | 8 | 2 | 0 | 0 |
| gpt-oss | 4 | 5 | 2 | 0 |
| DeepSeek-V3 | 0 | 0 | 0 | – |
| BERT, cased | 0 | 0 | 0 | – |
| BERT, uncased | 0 | 0 | 0 | – |
Table S10: CLongEval samples whose IDs from LoPT’s published rule differ from serial encoding. The last column reruns all three subsets for the three tokenizers with differences, with token IDs compared as well as positions; a dash marks a tokenizer that was not rerun because its published rule produced no difference.
| Index | Serial ID | LoPT ID | Serial bytes | LoPT bytes | Character span |
|---|---|---|---|---|---|
| 11,133 | 53930 | 53930 | U+4E4B | U+4E4B | 13,574–13,575 |
| 11,134 | 17401 | 50509 | 229 145 | 229 145 189 | 13,575–13,576 |
| 11,135 | 29941 | 29941 | 189, U+6570 | 189, U+6570 | 13,575–13,577 |
Table S11: The seam of one wrong output under Qwen3. Index 11,134 carries different tokens with the same span, so position matching accepts the seam, and the joined output contains byte 189 twice.
| 8 chunks | 16 chunks | |||||||
| Tokenizer | Input | Size | Serial | LoPT | Safe cuts | Serial | LoPT | Safe cuts |
| Qwen3 | Web text | 256K | 103 | 31.7 | 18.6 | 101 | 32.9 | 19.4 |
| 1M | 420 | 121 | 73.7 | 417 | 123 | 77.9 | ||
| 4M | 1,784 | 499 | 286 | 1,789 | 508 | 305 | ||
| Code | 256K | 121 | 39.0 | 22.2 | 120 | 40.2 | 19.7 | |
| 1M | 483 | 154 | 87.2 | 487 | 150 | 88.2 | ||
Table S12: Median encode time in milliseconds on eight cores, with the Hugging Face tokenizer as the encoder in every column. Serial encoding was measured again in each configuration. Each 8-chunk cell has 20 inputs, and each 16-chunk cell 10.
| 100K-token request form | Reference | Fast stateless | Delta Tokenizer |
|---|---|---|---|
| Clean raw text | 140.64 | 54.21 | 53.17 |
| Raw text, one decomposed accent | 137.94 | 145.66 | 53.64 |
| Chat completion | 144.80 | 150.07 | 59.61 |
Table S13: Median TTFT in milliseconds. These are the unrounded values behind the request-form comparison of the main paper.
| Parameter | Pool default | In the experiments |
|---|---|---|
| Queue order | shortest-first | shortest-first; FIFO in the four-worker FIFO comparison |
| Aging weight | 1.0 ms per ms waited | 1.0 |
| Cutting threshold | 131,072 characters | 65,536 in the replay, the SGLang pool runs, and the microbenchmark; 131,072 in the testbed |
| Target chunk size | 65,536 characters | 65,536; also 32,768 and one chunk per worker on the 128-core Zen 3 node |
| Chunk count | testbed rounds to the nearest integer | |
| Chunk cap | 16 | 16; equal to the number of workers on the 128-core Zen 3 node |
Table S14: Scheduler and chunking parameters. The pool default is what the adapter uses when nothing is configured.
| Call | 100K characters | 1M characters |
|---|---|---|
| Reference tokenizer, serial | 22.79 | 325.31 |
| Delta Tokenizer serial, stateless | 1.98 | 30.80 |
| Delta Tokenizer serial, session append of 29 characters | 0.68 | 5.34 |
| Pool of 4, same session append | 1.51 | 11.07 |
| Pool of 8, same session append | 1.46 | 11.29 |
| Pool of 4, stateless, one task | 1.71 | – |
Table S15: Cost of a single call with the Qwen3 tokenizer, in milliseconds to two decimals, measured on an otherwise idle host, each the median of five calls, or of three for the reference. The cutting threshold is 65,536 characters here, so 100K-character requests are cut. The one-task rows use the first 60,000 characters of the same document, which stay below the threshold and run as a single task. The session rows time the first call that appends 29 characters to a session holding the text, in a fresh session for each repetition; a second call with the same text, which finds it stored and only returns the IDs, costs about the same (0.75 and 6.44 ms serial). Every answer of the pool equals the reference IDs.
| 100K-character contexts | |||||||
|---|---|---|---|---|---|---|---|
| Achieved load (requests/s) | 307 | 611 | 812 | 962 | 1,117 | 1,399 | |
| Per-worker FIFO | 8.4 | 12.0 | 22.0 | 69.3 | 4,108† | – | |
| Two lanes, tuned | 8.9 | 14.0 | 65.0 | 4,584† | – | – | |
| Shared shortest-first | 8.4 | 10.5 | 17.0 | 36.1 | 2,162 | 8,633† | |
| Chunks, FIFO | 4.4 | 6.4 | 10.5 | 24.0 | 675 | 7,650† | |
| Chunks, shortest-first | 4.5 | 6.8 | 10.3 | 23.6 | 371 | 6,876† | |
Table S16: P99 latency of the five pool designs on four workers, with every request encoded in full. The two-lane design sends requests of at least 256K characters to one worker and the rest to three at 100K, and requests of at least 128K characters to three workers and the rest to one at 1M. A dagger marks a backlogged run. The column at 1,117 requests/s sits at the edge of saturation, where a repeated run moves an entry by about 20% but leaves the order of the designs unchanged.
| Threshold | Small:large = 1:3 | 2:2 | 3:1 |
|---|---|---|---|
| 100K-character contexts, loads of 611 and 812 requests/s | |||
| 8K | 15.5 / 510 | † | † |
| 32K | 15.5 / 506 | † | † |
| 128K | 34.3 / † | 22.8 / 358 | † |
| 256K | † | 738 / † | 14.0 / 55.2 |
| 1M-character contexts, loads of 52.5 and 71.1 requests/s | |||
Table S17: P99 latency of the twelve two-lane configurations at each context size, given as P99 at the lower load / P99 at the higher load. Columns give how many of the four workers serve the small and the large lane, and K is 1,024 characters. A dagger alone means the configuration was backlogged at the lower load and not run at the higher one. The configuration in bold is the one used in Table S16 .
| Appends to 1M-character contexts, P99 of all requests | |||||
|---|---|---|---|---|---|
| Achieved load (requests/s) | 88.4 | 168.2 | 225.4 | 264.2 | |
| Per-worker FIFO | 404 | 497 | 3,261† | – | |
| Two lanes | 410 | 681 | 8,734† | – | |
| Shared shortest-first | 397 | 480 | 1,637 | 6,355† | |
| Chunks, FIFO | 400 | 494 | 3,576† | – | |
| Chunks, shortest-first | 400 | 476 | 1,697 | 6,253† | |
Table S18: The two mixes with session reuse on four workers. The two-lane design uses a 32K threshold with three small-lane workers. A dagger marks a backlogged run.
| Rate | Reference | Delta Tokenizer serial | In-process chunks | Pool of 2 | Pool of 4 | Pool of 8 |
|---|---|---|---|---|---|---|
| 100K-character sessions | ||||||
| 5 | 36.2 / 73.9 | 1.7 / 4.9 | 1.6 / 4.7 | 2.9 / 4.7 | 2.7 / 5.7 | 2.8 / 5.5 |
| 10 | 39.3 / 108 | 1.6 / 4.5 | 1.7 / 4.7 | 2.6 / 6.0 | 2.7 / 7.6 | 2.8 / 6.9 |
| 20 | 80.8 / 524 | 1.6 / 4.4 | 1.6 / 4.5 | 2.7 / 6.1 | 2.6 / 6.2 | 2.7 / 6.9 |
| 40 | 7,161† / 20,387† | 1.7 / 5.4 | 1.7 / 5.5 | 2.8 / 6.1 | 2.8 / 6.5 | 2.8 / 7.0 |
| 80 | – | 1.7 / 4.9 | 1.8 / 5.2 | 2.8 / 6.1 | 2.8 / 6.3 | 2.9 / 6.8 |
Table S19: Median / P99 latency from scheduled arrival in the replay of SGLang’s tokenizer stage, by offered rate in requests/s. No configuration was run above the first rate at which it backlogged. A dagger marks a backlogged run.
| Rate | Reference | Delta Tokenizer serial | In-process chunks | Pool of 4 | Pool of 8 | Documents |
|---|---|---|---|---|---|---|
| 20 | 17,160† / 12,802† | 377 / 397 | 54.7 / 111 | 32.4 / 142 | 14.5 / 94.2 | 40 |
| 40 | – | 325 / 478 | 38.5 / 252 | 20.5 / 119 | 10.0 / 72.0 | 56 |
| 80 | – | 654 / 764 | 53.0 / 73.1 | 31.7 / 110 | 15.1 / 68.4 | 48 |
| 160 | – | 542 / 728 | 48.8 / 81.1 | 28.9 / 121 | 16.0 / 72.8 | 48 |
| 320 | – | 530 / 525 | 200 / 269 | 49.2 / 132 | 21.0 / 83.1 | 24 |
Table S20: P99 of session turns / P99 of documents in the burst workload of the tokenizer-stage replay, by offered turn rate in requests/s. The last column counts the documents in the measured window. A dagger marks a backlogged run.
| Offered rate | Achieved rate | Whole requests | Chunks |
|---|---|---|---|
| 13 | 12.4 | 32.0 / 346 | 13.6 / 95.7 |
| 26 | 26.8 | 32.1 / 374 | 14.4 / 103 |
| 39 | 41.4 | 36.3 / 409 | 16.5 / 117 |
| 52 | 52.4 | 40.6 / 451 | 19.5 / 132 |
Table S21: Median / P99 latency with one shared FIFO queue on four workers and stateless million-character requests, with rates in requests/s. The arrivals are the same in both columns.
| Workers | Configuration | Passes in every run | Mixed | Fails in every run |
|---|---|---|---|---|
| 1M-character requests | ||||
| 16 | Chunks, 64K target | 244.4 (44.8, 44.8) | – | 290.2 (59.5–62.2, 3 runs) |
| 16 | Per-worker FIFO | – | – | 91.7 (1,421) |
| 32 | Chunks, 64K target | 411.5 (48.9, 49.5) | – | 488.7 (53.7–58.9, 3 runs) |
| 32 | Chunks, one per worker | 530 (30.1, 38.9) | 565.8 (45.3, 50.4) | 620 (198, 1,398) |
| 32 | Per-worker FIFO | – | – | 154.3 (4,878†) |
Table S22: Offered rates in requests/s around a 50 ms P99 objective on the 128-core Zen 3 node, with the P99 of each run in parentheses. The columns give the highest rate that passed in every run, a rate that passed in one run and failed in another, and the lowest rate above it that failed in every run. Several values are runs on different hosts. A dash means that no tested rate falls in that column. All rows use the earlier version of the testbed’s dispatcher.
| Workers | Configuration | Seed 1 | Seed 2 | Seed 3 | All seeds |
|---|---|---|---|---|---|
| 16 | Chunks, 64K target | 270 290.2 | 270 290.2 | 244.4 270 | 244.4 290.2 |
| 32 | Chunks, one per worker | 530 565.8 | 565.8 620 | 565.8 620 | 530 620 |
| 64 | Chunks, 32K target | 750 812 | 812 875 | 812 875 | 750 875 |
Table S23: Offered rates in requests/s of million-character stateless requests on the dedicated 128-core Zen 3 node, by arrival seed. Each entry is the highest tested rate whose P99 stayed at or below 50 ms, followed by the lowest tested rate above it that did not. The last column is the union over seeds. A run is valid if it exits cleanly, every scheduled arrival is either completed or still pending at the end, and the host did not stall. Every run was valid in this sense, and 24,475 shadow comparisons found no difference.
| Target | Chunks | 250 | 375 | 500 | 625 | 750 | 875 | 1,000 | Dispatcher CPU at 750 |
| Earlier dispatcher | |||||||||
| 64K | 15–20 | 38.5 | 39.2 | 46.0 | 56.6 | 78.4 | 101 | – | 0.38 |
| 32K | 31–39 | 20.5 | 36.5 | 23.4 | 24.5 | 29.1 | 104 | 3,057† | 0.62 |
| 32K, repeat | 31–39 | 20.4 | 21.7 | 22.0 | 36.7 | 27.8 | 69.9 | 3,252† | 0.62 |
| One per worker | 61–64 | 13.7 | 14.4 | 20.9 | 39.1 | 4,078† | – | – | 0.93 |
| One per worker, repeat | 61–64 | – | – | – | 93.5 | 5,095† | – | – | 0.93 |
Table S24: P99 latency on 64 workers with million-character requests, by target chunk size and offered rate in requests/s. The second column gives the range of chunks per request. The last column is the dispatcher’s CPU time divided by the run’s duration. The two repeats with one chunk per worker ran on the same host with the same allocation. With the 64K target and the earlier dispatcher, 1,094 requests/s is backlogged (8,182 ms), and with the 32K target and the later dispatcher, 1,125 requests/s is backlogged (5,743 ms). A dagger marks a backlogged run.
| Workers | Design | Seed 1 | Seed 2 | Seed 3 | All seeds |
|---|---|---|---|---|---|
| 16 | Per-worker FIFO | 1,887 2,200 | 1,887 2,200 | 1,887 2,200 | 1,887 2,200 |
| 16 | Shared queue | 2,516 2,750 | 2,516 2,750 | 2,516 2,750 | 2,516 2,750 |
| 32 | Per-worker FIFO | 3,316 3,850 | 2,750 3,316 | 2,750 3,316 | 2,750 3,850 |
| 32 | Shared queue | 4,900 5,251 | 4,422 4,900 | 4,422 4,900 | 4,422 5,251 |
| 64 | Per-worker FIFO | 2,953 3,900 | 2,953 3,900 | 2,953 3,900 | 2,953 3,900 |
| 64 | Shared queue | 5,907 6,500 | 7,656 9,570 | 6,500 7,656 | 5,907 9,570 |
Table S25: Offered rates in requests/s of 100K-character stateless requests on the dedicated 128-core Zen 3 node, by arrival seed, with the 64K chunk target, which leaves these requests whole. Each entry is the highest tested rate whose P99 stayed at or below 50 ms, followed by the lowest tested rate above it that did not; the last column is the union over seeds. In seed 1 on 64 workers the P99 crossed 50 ms at 6,500 requests/s (64.8 ms) and fell below it again at 7,656 (45.6 ms) before the run at 9,570 overloaded; the bracket takes the first crossing.
| Four workers, every call encoded in full | ||||
|---|---|---|---|---|
| Achieved load (requests/s) | 52.6 | 105.1 | 210.0 | 420.4 |
| Per-worker FIFO | 6.0 | 4.3 | 3.6 | 4.5 |
| Shared shortest-first | 5.5 | 4.1 | 3.5 | 3.5 |
| Chunks, FIFO | 4.1 | 3.1 | 3.0 | 3.0 |
| Four workers, with session reuse | ||||
| Achieved load (requests/s) | 105.1 | 210.0 | 420.4 | 841.8 |
Table S26: P99 latency in the replay whose sessions follow real agent traces in their arrival times and request sizes, filled with web text. The workers are at most 7.9% busy in the first block, 14.9% in the second, and 5.5% in the third.
| Setting | Sessions | Runs | Reference | Delta Tokenizer serial | Change |
|---|---|---|---|---|---|
| Closed loop | 1 | 3 | 177 / 309 | 158 / 276 | 10.5 / 11.0% |
| Closed loop | 4 | 3 | 520 / 878 | 568 / 887 | 9.2 / 1.0% |
| Closed loop | 8 | 3 | 870 / 1,660 | 941 / 1,596 | 8.1 / 3.8% |
| Paced | 4 | 2 | 198 / 481 | 172 / 383 | 13.0 / 20.4% |
| Paced | 8 | 2 | 227 / 762 | 201 / 719 | 11.5 / 5.6% |
| Small appends, closed loop | 4 | 2 | 101 / 147 | 94.7 / 106 | 6.4 / 27.5% |
Table S27: Warm-turn TTFT in SGLang, median / P99 in milliseconds, with the reference tokenizer and with Delta Tokenizer serial. The last column gives the change of the median and of the P99. The Poisson row measures from each turn’s scheduled arrival.
| Tokenizer | Cores | 4 turns/s | 8 turns/s | 12 turns/s |
|---|---|---|---|---|
| Reference | 1 | 146 / 839 / 0.16 | 293 / 1,614 / 116 | 3,385 / 16,492 / 3,180 |
| Reference | 8 | 147 / 849 / 0.17 | 296 / 1,632 / 114 | 3,284 / 15,845 / 3,067 |
| Delta Tokenizer | 1 | 59.4 / 517 / 0.18 | 85.8 / 1,041 / 0.18 | 1,496 / 9,842 / 1,386 |
| Delta Tokenizer | 8 | 59.3 / 516 / 0.17 | 87.0 / 1,050 / 0.20 | 1,483 / 9,783 / 1,379 |
| fastokens | 1 | 62.7 / 534 / 0.17 | 89.0 / 1,039 / 0.20 | 1,574 / 10,577 / 1,454 |
Table S28: Median TTFT / P99 TTFT / median wait behind the session’s previous request, in milliseconds from scheduled arrival, when six sessions replay 250K-character transcripts open-loop in SGLang. The cores column is the budget of SGLang’s HTTP and tokenizer process.
| Reference | Delta Tokenizer serial | Pool of 4 | Pool of 8 | |||||
|---|---|---|---|---|---|---|---|---|
| Rate | Turns | Doc. | Turns | Doc. | Turns | Doc. | Turns | Doc. |
| 4 | 52.0 / 961 / 1,297 | 995 | 25.6 / 49.1 / 71.7 | 128 | 26.7 / 46.6 / 62.9 | 32.4 | 26.8 / 47.4 / 64.3 | 31.5 |
| 8 | 58.7 / 1,242 / 1,441 | 1,013 | 26.0 / 53.8 / 166 | 125 | 26.5 / 48.7 / 62.7 | 33.2 | 26.6 / 49.0 / 62.6 | 32.9 |
| 12 | 71.5 / 1,324 / 1,561 | 976 | 26.4 / 53.7 / 138 | 125 | 27.1 / 51.0 / 65.1 | 35.0 | 27.0 / 51.1 / 67.1 | 32.5 |
Table S29: SGLang with the model in the loop and clusters of million-character documents that are tokenized and then declined. The turn columns give warm-turn TTFT from scheduled arrival as P50 / P95 / P99, and the document columns give the median time until a document is answered. Rates are warm turns per second.
| Token | Reference | fastokens |
|---|---|---|
| 9,020 | _state | _state |
| 9,021 | newline, 6 spaces, newline (35117) | newline (198) |
| 9,022 | 13 spaces (1060) | 6 spaces, newline (6926) |
| 9,023 | space, for (369) | 13 spaces (1060) |
| 9,024 | space, method (1714) | space, for (369) |
Table S30: The first difference in the fastokens replay, at turn 16 of one session, with token IDs in parentheses. Turn 15 ended after the first newline and six spaces, so the reference’s token 9,021 spans the boundary between the two turns.
| Execution | Clean | Guarded |
|---|---|---|
| Whole input, one core | 26.5 | 276 |
| 16 chunks, one core | 24.7 | 45.0 |
| 16 chunks, 16 cores | 5.1 | 17.4 |
Table S31: Median encode time in milliseconds for English documents of one million characters, 16 clean and 4 that trigger the fast encoder’s guard. Chunking on one core confines the fallback to one chunk.
| Text | Size | Serial | 8 | 16 | 32 | 64 | Best, list | Best, bytes | Guarded | Fallback | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Web | 1M | 50.5 | 15.6 | 12.9 | 10.3 | 7.8 | 8.1 | 6.47 (32) | 8.55 (32) | 4 of 20 | 1 |
| Web | 4M | 268 | 93.7 | 65.5 | 43.5 | 35.7 | 36.2 | 7.50 (32) | 12.93 (64) | 7 of 20 | 2 |
| Web | 10M | 4,514 | 1,405 | 678 | 333 | 194 | 138 | 32.73 (64) | 47.84 (64) | 7 of 10 | 4 |
| Code | 1M | 41.8 | 17.9 | 11.7 | 9.5 | 8.0 | 7.1 | 5.88 (64) | 8.50 (64) | 0 of 20 | 0 |
| Code | 4M | 255 | 86.7 | 57.8 | 49.1 | 39.7 | 38.3 | 6.66 (64) | 14.33 (64) | 1 of 20 | 2 |
| Code | 10M | 552 | 272 | 182 | 137 | 111 | 107 | 5.17 (64) | 11.31 (64) | 2 of 10 | 2 |
Table S32: Median time in milliseconds of the Rust prototype with the Qwen3 tokenizer on 66 cores of the rented 128-core Zen 3 node, by chunk count with the list return, and the best speedup over serial encoding, to two decimals, with the chunk count that gives it. Guarded counts the documents whose serial encode falls back to the reference for the whole document. Fallback is the largest number of the 64 chunks of one document that fall back.