TokenRouter: Efficient Serving System for Token-Level LLM Routing

Tianyu Fu; Tengxuan Liu; Ruoxi Wang; Yixin Dong; Yi Ge; Yichen You; Yu Wang

cs.CLOct 8, 2026

TokenRouter: Efficient Serving System for Token-Level LLM Routing

Authors: Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong, Yi Ge, Yichen You, Yu Wang

Organizations: Tsinghua University · Carnegie Mellon University

Abstract

Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing can yield substantial efficiency and quality gains. However, efficiently serving token-level routed inference poses significant challenges to existing systems. Built on single-LLM assumptions, current systems suffer from severe step desynchronization and frequent batch admission delays under token-level routing, and they also impose high implementation complexity on developers. To address these challenges, we design TokenRouter, an efficient and developer-friendly serving system for token-level routed LLM inference. TokenRouter follows the principle of request-centric programming, model-centric execution: developers describe routing logic from the perspective of a single request, while the runtime launches a subserver for each LLM and dispatches requests asynchronously. Each subserver employs a delayed-batching scheduler, whose optimal hyperparameters are derived from a mathematical throughput model of the system. Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01-64.15x higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing. Our code is available at https://github.com/thu-nics/TokenRouter.

Figures & tables

Panel of Figure 1 — Figure 1 : TokenRouter improves serving efficiency for token-level routing. (a) At concurrency 8, engineering optimizations yield a 1.71 $\times$ throughput gain, and TokenRouter achieves 2.76 $\times$ overall, relative to the official R2R implementation. (b) Throughput–speed trade-off. TokenRouter achieves 18.58 $\times$ higher throughput than R2R under a stricter SLO.

Panel of Figure 2 — Figure 2 : (a) Routing granularities in LLM serving. Token-level routing allows decisions at each token, whereas session- or query-level routing binds an entire request to one model. (b) TokenRouter’s model-centric runtime. Each cooperating LLM is hosted by an autonomous subserver that implements three user-defined routing functions (route, send, receive) and runs three decoupled scheduler loops, shown in purple (client-server), blue (decoding), and black (inter-model). The Server Interface exposes a single external endpoint to the client.

Algorithm	#Models	Model $i$	Model $j$
CITER [ 12 ]	2	low-confidence token	after one token
R2R [ 5 ]	2	predicted-divergent token	after one token
R-Stitch [ 27 ]	2	high-entropy token	low-entropy token
Co-LLM [ 10 ]	2	high-deferral token	after one token
ME [ 33 ]	$\geq$ 2	after every token, according to ensemble weights

Table 1: Routing behavior of the evaluated token-level routing algorithms. Each entry under Model

i

or Model

j

specifies the condition under which that model routes generation to another model.

Figure 4 : Scheduling strategies of token-level routing. (a) Step-Sync sets step barriers; the next step starts only after all models finish the current one. (b) Model-Sync runs one model at a time; the inactive model’s requests stall. (c) Eager-Async processes peer requests immediately, causing large batch admission delay. (d) TokenRouter preserves asynchronous scheduling but delays LLM requests until enough same-model requests accumulate, amortizing the cost across a wider batch.

Figure 5 : Throughput (token/s) comparison across serving systems at concurrency 4.

Figure 6 : Latency (s) comparison across serving systems at concurrency 4.

Algorithm	Implementation	Throughput/(token/s)	TTFT/s	Latency/s
R2R	LLM-only	145.30	0.13	411.98
	Official Code	89.62	0.11	751.15
	TokenRouter	244.56	0.11	270.19
CITER	LLM-only	123.72	0.083	2.68
	Official Code	17.16	0.036	7.64
	TokenRouter	149.31	0.036	0.48

Table 2 : TTFT, throughput, and end-to-end latency at concurrency 4, comparing across serving systems under original settings.

System	Model Pair	Benchmark
R2R	DeepSeek-R1-Distill-Qwen-1.5B / 32B	AIME [ 35 ]
CITER	Qwen2-1.5B / 72B	CommonsenseQA [ 37 ]
Co-LLM	LLaMA2-7B (tuned) / 70B	GSM8K [ 38 ]
R-Stitch	L1-1.5B-Short / QwQ-32B	AIME [ 35 ]

Table 4 : Model pairs and benchmarks used by different token-level routing algorithms.

Figure 8 : Throughput (token/s) comparison across serving systems at concurrency 1.

Figure 9 : Latency (s) comparison across serving systems at concurrency 1.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Algorithm	Implementation	Throughput/(token/s)	TTFT/s	Latency/s
R2R	LLM-only	40.17	0.074	388.12
	Official Code	54.77	0.12	330.69
	TokenRouter	105.81	0.11	144.10
CITER	LLM-only	34.40	0.055	2.42
	Official Code	14.26	0.031	2.41
	TokenRouter	50.63	0.031	0.36

Appendix

Table 5 : TTFT, throughput, and end-to-end latency at concurrency 1, comparing across serving systems under original settings.

Method	$N=8$	$N=4$	$N=1$
TokenRouter	372.48	210.61	73.02
- Delayed batching	296.86	161.26	73.02
- Async execution	230.79	143.88	73.02
- Router CUDA graph	228.10	142.33	72.62
- LLM CUDA graph	164.08	95.55	51.22
- SLM CUDA graph ( $\approx$ R2R)	132.78	75.54	34.20

Appendix

Table 6 : Ablation study of different optimization components. Throughput is reported in token/s.

Total	#SLM	#LLM	Overlap	$N{=}1$	$N{=}4$
2	1	1	$\times$	50.37	160.74
	1	2	$\checkmark$	70.83	197.40
	2	2	$\checkmark$	71.41	198.13
8	4	4	$\times$	93.23	284.62
	1	8	$\checkmark$	109.67	296.34
	2	8	$\checkmark$	106.02	289.08

Appendix

Table 7 : Throughput across parallel strategies. Throughput is reported in token/s.

Setup		Routing Algorithm
$N$	Deployment	R2R	R-Stitch	Co-LLM	CITER
1	Overlapping	73.27	43.06	47.46	102.29
1	Non-overlapping	74.90	47.53	48.25	108.65
4	Overlapping	205.05	108.42	165.96	281.27
4	Non-overlapping	228.94	116.14	170.20	308.36

Appendix

Table 8 : Throughput (token/s) comparison across routing strategies under different concurrency and overlap settings.

Setup		Routing Algorithm
$N$	Deployment	R-Stitch	R2R	CITER
1	Single-node	47.53	74.90	108.65
1	Multi-node	46.19	69.92	93.99
4	Single-node	116.14	228.94	308.36
4	Multi-node	104.20	202.43	277.56

Appendix

Table 9 : Throughput comparison across single-node and multi-node settings.

Operation	Latency (ms)	Ratio
Prefix matching	7.78	20.94%
Locking cache nodes	6.49	17.47%
SLM inference	1.56	4.20%
Releasing locks	5.13	13.81%
Update radix cache	12.12	32.62%
Others	4.08	10.98%

Appendix

Table 10 : Latency breakdown of the SLM-side execution per decoding step in Std. Serving.

Notation	Definition
$M$	Number of cooperating models.
$N$	Number of concurrent requests in the system.
$L_{i}$	Latency of one decoding step on model $i$ .
$p_{ij}$	Probability that a request is routed from model $i$ to model $j$ after a decoding step.
$c_{ij}$	Number of output tokens committed when a request is routed from model $i$ to model $j$ .
$B_{i}$	Delayed-batching threshold of model $i$ .

Appendix

Table 11: Key notations used throughout the analysis.

Figure 10 : Predicted and measured R2R throughput under different concurrency levels and delayed-batching thresholds. We fix the SLM threshold at $B_{0}=1$ and vary the LLM threshold $B_{1}$ . An asterisk (*) marks the highest throughput for each concurrency level in each panel.

Explore similar work

Aug 7, 2026cs.CL

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.

Tao Feng, Fangxu Yu, Haozhen Zhang +9

University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2

LLM Evaluation LLM Routing

Ask PDF

Jul 31, 2026cs.CL

TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving

LLM serving caches prompt KV state, yet most front ends still re-tokenize the full request on every call. Coding agents pay most: sessions repeatedly submit a long transcript after a small append, which can shift token boundaries near the end of the prior sequence. Across 153,951 calls the median append is ~1.4K characters; only 1.0-3.6% of calls start or rebuild a session, yet those carrymulti-million-character contexts. Fleet prompt-cache hit rate is 94.1%, and as it approaches 0.99, tokenization grows from 10% to 64% of time to first token (TTFT) in component measurements. TokTier is a stateful CPU+GPU tokenization service for this two-mode workload, under one contract: emitted token IDs are always identical to full reference tokenization. For session continuations it re-tokenizes a small window around the append and splices only when a per-request check finds a stable pre-tokenization boundary; failed checks widen the window or fall back to full reference tokenization. For calls without a reusable prefix it runs exact GPT-family regex pre-tokenization and BPE on a GPU. A sampled shadow verifier re-checks live traffic. Across 17 production tokenizer families, differential campaigns cover 1.5x10^10 split checks, a 12.4 TB real-text corpus, and 93,000+ replayed agent steps, with zero divergence. Incremental repair takes 0.5-1.1 ms from 100K to 3M characters, up to 437x faster than HF tokenization and 2.1x faster at 1M characters than the strongest cache-based baseline (Gigatoken) fully prewarmed. GPU tokenization encodes a 1M-character request in 0.87 ms, up to 491x below HF and 23.4x below the fastest published CPU method on the same protocol. With vLLM, median TTFT drops 16-34% and P99 TTFT 23% under recorded bursts. Under a 50 ms P99 objective, a four-core repair pool plus one GPU sustains 1,821 requests/s, where a 16-core stateless front end saturates at 40 requests/s.

Zhenyu Zhang, Zhichao Cao

Arizona State University · Tempe, Arizona, USA

LLM Serving LLM Inference Acceleration Subword Tokenization

Ask PDF

May 1, 2026cs.LG

Rethinking LLM Ensembling from the Perspective of Mixture Models

Model ensembling is a well-established technique for improving the performance of machine learning models. Conventionally, this involves averaging the output distributions of multiple models and selecting the most probable label. This idea has been naturally extended to large language models (LLMs), yielding improved performance but incurring substantial computational cost. This inefficiency stems from directly applying conventional ensemble implementation to LLMs, which require a separate forward pass for each model to explicitly compute the ensemble distribution. In this paper, we propose the Mixture-model-like Ensemble (ME). By reinterpreting the ensemble as a mixture model, ME stochastically selects a single model at each step to generate the next token, thereby avoiding the need to explicitly compute the full ensemble distribution. ME is mathematically equivalent to sampling from the ensemble distribution, but requires invoking only one model, making it 1.78x-2.68x faster than conventional ensembling. Furthermore, this perspective connects LLM ensembling and token-level routing methods, suggesting that LLM ensembling is a special case of routing methods. Our findings open new avenues for efficient LLM ensembling and motivate further exploration of token-level routing strategies for LLMs. Our code is available at https://github.com/Kamichanw/Mixture-model-like-Ensemble.

Jiale Fu, Yuchu Jiang, Peijun Wu +3

Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education · Southeast University · Qiyuan Tech +2

LLM Inference Acceleration LLM Routing Language Model Ensembles

Ask PDF