TokenRouter: Efficient Serving System for Token-Level LLM Routing

Tianyu Fu; Tengxuan Liu; Ruoxi Wang; Yixin Dong; Yi Ge; Yichen You; Yu Wang

cs.CLOct 8, 2026

TokenRouter: Efficient Serving System for Token-Level LLM Routing

Authors: Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong, Yi Ge, Yichen You, Yu Wang

Organizations: Tsinghua University · Carnegie Mellon University

Abstract

Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing can yield substantial efficiency and quality gains. However, efficiently serving token-level routed inference poses significant challenges to existing systems. Built on single-LLM assumptions, current systems suffer from severe step desynchronization and frequent batch admission delays under token-level routing, and they also impose high implementation complexity on developers. To address these challenges, we design TokenRouter, an efficient and developer-friendly serving system for token-level routed LLM inference. TokenRouter follows the principle of request-centric programming, model-centric execution: developers describe routing logic from the perspective of a single request, while the runtime launches a subserver for each LLM and dispatches requests asynchronously. Each subserver employs a delayed-batching scheduler, whose optimal hyperparameters are derived from a mathematical throughput model of the system. Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01-64.15x higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing. Our code is available at https://github.com/thu-nics/TokenRouter.

Figures & tables

Panel of Figure 1 — Figure 1 : TokenRouter improves serving efficiency for token-level routing. (a) At concurrency 8, engineering optimizations yield a 1.71 $\times$ throughput gain, and TokenRouter achieves 2.76 $\times$ overall, relative to the official R2R implementation. (b) Throughput–speed trade-off. TokenRouter achieves 18.58 $\times$ higher throughput than R2R under a stricter SLO.

Panel of Figure 2 — Figure 2 : (a) Routing granularities in LLM serving. Token-level routing allows decisions at each token, whereas session- or query-level routing binds an entire request to one model. (b) TokenRouter’s model-centric runtime. Each cooperating LLM is hosted by an autonomous subserver that implements three user-defined routing functions (route, send, receive) and runs three decoupled scheduler loops, shown in purple (client-server), blue (decoding), and black (inter-model). The Server Interface exposes a single external endpoint to the client.

Algorithm	#Models	Model $i$	Model $j$
CITER [ 12 ]	2	low-confidence token	after one token
R2R [ 5 ]	2	predicted-divergent token	after one token
R-Stitch [ 27 ]	2	high-entropy token	low-entropy token
Co-LLM [ 10 ]	2	high-deferral token	after one token
ME [ 33 ]	$\geq$ 2	after every token, according to ensemble weights

Table 1: Routing behavior of the evaluated token-level routing algorithms. Each entry under Model

i

or Model

j

specifies the condition under which that model routes generation to another model.

Figure 4 : Scheduling strategies of token-level routing. (a) Step-Sync sets step barriers; the next step starts only after all models finish the current one. (b) Model-Sync runs one model at a time; the inactive model’s requests stall. (c) Eager-Async processes peer requests immediately, causing large batch admission delay. (d) TokenRouter preserves asynchronous scheduling but delays LLM requests until enough same-model requests accumulate, amortizing the cost across a wider batch.

Figure 5 : Throughput (token/s) comparison across serving systems at concurrency 4.

Figure 6 : Latency (s) comparison across serving systems at concurrency 4.

Algorithm	Implementation	Throughput/(token/s)	TTFT/s	Latency/s
R2R	LLM-only	145.30	0.13	411.98
	Official Code	89.62	0.11	751.15
	TokenRouter	244.56	0.11	270.19
CITER	LLM-only	123.72	0.083	2.68
	Official Code	17.16	0.036	7.64
	TokenRouter	149.31	0.036	0.48

Table 2 : TTFT, throughput, and end-to-end latency at concurrency 4, comparing across serving systems under original settings.

System	Model Pair	Benchmark
R2R	DeepSeek-R1-Distill-Qwen-1.5B / 32B	AIME [ 35 ]
CITER	Qwen2-1.5B / 72B	CommonsenseQA [ 37 ]
Co-LLM	LLaMA2-7B (tuned) / 70B	GSM8K [ 38 ]
R-Stitch	L1-1.5B-Short / QwQ-32B	AIME [ 35 ]

Table 4 : Model pairs and benchmarks used by different token-level routing algorithms.

Figure 8 : Throughput (token/s) comparison across serving systems at concurrency 1.

Figure 9 : Latency (s) comparison across serving systems at concurrency 1.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Algorithm	Implementation	Throughput/(token/s)	TTFT/s	Latency/s
R2R	LLM-only	40.17	0.074	388.12
	Official Code	54.77	0.12	330.69
	TokenRouter	105.81	0.11	144.10
CITER	LLM-only	34.40	0.055	2.42
	Official Code	14.26	0.031	2.41
	TokenRouter	50.63	0.031	0.36

Appendix

Table 5 : TTFT, throughput, and end-to-end latency at concurrency 1, comparing across serving systems under original settings.

Method	$N=8$	$N=4$	$N=1$
TokenRouter	372.48	210.61	73.02
- Delayed batching	296.86	161.26	73.02
- Async execution	230.79	143.88	73.02
- Router CUDA graph	228.10	142.33	72.62
- LLM CUDA graph	164.08	95.55	51.22
- SLM CUDA graph ( $\approx$ R2R)	132.78	75.54	34.20

Appendix

Table 6 : Ablation study of different optimization components. Throughput is reported in token/s.

Total	#SLM	#LLM	Overlap	$N{=}1$	$N{=}4$
2	1	1	$\times$	50.37	160.74
	1	2	$\checkmark$	70.83	197.40
	2	2	$\checkmark$	71.41	198.13
8	4	4	$\times$	93.23	284.62
	1	8	$\checkmark$	109.67	296.34
	2	8	$\checkmark$	106.02	289.08

Appendix

Table 7 : Throughput across parallel strategies. Throughput is reported in token/s.

Setup		Routing Algorithm
$N$	Deployment	R2R	R-Stitch	Co-LLM	CITER
1	Overlapping	73.27	43.06	47.46	102.29
1	Non-overlapping	74.90	47.53	48.25	108.65
4	Overlapping	205.05	108.42	165.96	281.27
4	Non-overlapping	228.94	116.14	170.20	308.36

Appendix

Table 8 : Throughput (token/s) comparison across routing strategies under different concurrency and overlap settings.

Setup		Routing Algorithm
$N$	Deployment	R-Stitch	R2R	CITER
1	Single-node	47.53	74.90	108.65
1	Multi-node	46.19	69.92	93.99
4	Single-node	116.14	228.94	308.36
4	Multi-node	104.20	202.43	277.56

Appendix

Table 9 : Throughput comparison across single-node and multi-node settings.

Operation	Latency (ms)	Ratio
Prefix matching	7.78	20.94%
Locking cache nodes	6.49	17.47%
SLM inference	1.56	4.20%
Releasing locks	5.13	13.81%
Update radix cache	12.12	32.62%
Others	4.08	10.98%

Appendix

Table 10 : Latency breakdown of the SLM-side execution per decoding step in Std. Serving.

Notation	Definition
$M$	Number of cooperating models.
$N$	Number of concurrent requests in the system.
$L_{i}$	Latency of one decoding step on model $i$ .
$p_{ij}$	Probability that a request is routed from model $i$ to model $j$ after a decoding step.
$c_{ij}$	Number of output tokens committed when a request is routed from model $i$ to model $j$ .
$B_{i}$	Delayed-batching threshold of model $i$ .

Appendix

Table 11: Key notations used throughout the analysis.

Figure 10 : Predicted and measured R2R throughput under different concurrency levels and delayed-batching thresholds. We fix the SLM threshold at $B_{0}=1$ and vary the LLM threshold $B_{1}$ . An asterisk (*) marks the highest throughput for each concurrency level in each panel.

Explore similar work

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
Aug 7, 2026Tao Feng, Fangxu Yu, Haozhen Zhang +9LLM Evaluation LLM Routing
University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2
TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving
Jul 31, 2026Zhenyu Zhang, Zhichao CaoLLM Serving LLM Inference Acceleration
Arizona State University · Tempe, Arizona, USA
Rethinking LLM Ensembling from the Perspective of Mixture Models
May 1, 2026Jiale Fu, Yuchu Jiang, Peijun Wu +3LLM Inference Acceleration LLM Routing
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education · Southeast University · Qiyuan Tech +2

Aug 7, 2026, cs.CLJ/K move · Enter open · S save

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

Tao Feng, Fangxu Yu, Haozhen Zhang, Zhongjie Dai+8

University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University+2

LLM Evaluation LLM Routing

Open paper AskPDF arXiv

No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.

Abstract

Explore similar work

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving

Rethinking LLM Ensembling from the Perspective of Mixture Models