In a multi-turn conversation, the choice of language model at one turn can affect which model should answer the next. The selected model's reply shapes the context for the next query, and its cached prompt prefix can reduce the cost of using that model again. We introduce MultiTurnRoute, a platform for building and evaluating multi-turn LLM routers. A shared interface gives built-in and user-defined routers the current query, dialogue history, and an estimate of each model's reusable prefix length. Our routing algorithm, Chain-of-Route (CoR), uses a learned gate over recent and recurrent dialogue history to predict response quality and estimates cost from each model's reusable prefix. MultiTurnRoute Arena ranks routers on recorded conversations and compares session quality and cost in interactive runs where a simulated user responds to each reply. On a benchmark drawn from WildChat and LMSYS-Chat-1M with per-turn quality annotations for five LLMs, CoR reduces the normalized cost required to match the strongest model's quality by up to 20.3% relative to the strongest routing baseline, while matching or improving peak response quality.
Figures & tables
Figure 1: MultiTurnRoute is a multi-turn LLM router platform. A registered router reads the query and session state to select a model. The reply and token counts update the state for the next turn. The Arena ranks routers by area under the deferral curve (AUDC) in static replay and compares quality and cost in interactive sessions.
Figure 2: Challenges of adapting single-turn LLM routing to multi-turn conversations regarding dialogue state. (a) Routing using only the current query ignores conversational context. (b) Routing using the full context can let dominant prior content overshadow the current query’s intent ( context overshadowing ). (c) Routing using selectively retrieved context misses implicit dependencies when users refer back without repeating keywords.
Figure 3: Ignoring serving state produces misleading cost rankings.
Figure 4: Overview of Chain-of-Route (CoR). At each turn, the router receives the current query Qt , the dialogue state Ht , and the serving state Kt . DSP conditions on Ht to estimate quality Q^ , while SST conditions on Kt to estimate cost C^ . The model maximizing (1−λ)⋅Q^−λ⋅C^ is selected, and both states are updated before propagating to the next turn.
Figure 5: Overview of the Dialogue State Propagator (DSP). (a) Context-dependency Gate: query Q and dialogue state H produce a gate αt , which scales H before the quality estimator fQ . (b) Local History Update: a FIFO buffer (e.g., of size W=2 ) evicts the oldest pair (Q1,A1) upon each new arrival (Q3,A3) . (c) Global History Update: a GRU updates the global state at each turn (e.g., H3→H4 ), accumulating conversation-level trends.
Figure 6: Overview of SST. Taking turn 3 as an example, Qwen caches (Q1,A1) while GPT caches (Q1,A1,Q2,A2) . (Left) Each model’s serving state and cached token count Lcache . (Middle) Static cost estimation yields a lower estimate for Qwen; cache dividend estimation computes each model’s DIVcache=dcache×Lcache . (Right) After subtracting DIVcache , GPT’s larger cache advantage reverses the ranking, making it the truly more cost-efficient option.
Method
WildChat
LMSYS-Chat-1M
AUDC ↑
QNC ↓
Peak ↑
AUDC ↑
QNC ↓
Peak ↑
Random
0.49
0.97
0.87
0.55
0.93
0.78
RouteLLM [ 1 ]
0.56
0.98
0.94
0.62
0.93
0.83
CARROT [ 28 ]
0.59
0.95
0.94
0.65
0.91
0.83
CSCR [ 27 ]
0.68
0.85
0.94
0.65
0.87
0.83
GMTRouter [ 31 ]
0.65
0.87
0.94
0.62
0.89
0.84
Table 1: Quality-cost tradeoff on WildChat and LMSYS-Chat-1M.
Figure 7: (a) Deferral curves on WildChat as the cost-sensitivity parameter λ is swept; “Oracle” is the per-turn Myopic Oracle defined in Section 4 . (b) Quality prediction MSE for four history strategies, broken down by query type. (c) Cost estimation MAPE as a function of conversation turn for cache-unaware estimation vs. SST.
Query type
mean αt
Context-dependent
0.78
Context-independent
0.28
Table 2: Context-dependency gate αt behavior.
Turn bucket
#conv.
AUDC ↑
QNC ↓
Peak ↑
CSCR
CoR
CSCR
CoR
CSCR
CoR
2 turns
∼ 1,680
0.69
0.71
0.86
0.65
0.96
0.94
3–4 turns
∼ 1,240
0.67
0.74
0.85
0.58
0.94
0.95
5–7 turns
∼ 0 700
0.65
0.75
0.88
0.53
0.92
0.96
8+ turns
∼ 0 380
0.63
0.78
0.92
0.45
0.90
0.97
Δ (8+ vs 2)
—
−0.06
+0.07
+0.06
−0.20
−0.06
+0.03
Table 3: Quality-cost performance stratified by conversation length on WildChat. Buckets group test conversations by total turn count. CoR’s gain over the strongest single-turn baseline (CSCR) widens as conversations grow longer, consistent with SST exploiting more accumulated cache.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Total params
cprefill
ccached
cgen
Qwen3-0.6B
0.6B
$0.11
$0.022
$0.42
Qwen3-8B
8B
$0.18
$0.036
$0.70
Qwen3-14B
14B
$0.35
$0.070
$1.40
Qwen3-32B
32B
$0.16
$0.032
$0.64
Qwen3-Next-80B-A3B-Instruct
80B
$0.15
$0.030
$1.20
Appendix
Table 4: Per-token prices for the five Qwen3 models in our pool, in USD per 1M tokens. Full-prefill ( cprefill ) and generation ( cgen ) prices are taken from publicly listed OpenRouter pricing (snapshot retrieved during paper preparation); the cached-input price is set to ccached=0.2cprefill following Alibaba Cloud Model Studio’s context-cache convention. Note that posted prices are non-monotone in model size (e.g., Qwen3-32B at 0.16/MischeaperthanQwen3−14Bat0.35/M) because providers set prices by tier rather than by size; we use posted prices as-is rather than imputing a monotone schedule.
Variant
AUDC ↑
QNC ↓
(a) Local-history window size W (DSP)
W=1
0.72
0.65
W=2 (default)
0.74
0.59
W=3
0.74
0.61
W=5
0.73
0.64
(b) Global state architecture (DSP)
Appendix
Table 5: Sensitivity of CoR to design choices on WildChat. Default configuration in italics .
K
Pool composition
CoR (Ours)
CSCR
AUDC ↑
QNC ↓
AUDC ↑
QNC ↓
2
Qwen3-0.6B + Qwen3-Next-80B-A3B-Instruct
0.78
0.49
0.74
0.74
3
+ Qwen3-32B
0.77
0.53
0.71
0.79
4
+ Qwen3-14B
0.75
0.56
0.70
0.81
5
+ Qwen3-8B (default)
0.74
0.59
0.68
0.85
6
+ Qwen3-4B
0.73
0.61
0.66
0.87
Appendix
Table 6: Robustness to candidate-pool composition on WildChat. Top: pool size K varied from 2 to 6 over Qwen3 models. Bottom: cross-family pool at K=5 that replaces three Qwen3 models with Llama-3.1 and Mistral to test family generalization.
pevict
CoR (Ours)
Bao et al.
AUDC ↑
QNC ↓
Peak ↑
AUDC ↑
QNC ↓
Peak ↑
0% (default, main text)
0.74
0.59
0.95
0.70
0.74
0.95
10%
0.72
0.63
0.95
0.69
0.76
0.95
30%
0.69
0.71
0.95
0.67
0.80
0.95
Appendix
Table 7: Sensitivity of CoR to per-turn cache eviction. The main experiments assume that cache from the current conversation is retained across consecutive turns of the same conversation, which is the typical behavior under prefix-caching engines (vLLM, SGLang) when serving traffic permits cache retention. In this experiment, SST observed the surviving prefix length of each candidate before CoR selected a model. We vary the eviction probability pevict∈{0%,10%,30%} on WildChat. CoR retains its lead over the strongest baseline (Bao et al.) at every eviction rate.
Training judge
Test judge
Method
AUDC ↑
QNC ↓
Peak ↑
DeepSeek-V3.1
DeepSeek-V3.1 (Table 1 )
CSCR
0.68
0.85
0.94
Bao et al.
0.70
0.74
0.95
CoR
0.74
0.59
0.95
DeepSeek-V4-Pro
Claude Opus 4.8
CSCR
0.664
0.873
0.836
Bao et al.
0.693
0.761
0.875
CoR
0.719
0.642
0.902
Appendix
Table 8: Cross-judge evaluation on WildChat. All methods are re-trained on labels from the training judge and evaluated with the test judge.
Variant
Turns 1–2 ↓
Turns 3–4 ↓
Turns 5–7 ↓
Turns 8+ ↓
Δ (8+ vs. 1–2) ↓
Query-only
0.154
0.169
0.181
0.193
+0.039
Local-only
0.115
0.106
0.112
0.123
+0.008
Global-only
0.128
0.119
0.118
0.127
−0.001
DSP (Ours)
0.094
0.081
0.079
0.088
−0.006
Appendix
Table 9: Quality-prediction MSE by conversation depth on WildChat.
Test length
Quality MSE ↓
BERT + SST
DSP + SST
BERT
DSP
AUDC ↑
QNC ↓
AUDC ↑
QNC ↓
2–4 turns (in-distribution)
0.086
0.092
0.735
0.605
0.730
0.613
5–7 turns
0.121
0.096
0.701
0.668
0.737
0.576
8+ turns
0.183
0.098
0.653
0.739
0.742
0.552
Appendix
Table 10: Dialogue-length shift: both quality predictors are trained on 2–4-turn WildChat conversations only.
Train → Test
BERT + SST
DSP + SST
AUDC ↑
Transfer drop
AUDC ↑
Transfer drop
WildChat (writing) → LMSYS (coding)
0.611
−0.073
0.680
−0.020
LMSYS (coding) → WildChat (writing)
0.648
−0.075
0.710
−0.030
Appendix
Table 11: Topic-distribution transfer without target-corpus training. Drops are relative to target-trained models (BERT + SST / DSP + SST: 0.684 / 0.700 on LMSYS-Chat-1M; 0.723 / 0.740 on WildChat).
Method
AUDC ↑
QNC ↓
Peak ↑
Billed cost (/1Kturns)\downarrow$
Cache-hit rate ↑
CSCR
0.664
0.869
0.887
1.646
11.0%
Bao et al.
0.699
0.758
0.912
1.422
19.4%
CoR
0.734
0.616
0.941
1.142
31.8%
Appendix
Table 12: End-to-end billing on WildChat under finite commercial cache policies (heterogeneous Claude / Gemini / Qwen pool via OpenRouter).
λ / regime
Method
0.6B
4B
8B
14B
32B
80B-A3B
0.1 / quality-focused
CoR
3.4
5.8
8.1
11.5
23.9
47.3
Bao et al.
2.9
4.8
7.4
10.6
21.7
52.6
0.5 / balanced
CoR
12.6
17.4
13.8
9.1
30.4
16.7
Bao et al.
21.8
11.3
10.1
6.8
24.7
25.3
0.9 / cost-focused
CoR
56.8
26.1
4.7
1.3
9.4
1.7
Bao et al.
75.9
12.7
3.1
1.0
5.2
2.1
Appendix
Table 13: Routing shares (%) in the six-model Qwen3 pool on WildChat.
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.
Department of Big Data Management and Application, Shanghai Jiao Tong University, Shanghai, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Baidu Inc., Beijing, China +3