cs.AIOct 11, 2026

Handoff Curves: Initial Selection, Later Handoff, and Supervision Targets in Multi-turn LLM Routing

Authors: Daniel Matos, Lu Liu, Yanfeng Zhang

Organizations: Riemann Lab, Huawei Technologies

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study how initial model selection, later handoff, and training targets affect success and cost in LLM tasks that span multiple turns. We compare several router architectures using handoff curves, which record task success and cost as the handoff step changes. Our dataset contains 34,204 complete executions of three Qwen models across BFCL, MultiWOZ, and ττ-retail, providing offline supervision for routing decisions. In our primary comparison, with Qwen3.5 35B-A3B as the starting model, allowing a later handoff does not establish an improvement in utility over initial selection on BFCL or MultiWOZ, and increases cost on BFCL across a range of cost weights. Changing the supervision targets improves router performance on both our dataset and TwinRouterBench. On our dataset, replacing noisy training success curves with monotone fits raises success by 0.2 to 4.1 percentage points at the same cost across the reported operating points. On TwinRouterBench, we construct targets that describe the model capability needed for the remaining work and difficult steps ahead. Compared with training on the benchmark's per-step labels, these targets raise trajectory pass scores by 7.8 percentage points and cost savings by 6.3 percentage points on the full static bank. A live deployment of the router trained with these targets resolves 79 of 100 instances at a router cost of 0.59 US dollars per instance.

Figures & tables

Figure 1: Supervision targets (illustrative values). (a) Raw success of an upward handoff and its projected label. (b) Benchmark labels rjr_{j} and forward targets Rk=max⁡j≥krjR_{k}=\max_{j\geq k}r_{j} .
RouterDataSupervision targetModel
DirectOursRaw or projectedMLP regressor
FactoredOursProjected (timing), raw (difficulty)Two MLP regressors
Burst-awareTRBBenchmark or forwardBoosted-tree classifiers
SR-KNNBothRaw (ours), benchmark (TRB)kk -NN ( k=5k=5 )
Table 1: Routers, the data they run on, what they learn to predict, and their model, see Section 3.2 for the supervision targets. Ours: our dataset. TRB: TwinRouterBench.
DatasetUpwardDownwardStep unit
BFCL1–61–6User turn
MultiWOZ1–111–11Dialogue turn
τ\tau -retail1, 2, 4, 6, 82, 4, 6, 8, 10Agent action
Table 2: Collected handoff steps by dataset and direction. At step kk , the starting model has completed kk steps and the receiving model takes over. Initial selection at k=0k=0 uses solo executions. Evaluation considers reachable collected steps before the training-derived cutoff described in Appendix C.3 .
   Evaluation tasks
DatasetCellsExecutions8B35B397B
BFCL2,2474,494102102102
MultiWOZ12,50425,008117117117
τ\tau -retail2,3514,702411717
Total17,10234,204   
Table 3: Dataset size. Cells and executions cover all tasks (training, screening, and evaluation). The last three columns give the evaluation tasks for each starting model.
Figure 2: Effect of allowing later handoff for the direct router at the 35B start. Each panel shows later handoff minus initial selection, so values above zero mean higher success, cost, or utility with later handoff. Bars are simultaneous bands that cover all cost weights and metrics at once, at 97.5% per dataset and 95% over both datasets, and filled markers show bands that exclude 0. Success differences are in percentage points and cost differences in 10−310^{-3} $ per task.
Figure 3: Initial selection at the 35B start. Each marker is a router’s operating point at one of the 15 cost weights, and lines join the points in order of increasing λ\lambda . τ\tau -retail has 17 evaluation tasks at this start.
 Projected labelsRaw labelsProjected −- Raw
λ\lambdaSuccessCostSuccessCostΔ\Delta SuccessΔ\Delta UtilityΔ\Delta Success at same cost
 (%)( 10−310^{-3} $)(%)( 10−310^{-3} $)(%) (%)
BFCL
086.612.5986.410.750.2 [−2.6,3.3]0.002 [−0.026,0.033]0.2
0.273.74.9371.34.992.4 [−0.1,5.0]0.027 [0.003,0.051]2.7
0.569.03.7464.03.715.1 [2.3,8.0]0.047 [0.021,0.075]4.1
Table 4: Direct router at the 35B start under initial selection, trained on projected or raw labels, see Section 3.2 , and scored on the same raw outcomes. Success in %, success differences in percentage points, cost in 10−310^{-3} pertask.per task.\DeltaSuccessandSuccess and\DeltaUtilitycomparethetwolabeltypesatthesameUtility compare the two label types at the same\lambda,withunadjusted95, with unadjusted 95% intervals, and bold utility differences have intervals that exclude zero.\Delta$ Success at same cost compares each projected operating point with the Pareto front of the raw-label operating points over all 15 weights. Linear interpolation between front points gives the raw-label success at the projected point’s cost, and beyond the most expensive front point the front keeps its highest success.
PolicyRowPassRowExactTrajPassCostSaveCombined
Full bank
Always strongest100.017.5100.00.054.4
Burst-aware, benchmark labels95.120.190.354.364.9
Burst-aware, forward targets99.326.198.160.671.0
SR-KNN95.776.893.055.480.2
SWE
Table 5: TwinRouterBench static scores (%). Our burst-aware router is trained on the benchmark labels or on our forward targets, see Section 3.2 . Benchmark labels are TwinRouterBench’s tier labels for each step, and forward targets give each step the strongest tier needed from that step to the end of the instance. The full bank has 520 instances with 970 labelled steps, and SWE is its subset of 40 SWE-bench Verified instances. Always strongest, in grey, is a reference. Among the three router rows, the highest value in each column is in bold and the second highest is underlined.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Initial selection (solid, filled) and later handoff (dashed, open) at every starting model over the 15 cost weights. τ\tau -retail has 17 evaluation tasks at the 35B and 397B starts and 41 at the 8B start, see Table 3 .
StartRouterΔ\Delta Success (%)Δ\Delta Cost ( 10−310^{-3} $)Δ\Delta Utility
BFCL
8BDirect2.5 [1.1,4.0]0.03 [−0.01,0.07]0.023 [0.009,0.038]
 Factored1.6 [0.4,3.1]0.02 [−0.01,0.06]0.015 [0.002,0.031]
 SR-KNN1.0 [0.0,2.9]0.08 [−0.01,0.22]0.005 [−0.002,0.018]
35BDirect−4.2 [−8.0,−0.7]0.87 [0.60,1.16]−0.088 [−0.127,−0.052]
 Factored1.4 [−0.6,3.6]0.78 [0.46,1.15]−0.027 [−0.047,−0.009]
Appendix
Table 6: Later handoff minus initial selection for each router at λ=0.2\lambda=0.2 , with success differences in percentage points and cost differences in 10−310^{-3} pertask.Intervalsareunadjusted95per task. Intervals are unadjusted 95%, and rows of zeros mean that the two policies make identical choices.\tau$ -retail has 41 evaluation tasks at the 8B start and 17 at the 35B and 397B starts, see Table 3 .
λ\lambdaΔ\Delta Success (%)Δ\Delta Cost ( 10−310^{-3} $)Δ\Delta Utility
BFCL
0−0.7 [−5.2,3.8]1.22 [0.65,1.79]−0.007 [−0.052,0.038]
0.05−1.7 [−8.4,5.0]1.58 [0.82,2.33]−0.038 [−0.103,0.027]
0.1−2.4 [−8.8,4.1]1.31 [0.70,1.93]−0.058 [−0.122,0.006]
0.2−4.2 [−10.4,2.1]0.87 [0.39,1.34]−0.088 [−0.151,−0.024]
0.35−5.1 [−10.7,0.6]0.75 [0.22,1.28]−0.120 [−0.192,−0.049]
Appendix
Table 7: Later handoff minus initial selection for the direct router at the 35B start, at every cost weight, with success differences in percentage points and cost differences in 10−310^{-3} $ per task. Intervals are the primary simultaneous bands of Figure 2 , covering all cost weights and metrics at 97.5% per dataset and 95% over both datasets.
λ\lambdaΔ\Delta Success (%)Δ\Delta Cost ( 10−310^{-3} $)Δ\Delta Utility
BFCL
00.2 [−2.6,3.3]1.83 [1.34,2.36]0.002 [−0.026,0.033]
0.054.7 [2.3,7.3]1.68 [1.13,2.27]0.025 [0.002,0.049]
0.14.0 [1.4,6.7]0.85 [0.39,1.34]0.017 [−0.006,0.041]
0.22.4 [−0.1,5.0]−0.06 [−0.42,0.27]0.027 [0.003,0.051]
0.354.4 [1.8,7.3]−0.01 [−0.05,0.03]0.045 [0.019,0.072]
Appendix
Table 8: Projected minus raw labels for the direct router at the 35B start under initial selection, at every cost weight, with success differences in percentage points and cost differences in 10−310^{-3} $ per task. Intervals are unadjusted 95%.
More in this topic

Adaptive Model Routing

Adaptive model routing selects models or specialized experts per input to balance prediction quality, computational cost, latency, and resource constraints.

See all 91 papers

Explore similar work

CardsList
Aug 7, 2026

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.

Tao Feng, Fangxu Yu, Haozhen Zhang +9

University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2

LLM EvaluationLLM Routing
AskPDF
Apr 26, 2026

MTRouter: Cost-Aware Multi-Turn LLM Routing with History-Model Joint Embeddings

Multi-turn, long-horizon tasks are increasingly common for large language models (LLMs), but solving them typically requires many sequential model invocations, accumulating substantial inference costs. Here, we study cost-aware multi-turn LLM routing: selecting which model to invoke at each turn from a model pool, given a fixed cost budget. We propose MTRouter, which encodes the interaction history and candidate models into joint history-model embeddings, and learns an outcome estimator from logged trajectories to predict turn-level model utility. Experiments show that MTRouter improves the performance-cost trade-off: on ScienceWorld, it surpasses GPT-5 while reducing total cost by 58.7%; on Humanity's Last Exam (HLE), it achieves competitive accuracy while reducing total cost by 43.4% relative to GPT-5, and these gains even carry over to held-out tasks. Further analyses reveal several mechanisms underlying its effectiveness: relative to prior multi-turn routers, MTRouter makes fewer model switches, is more tolerant to transient errors, and exhibits emergent specialization across models. Code: https://github.com/ZhangYiqun018/MTRouter

Yiqun Zhang, Hao Li, Zihan Wang +6

School of Computer Science and Engineering, Northeastern University Shenyang 110819, China · Shanghai Artificial Intelligence Laboratory

Cost-Aware InferenceLLM Routing
AskPDF
Oct 20, 2026

COMPASS: Routing via Capability-Oriented Model Profiling with Adaptive Search, Selection, and Tool Use

LLM routing aims to select a capable model at low cost, but specialized routers can require updates as models and tasks change. We propose COMPASS, an agentic router in which an LLM follows a routing recipe, gathers performance and cost evidence through tools, and optionally inspects candidate answers before selecting a model. Our evaluations on LLMRouterBench and DeepSWE show that agentic routing can improve accuracy and lower model-call cost relative to specialized and trained routing baselines, without routing-specific training. On LLMRouterBench, COMPASS reaches 69.85% accuracy at $139.57, compared with RLCascadeRouter's 69.14% at $180.22. On DeepSWE under fixed five-fold cross-validation, it reaches 78.1% at $215, compared with 74.1% at $500 for the best single model. COMPASS also supports routing across benchmarks and adapts to an expanded model pool by reading new evaluation records, without changing the router's weights or prompt.

Amin Rezaei, Pouya Parsa

University of Minnesota

LLM RoutingLLM Agent RoutingTool-Augmented Language Model Agents
AskPDF