Response-Informed Value Estimation for Cost-Aware Multi-Turn LLM Routing

Haotian Yan; Jiaqi Liu; Bingyi Jing; Tianjun Yao

cs.LGOct 16, 2026

Response-Informed Value Estimation for Cost-Aware Multi-Turn LLM Routing

Authors: Haotian Yan, Jiaqi Liu, Bingyi Jing, Tianjun Yao

Organizations: Southwest University of Finance and Economics · Shenzhen Loop Area Institute · Dalian University of Technology · The Chinese University of Hong Kong, Shenzhen

Abstract

Multi-turn LLM routing selects a model at every turn before its response is available, yet task success is revealed only when the interaction ends, leaving each intermediate choice without a label of its own. Logged trajectories, however, retain the response each call produced, which exposes what each turn required and how well the chosen model served it, evidence that outcomes alone hide. We introduce RIVET (Response-Informed Value Estimation for Turn-Level Routing), which exploits this post-call evidence during training while keeping routing strictly pre-call at deployment. RIVET jointly trains two value estimators over a shared state-model representation: a deployable Routing Value Estimator that scores candidate models from pre-call information, and a training-only Response-Informed Value Estimator that additionally observes the realized response. Conditioning on the response turns response-induced return variation from residual noise into an observed covariate, and we show that this reduces the leading asymptotic estimation error of the shared representation. On the MINT benchmark, RIVET solves more held-out tasks than the strongest multi-turn baseline while using 74.7% less total call cost.

Figures & tables

Figure 1: Intermediate responses reveal what outcomes hide. Three candidate models on a four-turn recommendation task. Each turn demands a different ability (bottom labels), and the response exposes whether the model met it: check and cross mark acceptable and unacceptable responses, and the remaining icons mark responses that are vague, over-deliberated, or needlessly verbose. The green path is the cost-efficient route and switches models across turns. Illustrative; each logged turn observes only the selected model’s response.

Figure 2: RIVET separates pre-call deployment from post-call training. Both estimators read the same trainable state–model representation $z_{\theta}(s_{t},m)$ . The Routing Value Estimator (RVE, deployed) scores every candidate model from pre-call information and makes the routing decision; the Response-Informed Value Estimator (RIVE, training only) additionally conditions on the executed response and shapes the shared representation through its own regression loss. Both estimators regress to the same scalar return target; deployment uses RVE alone.

Panel of Table 1 — Table 1: In-distribution and OOD performance. MINT Test and MMLU-Pro report accuracy; MuSiQue and DROP report exact match. Cost is total evaluation API cost relative to DeepSeek V4-Pro (1.000) on the same dataset; annotations give score changes in points and relative cost changes from it. Bold marks the best router per column.

Configuration	RIVE	Response input	CQL	ACC $\uparrow$	Cost $\downarrow$
Ours	$\checkmark$	$\checkmark$	$\checkmark$	87.94%	1.000
- w/o CQL	$\checkmark$	$\checkmark$	$\times$	85.82% $\downarrow$ 2.12	2.163 $\uparrow$ 116.4%
- w/o RIVE	$\times$	$\times$	$\checkmark$	80.14% $\downarrow$ 7.80	1.172 $\uparrow$ 17.2%
- w/o response	$\checkmark$	$\times$	$\checkmark$	80.14% $\downarrow$ 7.80	2.454 $\uparrow$ 145.4%

Table 2: Component ablations on MINT Test-141. All configurations deploy RVE alone; cost is relative to Ours.

Policy	ACC $\uparrow$	Cost $\downarrow$
Original	84.75%	1.000
Stay	72.88%	0.825
Original	82.61%	1.000
Switch	84.78%	1.429

Table 3: Routing interventions on MINT evaluate retaining the first-turn model in episodes where RIVET originally switches and introducing a switch in episodes where it originally uses one model. Original: learned RIVET policy. Stay: retain the first-turn model throughout the episode. Switch: select the second-highest RVE-scoring model at turn two, then resume greedy routing. Cost is normalized to Original separately for each intervention.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	Composition	Sample / Source size
MINT Test	Multi-task reasoning	GSM8K	21 / 48
		HotpotQA	19 / 43
		MATH	45 / 100
		MMLU	34 / 76
		TheoremQA	22 / 49
MuSiQue	Multi-hop QA	2-hop	60 / 1,252

Appendix

Table 4: Dataset domains and sampling distributions. The final column gives the sampled count and source-pool size within each composition.

Panel of Table 5 — Table 5: Routing stability under candidate-model unavailability on MINT Test-141. Each model pool removes the indicated model and uses RIVET Online as the reference. Colored annotations show accuracy changes from RIVET Online in percentage points and relative cost changes from RIVET Online, whose cost is set to 1.000. Bold values denote the best router result in each model pool.

Category	Matched error signatures
Non-executable action	UnsafeCodeError; Invalid action; Illegal action; Unknown action; Action not allowed
Formatting error	SyntaxError; JSONDecodeError; Parse Error; FormatError; Malformed action
Non-network tool failure	TypeError; NameError; AttributeError; local-tool TimeoutError messages

Appendix

Table 6: Error-event categories used in the post-error routing analysis.

Method	Error-event rate	Switch rate	Switch cost $\downarrow$	Next-turn recovery
RIVET Online	3.96%	26.3%	0.761	93.3%
MTRouter	8.48%	4.2%	1.000	100.0%

Appendix

Table 7: Post-error routing behavior pooled across MINT Test, MuSiQue, DROP, and MMLU-Pro canonical traces. The MTRouter switch cost is 1.000.

Stage	Classification rule
Analyze (outer ring)	Planning; task interpretation; task decomposition; blank pre-solution output (no substantive computation or result).
Solve (middle ring)	Tool calls; code execution; tool feedback; calculations; evidence; intermediate conclusions; verification.
Submit (inner ring)	Solution action; submitted solution field; forced final answer; solution tag.

Appendix

Table 8: Turn-stage definitions used in the stage-wise routing analysis.

Figure 5: Stage-wise model-selection preferences for seven routers on MINT Test-141. Each panel shows the distribution of selected models within Analyze (outer ring), Solve (middle ring), and Submit (inner ring) turns; each ring is normalized independently to 100%. Segment colors denote candidate models.

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 351 papers

MTRouter: Cost-Aware Multi-Turn LLM Routing with History-Model Joint Embeddings
Apr 26, 2026Yiqun Zhang, Hao Li, Zihan Wang +6Cost-Aware Inference LLM Routing
School of Computer Science and Engineering, Northeastern University Shenyang 110819, China · Shanghai Artificial Intelligence Laboratory
Handoff Curves: Initial Selection, Later Handoff, and Supervision Targets in Multi-turn LLM Routing
Oct 11, 2026Daniel Matos, Lu Liu, Yanfeng ZhangAdaptive Model Routing LLM Routing
Riemann Lab, Huawei Technologies
LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
Aug 7, 2026Tao Feng, Fangxu Yu, Haozhen Zhang +9LLM Evaluation LLM Routing
University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2

Apr 26, 2026J/K move · Enter open · S save

MTRouter: Cost-Aware Multi-Turn LLM Routing with History-Model Joint Embeddings

Yiqun Zhang, Hao Li, Zihan Wang, Shi Feng+5

School of Computer Science and Engineering, Northeastern University Shenyang 110819, China · Shanghai Artificial Intelligence Laboratory

Cost-Aware Inference LLM Routing

Open paper AskPDF arXiv

Multi-turn, long-horizon tasks are increasingly common for large language models (LLMs), but solving them typically requires many sequential model invocations, accumulating substantial inference costs. Here, we study cost-aware multi-turn LLM routing: selecting which model to invoke at each turn from a model pool, given a fixed cost budget. We propose MTRouter, which encodes the interaction history and candidate models into joint history-model embeddings, and learns an outcome estimator from logged trajectories to predict turn-level model utility. Experiments show that MTRouter improves the performance-cost trade-off: on ScienceWorld, it surpasses GPT-5 while reducing total cost by 58.7%; on Humanity's Last Exam (HLE), it achieves competitive accuracy while reducing total cost by 43.4% relative to GPT-5, and these gains even carry over to held-out tasks. Further analyses reveal several mechanisms underlying its effectiveness: relative to prior multi-turn routers, MTRouter makes fewer model switches, is more tolerant to transient errors, and exhibits emergent specialization across models. Code: https://github.com/ZhangYiqun018/MTRouter

Abstract

LLM Routing

Explore similar work

MTRouter: Cost-Aware Multi-Turn LLM Routing with History-Model Joint Embeddings

Handoff Curves: Initial Selection, Later Handoff, and Supervision Targets in Multi-turn LLM Routing

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers