SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing

Vasilis Perifanis; Nikolaos Pavlidis; Symeon Symeonidis

cs.AISep 28, 2026

SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing

Authors: Vasilis Perifanis, Nikolaos Pavlidis, Symeon Symeonidis

Organizations: Indigma Innovations · Democritus University of Thrace · Athena Research Center

Abstract

Large language model (LLM) routing aims to select the most suitable model for each incoming query. Most existing routers learn this decision directly from query embeddings, model representations, preference data, or clusters of similar examples. Such approaches can be effective, yet the representation used for routing rarely states what a query actually requires. We introduce SeLMRoute, a routing framework that separates the extraction of candidate-independent semantic evidence from the learning of candidate performance and the application of deployment objectives. A decision model first evaluates a set of interpretable questions about the query, such as its reasoning requirements and use of external knowledge, with each judgment retained as a probability distribution. The resulting probabilistic semantic state is used by a lightweight supervised router to estimate candidate model performance. Routing objectives are applied after performance estimation, which allows the same semantic state to support performance-oriented and cost-aware decisions. On the LLMRouterBench (15 datasets, 20 candidate models, 11,481 queries), SeLMRoute achieves an average accuracy of $72.08\% \pm 0.45$ , while grouped five-fold out-of-fold evaluation reaches $72.64\%$ , compared with $69.23\%$ for the strongest fixed candidate. The representation achieves the highest mean performance among the evaluated semantic, dense, lexical, and domain-level representations. In a separate 13-model performance-cost setting, SeLMRoute improves performance in all five grouped splits, with a mean PerfGain of $2.66\%$ . Our code is available at https://github.com/Indigma-Innovations/SeLMRoute.

Figures & tables

Figure 1: Overview of SeLMRoute. A query is first transformed into an interpretable probabilistic semantic state by a typed decision model. A learned performance model maps the semantic state to predicted candidate model performance. The routing objective then selects a model according to user-defined preferences such as predicted quality or a quality-cost trade-off.

Probe	Type	Semantic question
math_reasoning	Noul	Whether correctness requires mathematical calculation, symbolic manipulation, or quantitative reasoning.
code_reasoning	Noul	Whether correctness requires writing, modifying, debugging, or reasoning about executable code.
formal_logic	Noul	Whether the task materially requires formal, combinatorial, rule-based, or constraint-satisfaction reasoning.
factual_recall	Noul	Whether the request can be solved primarily through factual knowledge with little derivation.
social_affective	Noul	Whether correctness depends on emotion, intention, interpersonal meaning, or social context.
tool_interaction	Noul	Whether completing the task requires interaction with an external tool, API, software environment, or simulator.

Table 1: Fine-grained semantic probes used by SeLMRoute. Noul probes return a probability for a binary judgment. Score probes return probability mass over four ordered levels.

Dataset	Queries
AIME	60
BBH	1,080
EmoryNLP	697
FinQA	1,147
GPQA	198
HumanEval	164

Table 2: Performance-oriented benchmark used for the main SeLMRoute evaluation.

Method	AvgAcc	Gain@R (%)	Gain@B (%)	Gap@O (%)
SeLMRoute ProbabilityMass	$\mathbf{72.08\pm 0.45}$	$\mathbf{56.08\pm 1.62}$	$\mathbf{5.72\pm 0.67}$	$\mathbf{22.00\pm 0.67}$
SeLMRoute Full	$71.30\pm 0.18$	$54.11\pm 1.40$	$4.58\pm 0.52$	$22.84\pm 0.44$
SeLMRoute Hard	$71.20\pm 0.59$	$54.61\pm 2.28$	$4.41\pm 1.11$	$22.94\pm 0.87$
GTE-Qwen2 + CatBoost	$71.16\pm 0.74$	$53.32\pm 2.71$	$4.34\pm 0.93$	$23.01\pm 0.90$
DomainOnly	$71.08\pm 1.03$	$53.45\pm 2.19$	$4.45\pm 0.82$	$23.16\pm 1.13$
TF-IDF	$70.87\pm 0.78$	$53.25\pm 1.18$	$3.82\pm 1.05$	$23.29\pm 1.25$

Table 3: Grouped performance-oriented evaluation. Results are mean

\pm

standard deviation over five 70/30 grouped splits.

Router	AvgAcc
RouterDC	61.33
GraphRouter	70.29
EmbedLLM	71.24
MODEL-SAT	71.88
Avengers	71.94
SeLMRoute ProbabilityMass	$\mathbf{72.08\pm 0.45}$

Table 4: AvgAcc compared with published LLMRouterBench performance-oriented results. Published methods use the benchmark protocol reported in [ 1 ] ; SeLMRoute uses the duplicate-query-safe grouped protocol.

Removed probe	Drop (pp)
Constraint density	+0.604
Context integration	+0.511
Current information	+0.487
Tool interaction	+0.314
Formal logic	+0.212
Factual recall	+0.154

Table 5: Leave-one-probe-out results under grouped evaluation. Positive drop indicates lower AvgAcc after removing the probe.

Seed	GPT-5 AvgAcc	SeLMRoute AvgAcc	PerfGain	GPT-5 Cost	$\theta^{*}$ Cost	Strict CostSave
42	65.63	67.85	+3.37%	$125.29	$133.30	N/A
3407	65.91	68.93	+4.58%	$127.96	$128.09	-0.10%
0	64.38	66.92	+3.94%	$122.68	$112.83	N/A
1	64.92	65.48	+0.87%	$127.72	$108.27	N/A
2	66.78	67.12	+0.52%	$122.23	$124.16	-1.58%
Mean PerfGain			$\mathbf{+2.66\pm 1.85\%}$

Table 6: Grouped performance-cost evaluation. Monetary values refer to candidate model inference costs on the held-out test partition. Strict CostSave is reported only when the validation-selected

\theta^{\dagger}

also satisfies the test accuracy requirement. N/A indicates that no configuration qualified under the strict protocol.

Config	Mean input	Total input	$/1K queries	Total $
PM-16	1,606.7	18.45M	0.0675	0.775
Lite-12	1,062.7	12.20M	0.0446	0.512

Table 7: Estimated semantic extraction cost on the 11,481-query performance-oriented benchmark. PM-16 uses all sixteen semantic probes and the 40-feature ProbabilityMass representation. Candidate model inference costs are excluded.

Explore similar work

Aug 7, 2026cs.CL

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.

Tao Feng, Fangxu Yu, Haozhen Zhang +9

University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2

Large Language Model Routing Inference Cost Sentence Encoders Infrastructure

Ask PDF

Sep 29, 2026cs.AI

Pretrain Once, Route Anywhere: Towards a Foundation Model for LLM Routing

Large language model (LLM) routing aims to assign each query to the most suitable model from a heterogeneous candidate pool, improving the quality--efficiency trade-off of LLM inference. Existing routers are typically learned through local fitting: a router is optimized for a particular query workload and candidate pool, and often requires additional supervision or retraining as the routing environment changes. We ask whether LLM routing can instead be approached from a foundation-model perspective, learning a reusable routing capability that generalizes across tasks, candidate models, and deployment conditions. To this end, we introduce RouteFM, which learns to characterize anonymous candidate models from behavioral context and infer their target-specific capabilities, rather than binding routing decisions to fixed model identities or a single environment. Through episodic pretraining across heterogeneous routing environments, this capability can be reused by a frozen router and adapted to new environments through context alone. Experiments demonstrate transfer across changes in domains, modalities, candidate pools, and context budgets, with the largest gains when behavioral evidence is limited. On MMR-Bench, which is excluded from pretraining, RouteFM outperforms the strongest baseline by 2.23 quality points with only eight observations per candidate. These results support moving LLM routing from repeated local fitting toward a pretrain once, route anywhere paradigm. Our code is publicly available at https://github.com/LAMDA-Model-Reuse/RouteFM.

Guannan Lai, Han-Jia Ye

School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University

Large Language Model Routing LLM Inference Optimization Pretraining Foundation Model

Ask PDF

Jul 20, 2026cs.CL

VDAR-Router: Adaptive LLMs Routing via Verbalized Query Difficulty Analysis Retrieval

Large language models are increasingly used in practical systems, making efficient model selection important for reducing deployment cost. LLM routing has emerged as a practical solution for allocating each input query to an appropriate model under a desired cost-performance trade-off. Existing routing methods often estimate model suitability from the surface semantics or embedding similarity of the input query. However, such methods may ignore the underlying difficulty of a query, leading to suboptimal routing decisions. To address the challenge, we propose VDAR-Router, a difficulty-aware retrieval-based routing framework. For each input query, VDAR-Router first generates an explicit difficulty analysis. It then retrieves historical examples with similar difficulty profiles. Based on the retrieved records, it estimates candidate model suitability and selects the model using a reward function that considers both performance and cost. Experiments on three datasets show that VDAR-Router consistently achieves better cost-performance trade-offs than existing baselines. These results demonstrate the effectiveness of difficulty-aware retrieval for training-free LLM routing. Case studies further show that explicit query analysis helps retrieve more relevant examples and supports more reliable routing decisions.

Yu-Chien Tang, Jun-Chen Hung, Wen-Chih Peng +1

National Yang Ming Chiao Tung University · Department of Computer Science Hsinchu, Taiwan

Large Language Model Routing Strong Large Language Model Trade-Off Queries

Ask PDF