cs.CLOct 5, 2026

Breaking the Tie: A Cluster-Aware Routing Framework for Large Language Models

Authors: Yao Lu, Zhaiyuan Ji, Yaxin Gao, Zeyu Wang, Zhe Tang, Jiaheng Wei, Zhaowei Zhu, Shanqing Yu, +1 more

Organizations: Institute of Cyberspace Security, Zhejiang University of Technology · Binjiang Institute of Artificial Intelligence, Zhejiang University of Technology · Hong Kong University of Science and Technology (Guangzhou)

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

With the rapid development of artificial intelligence, the emergence of various Large Language Models (LLMs) has created a rich model ecosystem. However, this also brings a key challenge: how to select the optimal model for a specific user query. LLM routing addresses this need by dynamically assigning queries to the most suitable expert in the pool of candidate models. However, existing routing frameworks often simplify this process to a standard classification task; thus, a critical vulnerability is exposed when multiple candidate models correctly answer the same query. We formalize this capability overlap as routing noise, which misleads the router with arbitrarily correct candidate models, ultimately leading to routing collapse (a severe decline in generalization ability on unseen tasks). To address this problem, we propose a novel Cluster-Aware Soft-Labeling Routing (CASLR) framework. CASLR shifts the evaluation paradigm from the success of a single query to macro-domain consensus by replacing traditional one-hot vectors with a masked softmax mechanism. Specifically, for experts who answer incorrectly, we penalize their target probability to zero; for the remaining candidates, we directly compute continuous fine-grained soft labels based on their global clustering utility scores. We then use these refined soft labels to supervise a lightweight router. Specifically, the framework not only demonstrates superior accuracy on multiple benchmarks, but also outperforms Llama-3.3-70B-Instruct by 7.80% in overall average performance. Furthermore, the extremely low routing inference latency of only 1.13s further confirms that CASLR can achieve efficient system scheduling with almost zero additional overhead, while ensuring high response quality.

Figures & tables

Figure 1: Visualization of routing noise and routing collapse.
Figure 2: Overview of the CASLR framework.
TaskMathCodingReasoningAverage
Models\DatasetMATHGSM-SymbolicAIME1983-2025HumanEvalMbppMMLUHellaSwag
Oracle89.40%97.10%78.06%93.94%89.11%93.04%89.36%90.00%
Random70.12%81.11%39.99%70.35%60.96%69.95%58.17%64.38%
Random_Oracle_0.579.79%89.17%59.07%82.33%75.16%81.43%73.76%77.24%
PRknn76.00%91.90%67.35%78.79%68.00%76.36%65.90%74.90%
LinearR79.76%91.30%60.71%90.91%72.00%82.38%69.85%78.13%
Table 1: Performance comparison with state-of-the-art LLM routing methods. We select the first 55 models in Table A1 as experts.
Figure 3: Case studies of model predictions: Ours vs. existing LLM models.
TaskMathCodingReasoningAverage
Models\DatasetMATHGSM-SymbolicAIME1983-2025HumanEvalMbppMMLUHellaSwag
Qwen3-30B-A3B72.76%74.20%79.59%69.70%71.00%85.29%74.75%75.33%
Qwen3-30B-A3B-Instruct-250783.24%76.40%81.63%90.91%71.00%85.18%78.15%80.93%
Mixtral-8x7B-Instruct-v0.123.32%62.80%1.53%54.55%41.00%68.89%63.10%45.03%
Qwen3-Coder-30B-A3B-Instruct82.20%91.10%49.49%90.91%71.00%82.05%71.40%76.88%
Llama-2-13B3.64%26.00%0.51%9.09%22.00%47.48%52.55%23.04%
Table 2: Performance comparison with large-scale LLM models. Experiments are conducted using the 5 best-performing models.
Figure 4: Percentage of responses from different models across various tasks.
TaskMathCodingReasoningAverage
Models\DatasetMathGSM-SymbolicAIME1983-2025HumanEvalMbppMMLUHellaSwag
Adding a high-performance model
Base (1+2+3)77.00%92.10%26.02%69.70%61.00%73.76%65.85%66.49%
Adding DeepSeek-R1-Distill-Llama-8B77.76%91.50%54.08%78.79%54.00%73.90%65.70%70.82% (+4.33%)
Adding DeepSeek-R1-Distill-Qwen-14B78.44%93.10%64.29%93.94%68.00%82.44%72.05%78.89% (+12.40%)
Adding Llama-3.3-70B-Instruct80.60%95.10%55.10%81.82%64.00%86.30%79.25%77.45% (+10.96%)
Table 3: The impact of the base models’ performance on the proposed algorithm.
TaskMathCodingReasoningAverage
Models\DatasetMathGSM-SymbolicAIME1983-2025HumanEvalMbppMMLUHellaSwag
3 experts (1-3)77.00%92.10%26.02%69.70%61.00%73.76%65.85%66.49%
4 experts (1-4)77.76%91.50%54.08%78.79%54.00%73.90%65.70%70.82%
5 experts (1-5)81.44%93.20%66.33%93.94%69.00%82.52%72.10%79.79%
6 experts (1-6)81.52%94.20%67.35%90.91%69.00%82.48%72.50%79.71%
7 experts (1-7)82.80%96.00%65.82%90.91%64.00%86.88%79.25%80.81%
Table 4: The impact of the number of experts on the proposed algorithm.
ModelDeepSeek-R1-Distill-Qwen-14BLlama-3.3-70B-InstructQwen3-30B-A3BQwen3-30B-A3B-Instruct-2507Qwen3-Coder-30B-A3B-InstructRouter
Time Cost3325.10s8771.70s13724.83s9434.59s10150.45s1.13s
Table 5: The computational overhead of routers compared to existing LLMs.
TaskTypeMathCodingReasoningAverage
Models\DatasetMathGSM-SymbolicAIME1983-2025Human-evalMbppMMLUhellaswag
Select the best expertHard Label78.32%92.10%54.08%87.88%46.00%70.62%70.10%71.30%
OursSoft Label81.44%93.20%66.33%93.94%69.00%82.52%72.10%79.79%
Table 6: Performance comparison between the naive hard-labeling and our soft-labeling mechanism across diverse benchmarks.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

IDTaskMathCodingReasoningAverage
Models\DatasetMathGSM-SymbolicHumanEvalMbppMMLUAIME1983-2025HellaSwag
1Qwen2.5-7B-Instruct71.56%87.40%69.70%54.00%73.54%23.47%65.70%63.62%
2Qwen2.5-Coder-7B-Instruct64.40%74.80%75.76%61.00%66.11%14.80%57.95%59.26%
3Qwen2.5-Math-7B-Instruct76.92%89.80%30.30%35.00%52.70%26.02%39.80%50.08%
4DeepSeek-R1-Distill-Llama-8B64.16%71.60%81.82%50.00%71.41%68.37%59.35%66.67%
5DeepSeek-R1-Distill-Qwen-14B71.68%84.90%93.94%68.00%82.66%67.35%72.05%77.23%
Appendix
Table A1: Performance of the base models.
Figure A1: Percentage of responses from different models across various tasks.
Figure A2: Percentage of responses from different models across various tasks.
Figure A3: Percentage of responses from different models across various tasks.
Figure A4: Percentage of responses from different models across various tasks.

Explore similar work

CardsList
Sep 28, 2026cs.AI

SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing

Large language model (LLM) routing aims to select the most suitable model for each incoming query. Most existing routers learn this decision directly from query embeddings, model representations, preference data, or clusters of similar examples. Such approaches can be effective, yet the representation used for routing rarely states what a query actually requires. We introduce SeLMRoute, a routing framework that separates the extraction of candidate-independent semantic evidence from the learning of candidate performance and the application of deployment objectives. A decision model first evaluates a set of interpretable questions about the query, such as its reasoning requirements and use of external knowledge, with each judgment retained as a probability distribution. The resulting probabilistic semantic state is used by a lightweight supervised router to estimate candidate model performance. Routing objectives are applied after performance estimation, which allows the same semantic state to support performance-oriented and cost-aware decisions. On the LLMRouterBench (15 datasets, 20 candidate models, 11,481 queries), SeLMRoute achieves an average accuracy of 72.08%±0.4572.08\% \pm 0.45, while grouped five-fold out-of-fold evaluation reaches 72.64%72.64\%, compared with 69.23%69.23\% for the strongest fixed candidate. The representation achieves the highest mean performance among the evaluated semantic, dense, lexical, and domain-level representations. In a separate 13-model performance-cost setting, SeLMRoute improves performance in all five grouped splits, with a mean PerfGain of 2.66%2.66\%. Our code is available at https://github.com/Indigma-Innovations/SeLMRoute.

Vasilis Perifanis, Nikolaos Pavlidis, Symeon Symeonidis

Indigma Innovations · Democritus University of Thrace · Athena Research Center

Large Language Model Routing
AskPDF
Aug 7, 2026cs.CL

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.

Tao Feng, Fangxu Yu, Haozhen Zhang +9

University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2

Large Language Model RoutingInference CostSentence EncodersInfrastructure
AskPDF
Jul 20, 2026cs.CL

VDAR-Router: Adaptive LLMs Routing via Verbalized Query Difficulty Analysis Retrieval

Large language models are increasingly used in practical systems, making efficient model selection important for reducing deployment cost. LLM routing has emerged as a practical solution for allocating each input query to an appropriate model under a desired cost-performance trade-off. Existing routing methods often estimate model suitability from the surface semantics or embedding similarity of the input query. However, such methods may ignore the underlying difficulty of a query, leading to suboptimal routing decisions. To address the challenge, we propose VDAR-Router, a difficulty-aware retrieval-based routing framework. For each input query, VDAR-Router first generates an explicit difficulty analysis. It then retrieves historical examples with similar difficulty profiles. Based on the retrieved records, it estimates candidate model suitability and selects the model using a reward function that considers both performance and cost. Experiments on three datasets show that VDAR-Router consistently achieves better cost-performance trade-offs than existing baselines. These results demonstrate the effectiveness of difficulty-aware retrieval for training-free LLM routing. Case studies further show that explicit query analysis helps retrieve more relevant examples and supports more reliable routing decisions.

Yu-Chien Tang, Jun-Chen Hung, Wen-Chih Peng +1

National Yang Ming Chiao Tung University · Department of Computer Science Hsinchu, Taiwan

Large Language Model RoutingStrong Large Language ModelTrade-OffQueries
AskPDF