LLM Routing

LLM: Large Language Model

Ask about this topic

Overview

Large language model routing studies how to assign a query, workflow step, or token to an appropriate model, expert, tool, retrieval path, or amount of computation. Common approaches use query features, confidence or uncertainty, hidden-state signals, and response disagreement to choose a capable low-cost option or escalate when needed, using classifiers, contextual bandits, reinforcement learning, and calibrated cascades. Systems work also schedules heterogeneous models and GPUs around budgets, latency, energy, cache reuse, privacy, and service guarantees, with evaluations spanning reasoning, coding, question answering, retrieval-augmented generation, agents, and production workloads. A central tension is that oracle model choices can show substantial gains, while deployable routers depend on reliable quality estimates and supervision and can falter under distribution shifts, weak model specialization, noisy evaluation, or routing overhead.

Latest papers

CardsList

355 papers54 papers in the last four weeks, up 35% on the four weeks before. 0.3% of all new papers.

  1. DolphinRouting: Cluster-Level Binary Routing for Disaggregated LLM Inference Serving

    Oct 29, 2026Yishuo Yuan, Shaoyu Yang, Chunrong FangDisaggregated LLM ServingLLM Routing

    Nanjing University, Nanjing, China.

  2. Upcycling Language Models for Multiple Specialized Domains

    Oct 29, 2026Mingzi Cao, Xi Wang, Nikolaos AletrasDomain Adaptation for LLMsDomain Adaptation

    School of Computer Science, University of Sheffield, United Kingdom

  3. Governing Enterprise AI Traffic at Scale: Design and Operation of a Policy-Driven, Multi-Vendor LLM Gateway on Envoy

    Oct 26, 2026Mithun Sasidharan, Srinivasan Manoharan, Rupesh Thakur +3LLM RoutingLLM Serving

    PayPal Inc.

  4. Context Is Not Free: Carbon-Aware LLM Routing for Multi-Turn LLM Web Services across Geo-Distributed Data Centers

    Oct 26, 2026Xiaoyang Zhang, Yuejun Yan, Qingang Zhang +1Carbon-Aware ComputingLLM Routing

    Nanyang Technological University · Alibaba Group

  5. The Coordination Tax and the Heterogeneity Dividend: Break-Even Conditions for Multi-Agent LLM Pipelines

    Oct 25, 2026Andreas Zettl, Diego GosmarMulti-Agent LLMsLLM Agent Efficiency

    Member, VoiceInteroperability.ai Linux Foundation AI & Data Berlin, Germany · Member, VoiceInteroperability.ai Linux Foundation AI & Data Turin, Italy

  6. The Hitchhiker's Guide to System 1 Models: Fast Decision-Making Beyond LLMS

    Oct 24, 2026Haggai RoitmanLanguage Model CalibrationLLM Decision-Making

  7. Predictable performance is suboptimal, and what it means for model routing

    Oct 22, 2026Guanhua Zhang, Moritz HardtLLM RoutingLanguage Model Performance Prediction

    Max Planck Institute for Intelligent Systems, Tübingen · Tübingen AI Center

  8. Route-and-Generate: KL-Regularized Router Improvement for Reasoning Routing in LLMs

    Oct 21, 2026Tien-Phat Nguyen, Truong Nguyen, Thanh-Thien Le +4Adaptive Reasoning in Language ModelsEfficient Language Model Reasoning

    Hanoi University of Science and Technology, Vietnam · Monash University, Australia · New York University, USA +1

  9. COMPASS: Routing via Capability-Oriented Model Profiling with Adaptive Search, Selection, and Tool Use

    Oct 20, 2026Amin Rezaei, Pouya ParsaLLM RoutingLLM Agent Routing

    University of Minnesota

  10. RouterWorld: Scaling Conversational AI with Latency-Aware LLM Routing

    Oct 19, 2026Jinyang Li, Denis Emelin, Simone Tedeschi +6LLM RoutingAI Agent Benchmarks

    The University of Hong Kong · Amazon · Centre for Artificial Intelligence, University College London

  11. Benefit-Targeted Routing and Bounded Revision for LLM-Based Evidence Screening in Manufacturing Supplier Discovery

    Oct 19, 2026Yijiashun Qi, Hanzhe Guo, Yuxuan LiLLM RoutingEvidence Selection

    University of Michigan, Ann Arbor, MI 48109, USA · University of Pennsylvania, Philadelphia, PA 19104, USA

  12. Alliance of Stochastic Routing and Reinforcement Fusion for Multi-Model Reasoning

    Oct 16, 2026Selim Furkan Tekin, Ling LiuMultimodal Model RoutingAdaptive Model Routing

    Georgia Institute of Technology

  13. MARS: Memory-Aware Routing for Session-Level LLM Assistants

    Oct 16, 2026Minghan Wang, Jinhang Zuo, Zhongxiang Dai +2Memory Management for LLM AgentsLLM Routing

    Southern University of Science and Technology · City University of Hong Kong · The Chinese University of Hong Kong, Shenzhen +1

  14. ScoutRoute: Efficient Mixture-of-Experts Routing Using Internal Language Model Signals

    Oct 16, 2026Rajdeep Pandhare, Delzaan BharuchaLLM RoutingMixture-of-Experts Inference

    Department of Artificial Intelligence and Data Science, Rajiv Gandhi Institute of Technology, Mumbai, Maharashtra, India

  15. Who Pays for Efficient Inference? Confidence-Based Small-to-Large LLM Routing Across Tasks and Languages

    Oct 16, 2026Md Sabbir Hossen, Anichur Rahman, Pabon Shaha +3LLM RoutingLLM Inference Acceleration

    Bangladesh University, Bangladesh · Georgia Southern University, USA · University of Southern Mississippi, USA

  16. Response-Informed Value Estimation for Cost-Aware Multi-Turn LLM Routing

    Oct 16, 2026Haotian Yan, Jiaqi Liu, Bingyi Jing +1LLM RoutingCost-Aware Inference

    Southwest University of Finance and Economics · Shenzhen Loop Area Institute · Dalian University of Technology +1

  17. Chance-Constrained Personalized LLM Routing with Online Preference Learning

    Oct 15, 2026Ameya Anjarlekar, Rasoul Etesami, R SrikantLLM RoutingCost-Aware Inference

  18. Disagreement-Guided Weak-to-Strong Routing

    Oct 15, 2026Sima Noorani, Shayan Kiyani, George Pappas +1LLM Routing

    University of Pennsylvania

  19. RouteWeaver: Weaving Mode Selection and Execution into Unified LLM Routing

    Oct 14, 2026Xiaohan Wang, Haozhen Zhang, Qingyuan Liu +2LLM RoutingRL for Language Models

    Nanyang Technological University · Huazhong University of Science and Technology · University of Illinois Urbana-Champaign

  20. Learning When to Trust Medical Knowledge: Regression-Safe Routing for Clinical Reasoning

    Oct 14, 2026Hui Zhang, Yujia Zeng, Yuhao XinMedical ReasoningLLM Routing

    Beijing Institute of Technology · Shanghai Jiao Tong University · Xi’an Jiaotong University

  21. Whose turn is it? Context-guided model routing for stateful agents

    Oct 13, 2026Anikesh Parashar, Avik Dutta, Vashu Chauhan +6LLM RoutingLLM Agent Routing

    Microsoft

  22. Embedding Geometry Shapes LLM Routing: Tracing Failure Modes and Improving Generalization

    Oct 13, 2026Leonardo Torti, Emanuele Rimoldi, Tomer Galanti +2LLM RoutingRepresentation Geometry in Language Models

    Massachusetts Institute of Technology. · Texas A&M University.

  23. Verification Turns Capability Gaps into Routing Decisions: Small Language Models for Clinical Documentation

    Oct 13, 2026Nina Fatehi, Meysam Ghaffari, Animesh Agarwal +2Medical Report GenerationMedical LLMs

    Optum

  24. Rethinking Quality in Large Language Model Routing: Security-Conditioned Model-and-Budget Selection

    Oct 13, 2026Yichi Zhang, Zhenxiang Huang, Yuyang Zhang +1LLM RoutingLLM Security

    The Pennsylvania State University · The University of Hong Kong

  25. Handoff Curves: Initial Selection, Later Handoff, and Supervision Targets in Multi-turn LLM Routing

    Oct 11, 2026Daniel Matos, Lu Liu, Yanfeng ZhangAdaptive Model RoutingLLM Routing

    Riemann Lab, Huawei Technologies

  26. TurnCost: Turns, Not Observations: Cache-Aware Loop Engineering for Coding Agents

    Oct 10, 2026Pramod Kumar, Raj Kumar PandeyLLM Agent Workflow OptimizationCoding Agents

    The Pennsylvania State University, University Park, USA

  27. CopRoute: Dependence-Aware Conformal Routing for Language Models

    Oct 9, 2026Siva Rajesh Kasa, Santhosh Kumar Kasa, Karan GuptaLLM RoutingEfficient Language Model Inference

  28. TokenRouter: Efficient Serving System for Token-Level LLM Routing

    Oct 8, 2026Tianyu Fu, Tengxuan Liu, Ruoxi Wang +4LLM ServingLLM Routing

    Tsinghua University · Carnegie Mellon University

  29. Sensitive-Topic Leakage Through LLM Routing Metadata: Measurement and Mitigation

    Oct 7, 2026Teng-Ruei ChenPrivacy Leakage in Language ModelsLLM Routing

    Krixvon, Taipei 100, Taiwan

  30. Evaluating Trajectory Features for Routing Final-Layer Attention

    Oct 7, 2026Yupeng YaoAttention RoutingLLM Routing