LLM Routing

LLM: Large Language Model

Overview

Large language model routing studies how to assign a query, workflow step, or token to an appropriate model, expert, tool, retrieval path, or amount of computation. Common approaches use query features, confidence or uncertainty, hidden-state signals, and response disagreement to choose a capable low-cost option or escalate when needed, using classifiers, contextual bandits, reinforcement learning, and calibrated cascades. Systems work also schedules heterogeneous models and GPUs around budgets, latency, energy, cache reuse, privacy, and service guarantees, with evaluations spanning reasoning, coding, question answering, retrieval-augmented generation, agents, and production workloads. A central tension is that oracle model choices can show substantial gains, while deployable routers depend on reliable quality estimates and supervision and can falter under distribution shifts, weak model specialization, noisy evaluation, or routing overhead.

Latest papers

CardsList

383 papers30 papers in the last four weeks, down 52% on the four weeks before. 0.2% of all new papers.

  1. TableRoute: Capability-Aware Routing for Scalable Table Question Answering

    Nov 19, 2026Yuxiang Wang, Junhao Gan, Jianzhong QiTable QALLM Routing

  2. What MoE Routing Controls: Causal Interventions on Expert Trajectories

    Nov 17, 2026Mukund Athmaram, Manoj Rao, Liang MaMixture-of-Experts Language ModelsLanguage Model Steering

  3. Call-Efficient ICD-10 Code Prediction via Supervised Candidate Generation and Selective LLM Invocation

    Nov 16, 2026Ilia Kopanichuk, Daniil Kartashev, Yuri Bolkonsky +4Automated Medical CodingLLM Routing

  4. Routing Beyond Task Categories: A Cost-Aware LAB-Bench Study

    Nov 16, 2026Ahmed Tariq, Leo GuinanLLM RoutingCost-Aware Inference

  5. When the Cheapest Feasible Router Fails: Auditing Deployment Commitments in Cost-Aware LLM Routing

    Nov 16, 2026Minxuan Hu, Qishi Zhan, Ziheng Chen +4LLM RoutingCost-Aware Inference

  6. CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs

    Nov 15, 2026Byeongju Kim, Hoonki Lee, Byungjun Kim +4On-Device Language Model InferenceMixture-of-Experts Inference

  7. SpecRelay: Amortize or Avoid the Round Trip in Edge-Cloud Speculative Decoding

    Nov 13, 2026Jai Jain, Chihyo Ahn, Yong Liu +10Speculative DecodingEdge-Cloud Collaborative Inference

  8. Ballad: Bandit-Based LLM Routing for Automated Heuristic Discovery

    Nov 12, 2026Samidha Verma, Ankit Anand, Sayan RanuLLM RoutingAutomated Heuristic Design

  9. Smart Model Routing for Order-of-Magnitude LLM Cost Reduction

    Nov 11, 2026Hesham Elbakoury, Deep Shah, Ankur SharmaLLM RoutingCost-Aware Inference

  10. ShadowBench-MCP: Measuring Shadow Invocations in MCP-Style Tool Routing

    Nov 9, 2026Eyhab Al-Masri, Wenjun YangLLM Agent SecurityLLM Routing

  11. RADE: Risk-Adjusted Deployment Efficiency, A Framework for Routing, Memory, and Escalation in Safe LLM Systems

    Nov 8, 2026Chaymaa Abbas, Kareem Hassani, Mariette AwadLLM SafetySafety-Utility Trade-Off in Language Models

  12. MultiTurnRoute: A Platform for Building and Evaluating Multi-Turn LLM Routers

    Nov 7, 2026Jiaqi Xue, Mengxin Zheng, Heng Huang +1LLM Routing

  13. Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference

    Nov 6, 2026Saul Urso, Emanuele Carlini, Patrizio Dazzi +1LLM RoutingCost-Aware Inference

  14. Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill

    Nov 6, 2026Aristides Milios, Christopher PalLLM Output Length PredictionLLM Routing

  15. RouteAlign: Aligning Task Requirements with Model Profiles for LLM Service Selection

    Nov 5, 2026Jihao Zhao, Ding Chen, Yanfang Chen +4LLM RoutingLanguage Model Performance Prediction

  16. Pathwise-Budgeted LLM Routing: Online Allocation with K-Bit Tail-Cost Learning

    Nov 4, 2026Hanzhang Qin, Sophie H. Yu, Yuewei Zhang +1LLM RoutingCost-Aware Inference

  17. Creating Power Flexibility in Datacenters via Query Routing

    Nov 3, 2026Shivam Patel, Bingqing Chen, Jared Fernandez +2Data Center Energy ManagementEnergy-Aware LLM Inference

  18. When Strong Reasoners Join the Pool: Small Union Premiums and Costly Runaway Generations in LLM Routing

    Nov 3, 2026Teng-Ruei ChenLLM RoutingLanguage Model Reasoning Evaluation

  19. Local or Escalate? Post-Training Expert Routing for Real-Time Duplex Speech Models

    Nov 1, 2026Ruoyu He, Jisen Li, Anthony Ji +5LLM RoutingFull-Duplex Spoken Dialogue Systems

  20. Capacity Planning for Resource-Constrained Clinical LLM Serving

    Oct 31, 2026Yi Tang, Geoffrey Martin, Yiye Zhang +1LLM ServingLLM Routing

  21. DolphinRouting: Cluster-Level Binary Routing for Disaggregated LLM Inference Serving

    Oct 29, 2026Yishuo Yuan, Shaoyu Yang, Chunrong FangDisaggregated LLM ServingLLM Routing

  22. Upcycling Language Models for Multiple Specialized Domains

    Oct 29, 2026Mingzi Cao, Xi Wang, Nikolaos AletrasDomain Adaptation for LLMsDomain Adaptation

  23. Context Is Not Free: Carbon-Aware LLM Routing for Multi-Turn LLM Web Services across Geo-Distributed Data Centers

    Oct 26, 2026Xiaoyang Zhang, Yuejun Yan, Qingang Zhang +1Carbon-Aware ComputingLLM Routing

  24. The Coordination Tax and the Heterogeneity Dividend: Break-Even Conditions for Multi-Agent LLM Pipelines

    Oct 25, 2026Andreas Zettl, Diego GosmarMulti-Agent LLMsLLM Agent Efficiency