LLM Routing

LLM: Large Language Model

Large language model routing studies how to assign a query, workflow step, or token to an appropriate model, expert, tool, retrieval path, or amount of computation. Common approaches use query features, confidence or uncertainty, hidden-state signals, and response disagreement to choose a capable low-cost option or escalate when needed, using classifiers, contextual bandits, reinforcement learning, and calibrated cascades. Systems work also schedules heterogeneous models and GPUs around budgets, latency, energy, cache reuse, privacy, and service guarantees, with evaluations spanning reasoning, coding, question answering, retrieval-augmented generation, agents, and production workloads. A central tension is that oracle model choices can show substantial gains, while deployable routers depend on reliable quality estimates and supervision and can falter under distribution shifts, weak model specialization, noisy evaluation, or routing overhead.

Latest papers

355 papers54 papers in the last four weeks, up 35% on the four weeks before. 0.3% of all new papers.

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge
May 11, 2026Wenbo Zhang, Lijinghua Zhang, Liner Xiang +1LLM-as-a-Judge Cost-Aware Inference
Department of Statistics, University of California, Irvine, USA.
Route Before Retrieve: Activating Latent Routing Abilities of LLMs for RAG vs. Long-Context Selection
May 11, 2026Yiwen Chen, Kuan Li, Fuzhen Zhuang +6LLM Prompting Long-Context Modeling
Beihang University · HKUST · Alibaba Group +1
UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing
May 11, 2026Varun KotteCost-Aware Inference Language Model Calibration
Independent Researcher.
A Communication-Theoretic Framework for LLM Agents: Cost-Aware Adaptive Reliability
May 9, 2026Hamed Omidvar, Vahideh AkhlaghiLLM Inference Efficiency AI Agent Reliability
INTELLERCE LLC San Jose, CA, USA
Iterative Critique-and-Routing Controller for Multi-Agent Systems with Heterogeneous LLMs
May 9, 2026Wenzhi Fang, Liangqi Yuan, Guangchen Lan +2Multi-Agent LLM Systems Multi-Agent Coordination
Purdue University · Yonsei University
Large Language Models over Networks: Collaborative Intelligence under Resource Constraints
May 9, 2026Liangqi Yuan, Wenzhi Fang, Shiqiang Wang +2LLM Inference Efficiency On-Device Language Model Inference
Purdue University, IN, USA. · University of Exeter, UK. · Princeton University, USA.
Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts
May 8, 2026Saloni Garg, Amit SagtaniLLM Evaluation LLM-as-a-Judge
Independent Researcher San Francisco, California, USA · San Francisco State University San Francisco, California, USA
When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
May 8, 2026Youngsik Yoon, Siwei Wang, Wei Chen +1Mixture-of-Experts Language Models LLM Routing
Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea
Switchcraft: AI Model Router for Agentic Tool Calling
May 8, 2026Sharad Agarwal, Pooria Namyar, Alec Wolman +3Tool-Augmented Language Model Agents Cost-Aware Inference
Microsoft Research · Microsoft · Stanford
ModelLens: Finding the Best for Your Task from Myriads of Models
May 8, 2026Rui Cai, Weijie Jacky Mo, Xiaofei Wen +5Model Selection LLM Routing
University of California, Davis · Arizona State University · Morgan Stanley
Towards Distributed Inference of LLMs on a P2P Network
May 7, 2026Shabari S Nair, Krishanu SainiLLM Inference KV Caching
Department of Computer Science, The University of Texas at Austin.
Is Escalation Worth It? On the Depth of LLM Cascades
May 7, 2026Dylan BouchardCost-Aware Inference LLM Routing
A Regime Theory of Controller Class Selection for LLM Action Decisions
May 7, 2026Zhaoyang Jiang, Zhizhong Fu, Yunsoo Kim +4Selective Prediction Language Model-Based Control
University of Glasgow · UESTC, Chengdu, China · University College London
Policy-Guided Stepwise Model Routing for Cost-Effective Reasoning
May 7, 2026Wenwen Si, Insup Lee, Osbert BastaniCost-Aware Inference Efficient Language Model Reasoning
Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, USA
Adaptive Consensus in LLM Ensembles via Sequential Evidence Accumulation: Automatic Budget Identification and Calibrated Commit Signals
May 5, 2026Roberto E. MedinaLLM Inference Efficiency Multi-Agent Consensus
Independent Researcher
Complexity Horizons of Compressed Models in Analog Circuit Analysis
May 4, 2026Pacome Simon MbonimpaLLM Evaluation LLM Compression
Carnegie Mellon University-Africa
Zero-Shot Confidence Estimation for Small LLMs: When Supervised Baselines Aren't Worth Training
May 4, 2026Luong N. NguyenConfidence Estimation in Language Models Language Model Self-Assessment
Model Routing as a Trust Problem: Route Receipts for Adaptive AI Systems
May 3, 2026Vincent SchmalbachAI Accountability Adaptive Model Routing
Independent Researcher
Rethinking LLM Ensembling from the Perspective of Mixture Models
May 1, 2026Jiale Fu, Yuchu Jiang, Peijun Wu +3LLM Inference Acceleration LLM Routing
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education · Southeast University · Qiyuan Tech +2
RouteProfile: Graph-Based Profiling for Cold-Start LLM Routing
Apr 30, 2026Jingjun Xu, Hongji Pu, Tao Feng +3Adaptive Model Routing LLM Routing
University of Illinois Urbana-Champaign · Nanyang Technological University
Optimized Deferral for Imbalanced Settings
Apr 30, 2026Corinna Cortes, Anqi Mao, Mehryar Mohri +1Cost-Sensitive Learning Learning to Defer
Google Research, New York, NY · Courant Institute of Mathematical Sciences, New York, NY
When to Vote, When to Rewrite: Disagreement-Guided Strategy Routing for Test-Time Scaling
Apr 29, 2026Zhimin Lin, Yixin Ji, Jinpeng Li +5Test-Time Scaling LLM Routing
School of Computer Science and Technology, Soochow University · Department of Foundation Model, 2012 Labs, Huawei · 3Harbin Institute of Technology, Shenzhen (HITSZ)
Learning to Route Queries to Heads for Attention-based Re-ranking with Large Language Models
Apr 27, 2026Yuxing Tian, Fengran Mo, Zhiqi Huang +2Attention Head Analysis Learning to Rank
Université de Montréal · Capital One · McGill University & MILA
GraphPlanner: Graph Memory-Augmented Agentic Routing for Multi-Agent LLMs
Apr 26, 2026Tao Feng, Haozhen Zhang, Zijie Lei +2Multi-Agent LLM Systems LLM Agent Orchestration
Department of Computer Science University of Illinois Urbana-Champaign Urbana, IL, USA
RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization
Apr 26, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuLLM Serving Cost-Aware Inference
1The University of Hong Kong · 2Stellaris AI Limited
MTRouter: Cost-Aware Multi-Turn LLM Routing with History-Model Joint Embeddings
Apr 26, 2026Yiqun Zhang, Hao Li, Zihan Wang +6Cost-Aware Inference LLM Routing
School of Computer Science and Engineering, Northeastern University Shenyang 110819, China · Shanghai Artificial Intelligence Laboratory
Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
Apr 25, 2026Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park +6Expert Load Balancing Mixture-of-Experts Inference
1Georgia Institute of Technology, USA · 2Meta Platforms, Inc.
Mixture of Heterogeneous Grouped Experts for Language Modeling
Apr 25, 2026Zhicheng Ma, Xiang Liu, Zhaoxiang Liu +5Expert Load Balancing Mixture-of-Experts Language Models
1Data Science & Artificial Intelligence Research Institute, China Unicom · 2Unicom Data Intelligence, China Unicom
RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment
Apr 24, 2026Yingfeng Luo, Hongyu Liu, Dingyang Lin +6LLM Routing Machine Translation
School of Computer Science and Engineering, Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China
EverydayGPT: Confidence-Gated Routing for Efficient and Safe Hybrid GPT-RAG Conversational QA
Apr 24, 2026Jaspreet Singh NahalLLM Inference Efficiency Retrieval-Augmented Generation
B.Tech Computer Science Engineering (Artificial Intelligence) Dr. A.P.J. Abdul Kalam Technical University, Delhi, India

May 11, 2026J/K move · Enter open · S save

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Wenbo Zhang, Lijinghua Zhang, Liner Xiang, Hengrui Cai

Department of Statistics, University of California, Irvine, USA.

LLM-as-a-Judge Cost-Aware Inference LLM Routing

Open paper AskPDF arXiv

Reasoning-capable large language models (LLMs) have recently been adopted as automated judges, but their benefits and costs in LLM-as-a-Judge settings remain unclear. Through controlled comparisons between reasoning and non-reasoning judges, we show that explicit reasoning substantially improves judgment accuracy on tasks requiring structured verification (e.g., math and coding), while offering limited or even negative gains on simpler evaluations and incurring significantly higher computational cost. These findings motivate that reasoning should be used selectively rather than universally, with awareness of possible distribution shift. We propose a Robust Adaptive Cost-Efficient Routing (RACER), which dynamically selects between reasoning and non-reasoning judges under a fixed budget by formulating routing as a constrained distributionally robust optimization problem. RACER explicitly accounts for distribution shift via a KL-divergence uncertainty set, admits an efficient primal--dual algorithm, and enjoys theoretical guarantees including uniqueness of the optimal policy and linear convergence. Extensive experiments show that RACER achieves superior accuracy--cost trade-offs under distribution shift.