LLM Routing

LLM: Large Language Model

Large language model routing studies how to assign a query, workflow step, or token to an appropriate model, expert, tool, retrieval path, or amount of computation. Common approaches use query features, confidence or uncertainty, hidden-state signals, and response disagreement to choose a capable low-cost option or escalate when needed, using classifiers, contextual bandits, reinforcement learning, and calibrated cascades. Systems work also schedules heterogeneous models and GPUs around budgets, latency, energy, cache reuse, privacy, and service guarantees, with evaluations spanning reasoning, coding, question answering, retrieval-augmented generation, agents, and production workloads. A central tension is that oracle model choices can show substantial gains, while deployable routers depend on reliable quality estimates and supervision and can falter under distribution shifts, weak model specialization, noisy evaluation, or routing overhead.

Latest papers

355 papers54 papers in the last four weeks, up 35% on the four weeks before. 0.3% of all new papers.

RouteGoT: Node-Adaptive Routing for Cost-Efficient Graph of Thoughts Reasoning
Mar 6, 2026Yuhang Liu, Ruijie Wang, Yunlong Chu +4LLM Reasoning with Graphs Efficient Language Model Reasoning
School of New Media and Communication, Tianjin University, Tianjin, China · School of Computer Science and Engineering, Beihang University, Beijing, China · Shanghai Jiao Tong University, Shanghai, China
MoE Lens -- An Expert Is All You Need
Mar 6, 2026Marmik Chaudhari, Idhant Gulati, Nishkal Hundia +2Mixture-of-Experts Expert Specialization Mixture-of-Experts Models
The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA
Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Experts
Mar 3, 2026Sanae Lotfi, Lucas Caccia, Alessandro Sordoni +2Language Model Ensembles Language Model Merging
1New York University · 2Microsoft Research
Expected Reward Prediction, with Applications to Model Routing
Mar 3, 2026Kenan Hasanaliyev, Silas Alberti, Jenny Hamer +5LLM Routing Reward Modeling
Stanford University · Inception Labs · Cognition Labs +2
RealRoute: Dynamic Query Routing System via Retrieve-then-Verify Paradigm
Mar 2, 2026Jiahe Liu, Qinkai Yu, Jingcheng Niu +5Retrieval-Augmented Generation LLM Routing
Technical University of Denmark · University of Exeter · University of Toronto +4
Tiny-Critic RAG: Empowering Agentic Fallback with Parameter-Efficient Small Language Models
Mar 1, 2026Yichao Wu, Penghao Liang, Yafei Xiang +5Agentic RAG LLM Routing
Northeastern University Boston, USA, 02115 · Independent Researcher Milpitas, USA, 95035 · Independent Researcher Austin, USA, 78613 +2
ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inference
Feb 27, 2026Siyuan Ma, Bo Gao, Xiaojun Jia +6Efficient Language Model Reasoning LLM Routing
Nanyang Technological University · Carnegie Mellon University · Northeast University (Qinhuangdao Campus) +4
Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference
Feb 25, 2026Bo-Wei Chen, Chung-Chi Chen, An-Zi YenCost-Aware Inference LLM Routing
Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan · Artificial Intelligence Research Center, AIST, Japan
Sustainable LLM Inference using Context-Aware Model Switching
Feb 25, 2026Yuvarani, Akashdeep Singh, Zahra Fathanah +3LLM Inference Efficiency LLM Inference
Faculty of Computer Science and Information Technology, Universiti Malaya, 50603 Kuala Lumpur, Malaysia
Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey
Feb 23, 2026Yasmin Moslem, John D. KelleherLLM Routing Efficient Language Model Inference
ADAPT Centre School of Computer Science & Statistics Trinity College Dublin Dublin, Ireland
Structured Prompt Language: Declarative Context Management for LLMs
Feb 23, 2026Wen G. GongLLM Prompting LLM Context Management
vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models
Feb 23, 2026Xunzhuo Liu, Huamin Chen, Samzong Lu +30Multimodal Model Routing LLM Routing
Steve
Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference
Feb 23, 2026Arindam KhaledLLM Routing Cost-Aware Inference
Independent Researcher
RACER: Risk-Aware Calibrated Efficient Routing for Large Language Models
Feb 20, 2026Sai Hao, Hao Zeng, Hongxin Wei +1Set-Valued Prediction LLM Routing
Southern University of Science and Technology, China · The Chinese University of Hong Kong, Shenzhen, China · Shenzhen Loop Area Institute, China
Multi-Agentic AI for Fairness-Aware and Accelerated Multi-modal Large Model Inference in Real-world Mobile Edge Networks
Feb 6, 2026Haiyuan Li, Hari Madhukumar, Shuangyi Yan +2Efficient Multimodal Inference LLM Inference Scheduling
Smart Internet Lab, Department of Electrical and Electronic Engineering, University of Bristol, BS8 1QU, U.K.
Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration
Feb 4, 2026Sudipto Ghosh, Sujoy Nath, Sunny Manchanda +1Multi-Agent Orchestration LLM Routing
Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi, India · Department of Electrical Engineering, Indian Institute of Technology Delhi, India · DYSL-AI, Defence Research and Development Organisation, India
Learning to Route and Schedule LLMs from User Retrials via Contextual Queueing Bandits
Feb 2, 2026Seoungbin Bae, Junyoung Son, Dabeen LeeLLM Serving LLM Routing
Department of Industrial & Systems Engineering, KAIST · Graduate School of Data Science, KAIST · Department of Mathematical Sciences, Seoul National University
Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluations
Jan 29, 2026Baris Askin, Shivam Patel, Anupam Nayak +4LLM Routing Federated Learning
Carnegie Mellon University.
Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement
Dec 5, 2025Nils Strassenburg, Boris Glavic, Tilmann RablCost-Aware Inference Small Language Models
Hasso Plattner Institute, Uni Potsdam · University of Illinois Chicago
GMTRouter: Personalized LLM Router over Multi-turn User Interactions
Oct 29, 2025Yihang Sun, Encheng Xie, Tao Feng +1LLM Personalization LLM Routing
University of Illinois Urbana-Champaign
PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoning
Sep 17, 2025Jingyuan Fan, Purui Liu, Hengbo Xiao +4Numerical Reasoning in Language Models LLM Inference Acceleration
Peking University · Tsinghua University
VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing
Sep 5, 2025Jiahuan Yu, Aryan Taneja, Junfeng Lin +1LLM Serving Energy-Efficient ML
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Champaign, USA · Department of Precision Instrument, Tsinghua University, Beijing, China
FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data
Jul 14, 2025Tao Feng, Haozhen Zhang, Zijie Lei +5LLM Evaluation LLM Routing
University of Illinois Urbana-Champaign · Nanyang Technological University · Meta Monetization AI +2
MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning
May 30, 2025Jingyan Shen, Jiarui Yao, Rui Yang +5Pairwise Preference Learning Reward Modeling
New York University · University of Illinois Urbana-Champaign · Rice University
A Unified Approach to Routing and Cascading for LLMs
Oct 14, 2024Jasper Dekoninck, Maximilian Baader, Martin VechevLLM Routing Language Model Cascading
Department of Computer Science, ETH Zurich, Switzerland.

Mar 6, 2026J/K move · Enter open · S save

RouteGoT: Node-Adaptive Routing for Cost-Efficient Graph of Thoughts Reasoning

Yuhang Liu, Ruijie Wang, Yunlong Chu, Bing Hao+3

School of New Media and Communication, Tianjin University, Tianjin, China · School of Computer Science and Engineering, Beihang University, Beijing, China · Shanghai Jiao Tong University, Shanghai, China

LLM Reasoning with Graphs Efficient Language Model Reasoning LLM Routing LLM Inference Scheduling CoT Reasoning

Open paper AskPDF arXiv

Large Language Models (LLMs) excel at multi-step reasoning, yet increasing the structural complexity of inference does not consistently improve system-level returns. Methods such as Tree of Thoughts (ToT), Graph of Thoughts (GoT), and Adaptive Graph of Thoughts (AGoT) can boost accuracy on some benchmarks, but often introduce substantial overhead in token consumption and latency, and their gains can be unstable across task distributions-sometimes underperforming simpler Chain-of-Thought (CoT) or direct input-output prompting (IO). We attribute this inefficiency to stage-wise and node-wise heterogeneity inside GoT-style reasoning pipelines: high-quality planning and final synthesis are globally coupled and typically benefit from strong models, whereas many intermediate subtasks are localized and can be solved accurately by lighter models with far fewer tokens. Motivated by these observations, we propose RouteGoT, a budget-controllable, node-adaptive routing framework for graph-structured reasoning. RouteGoT performs in-graph routing by prioritizing strong models for planning and synthesis, while dynamically allocating lightweight models and cost-effective strategies to leaf subtasks based on predicted difficulty. It further integrates explicit budget constraints into a global inference scheduler to control graph expansion under a user-specified token budget, enabling predictable performance-cost trade-offs. Experiments across reasoning, retrieval, and multi-hop QA benchmarks show that RouteGoT matching or improving accuracy while substantially reducing token usage; specifically, it achieves an average 8.1 percentage points accuracy improvement and 79.1% output token reduction compared to AGoT. Furthermore, RouteGoT outperforms existing routing baselines by maintaining a superior cost-accuracy trade-off, demonstrating improved robustness under varying budget targets and tasks.