LLM Routing

LLM: Large Language Model

Large language model routing studies how to assign a query, workflow step, or token to an appropriate model, expert, tool, retrieval path, or amount of computation. Common approaches use query features, confidence or uncertainty, hidden-state signals, and response disagreement to choose a capable low-cost option or escalate when needed, using classifiers, contextual bandits, reinforcement learning, and calibrated cascades. Systems work also schedules heterogeneous models and GPUs around budgets, latency, energy, cache reuse, privacy, and service guarantees, with evaluations spanning reasoning, coding, question answering, retrieval-augmented generation, agents, and production workloads. A central tension is that oracle model choices can show substantial gains, while deployable routers depend on reliable quality estimates and supervision and can falter under distribution shifts, weak model specialization, noisy evaluation, or routing overhead.

Latest papers

355 papers54 papers in the last four weeks, up 35% on the four weeks before. 0.3% of all new papers.

Privacy Guard & Token Parsimony by Prompt and Context Handling and LLM Routing
Mar 30, 2026Alessio LangiuPrivacy Protection for LLM Agents Privacy Leakage in Language Models
National Research Council of Italy - Institute of Marine Sciences (CNR-ISMAR)
From Inference Routing to Agent Orchestration: Declarative Policy Compilation with Cross-Layer Verification
Mar 28, 2026Huamin Chen, Xunzhuo Liu, Bowei He +1LLM Routing LLM Agent Orchestration
vLLM Semantic Router Project · MBZUAI · McGill University +1
Self-Describing Structured Data with Dual-Layer Guidance: A Lightweight Alternative to RAG for Precision Retrieval in Large-Scale LLM Knowledge Navigation
Mar 28, 2026Hung Ming LiuLost-in-the-Middle Effect Information Retrieval
PARRAWA AI
MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference
Mar 27, 2026Joris Köster, Zixuan Liu, Siavash Khajavi +1LLM Inference Serving Cost-Aware Inference
Department of Computer Science, Aalto University, Finland · Department of Computer Science, Tulane University, LA, USA · Department of Industrial Engineering and Management, Aalto University, Finland
Runtime Burden Allocation for Structured LLM Routing in Agentic Expert Systems: A Full-Factorial Cross-Backend Methodology
Mar 26, 2026Zhou Hanlin, Chan Huah YongLLM Routing LLM Evaluation
School of Computer Sciences, Universiti Sains Malaysia (USM), Penang, Malaysia · Xiamen Software Vocational & Technical College, Xiamen, China
Evaluating Small Language Models for Front-Door Routing: A Harmonized Benchmark and Synthetic-Traffic Experiment
Mar 26, 2026Warren Johnson, Charles LeeLLM Routing Small Language Models
Plexor Labs, Sammamish, WA, USA · Project Autobots, Seattle, WA, USA
Cost-Aware Query Routing in RAG: Empirical Analysis of Retrieval Depth Tradeoffs
Mar 26, 2026Sanjay MishraRetrieval-Augmented Generation Adaptive RAG
Raleigh, NC, USA
Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints
Mar 25, 2026Jelena Markovic-Voronov, Kayhan Behdin, Yuanda Xu +3LLM Routing LLM Inference Scheduling
LinkedIn · MIT
ReLope: From Hidden-State Probing to a Decision Module for Multimodal LLM Routing
Mar 25, 2026Yaopei Zeng, Congchao Wang, Blake JianHang Chen +1Multimodal Large Language Models LLM Routing
Pennsylvania State University · Google · Amazon AGI
GeoRouter: Dynamic Paradigm Routing for Worldwide Image Geolocalization
Mar 25, 2026Pengyue Jia, Derong Xu, Yingyi Zhang +5LLM Routing
Department of Data Science, City University of Hong Kong
DAK-UCB: Diversity-Aware Prompt Routing for LLMs and Generative Models
Mar 24, 2026Donya Jafari, Farzan FarniaLLM Routing Contextual Bandits
Sharif University of Technology · The Chinese University of Hong Kong
Knowledge Access Beats Model Size: Memory Augmented Routing for Persistent AI Agents
Mar 24, 2026Xunzhuo Liu, Bowei He, Xue Liu +3Memory-Augmented Language Model Agents Conversational Memory
vLLM Semantic Router Project · MBZUAI · McGill University +3
KALAVAI: Predicting When Independent Specialist Fusion Works -- A Quantitative Model for Post-Hoc Cooperative LLM Training
Mar 24, 2026Ramchand KumaresanLanguage Model Merging Mixture-of-Experts Expert Specialization
Murai Labs
Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs
Mar 23, 2026Kangqi Ni, Wenyue Hua, Xiaoxiang Shi +3LLM Serving LLM Inference Scheduling
University of North Carolina, Chapel Hill · Microsoft · Carnegie Mellon University +2
The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project
Mar 22, 2026Huamin Chen, Xunzhuo Liu, Bowei He +5LLM Routing LLM Inference Serving
vLLM Semantic Router Project · MBZUAI · McGill University +3
ConsRoute:Consistency-Aware Adaptive Query Routing for Cloud-Edge-Device Large Language Models
Mar 22, 2026Haoyu Qiao, Hao Zhang, Shanwen Mao +2Edge-Cloud Collaborative Inference LLM Routing
the Department of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China · the National Key Laboratory of Smart Farm Technologies and Systems, Harbin 150001, China · the Department of Computer Science and Technology, Harbin Institute of Technology (ShenZhen), ShenZhen 518055, China
When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models
Mar 22, 2026Abhinaba BasuAttention Routing LLM Routing
National Institute of Electronics and Information Technology (NIELIT)
LLM Router: Rethinking Routing with Prefill Activations
Mar 21, 2026Tanay Varshney, Annie Surla, Michelle Xu +5LLM Routing Adaptive Model Routing
NVIDIA
RouterKGQA: Specialized--General Model Routing for Constraint-Aware Knowledge Graph Question Answering
Mar 20, 2026Bo Yuan, Hexuan Deng, Xuebo Liu +1Knowledge Graph Question Answering LLM Reasoning with Graphs
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Zhongguancun Academy, Beijing, China
SEAR: Schema-Based Evaluation and Routing for LLM Gateways
Mar 20, 2026Zecheng Zhang, Han Zheng, Yue XuLLM Routing LLM Evaluation
Strukto.AI, USA · Infron.AI, USA
Scalable Prompt Routing via Fine-Grained Latent Task Discovery
Mar 19, 2026Yunyi Zhang, Soji Adeshina, Sheng Guan +5LLM Routing
Amazon Web Services
TARo: Token-level Adaptive Routing for LLM Test-time Alignment
Mar 19, 2026Arushi Rai, Qiang Zhang, Hanqing Zeng +5Inference-Time Language Model Alignment LLM Reasoning
Meta · University of Pittsburgh · University of California, Berkeley
Conflict-Free Policy Languages for Probabilistic ML Predicates: A Framework and Case Study with the Semantic Router DSL
Mar 18, 2026Xunzhuo Liu, Hao Wu, Huamin Chen +2LLM Routing
vLLM Semantic Router Project · MBZUAI / McGill University
MoLoRA: Composable Specialization via Per-Token Adapter Routing
Mar 16, 2026Shrey Shah, Justin WagleLLM Routing Low-Rank Adaptation
Directional Routing in Transformers
Mar 16, 2026Kevin TaylorAttention Routing Transformer
LLM Routing as Reasoning: A MaxSAT View
Mar 13, 2026Son Nguyen, Xinyuan Liu, Ransalu SenanayakeLLM Routing Large Language Model-Based Recommendation
Arizona State University
Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference
Mar 13, 2026Huamin Chen, Xunzhuo Liu, Junchen Jiang +2LLM Inference Efficiency LLM Routing
vLLM Semantic Router Project · Tensormesh Inc / UChicago · MBZUAI / McGill University
Efficient and Interpretable Multi-Agent LLM Routing via Ant Colony Optimization
Mar 13, 2026Xudong Wang, Chaoning Zhang, Jiaquan Zhang +8LLM Agent Routing LLM Routing
School of Computing, Kyung Hee University, Yongin-si 17104, South Korea · School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China · School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu 610054, China +1
98 $\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router
Mar 13, 2026Xunzhuo Liu, Bowei He, Xue Liu +3LLM Routing LLM Serving
vLLM Semantic Router Project · MBZUAI · McGill University +2
Outcome-Aware Tool Selection for Semantic Routers: Latency-Constrained Learning Without LLM Inference
Mar 13, 2026Huamin Chen, Xunzhuo Liu, Junchen Jiang +2LLM Routing LLM Tool Use
vLLM Semantic Router Project · Tensormesh Inc / UChicago · MBZUAI / McGill University

Mar 30, 2026J/K move · Enter open · S save

Privacy Guard & Token Parsimony by Prompt and Context Handling and LLM Routing

Alessio Langiu

National Research Council of Italy - Institute of Marine Sciences (CNR-ISMAR)

Privacy Protection for LLM Agents Privacy Leakage in Language Models LLM Routing LLM Context Management Privacy-Preserving Language Models LLM Inference Efficiency+1

Open paper AskPDF arXiv

The large-scale adoption of Large Language Models (LLMs) forces a trade-off between operational cost (OpEx) and data privacy. Current routing frameworks reduce costs but ignore prompt sensitivity, exposing users and institutions to leakage risks towards third-party cloud providers. We formalise the "Inseparability Paradigm": advanced context management intrinsically coincides with privacy management. We propose a local "Privacy Guard" -- a holistic contextual observer powered by an on-premise Small Language Model (SLM) -- that performs abstractive summarisation and Automatic Prompt Optimisation (APO) to decompose prompts into focused sub-tasks, re-routing high-risk queries to Zero-Trust or NDA-covered models. This dual mechanism simultaneously eliminates sensitive inference vectors (Zero Leakage) and reduces cloud token payloads (OpEx Reduction). A LIFO-based context compacting mechanism further bounds working memory, limiting the emergent leakage surface. We validate the framework through a 2x2 benchmark (Lazy vs. Expert users; Personal vs. Institutional secrets) on a 1,000-sample dataset, achieving a 45% blended OpEx reduction, 100% redaction success on personal secrets, and -- via LLM-as-a-Judge evaluation -- an 85% preference rate for APO-compressed responses over raw baselines. Our results demonstrate that Token Parsimony and Zero Leakage are mathematically dual projections of the same contextual compression operator.