LLM Routing

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 88% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 224

All topics

Streaming Model Cascades for Semantic SQL
Apr 1, 2026Paweł Liskowski, Kyle SchmausCost-Aware Inference LLM Routing
Snowflake Inc. · Poznań, Poland · San Francisco, CA, USA
Self-Routing: Parameter-Free Expert Routing from Hidden States
Apr 1, 2026Jama Hussein Mohamud, Drew Wagner, Mirco RavanelliParameter-Free Mixture-of-Experts Routing LLM Routing
Mila - Quebec AI Institute · Universite de Montreal · Concordia University
ReLope: From Hidden-State Probing to a Decision Module for Multimodal LLM Routing
Mar 25, 2026Yaopei Zeng, Congchao Wang, Blake JianHang Chen +1Multimodal Large Language Models LLM Routing
Pennsylvania State University · Google · Amazon AGI
Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey
Feb 23, 2026Yasmin Moslem, John D. KelleherLLM Routing Efficient Language Model Inference
ADAPT Centre School of Computer Science & Statistics Trinity College Dublin Dublin, Ireland
Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration
Feb 4, 2026Sudipto Ghosh, Sujoy Nath, Sunny Manchanda +1Multi-Agent Orchestration LLM Routing
Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi, India · Department of Electrical Engineering, Indian Institute of Technology Delhi, India · DYSL-AI, Defence Research and Development Organisation, India
Learning to Route and Schedule LLMs from User Retrials via Contextual Queueing Bandits
Feb 2, 2026Seoungbin Bae, Junyoung Son, Dabeen LeeLLM Serving LLM Routing
Department of Industrial & Systems Engineering, KAIST · Graduate School of Data Science, KAIST · Department of Mathematical Sciences, Seoul National University
Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement
Dec 5, 2025Nils Strassenburg, Boris Glavic, Tilmann RablCost-Aware Inference Small Language Models
Hasso Plattner Institute, Uni Potsdam · University of Illinois Chicago
GMTRouter: Personalized LLM Router over Multi-turn User Interactions
Oct 29, 2025Yihang Sun, Encheng Xie, Tao Feng +1LLM Personalization LLM Routing
University of Illinois Urbana-Champaign
PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoning
Sep 17, 2025Jingyuan Fan, Purui Liu, Hengbo Xiao +4Numerical Reasoning in Language Models LLM Inference Acceleration
Peking University · Tsinghua University
VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing
Sep 5, 2025Jiahuan Yu, Aryan Taneja, Junfeng Lin +1LLM Serving Energy-Efficient ML
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Champaign, USA · Department of Precision Instrument, Tsinghua University, Beijing, China
FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data
Jul 14, 2025Tao Feng, Haozhen Zhang, Zijie Lei +5LLM Evaluation LLM Routing
University of Illinois Urbana-Champaign · Nanyang Technological University · Meta Monetization AI +2
MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning
May 30, 2025Jingyan Shen, Jiarui Yao, Rui Yang +5Pairwise Preference Learning Reward Modeling
New York University · University of Illinois Urbana-Champaign · Rice University
Learning What to Retain: Gated-Memory Routing for Efficient Collaboration in Multi-Agent LLM Systems
Date pendingRakibul Hasan Rajib, Mengxin Zheng, Qian LouMulti-Agent LLM Systems Multi-Agent System Optimization
University of Central Florida
PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Serving
Date pendingLin Huang, Yujuan Tan, Weisheng Li +4Retrieval-Augmented Generation Inference-Time Optimization
Chongqing University. No. 55, University Town South Road, Gaoxin District, Chongqing, 401331, China · Inspur Yunzhou Industrial Internet Co., Ltd. No.1036 Langchao Road, Lixia District, Jinan, Shandong, 250101, China. · Guoqi Zhimo (Chongqing) Technology Co., Ltd. 5th Floor, Building B15, Xiantao Data Valley, Yubei District, Chongqing, 401122, China. +4

Apr 1, 2026, cs.DBJ/K move · Enter open · S save

Streaming Model Cascades for Semantic SQL

Paweł Liskowski, Kyle Schmaus

Snowflake Inc. · Poznań, Poland · San Francisco, CA, USA

Cost-Aware Inference LLM Routing Streaming Algorithms

Open paper AskPDF arXiv

Modern data warehouses extend SQL with semantic operators that invoke large language models on each qualifying row, making per-row inference orders of magnitude more expensive than traditional SQL. Model cascades reduce this cost by routing most rows through a fast proxy model and delegating uncertain cases to an expensive oracle. Prior SUPG-style cascades, however, require a global proxy-score pass that is itself an LLM-inference workload and blocks output in pipelined query engines. They also target either precision or recall and cannot serve workloads that need both. We formalize the cascade routing problem for streaming semantic SQL with independent parallel workers and present two complementary algorithms within this model. SUPG-IT extends SUPG from single-pass, single-metric estimation to streaming execution by iteratively refining two thresholds as oracle labels accumulate across batches, and is the first streaming cascade with joint probabilistic guarantees on user-specified precision and recall at a chosen failure probability

δ

. GAMCAL replaces user-specified targets with a single tradeoff parameter

α

between classification error and oracle cost, and learns a monotone Generalized Additive Model that calibrates proxy scores to true-positive probabilities and supplies pointwise uncertainty for stochastic routing. On six classification, filtering, and join benchmarks evaluated in a production semantic SQL engine, both algorithms reach

F_1 \geq 0.95

at their best operating points. GAMCAL also leads all six datasets at a 20% delegation budget and reaches

F_1 \geq 0.95

with up to 58% fewer oracle calls than LOTUS's SUPG cascade. SUPG-IT attains the highest best-case

F_1

, with a mean of 0.989 across the six datasets.

Momentum

Related topics

Latest papers 224