LLM Routing
LLM: Large Language Model
Overview
Large language model routing studies how to assign a query, workflow step, or token to an appropriate model, expert, tool, retrieval path, or amount of computation. Common approaches use query features, confidence or uncertainty, hidden-state signals, and response disagreement to choose a capable low-cost option or escalate when needed, using classifiers, contextual bandits, reinforcement learning, and calibrated cascades. Systems work also schedules heterogeneous models and GPUs around budgets, latency, energy, cache reuse, privacy, and service guarantees, with evaluations spanning reasoning, coding, question answering, retrieval-augmented generation, agents, and production workloads. A central tension is that oracle model choices can show substantial gains, while deployable routers depend on reliable quality estimates and supervision and can falter under distribution shifts, weak model specialization, noisy evaluation, or routing overhead.
Latest papers
355 papers54 papers in the last four weeks, up 35% on the four weeks before. 0.3% of all new papers.