COMPASS: Routing via Capability-Oriented Model Profiling with Adaptive Search, Selection, and Tool Use

Amin Rezaei; Pouya Parsa

cs.AIOct 20, 2026

COMPASS: Routing via Capability-Oriented Model Profiling with Adaptive Search, Selection, and Tool Use

Authors: Amin Rezaei, Pouya Parsa

Organizations: University of Minnesota

Abstract

LLM routing aims to select a capable model at low cost, but specialized routers can require updates as models and tasks change. We propose COMPASS, an agentic router in which an LLM follows a routing recipe, gathers performance and cost evidence through tools, and optionally inspects candidate answers before selecting a model. Our evaluations on LLMRouterBench and DeepSWE show that agentic routing can improve accuracy and lower model-call cost relative to specialized and trained routing baselines, without routing-specific training. On LLMRouterBench, COMPASS reaches 69.85% accuracy at $139.57, compared with RLCascadeRouter's 69.14% at $180.22. On DeepSWE under fixed five-fold cross-validation, it reaches 78.1% at $215, compared with 74.1% at $500 for the best single model. COMPASS also supports routing across benchmarks and adapts to an expanded model pool by reading new evaluation records, without changing the router's weights or prompt.

Figures & tables

Figure 1: COMPASS queries evaluation records, optionally pays to preview an answer, and selects a model and reasoning-effort level. New model records can be added without retraining the router.

Tool	Input/output	Purpose
find_similar_tasks	$q\mapsto\{t_{i}\}$	Nearest visible tasks
get_task_description	$t\mapsto\text{text}$	Task text shown to models
lookup_scores	$(\{t\},\{m\})\mapsto\{(s,c)\}$	Outcomes and costs; optional grouping
list_configurations	$\emptyset\mapsto\{(m,e)\}$	Allowed candidates
neighbourhood_evidence	$q\mapsto$ table	All candidates’ neighbour outcomes and benchmark composition
reliability_table	$\emptyset\mapsto$ table	Overall and ecosystem pass rates, costs, frontier

Table 1: COMPASS tools. Evidence reads are free; answer previews add model-call cost.

Figure 2: Accuracy and model-call cost of COMPASS, single models, and routing baselines on LLMRouterBench.

Panel of Figure 3 — Figure 3: DeepSWE accuracy with in-domain evidence (left) and TerminalBench evidence (right). Error bars show available 95% task-bootstrap intervals.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Accuracy gains from different sources of routing evidence.

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 349 papers

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
Aug 7, 2026Tao Feng, Fangxu Yu, Haozhen Zhang +9LLM Evaluation LLM Routing
University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2
Agent-as-a-Router: Agentic Model Routing for Coding Tasks
Jun 22, 2026Pengfei Zhou, Zhiwei Tang, Yixing Ma +8AI Coding Agents AI Agent Benchmarks
1National University of Singapore · 2DAMO Academy, Alibaba Group · 3Hupan Lab +3
Dynamic LLM Routers are Often Misguided
Oct 2, 2026Sam Wang, Julia White, Sahibzada Allahyar +3LLM Routing Cost-Aware Inference
Fastino Labs

Aug 7, 2026J/K move · Enter open · S save

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

Tao Feng, Fangxu Yu, Haozhen Zhang, Zhongjie Dai+8

University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University+2

LLM Evaluation LLM Routing

Open paper AskPDF arXiv

No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.

Abstract

LLM Routing

Explore similar work

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

Agent-as-a-Router: Agentic Model Routing for Coding Tasks

Dynamic LLM Routers are Often Misguided