cs.AINov 16, 2026

Routing Beyond Task Categories: A Cost-Aware LAB-Bench Study

Authors: Ahmed Tariq, Leo Guinan

Organizations: Independent researcher · Idea Nexus Ventures

Abstract

We tested whether features of biology questions or of a model's first answer can improve accuracy within an inference budget. In a LAB-Bench pilot, we compared a category router with one that adds six structural features. Neither reached the primary 80% accuracy target on development. At the 50% and 60% secondary targets, both chose Haiku for all 120 held-out questions and scored 71/120. We then froze a new comparison before collecting responses on 144 additional questions. Semantic routing scored 86/144. A policy that could defer after reading Haiku's answer scored 87/144. Both stayed within the inference budget of USD 0.678 per 1,000 queries. Against a 95% Haiku and 5% GPT mixture, the accuracy gains were 0.66 percentage points for semantic routing and 1.35 points for deferral. The paired 95% intervals were [-0.28,2.25] and [-0.14,3.47] points. A simple rule that escalated Haiku nonanswers scored 92/144 at USD 0.710 per 1,000 queries, above the budget. A retrospective nonanswer rule with a calibration-selected escalation probability gave an expected 85.63/144. At the learned deferral policy's observed inference cost, post hoc interpolation gave an expected 86.14/144. In a later development check, Astra and Gemini each scored 30/40. The best tested policy in each router family matched that score. The pair covered 33/40 in hindsight. The routers missed the 32/40 criterion, so the proposed 120-question evaluation did not start.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Aug 24, 2026

Most of the LLM Routing Gap Is Task Type

An LLM router picks which model should answer each query. The appeal is that models fail on different questions. Whatever single model is best overall still gets some wrong, and another model in the pool gets many of those right. Getting that choice right every time is the ceiling, and a router is an attempt to approach it. However, recent work reports that routers do not get close. Across 21 routing methods on five benchmarks, sharply different designs land within a fraction of a point of each other, and all of them stay far below that ceiling. Learned routers often fail to beat simply always calling the strongest model. We ask what those missed questions have in common. We set fourteen models to answer all 294 questions, with 7 task types across 3 languages: Korean, English and Hindi. We ran the whole matrix twice, changing nothing, but 5.37% of the 4,116 model-question pairs came out scored differently anyway. Run-to-run movement like that is normal, and we argue that a small win does not show that routing did anything, ours or anyone else's. Counting an answer correct only when the model got it right in both runs, 29 questions on this matrix can be improved with routing. Every correct-answer count here is on that rule. Task type accounts for most of them: assigning each task type one model in advance, chosen once and never updated, improves 21 of the 29. Splitting each task type by language improves 2 more and leaves 6 of 294 unoptimized. That handful is what a learned router would have been built for, and it is smaller than the run-to-run movement above, which is a share of pairs rather than of questions. The static table we adopted answers 262 of 294 questions at $3.33 per run, against the best single model's 245 at $7.69. All of this is fitted and scored on the same 294 questions with no holdout.
Oct 2, 2026

Dynamic LLM Routers are Often Misguided

Dynamic LLM routers promise to cut inference costs by sending each query to the cheapest model that can answer it correctly. We analyze six commercial routers across 14 settings on a diverse benchmark spanning eight task categories, finding that none of them outperforms a router that randomly selects between two well-chosen models at matched cost. Some underperform by more than 10 percentage points. We trace this gap to four patterns prevalent across routers: difficulty blindness, length reversal, semantic matching, and roster suboptimality. We show that the first three are what the standard objective rewards: cost-accuracy Pareto efficiency on realized costs favors escalating moderately hard queries over the hardest ones, shorter queries over longer ones, and routing by a query's source over its difficulty. We also argue that the two assumptions that would justify large rosters, model granularity and model specialization, do not hold empirically. We propose an alternative evaluation methodology that does not reward these patterns, and as a proof of concept, we design a simple two-model router that avoids all four. Nevertheless, its gain over random routing is limited, because a well-chosen roster leaves little to route.
Apr 3, 2026

Lightweight Query Routing for Adaptive RAG: A Baseline Study on RAGRouter-Bench

Retrieval-Augmented Generation pipelines span a wide range of retrieval strategies that differ substantially in token cost and capability. Selecting the right strategy per query is a practical efficiency problem, yet no routing classifiers have been trained on RAGRouter-Bench \citep{wang2026ragrouterbench}, a recently released benchmark of 7,7277,727 queries spanning four knowledge domains, each annotated with one of three canonical query types: factual, reasoning, and summarization. We present the first systematic evaluation of lightweight classifier-based routing on this benchmark. Five classical classifiers are evaluated under three feature regimes, namely, TF-IDF, MiniLM sentence embeddings \citep{reimers2019sbert}, and hand-crafted structural features, yielding 15 classifier feature combinations. Our best configuration, TF-IDF with an SVM, achieves a macro-averaged F1 of 0.928\mathbf{0.928} and an accuracy of 93.2%\mathbf{93.2\%}, while simulating 28.1%\mathbf{28.1\%} token savings relative to always using the most expensive paradigm. Lexical TF-IDF features outperform semantic sentence embeddings by 3.13.1 macro-F1 points, suggesting that surface keyword patterns are strong predictors of query-type complexity. Domain-level analysis reveals that medical queries are hardest to route and legal queries most tractable. These results establish a reproducible query-side baseline and highlight the gap that corpus-aware routing must close.