Routing Beyond Task Categories: A Cost-Aware LAB-Bench Study
Organizations: Independent researcher · Idea Nexus Ventures
Abstract
We tested whether features of biology questions or of a model's first answer can improve accuracy within an inference budget. In a LAB-Bench pilot, we compared a category router with one that adds six structural features. Neither reached the primary 80% accuracy target on development. At the 50% and 60% secondary targets, both chose Haiku for all 120 held-out questions and scored 71/120. We then froze a new comparison before collecting responses on 144 additional questions. Semantic routing scored 86/144. A policy that could defer after reading Haiku's answer scored 87/144. Both stayed within the inference budget of USD 0.678 per 1,000 queries. Against a 95% Haiku and 5% GPT mixture, the accuracy gains were 0.66 percentage points for semantic routing and 1.35 points for deferral. The paired 95% intervals were [-0.28,2.25] and [-0.14,3.47] points. A simple rule that escalated Haiku nonanswers scored 92/144 at USD 0.710 per 1,000 queries, above the budget. A retrospective nonanswer rule with a calibration-selected escalation probability gave an expected 85.63/144. At the learned deferral policy's observed inference cost, post hoc interpolation gave an expected 86.14/144. In a later development check, Astra and Gemini each scored 30/40. The best tested policy in each router family matched that score. The pair covered 33/40 in hindsight. The routers missed the 32/40 criterion, so the proposed 120-question evaluation did not start.
Figures & tables
| Phase | Data | Decision and scope |
|---|---|---|
| Pilot | 60 development and 120 held-out questions | Neither router met the primary target on development. We evaluated the declared secondary policies on the holdout. |
| Prospective follow-up | 120 historical fitting, 60 historical calibration and 144 new evaluation questions | We locked the original 13 policies and analysis before collecting new answers. We added the randomized nonanswer comparison and decision audit after evaluation. |
| Stronger-model screen | Original 60 development questions, with 40 used for the routing check | The tested mixed routers fell short of 32/40 on development. We stopped before selecting the proposed 120 new questions. |
| Model | C | I | N | Accuracy (%) | USD/1,000 |
|---|---|---|---|---|---|
| Gemini 3.5 Flash-Lite | 56 | 37 | 27 | 46.67 | 2.063998 |
| Haiku 5.5 | 71 | 40 | 9 | 59.17 | 0.627299 |
| GPT-5.6 Luna | 82 | 31 | 7 | 68.33 | 0.790408 |
| Sonnet 5.5 | 76 | 37 | 7 | 63.33 | 8.240916 |
| Policy | Target | Available | Correct | Acc. (%) | USD/1,000 |
|---|---|---|---|---|---|
| Category | 0.50 | Yes | 71/120 | 59.17 | 0.62729921 |
| CRR | 0.50 | Yes | 71/120 | 59.17 | 0.62731007 |
| Category | 0.60 | Yes | 71/120 | 59.17 | 0.62729927 |
| CRR | 0.60 | Yes | 71/120 | 59.17 | 0.62731011 |
| Category | 0.70 | No | n/a | n/a | n/a |
| CRR | 0.70 | Yes | 65/120 | 54.17 | 2.95685522 |
| Target | accuracy (pp) | 95% CI (pp) | Saving | 95% CI |
|---|---|---|---|---|
| ( USD/query) | ||||
| 0.50 | 0.00 | [0.00, 0.00] † | -1.0858 | [-1.2980, -0.8863] |
| 0.60 | 0.00 | [0.00, 0.00] † | -1.0836 | [-1.2955, -0.8825] |
| Policy | Correct (out of 144) | Accuracy (%) | Standard inference (USD/1,000) | Inference + CPU (USD/1,000) |
|---|---|---|---|---|
| Fixed configurations | ||||
| Gemini 3.5 Flash-Lite | 72 | 50.00 | 2.022599 | 2.022599 |
| Haiku 5.5 | 85 | 59.03 | 0.637203 | 0.637203 |
| GPT-5.6 Luna | 86 | 59.72 | 0.825743 | 0.825743 |
| Sonnet 5.5 | 99 | 68.75 | 7.857681 | 7.857681 |
| Budget-selected baselines | ||||
| Operating point | Correct /144 | Inference (USD/1,000) | Inference + CPU (USD/1,000) | |
|---|---|---|---|---|
| Historical calibration costs | 0.0905 | 85.63 | 0.643763 | 0.643763 |
| Evaluation cost matched to learned deferral | 0.1627 | 86.14 | 0.648997 | 0.648997 |
| Evaluation cost set to the ceiling | 0.5635 | 88.94 | 0.678045 | 0.678045 |
| Learned deferral (observed) | 87 | 0.648997 | 0.650736 |
| Model | LitQA2 /20 | ProtocolQA /20 | SeqQA /20 | Total /60 |
|---|---|---|---|---|
| GPT-5.6 Luna | 11 | 11 | 17 | 39 |
| Gemini 3.8 Flash | 15 | 16 | 15 | 46 |
| Grok 4.6 | 9 | 16 | 12 | 37 |
| GPT-6 Astra | 14 | 18 | 16 | 48 |
| Development policy | Correct /40 | Known USD over 40 | Relative cost |
|---|---|---|---|
| Fixed Gemini | 30 | 0.12890325 | |
| Fixed Astra | 30 | 0.78981000 | |
| Best mixed category ( ) | 30 | 0.19067825 | |
| Best mixed structural CRR ( ) | 30 | 0.28599725 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Category Brier | CRR Brier |
|---|---|---|
| Gemini 3.5 Flash-Lite | 0.263169 | 0.232525 |
| Haiku 5.5 | 0.242593 | 0.224248 |
| GPT-5.6 Luna | 0.230247 | 0.227075 |
| Sonnet 5.5 | 0.263992 | 0.263086 |
| Model | Collection mode |
|---|---|
| Gemini 3.5 Flash-Lite | Synchronous |
| Haiku 5.5 | Synchronous |
| GPT-5.6 Luna | Batch |
| Sonnet 5.5 | Batch |
| Model | Input | Output | Cache read | Cache write |
|---|---|---|---|---|
| Gemini 3.5 Flash-Lite | 0.3 | 2.5 | 0.03 | n/a |
| Haiku 5.5 | 0.1 | 0.5 | 0.01 | 0.125 |
| GPT-5.6 Luna | 0.2 | 1.2 | 0.02 | 0.25 |
| Sonnet 5.5 | 2 | 10 | 0.1 | 2.5 |
| Configuration and price basis | Input | Inclusive output | Cache read | Cache write |
|---|---|---|---|---|
| Gemini 3.8 Flash, standard | 0.75 | 3.75 | 0.075 | 0.041667 |
| Grok 4.6, standard | 2.00 | 6.00 | 0.50 | 2.00 |
| GPT-6 Astra, standard | 10.00 | 50.00 | 1.00 | 12.50 |
| Model | SeqQA | ProtocolQA | LitQA2 | Overall |
|---|---|---|---|---|
| Gemini 3.5 Flash-Lite | 22/40 | 22/40 | 12/40 | 56/120 |
| Haiku 5.5 | 32/40 | 24/40 | 15/40 | 71/120 |
| GPT-5.6 Luna | 29/40 | 35/40 | 18/40 | 82/120 |
| Sonnet 5.5 | 35/40 | 28/40 | 13/40 | 76/120 |
| Fixed configuration | LitQA2 | ProtocolQA | SeqQA | Total |
|---|---|---|---|---|
| Gemini 3.5 Flash-Lite | 19/48 | 24/48 | 29/48 | 72/144 |
| Haiku 5.5 | 19/48 | 28/48 | 38/48 | 85/144 |
| GPT-5.6 Luna | 26/48 | 29/48 | 31/48 | 86/144 |
| Sonnet 5.5 | 27/48 | 33/48 | 39/48 | 99/144 |
| Policy | Changed | Rescues | Harms | Both right | Both wrong | Haiku nonanswers |
|---|---|---|---|---|---|---|
| Category | 0 | 0 | 0 | 0 | 0 | 0 |
| Structural CRR | 3 | 1 | 0 | 0 | 2 | 0 |
| Semantic gain | 4 | 1 | 0 | 1 | 2 | 0 |
| Output-aware deferral | 4 | 2 | 0 | 0 | 2 | 4 |
| Shuffled semantic gain | 16 | 1 | 1 | 5 | 9 | 3 |
| Full nonanswer cascade | 16 | 7 | 0 | 0 | 9 | 16 |
LLM Routing
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.
See all 382 papers