We tested whether features of biology questions or of a model's first answer can improve accuracy within an inference budget. In a LAB-Bench pilot, we compared a category router with one that adds six structural features. Neither reached the primary 80% accuracy target on development. At the 50% and 60% secondary targets, both chose Haiku for all 120 held-out questions and scored 71/120. We then froze a new comparison before collecting responses on 144 additional questions. Semantic routing scored 86/144. A policy that could defer after reading Haiku's answer scored 87/144. Both stayed within the inference budget of USD 0.678 per 1,000 queries. Against a 95% Haiku and 5% GPT mixture, the accuracy gains were 0.66 percentage points for semantic routing and 1.35 points for deferral. The paired 95% intervals were [-0.28,2.25] and [-0.14,3.47] points. A simple rule that escalated Haiku nonanswers scored 92/144 at USD 0.710 per 1,000 queries, above the budget. A retrospective nonanswer rule with a calibration-selected escalation probability gave an expected 85.63/144. At the learned deferral policy's observed inference cost, post hoc interpolation gave an expected 86.14/144. In a later development check, Astra and Gemini each scored 30/40. The best tested policy in each router family matched that score. The pair covered 33/40 in hindsight. The routers missed the 32/40 criterion, so the proposed 120-question evaluation did not start.
Figures & tables
Phase
Data
Decision and scope
Pilot
60 development and 120 held-out questions
Neither router met the primary target on development. We evaluated the declared secondary policies on the holdout.
Prospective follow-up
120 historical fitting, 60 historical calibration and 144 new evaluation questions
We locked the original 13 policies and analysis before collecting new answers. We added the randomized nonanswer comparison and decision audit after evaluation.
Stronger-model screen
Original 60 development questions, with 40 used for the routing check
The tested mixed routers fell short of 32/40 on development. We stopped before selecting the proposed 120 new questions.
Table 1: Data and decision points for each study phase.
Model
C
I
N
Accuracy (%)
USD/1,000
Gemini 3.5 Flash-Lite
56
37
27
46.67
2.063998
Haiku 5.5
71
40
9
59.17
0.627299
GPT-5.6 Luna
82
31
7
68.33
0.790408
Sonnet 5.5
76
37
7
63.33
8.240916
Table 2: Fixed-model held-out outcomes. C counts correct answers, I counts official incorrect-grade outcomes including refusals, and N counts abstentions. Accuracy uses all 120 assigned questions. Cost includes frozen-standard inference and recorded local CPU, in USD per 1,000 queries.
Figure 1: Fixed-model accuracy on the complete held-out set and its three 40-question subsets. Labels give correct/assigned counts. Every wrong answer, abstention and refusal remains in the denominator. The plot reports observed accuracies without adding model-level uncertainty intervals.
Policy
Target
Available
Correct
Acc. (%)
USD/1,000
Category
0.50
Yes
71/120
59.17
0.62729921
CRR
0.50
Yes
71/120
59.17
0.62731007
Category
0.60
Yes
71/120
59.17
0.62729927
CRR
0.60
Yes
71/120
59.17
0.62731011
Category
0.70
No
n/a
n/a
n/a
CRR
0.70
Yes
65/120
54.17
2.95685522
Table 3: Policy availability after development selection and subsequent held-out results. The primary target is 0.80. “No” means development supplies no policy. Both routers at 0.50 and 0.60 choose Haiku on every held-out question. Cost uses the same basis as Table 2 .
Target
Δ accuracy (pp)
95% CI (pp)
Saving
95% CI
( 10−8 USD/query)
0.50
0.00
[0.00, 0.00] †
-1.0858
[-1.2980, -0.8863]
0.60
0.00
[0.00, 0.00] †
-1.0836
[-1.2955, -0.8825]
Table 4: Paired secondary comparisons. Saving means category cost minus CRR cost, so negative values indicate higher CRR cost. All 5,000 resamples are valid. The dagger marks degenerate accuracy intervals because both policies select the same model on every observed question. These intervals omit repeated model responses and development-fit uncertainty.
Figure 2: Held-out accuracy and frozen-standard inference plus recorded local CPU cost. The horizontal axis uses a logarithmic scale and USD per 1,000 queries. Category and CRR at targets 0.50 and 0.60 nearly coincide with Haiku because all choose Haiku on every question. The plot uses exact coordinates without jitter. Uniform denotes an analytic expectation over model choice. Neither primary 0.80 policy supplies a point.
Policy
Correct (out of 144)
Accuracy (%)
Standard inference (USD/1,000)
Inference + CPU (USD/1,000)
Fixed configurations
Gemini 3.5 Flash-Lite
72
50.00
2.022599
2.022599
Haiku 5.5
85
59.03
0.637203
0.637203
GPT-5.6 Luna
86
59.72
0.825743
0.825743
Sonnet 5.5
99
68.75
7.857681
7.857681
Budget-selected baselines
Table 6: Accuracy and cost for all 13 frozen follow-up policies. Costs are in USD per 1,000 queries. Mixture rows give expected counts. Both budget rules use the USD 0.678045 inference ceiling and give identical routes, accuracy and inference costs.
Operating point
p
Correct /144
Inference (USD/1,000)
Inference + CPU (USD/1,000)
Historical calibration costs
0.0905
85.63
0.643763
0.643763
Evaluation cost matched to learned deferral
0.1627
86.14
0.648997
0.648997
Evaluation cost set to the ceiling
0.5635
88.94
0.678045
0.678045
Learned deferral (observed)
87
0.648997
0.650736
Table 7: Retrospective comparisons on the recorded 144 questions. Randomized rows show expected outcomes. The second and third rows use evaluation-cohort costs. The CPU proxy for the randomized rows uses the measured nonanswer check and route selection from the full cascade.
Model
LitQA2 /20
ProtocolQA /20
SeqQA /20
Total /60
GPT-5.6 Luna
11
11
17
39
Gemini 3.8 Flash
15
16
15
46
Grok 4.6
9
16
12
37
GPT-6 Astra
14
18
16
48
Table 8: Fixed-model scores on the same 60 development questions. Luna was the strongest fixed model in the original pilot’s development set.
Development policy
Correct /40
Known USD over 40
Relative cost
Fixed Gemini
30
0.12890325
1.00×
Fixed Astra
30
0.78981000
6.13×
Best mixed category ( t=0.70 )
30
0.19067825
1.48×
Best mixed structural CRR ( t=0.65 )
30
0.28599725
2.22×
Table 9: Standard inference costs for the two-task development replay. Fixed Astra’s cost omits two failed attempts with unknown charges. Relative costs use known costs and fixed Gemini as the reference.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Category Brier
CRR Brier
Gemini 3.5 Flash-Lite
0.263169
0.232525
Haiku 5.5
0.242593
0.224248
GPT-5.6 Luna
0.230247
0.227075
Sonnet 5.5
0.263992
0.263086
Appendix
Table 10: Recorded development out-of-fold Brier scores for each model’s probability predictor. Both methods use the same grouped folds. The table preserves the frozen fit diagnostics and supplies no separate calibration guarantee.
Model
Collection mode
Gemini 3.5 Flash-Lite
Synchronous
Haiku 5.5
Synchronous
GPT-5.6 Luna
Batch
Sonnet 5.5
Batch
Appendix
Table 11: Model versions and collection modes in the pilot.
Model
Input
Output
Cache read
Cache write
Gemini 3.5 Flash-Lite
0.3
2.5
0.03
n/a
Haiku 5.5
0.1
0.5
0.01
0.125
GPT-5.6 Luna
0.2
1.2
0.02
0.25
Sonnet 5.5
2
10
0.1
2.5
Appendix
Table 12: Frozen standard prices in USD per million tokens. Inclusive output contains reasoning. The observed Gemini cache-write counts equal zero, so the analysis needs no Gemini cache-write price.
Configuration and price basis
Input
Inclusive output
Cache read
Cache write
Gemini 3.8 Flash, standard
0.75
3.75
0.075
0.041667
Grok 4.6, standard
2.00
6.00
0.50
2.00
GPT-6 Astra, standard
10.00
50.00
1.00
12.50
Appendix
Table 13: Frozen stronger-screen prices in USD per million tokens. The two-task gate uses standard prices for both models. The Gemini cache-write rate is rounded here.
Model
SeqQA
ProtocolQA
LitQA2
Overall
Gemini 3.5 Flash-Lite
22/40
22/40
12/40
56/120
Haiku 5.5
32/40
24/40
15/40
71/120
GPT-5.6 Luna
29/40
35/40
18/40
82/120
Sonnet 5.5
35/40
28/40
13/40
76/120
Appendix
Table 14: Fixed-model correct/assigned counts. Each subset supplies 40 held-out questions. The aggregate denominator includes all 120 questions and gives each subset equal weight.
Fixed configuration
LitQA2
ProtocolQA
SeqQA
Total
Gemini 3.5 Flash-Lite
19/48
24/48
29/48
72/144
Haiku 5.5
19/48
28/48
38/48
85/144
GPT-5.6 Luna
26/48
29/48
31/48
86/144
Sonnet 5.5
27/48
33/48
39/48
99/144
Appendix
Table 15: Fixed-model correct counts on the follow-up. Each task has 48 assigned questions. The total uses all 144, including refusals and abstentions.
Policy
Changed
Rescues
Harms
Both right
Both wrong
Haiku nonanswers
Category
0
0
0
0
0
0
Structural CRR
3
1
0
0
2
0
Semantic gain
4
1
0
1
2
0
Output-aware deferral
4
2
0
0
2
4
Shuffled semantic gain
16
1
1
5
9
3
Full nonanswer cascade
16
7
0
0
9
16
Appendix
Table 16: Paired correctness among changed decisions relative to fixed Haiku on the 144 questions. A rescue changes an incorrect result to correct. A harm changes a correct result to incorrect. The reviewer files include task-level counts and costs.
Figure 3: Cost and accuracy for randomized escalation on the 144 recorded questions. The line gives expectations from zero to full escalation. The three open circles mark the calibration-selected probability and the two probabilities computed from evaluation costs. The triangles mark observed frozen policies.
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.