Retrieval-based classifiers efficiently route queries to specialised services, but they typically select a candidate even when no suitable service exists. We introduce Adaptive Decision and Abstention with Pool-Size-Aware Thresholds for Service Routing (ADAPT), a lightweight routing architecture with an explicit fallback option that enables queries to abstain from specialist selection. ADAPT uses the separation between the highest relevance score and the candidate score distribution as routing confidence. Experiments across retrieval models and increasing numbers of services reveal a consistent empirical behaviour: the confidence distribution changes systematically with candidate-pool size K, making fixed abstention thresholds increasingly miscalibrated. ADAPT addresses this effect with a K-aware threshold that adapts as services are added without classifier retraining. Evaluation across two specialist datasets shows that this simple mechanism provides an effective accuracy-cost trade-off compared with generative LLM routing, supporting lightweight and dynamically extensible service-routing architectures.
Figures & tables
Figure 1: Gap between the highest and the median specialist score for in-scope (blue, 100 MMLU test queries per specialist) and off-topic (orange, 300 CLINC-OOS and BANKING77 queries) test queries at K=3 , 8 , 10 and 14 , before any gate is applied. The four representative models are two cross-encoders (Qwen3-Reranker-4B, mxbai-rerank-large-v2), a bi-encoder (bge-large-en-v1.5) and a ColBERT model (Reason-ModernColBERT).
Figure 2: Fitted T(K)=c0+c1lnK for one example model from each family, fitted on K=3,…,8 and tested on K=10 , 12 and 14 .
Figure 3: Rescaled measured cutoffs of the 14 retrieval models for K=3,…,8 , coloured by family. The dashed line marks exactly logarithmic cutoffs.
Pool size
Specialists
3
Abstract algebra, anatomy, astronomy
5
Pool of 3 plus business ethics, college computer science
8
Pool of 5 plus high school chemistry, high school geography, jurisprudence
10
Pool of 8 plus world religions, formal logic
12
Pool of 10 plus marketing, sociology
14
Pool of 12 plus high school psychology, prehistory
Table 1: Specialists in each pool. Each pool adds specialists to the previous one. Every specialist has 100 test queries, and the same 300 off-topic queries are used for every pool.
Family
Router
K=3
K=5
K=8
K=10
K=12
K=14
Bi-encoder
bge-large-en-v1.5
91.67
89.00
82.82
80.77
71.27
69.00
Qwen3-Embedding-0.6B
83.00
78.75
71.27
69.77
64.67
62.18
Qwen3-Embedding-4B
89.67
75.12
65.64
64.46
57.53
54.76
Qwen3-Embedding-8B
88.67
83.00
74.55
71.69
66.13
63.06
all-MiniLM-L6-v2
88.83
80.50
74.73
73.62
69.53
66.82
ColBERT
LFM2-ColBERT-350M
93.33
85.12
81.45
77.38
71.60
69.59
Table 2: Routing accuracy (%) of each model for each number of specialist services K with fallback. Bold marks the best retrieval model and the best LLM in each column.
Figure 4: Specialist recall (x-axis) and general recall (y-axis) of all 20 models at K=3 (left) and K=14 (right). The shaded zone marks both recalls above 80%.
Figure 5: Routing accuracy against median latency (left) and mean energy (right) per query at K=14 for all 20 models (log scale).
Figure 6: Median latency per query as the number of services K grows, for one model from each family: the LLM phi4:14b, the cross-encoder mxbai-rerank-large-v2, the ColBERT model Reason-ModernColBERT and the bi-encoder bge-large-en-v1.5.
Acc.
Spec. Recall
Gen. Recall
K=3
K=5
K=7
K=9
K=3
K=9
K=3
K=9
Bi-encoder
bge-large-en-v1.5
72.71
67.50
61.39
58.57
33.89
35.93
96.00
99.33
Qwen3-Embedding-8B
65.83
65.17
61.81
61.43
47.22
42.78
77.00
95.00
ColBERT
Reason-ModernColBERT
63.33
64.33
61.67
62.50
52.22
55.93
70.00
74.33
LFM2-ColBERT-350M
72.92
70.67
63.33
62.50
58.33
54.26
81.67
77.33
Cross-encoder
mxbai-rerank-large-v2
80.42
80.00
74.17
73.33
65.56
62.78
89.33
92.33
Table 3: MedMCQA specialists with MMLU general queries (%), using the constants fitted on MMLU without refitting. Bold marks the strongest retrieval model.
K=3
K=9
Spec.
Gen.
Spec.
Gen.
Model
Med.
MMLU
Δ
Med.
MMLU
Δ
Med.
MMLU
Δ
Med.
MMLU
Δ
bge-large-en-v1.5
47.78
33.89
−13.9
87.00
96.00
+9.0
48.33
35.93
−12.4
91.00
99.33
+8.3
Qwen3-Embedding-8B
47.22
47.22
0.0
77.00
77.00
0.0
39.26
42.78
+3.5
96.67
95.00
−1.7
Reason-ModernColBERT
52.22
52.22
0.0
68.00
70.00
+2.0
54.26
55.93
+1.7
81.00
74.33
−6.7
LFM2-ColBERT-350M
58.89
58.33
−0.6
81.33
81.67
+0.3
55.19
54.26
−0.9
70.67
77.33
+6.7
Table 4: Specialist and general recall (%) on MedMCQA with constants fitted on MedMCQA (Med.) and on MMLU (applied without refitting). Δ is MMLU minus Med., in points.
Fixed
K -aware
Δ
Family
Router
K
Acc.
Spec.
Gen.
Acc.
Spec.
Gen.
Acc.
Spec.
Gen.
Bi-encoder
bge-large-en-v1.5
3
91.83
89.33
94.33
91.67
89.33
94.00
−0.16
0.00
−0.33
5
88.50
88.60
88.33
89.00
86.80
92.67
+0.50
−1.80
+4.34
8
80.00
83.12
71.67
82.82
80.38
89.33
+2.82
−2.74
+17.66
10
78.15
83.10
61.67
80.77
78.20
89.33
+2.62
−4.90
+27.66
12
68.93
76.25
39.67
71.27
70.75
73.33
+2.34
−5.50
+33.66
Table 5: Fixed versus K -aware threshold for each number of specialist services K . Acc., Spec. and Gen. are the overall accuracy, specialist recall and general recall (%). The fixed threshold is calibrated once at K=3 and reused for every K . The K -aware constants are fitted on K=3 to K=8 , so K=10 , 12 and 14 are extrapolation. Δ is K -aware minus fixed, in points. Bold marks the higher accuracy.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
p
Specialist recall
General recall
Balanced accuracy
85
0.868
0.863
0.865
90
0.852
0.921
0.886
95
0.821
0.968
0.894
99
0.748
0.991
0.870
Appendix
Table 6: Validation results for each percentile p , averaged over 14 models and K=3,5,8,10 . Balanced accuracy is the mean of specialist and general recall.
Adaptive model routing selects models or specialized experts per input to balance prediction quality, computational cost, latency, and resource constraints.