Embedding Geometry Shapes LLM Routing: Tracing Failure Modes and Improving Generalization

Leonardo Torti; Emanuele Rimoldi; Tomer Galanti; Tomaso Poggio; Pierfrancesco Beneventano

cs.AIOct 13, 2026

Embedding Geometry Shapes LLM Routing: Tracing Failure Modes and Improving Generalization

Authors: Leonardo Torti, Emanuele Rimoldi, Tomer Galanti, Tomaso Poggio, Pierfrancesco Beneventano

Organizations: Massachusetts Institute of Technology. · Texas A&M University.

Abstract

With the growing number of LLMs available, selecting which one to use for a prompt has become a key question. A large part of the routing literature relies on semantically embedding prompts into a vector space and then applying a lightweight classifier to the embedded representations. These methods perform robustly in-distribution but they struggle to generalize to unseen tasks. By studying the geometry of semantic embeddings, we trace a clear failure mode: features that determine the neighbors of unseen prompts are an unreliable proxy for LLM selection. We therefore replace the semantic embedding with the following map: we run a partial forward pass of Qwen2.5-7B-Instruct on the prompt and use the hidden state of its last token at a deep layer as the representation. When applying kNN and other simple classifiers to this new representation, we obtain a clear result in generalization: on average across cost penalties, their performance improves. Moreover, kNN and another simple classifier that we propose outperform every evaluated semantic-embedding baseline. Our two simple methods also beat a recent SoTA hidden-states based router on RouterBench.

Figures & tables

Figure 1: Compute-aware routing: a prompt is mapped to a representation, on which a standard classifier selects the model to use.

Figure 2: In-distribution reward–cost frontiers with GTE. Neighborhood methods and group lookup remain competitive with more elaborate routers.

Figure 3: GTE geometry on RouterBench: families, subjects within MMLU, and difficulty within professional law. Each panel uses a separate t-SNE projection.

Figure 4: Changing the representation changes which training families supply held-out queries’ neighbors. Each matrix reports the family composition of the 100 nearest training queries under held-out-family evaluation.

	RouterBench		SPROUT
Method	GTE	L21	GTE	L21
Best single model	0.6618		0.7575
kNN-100	0.6615	0.7007	0.7529	0.7731
Validated kNN	0.6566	0.7034	0.7534	0.7739
Family Adaptation	0.7006	0.7157	0.7603	0.7767
Avengers-Pro	0.6662	0.6581	0.7439	0.7714

Table 1: L21 raises mean utility for every evaluated routing rule on both benchmarks except Avengers-Pro on RouterBench. Values average all 13 cost penalties under held-out-family evaluation; bold marks the higher value within each encoder pair. Full results with 95% confidence intervals are reported in Appendix D .

Method	RouterBench	SPROUT
L21 kNN-100	0.7007	0.7731
L21 Family Adaptation	0.7157	0.7767
SharedTrunkNet	0.6949	0.7829

Table 2: Common-encoder comparison with utility-adapted SharedTrunkNet. The simple L21 routers have higher mean utility on RouterBench; SharedTrunkNet has higher mean utility on SPROUT. Values average all 13 cost penalties under held-out-family evaluation; bold marks the highest mean. Full results with 95% confidence intervals are reported in Appendix D.2 .

Embedding / retrieval	RouterBench	SPROUT
GTE	0.6615	0.7529
GTE, same answer format	0.6974	0.7584
L21	0.7007	0.7731

Table 3: Restricting GTE retrieval by answer format recovers most of L21’s mean utility gain on RouterBench and less on SPROUT. The control also changes source-family composition. Values average held-out-family kNN-100 utility across 13 cost penalties; full results and 95% intervals appear in Appendix F.2 .

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Input	Output
Claude-3.5-Sonnet-v1	3.00	15.00
Titan Text Premier v1	0.50	1.50
GPT-4o	2.50	10.00
GPT-4o-mini	0.15	0.60
Granite-3-2B Instruct	0.10	0.10
Granite-3-8B Instruct	0.20	0.20

Appendix

Table 6: Fixed SPROUT prices in USD per million tokens.

Setting	Values
Hidden layers	$(100,100)$ or $(100,100,100)$ , with ReLU
L2 regularization $\alpha$	$10^{-4},\;10^{-2},\;1,\;10,\;100$
Training objective	Squared error with L2 weight regularization
Optimizer	Adam; learning rate $10^{-3}$
Adam parameters	$\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $\epsilon=10^{-8}$
Batch size / maximum epochs	200 / 200

Appendix

Table 7: MLP candidate grid and fixed training settings.

Benchmark	Neighbors	Random selection	Relative reduction
RouterBench	0.2463	0.2565	4.01%
SPROUT	0.1861	0.2467	24.58%

Appendix

Table 8: Mean absolute difference in solve fraction within family-cells.

Benchmark	Nearest neighbors	Random selection
RouterBench	95.51%	29.55%
SPROUT	97.30%	31.88%

Appendix

Table 9: Full-data GTE same-family share among 100 neighbors.

			Full-data GTE		Matched subset NMI
Benchmark	Family	Subjects	NMI	ARI	GTE	MiniLM
RouterBench	MMLU	57	0.722	0.418	0.726	0.640
SPROUT	MATH	7	0.327	0.246	0.334	0.296
SPROUT	MMLU-Pro	14	0.286	0.172	0.280	0.505

Appendix

Table 10: Recovery of native subjects within families on test queries. Matched columns use the same MiniLM-compatible queries and splits.

Benchmark	GTE	MiniLM	Random selection
RouterBench	95.48%	90.43%	29.49%
SPROUT	97.33%	90.74%	34.53%

Appendix

Table 11: Same-family share among 100 nearest training queries, on the matched subset. Random selection samples uniformly from all training queries.

Benchmark	GTE	L21
RouterBench	78.4%	72.7%
SPROUT	84.1%	92.0%

Appendix

Table 12: Mean largest-family share among 100 neighbors, with the query’s family excluded.

Benchmark	Randomly selected queries	GTE	L21
RouterBench	51.2%	76.4%	99.9%
SPROUT	38.9%	58.9%	90.2%

Appendix

Table 13: Percentage of held-out neighbors sharing the query’s answer-format label.

	RouterBench (six sources)		SPROUT (five sources)
$\lambda$	Mean rank	Top-1	Mean rank	Top-1
0	3.14	0/7	3.00	2/6
0.5	n/a	n/a	3.00	2/6
1	3.14	0/7	3.00	2/6
2	3.14	0/7	3.00	2/6
5	3.36	0/7	3.00	2/6

Appendix

Table 14: Retrieval rank of the family with the lowest relative-utility prediction error. Top-1 counts held-out families for which it is also the most retrieved. Each benchmark uses its prescribed penalty grid.

Figure 6: Correct utility ordering across candidate-model pairs, evaluated at every penalty. At

\lambda=0

, recorded reward ties change the set of evaluated pairs.

Figure 7: Held-out utility at every penalty for kNN-100 and Family Adaptation. Unlike reward–cost frontiers, these curves retain every evaluated policy.

Dataset	Lookup	Different choices	Gain on disagreements	Overall gain
RouterBench	Family	30.04% [-1pt] [29.80, 30.29]	$+$ 0.0233 [-1pt] [ $+$ 0.0178, $+$ 0.0290]	$+$ 0.0070 [-1pt] [ $+$ 0.0054, $+$ 0.0087]
RouterBench	Family-cell	28.78% [-1pt] [28.54, 29.03]	$+$ 0.0186 [-1pt] [ $+$ 0.0131, $+$ 0.0245]	$+$ 0.0054 [-1pt] [ $+$ 0.0038, $+$ 0.0071]
SPROUT	Family	45.50% [-1pt] [45.17, 45.79]	$+$ 0.0199 [-1pt] [ $+$ 0.0166, $+$ 0.0233]	$+$ 0.0090 [-1pt] [ $+$ 0.0076, $+$ 0.0106]
SPROUT	Family-cell	47.75% [-1pt] [47.44, 48.07]	$+$ 0.0135 [-1pt] [ $+$ 0.0101, $+$ 0.0170]	$+$ 0.0065 [-1pt] [ $+$ 0.0048, $+$ 0.0081]

Appendix

Table 15: GTE kNN-100 compared with group lookup across penalties, with 95% paired intervals. Positive gains favor kNN.

RouterBench
	Against family lookup Overall gain	Against family-cell lookup
$\lambda$		Different choices	Gain on disagreements	Overall gain
0	0.0000 [-1pt] [ $-$ - 0.0018, $+$ 0.0019]	23.55% [-1pt] [23.06, 24.03]	$+$ 0.0038 [-1pt] [ $-$ - 0.0039, $+$ 0.0120]	$+$ 0.0009 [-1pt] [ $-$ - 0.0009, $+$ 0.0029]
1	$+$ 0.0004 [-1pt] [ $-$ - 0.0015, $+$ 0.0023]	24.92% [-1pt] [24.42, 25.42]	$+$ 0.0057 [-1pt] [ $-$ - 0.0018, $+$ 0.0136]	$+$ 0.0014 [-1pt] [ $-$ - 0.0005, $+$ 0.0034]
2	$+$ 0.0004 [-1pt] [ $-$ - 0.0015, $+$ 0.0022]	25.51% [-1pt] [25.03, 25.99]	$+$ 0.0062 [-1pt] [ $-$ - 0.0012, $+$ 0.0140]	$+$ 0.0016 [-1pt] [ $-$ - 0.0003, $+$ 0.0036]
5	$+$ 0.0013 [-1pt] [ $-$ - 0.0008, $+$ 0.0033]	29.27% [-1pt] [28.77, 29.79]	$+$ 0.0078 [-1pt] [ $+$ 0.0007, $+$ 0.0149]	$+$ 0.0023 [-1pt] [ $+$ 0.0002, $+$ 0.0043]

Appendix

Table 16: kNN-100 utility gains at every penalty, with 95% paired intervals.

Benchmark	Label	Accuracy	Known-label utility	Predicted-label utility	$\Delta$
RouterBench	Family	99.97% [-1pt] [99.95, 99.99]	0.72124 [-1pt] [0.71720, 0.72523]	0.72117 [-1pt] [0.71714, 0.72520]	$-$ - 0.00007 [-1pt] [ $-$ - 0.00018, $+$ 0.00001]
RouterBench	Cell	93.85% [-1pt] [93.56, 94.14]	0.72288 [-1pt] [0.71868, 0.72713]	0.72232 [-1pt] [0.71819, 0.72646]	$-$ - 0.00056 [-1pt] [ $-$ - 0.00095, $-$ - 0.00020]
SPROUT	Family	99.91% [-1pt] [99.88, 99.94]	0.80966 [-1pt] [0.80609, 0.81320]	0.80955 [-1pt] [0.80597, 0.81312]	$-$ - 0.00012 [-1pt] [ $-$ - 0.00026, 0.00000]
SPROUT	Cell	91.68% [-1pt] [91.38, 91.96]	0.81225 [-1pt] [0.80881, 0.81584]	0.81182 [-1pt] [0.80843, 0.81533]	$-$ - 0.00042 [-1pt] [ $-$ - 0.00100, $+$ 0.00020]

Appendix

Table 17: Lookup with predicted labels, with 95% intervals. Utilities average five splits and thirteen penalties;

\Delta

is predicted-label minus known-label utility.

GTE embeddings
Method	RouterBench	SPROUT
Best single model	0.6618 [-1pt] [0.6586, 0.6652]	0.7575 [-1pt] [0.7544, 0.7604]
kNN-100	0.6615 [-1pt] [0.6583, 0.6651]	0.7529 [-1pt] [0.7498, 0.7559]
Validated kNN	0.6566 [-1pt] [0.6534, 0.6601]	0.7534 [-1pt] [0.7504, 0.7564]
Avengers-Pro	0.6662 [-1pt] [0.6631, 0.6698]	0.7439 [-1pt] [0.7408, 0.7470]
KM-Prox (fixed)	0.6764 [-1pt] [0.6732, 0.6797]	0.7460 [-1pt] [0.7429, 0.7491]

Appendix

Table 18: Mean held-out utility across thirteen cost penalties, with 95% intervals. Bold marks the highest mean among the listed methods.

Method / $\lambda$	0	1	2	5	10	Mean
GTE / Avengers-Pro	0.7913 [-1pt] [0.7869, 0.7957]	0.7888 [-1pt] [0.7844, 0.7932]	0.7863 [-1pt] [0.7819, 0.7907]	0.7788 [-1pt] [0.7744, 0.7833]	0.7664 [-1pt] [0.7618, 0.7709]	0.7176 [-1pt] [0.7134, 0.7219]
GTE / Best single model	0.7922 [-1pt] [0.7879, 0.7967]	0.7890 [-1pt] [0.7846, 0.7934]	0.7857 [-1pt] [0.7813, 0.7902]	0.7759 [-1pt] [0.7715, 0.7805]	0.7596 [-1pt] [0.7550, 0.7641]	0.6994 [-1pt] [0.6951, 0.7035]
GTE / Family lookup	0.7925 [-1pt] [0.7880, 0.7969]	0.7895 [-1pt] [0.7850, 0.7939]	0.7867 [-1pt] [0.7822, 0.7912]	0.7785 [-1pt] [0.7739, 0.7830]	0.7708 [-1pt] [0.7661, 0.7753]	0.7212 [-1pt] [0.7172, 0.7252]
GTE / Family-cell lookup	0.7916 [-1pt] [0.7870, 0.7960]	0.7884 [-1pt] [0.7839, 0.7929]	0.7855 [-1pt] [0.7810, 0.7900]	0.7775 [-1pt] [0.7730, 0.7820]	0.7697 [-1pt] [0.7650, 0.7742]	0.7229 [-1pt] [0.7187, 0.7271]
GTE / KM-Prox	0.7936 [-1pt] [0.7892, 0.7980]	0.7906 [-1pt] [0.7861, 0.7950]	0.7876 [-1pt] [0.7831, 0.7920]	0.7799 [-1pt] [0.7754, 0.7843]	0.7733 [-1pt] [0.7687, 0.7779]	0.7287 [-1pt] [0.7246, 0.7329]
GTE / KM-Prox (validated)	0.7949 [-1pt] [0.7904, 0.7993]	0.7914 [-1pt] [0.7870, 0.7960]	0.7884 [-1pt] [0.7840, 0.7928]	0.7807 [-1pt] [0.7762, 0.7852]	0.7735 [-1pt] [0.7690, 0.7780]	0.7289 [-1pt] [0.7248, 0.7330]

Appendix

Table 19: RouterBench in-distribution utility with 95% intervals, part 1 of 3. Mean averages all thirteen penalties.

Method / $\lambda$	20	30	50	75	Mean
GTE / Avengers-Pro	0.7407 [-1pt] [0.7360, 0.7453]	0.7188 [-1pt] [0.7139, 0.7237]	0.7022 [-1pt] [0.6974, 0.7071]	0.6937 [-1pt] [0.6887, 0.6987]	0.7176 [-1pt] [0.7134, 0.7219]
GTE / Best single model	0.7269 [-1pt] [0.7221, 0.7316]	0.6942 [-1pt] [0.6893, 0.6991]	0.6537 [-1pt] [0.6482, 0.6591]	0.6490 [-1pt] [0.6435, 0.6545]	0.6994 [-1pt] [0.6951, 0.7035]
GTE / Family lookup	0.7568 [-1pt] [0.7522, 0.7614]	0.7429 [-1pt] [0.7384, 0.7474]	0.7157 [-1pt] [0.7109, 0.7204]	0.6958 [-1pt] [0.6908, 0.7006]	0.7212 [-1pt] [0.7172, 0.7252]
GTE / Family-cell lookup	0.7561 [-1pt] [0.7515, 0.7608]	0.7424 [-1pt] [0.7378, 0.7470]	0.7172 [-1pt] [0.7123, 0.7219]	0.6993 [-1pt] [0.6941, 0.7043]	0.7229 [-1pt] [0.7187, 0.7271]
GTE / KM-Prox	0.7615 [-1pt] [0.7570, 0.7659]	0.7508 [-1pt] [0.7461, 0.7554]	0.7301 [-1pt] [0.7253, 0.7346]	0.7082 [-1pt] [0.7033, 0.7128]	0.7287 [-1pt] [0.7246, 0.7329]
GTE / KM-Prox (validated)	0.7610 [-1pt] [0.7565, 0.7655]	0.7510 [-1pt] [0.7463, 0.7555]	0.7313 [-1pt] [0.7266, 0.7359]	0.7080 [-1pt] [0.7032, 0.7126]	0.7289 [-1pt] [0.7248, 0.7330]

Appendix

Table 20: RouterBench in-distribution utility with 95% intervals, part 2 of 3. Mean averages all thirteen penalties.

Method / $\lambda$	100	150	250	500	Mean
GTE / Avengers-Pro	0.6824 [-1pt] [0.6773, 0.6875]	0.6620 [-1pt] [0.6571, 0.6672]	0.6376 [-1pt] [0.6325, 0.6427]	0.5794 [-1pt] [0.5740, 0.5850]	0.7176 [-1pt] [0.7134, 0.7219]
GTE / Best single model	0.6443 [-1pt] [0.6388, 0.6498]	0.6350 [-1pt] [0.6295, 0.6405]	0.6164 [-1pt] [0.6108, 0.6220]	0.5697 [-1pt] [0.5641, 0.5754]	0.6994 [-1pt] [0.6951, 0.7035]
GTE / Family lookup	0.6753 [-1pt] [0.6704, 0.6802]	0.6539 [-1pt] [0.6486, 0.6593]	0.6292 [-1pt] [0.6238, 0.6348]	0.5885 [-1pt] [0.5831, 0.5941]	0.7212 [-1pt] [0.7172, 0.7252]
GTE / Family-cell lookup	0.6840 [-1pt] [0.6788, 0.6890]	0.6598 [-1pt] [0.6547, 0.6650]	0.6323 [-1pt] [0.6269, 0.6376]	0.5936 [-1pt] [0.5879, 0.5992]	0.7229 [-1pt] [0.7187, 0.7271]
GTE / KM-Prox	0.6915 [-1pt] [0.6868, 0.6965]	0.6660 [-1pt] [0.6609, 0.6712]	0.6399 [-1pt] [0.6346, 0.6452]	0.6002 [-1pt] [0.5949, 0.6057]	0.7287 [-1pt] [0.7246, 0.7329]
GTE / KM-Prox (validated)	0.6912 [-1pt] [0.6864, 0.6960]	0.6659 [-1pt] [0.6610, 0.6708]	0.6391 [-1pt] [0.6339, 0.6441]	0.5994 [-1pt] [0.5941, 0.6050]	0.7289 [-1pt] [0.7248, 0.7330]

Appendix

Table 21: RouterBench in-distribution utility with 95% intervals, part 3 of 3. Mean averages all thirteen penalties.

Method / $\lambda$	0	1	2	5	10	Mean
GTE / Avengers-Pro	0.7905 [-1pt] [0.7871, 0.7942]	0.7873 [-1pt] [0.7839, 0.7910]	0.7841 [-1pt] [0.7807, 0.7878]	0.7743 [-1pt] [0.7709, 0.7780]	0.7581 [-1pt] [0.7547, 0.7618]	0.6662 [-1pt] [0.6631, 0.6698]
GTE / Best single model	0.7921 [-1pt] [0.7887, 0.7958]	0.7889 [-1pt] [0.7854, 0.7926]	0.7856 [-1pt] [0.7822, 0.7893]	0.7758 [-1pt] [0.7724, 0.7795]	0.7596 [-1pt] [0.7561, 0.7632]	0.6618 [-1pt] [0.6586, 0.6652]
GTE / KM-Prox	0.7921 [-1pt] [0.7887, 0.7958]	0.7889 [-1pt] [0.7854, 0.7926]	0.7856 [-1pt] [0.7822, 0.7893]	0.7758 [-1pt] [0.7724, 0.7795]	0.7595 [-1pt] [0.7560, 0.7632]	0.6764 [-1pt] [0.6732, 0.6797]
GTE / KM-Prox (validated)	0.7921 [-1pt] [0.7887, 0.7958]	0.7889 [-1pt] [0.7854, 0.7926]	0.7856 [-1pt] [0.7822, 0.7893]	0.7758 [-1pt] [0.7724, 0.7795]	0.7576 [-1pt] [0.7542, 0.7613]	0.6764 [-1pt] [0.6732, 0.6798]
GTE / MLP	0.7921 [-1pt] [0.7887, 0.7958]	0.7889 [-1pt] [0.7854, 0.7926]	0.7856 [-1pt] [0.7822, 0.7893]	0.7758 [-1pt] [0.7724, 0.7795]	0.7594 [-1pt] [0.7559, 0.7631]	0.6755 [-1pt] [0.6722, 0.6789]
GTE / Realized oracle	0.9189 [-1pt] [0.9171, 0.9206]	0.9186 [-1pt] [0.9169, 0.9203]	0.9184 [-1pt] [0.9167, 0.9201]	0.9177 [-1pt] [0.9160, 0.9194]	0.9166 [-1pt] [0.9148, 0.9182]	0.9030 [-1pt] [0.9012, 0.9047]

Appendix

Table 22: RouterBench out-of-distribution utility with 95% intervals, part 1 of 3. Mean averages all thirteen penalties.

Method / $\lambda$	20	30	50	75	Mean
GTE / Avengers-Pro	0.7250 [-1pt] [0.7216, 0.7288]	0.6259 [-1pt] [0.6219, 0.6303]	0.5724 [-1pt] [0.5684, 0.5769]	0.5119 [-1pt] [0.5077, 0.5163]	0.6662 [-1pt] [0.6631, 0.6698]
GTE / Best single model	0.7270 [-1pt] [0.7235, 0.7306]	0.6484 [-1pt] [0.6445, 0.6526]	0.5445 [-1pt] [0.5399, 0.5489]	0.4959 [-1pt] [0.4911, 0.5002]	0.6618 [-1pt] [0.6586, 0.6652]
GTE / KM-Prox	0.7267 [-1pt] [0.7232, 0.7304]	0.6935 [-1pt] [0.6901, 0.6972]	0.6267 [-1pt] [0.6233, 0.6305]	0.5481 [-1pt] [0.5443, 0.5520]	0.6764 [-1pt] [0.6732, 0.6797]
GTE / KM-Prox (validated)	0.7243 [-1pt] [0.7208, 0.7281]	0.6923 [-1pt] [0.6888, 0.6961]	0.6292 [-1pt] [0.6256, 0.6330]	0.5527 [-1pt] [0.5489, 0.5567]	0.6764 [-1pt] [0.6732, 0.6798]
GTE / MLP	0.7216 [-1pt] [0.7180, 0.7254]	0.6778 [-1pt] [0.6741, 0.6816]	0.5894 [-1pt] [0.5853, 0.5936]	0.5258 [-1pt] [0.5212, 0.5302]	0.6755 [-1pt] [0.6722, 0.6789]
GTE / Realized oracle	0.9142 [-1pt] [0.9125, 0.9159]	0.9120 [-1pt] [0.9103, 0.9138]	0.9080 [-1pt] [0.9063, 0.9098]	0.9037 [-1pt] [0.9019, 0.9054]	0.9030 [-1pt] [0.9012, 0.9047]

Appendix

Table 23: RouterBench out-of-distribution utility with 95% intervals, part 2 of 3. Mean averages all thirteen penalties.

Method / $\lambda$	100	150	250	500	Mean
GTE / Avengers-Pro	0.6037 [-1pt] [0.5995, 0.6083]	0.5994 [-1pt] [0.5948, 0.6041]	0.5800 [-1pt] [0.5754, 0.5846]	0.5483 [-1pt] [0.5437, 0.5529]	0.6662 [-1pt] [0.6631, 0.6698]
GTE / Best single model	0.5983 [-1pt] [0.5936, 0.6028]	0.5884 [-1pt] [0.5836, 0.5928]	0.6167 [-1pt] [0.6124, 0.6212]	0.4822 [-1pt] [0.4776, 0.4870]	0.6618 [-1pt] [0.6586, 0.6652]
GTE / KM-Prox	0.4796 [-1pt] [0.4755, 0.4837]	0.6351 [-1pt] [0.6308, 0.6397]	0.6139 [-1pt] [0.6094, 0.6185]	0.5669 [-1pt] [0.5624, 0.5715]	0.6764 [-1pt] [0.6732, 0.6797]
GTE / KM-Prox (validated)	0.4747 [-1pt] [0.4706, 0.4790]	0.6347 [-1pt] [0.6302, 0.6391]	0.6163 [-1pt] [0.6119, 0.6208]	0.5690 [-1pt] [0.5646, 0.5735]	0.6764 [-1pt] [0.6732, 0.6798]
GTE / MLP	0.6171 [-1pt] [0.6126, 0.6218]	0.6120 [-1pt] [0.6075, 0.6167]	0.5921 [-1pt] [0.5876, 0.5966]	0.5432 [-1pt] [0.5385, 0.5479]	0.6755 [-1pt] [0.6722, 0.6789]
GTE / Realized oracle	0.8997 [-1pt] [0.8979, 0.9015]	0.8921 [-1pt] [0.8903, 0.8939]	0.8771 [-1pt] [0.8751, 0.8789]	0.8414 [-1pt] [0.8393, 0.8435]	0.9030 [-1pt] [0.9012, 0.9047]

Appendix

Table 24: RouterBench out-of-distribution utility with 95% intervals, part 3 of 3. Mean averages all thirteen penalties.

Method / $\lambda$	0	0.5	1	2	5	Mean
GTE / Avengers-Pro	0.8648 [-1pt] [0.8612, 0.8683]	0.8632 [-1pt] [0.8596, 0.8667]	0.8618 [-1pt] [0.8582, 0.8654]	0.8593 [-1pt] [0.8558, 0.8628]	0.8536 [-1pt] [0.8500, 0.8572]	0.8138 [-1pt] [0.8104, 0.8173]
GTE / Best single model	0.8434 [-1pt] [0.8395, 0.8471]	0.8409 [-1pt] [0.8370, 0.8446]	0.8385 [-1pt] [0.8346, 0.8422]	0.8336 [-1pt] [0.8296, 0.8373]	0.8188 [-1pt] [0.8148, 0.8227]	0.7868 [-1pt] [0.7831, 0.7908]
GTE / Family lookup	0.8614 [-1pt] [0.8577, 0.8650]	0.8596 [-1pt] [0.8559, 0.8632]	0.8580 [-1pt] [0.8544, 0.8616]	0.8546 [-1pt] [0.8509, 0.8583]	0.8504 [-1pt] [0.8464, 0.8543]	0.8097 [-1pt] [0.8061, 0.8132]
GTE / Family-cell lookup	0.8626 [-1pt] [0.8589, 0.8662]	0.8605 [-1pt] [0.8568, 0.8641]	0.8591 [-1pt] [0.8554, 0.8627]	0.8553 [-1pt] [0.8515, 0.8591]	0.8510 [-1pt] [0.8473, 0.8549]	0.8122 [-1pt] [0.8088, 0.8158]
GTE / KM-Prox	0.8635 [-1pt] [0.8599, 0.8670]	0.8620 [-1pt] [0.8585, 0.8655]	0.8604 [-1pt] [0.8567, 0.8638]	0.8584 [-1pt] [0.8549, 0.8618]	0.8532 [-1pt] [0.8495, 0.8569]	0.8148 [-1pt] [0.8114, 0.8183]
GTE / KM-Prox (validated)	0.8651 [-1pt] [0.8616, 0.8685]	0.8642 [-1pt] [0.8606, 0.8676]	0.8625 [-1pt] [0.8590, 0.8659]	0.8611 [-1pt] [0.8575, 0.8646]	0.8548 [-1pt] [0.8512, 0.8585]	0.8166 [-1pt] [0.8132, 0.8200]

Appendix

Table 25: SPROUT in-distribution utility with 95% intervals, part 1 of 3. Mean averages all thirteen penalties.

Method / $\lambda$	10	25	50	75	Mean
GTE / Avengers-Pro	0.8446 [-1pt] [0.8410, 0.8482]	0.8337 [-1pt] [0.8298, 0.8375]	0.8188 [-1pt] [0.8150, 0.8228]	0.8068 [-1pt] [0.8030, 0.8108]	0.8138 [-1pt] [0.8104, 0.8173]
GTE / Best single model	0.8015 [-1pt] [0.7975, 0.8059]	0.7973 [-1pt] [0.7929, 0.8018]	0.7889 [-1pt] [0.7845, 0.7934]	0.7804 [-1pt] [0.7760, 0.7850]	0.7868 [-1pt] [0.7831, 0.7908]
GTE / Family lookup	0.8436 [-1pt] [0.8395, 0.8476]	0.8269 [-1pt] [0.8229, 0.8310]	0.8152 [-1pt] [0.8112, 0.8194]	0.8037 [-1pt] [0.7997, 0.8079]	0.8097 [-1pt] [0.8061, 0.8132]
GTE / Family-cell lookup	0.8448 [-1pt] [0.8411, 0.8488]	0.8335 [-1pt] [0.8296, 0.8374]	0.8205 [-1pt] [0.8165, 0.8245]	0.8063 [-1pt] [0.8024, 0.8104]	0.8122 [-1pt] [0.8088, 0.8158]
GTE / KM-Prox	0.8466 [-1pt] [0.8429, 0.8504]	0.8342 [-1pt] [0.8304, 0.8382]	0.8166 [-1pt] [0.8126, 0.8207]	0.8056 [-1pt] [0.8016, 0.8097]	0.8148 [-1pt] [0.8114, 0.8183]
GTE / KM-Prox (validated)	0.8477 [-1pt] [0.8440, 0.8515]	0.8370 [-1pt] [0.8332, 0.8408]	0.8204 [-1pt] [0.8165, 0.8243]	0.8066 [-1pt] [0.8027, 0.8108]	0.8166 [-1pt] [0.8132, 0.8200]

Appendix

Table 26: SPROUT in-distribution utility with 95% intervals, part 2 of 3. Mean averages all thirteen penalties.

Method / $\lambda$	100	150	250	500	Mean
GTE / Avengers-Pro	0.7943 [-1pt] [0.7904, 0.7983]	0.7731 [-1pt] [0.7693, 0.7773]	0.7367 [-1pt] [0.7325, 0.7411]	0.6689 [-1pt] [0.6643, 0.6738]	0.8138 [-1pt] [0.8104, 0.8173]
GTE / Best single model	0.7720 [-1pt] [0.7676, 0.7766]	0.7551 [-1pt] [0.7507, 0.7598]	0.7214 [-1pt] [0.7168, 0.7262]	0.6371 [-1pt] [0.6321, 0.6423]	0.7868 [-1pt] [0.7831, 0.7908]
GTE / Family lookup	0.7927 [-1pt] [0.7886, 0.7970]	0.7709 [-1pt] [0.7667, 0.7753]	0.7313 [-1pt] [0.7270, 0.7358]	0.6574 [-1pt] [0.6524, 0.6624]	0.8097 [-1pt] [0.8061, 0.8132]
GTE / Family-cell lookup	0.7955 [-1pt] [0.7915, 0.7998]	0.7752 [-1pt] [0.7711, 0.7795]	0.7364 [-1pt] [0.7322, 0.7408]	0.6584 [-1pt] [0.6536, 0.6635]	0.8122 [-1pt] [0.8088, 0.8158]
GTE / KM-Prox	0.7948 [-1pt] [0.7908, 0.7989]	0.7764 [-1pt] [0.7724, 0.7807]	0.7434 [-1pt] [0.7391, 0.7478]	0.6780 [-1pt] [0.6735, 0.6832]	0.8148 [-1pt] [0.8114, 0.8183]
GTE / KM-Prox (validated)	0.7951 [-1pt] [0.7911, 0.7993]	0.7766 [-1pt] [0.7726, 0.7809]	0.7466 [-1pt] [0.7424, 0.7510]	0.6777 [-1pt] [0.6731, 0.6826]	0.8166 [-1pt] [0.8132, 0.8200]

Appendix

Table 27: SPROUT in-distribution utility with 95% intervals, part 3 of 3. Mean averages all thirteen penalties.

Method / $\lambda$	0	0.5	1	2	5	Mean
GTE / Avengers-Pro	0.7891 [-1pt] [0.7857, 0.7924]	0.7883 [-1pt] [0.7849, 0.7916]	0.7875 [-1pt] [0.7841, 0.7908]	0.7874 [-1pt] [0.7840, 0.7907]	0.7830 [-1pt] [0.7796, 0.7864]	0.7439 [-1pt] [0.7408, 0.7470]
GTE / Best single model	0.8445 [-1pt] [0.8414, 0.8475]	0.8420 [-1pt] [0.8389, 0.8451]	0.8040 [-1pt] [0.8008, 0.8070]	0.8010 [-1pt] [0.7978, 0.8041]	0.7669 [-1pt] [0.7634, 0.7704]	0.7575 [-1pt] [0.7544, 0.7604]
GTE / KM-Prox	0.7887 [-1pt] [0.7854, 0.7921]	0.7873 [-1pt] [0.7840, 0.7906]	0.7861 [-1pt] [0.7828, 0.7894]	0.7837 [-1pt] [0.7803, 0.7869]	0.7808 [-1pt] [0.7774, 0.7841]	0.7460 [-1pt] [0.7429, 0.7491]
GTE / KM-Prox (validated)	0.7968 [-1pt] [0.7934, 0.8001]	0.7948 [-1pt] [0.7914, 0.7981]	0.7929 [-1pt] [0.7896, 0.7962]	0.7889 [-1pt] [0.7855, 0.7921]	0.7765 [-1pt] [0.7731, 0.7798]	0.7483 [-1pt] [0.7453, 0.7513]
GTE / MLP	0.8088 [-1pt] [0.8055, 0.8121]	0.8056 [-1pt] [0.8022, 0.8089]	0.8125 [-1pt] [0.8092, 0.8158]	0.8071 [-1pt] [0.8037, 0.8104]	0.7971 [-1pt] [0.7936, 0.8005]	0.7564 [-1pt] [0.7536, 0.7592]
GTE / Realized oracle	0.9766 [-1pt] [0.9753, 0.9778]	0.9764 [-1pt] [0.9751, 0.9776]	0.9762 [-1pt] [0.9750, 0.9774]	0.9758 [-1pt] [0.9746, 0.9770]	0.9747 [-1pt] [0.9734, 0.9759]	0.9601 [-1pt] [0.9588, 0.9613]

Appendix

Table 28: SPROUT out-of-distribution utility with 95% intervals, part 1 of 3. Mean averages all thirteen penalties.

Method / $\lambda$	10	25	50	75	Mean
GTE / Avengers-Pro	0.7822 [-1pt] [0.7789, 0.7856]	0.7799 [-1pt] [0.7765, 0.7832]	0.7663 [-1pt] [0.7629, 0.7698]	0.7548 [-1pt] [0.7514, 0.7582]	0.7439 [-1pt] [0.7408, 0.7470]
GTE / Best single model	0.7576 [-1pt] [0.7542, 0.7611]	0.7641 [-1pt] [0.7607, 0.7676]	0.7546 [-1pt] [0.7511, 0.7581]	0.7451 [-1pt] [0.7416, 0.7486]	0.7575 [-1pt] [0.7544, 0.7604]
GTE / KM-Prox	0.7796 [-1pt] [0.7762, 0.7829]	0.7725 [-1pt] [0.7690, 0.7759]	0.7579 [-1pt] [0.7543, 0.7614]	0.7437 [-1pt] [0.7400, 0.7471]	0.7460 [-1pt] [0.7429, 0.7491]
GTE / KM-Prox (validated)	0.7657 [-1pt] [0.7622, 0.7690]	0.7699 [-1pt] [0.7665, 0.7732]	0.7577 [-1pt] [0.7543, 0.7611]	0.7443 [-1pt] [0.7409, 0.7478]	0.7483 [-1pt] [0.7453, 0.7513]
GTE / MLP	0.7908 [-1pt] [0.7873, 0.7940]	0.7744 [-1pt] [0.7712, 0.7776]	0.7452 [-1pt] [0.7416, 0.7487]	0.7354 [-1pt] [0.7318, 0.7389]	0.7564 [-1pt] [0.7536, 0.7592]
GTE / Realized oracle	0.9730 [-1pt] [0.9717, 0.9742]	0.9692 [-1pt] [0.9680, 0.9705]	0.9642 [-1pt] [0.9629, 0.9655]	0.9595 [-1pt] [0.9582, 0.9608]	0.9601 [-1pt] [0.9588, 0.9613]

Appendix

Table 29: SPROUT out-of-distribution utility with 95% intervals, part 2 of 3. Mean averages all thirteen penalties.

Method / $\lambda$	100	150	250	500	Mean
GTE / Avengers-Pro	0.7423 [-1pt] [0.7390, 0.7458]	0.7175 [-1pt] [0.7141, 0.7211]	0.6413 [-1pt] [0.6375, 0.6449]	0.5510 [-1pt] [0.5469, 0.5550]	0.7439 [-1pt] [0.7408, 0.7470]
GTE / Best single model	0.7356 [-1pt] [0.7321, 0.7392]	0.7166 [-1pt] [0.7131, 0.7203]	0.7218 [-1pt] [0.7181, 0.7255]	0.5930 [-1pt] [0.5890, 0.5971]	0.7575 [-1pt] [0.7544, 0.7604]
GTE / KM-Prox	0.7319 [-1pt] [0.7282, 0.7354]	0.7123 [-1pt] [0.7086, 0.7159]	0.6822 [-1pt] [0.6785, 0.6859]	0.5908 [-1pt] [0.5866, 0.5951]	0.7460 [-1pt] [0.7429, 0.7491]
GTE / KM-Prox (validated)	0.7330 [-1pt] [0.7295, 0.7364]	0.7215 [-1pt] [0.7180, 0.7251]	0.6907 [-1pt] [0.6870, 0.6944]	0.5951 [-1pt] [0.5909, 0.5990]	0.7483 [-1pt] [0.7453, 0.7513]
GTE / MLP	0.7313 [-1pt] [0.7277, 0.7348]	0.7164 [-1pt] [0.7129, 0.7201]	0.7218 [-1pt] [0.7181, 0.7255]	0.5863 [-1pt] [0.5821, 0.5905]	0.7564 [-1pt] [0.7536, 0.7592]
GTE / Realized oracle	0.9549 [-1pt] [0.9535, 0.9562]	0.9463 [-1pt] [0.9449, 0.9477]	0.9316 [-1pt] [0.9300, 0.9330]	0.9026 [-1pt] [0.9008, 0.9042]	0.9601 [-1pt] [0.9588, 0.9613]

Appendix

Table 30: SPROUT out-of-distribution utility with 95% intervals, part 3 of 3. Mean averages all thirteen penalties.

Method	RouterBench	SPROUT
SharedTrunkNet: fixed L21	0.6841 [-1pt] [0.6809, 0.6875]	0.7812 [-1pt] [0.7783, 0.7841]
SharedTrunkNet: common selected layer	0.6803 [-1pt] [0.6771, 0.6837]	0.7806 [-1pt] [0.7777, 0.7835]
SharedTrunkNet: per-model selected layers	0.6949 [-1pt] [0.6918, 0.6983]	0.7829 [-1pt] [0.7800, 0.7857]
L21 kNN-100	0.7007 [-1pt] [0.6972, 0.7043]	0.7731 [-1pt] [0.7701, 0.7760]
L21 Family Adaptation	0.7157 [-1pt] [0.7125, 0.7191]	0.7767 [-1pt] [0.7737, 0.7797]

Appendix

Table 31: Out-of-distribution mean utility over thirteen penalties, with 95% intervals.

Comparison	RouterBench	SPROUT
L21 kNN-100 minus SharedTrunkNet	$+$ 0.0058 [-1pt] [ $+$ 0.0043, $+$ 0.0072]	$-$ - 0.0097 [-1pt] [ $-$ - 0.0112, $-$ - 0.0083]
L21 Family Adaptation minus SharedTrunkNet	$+$ 0.0208 [-1pt] [ $+$ 0.0195, $+$ 0.0221]	$-$ - 0.0062 [-1pt] [ $-$ - 0.0073, $-$ - 0.0050]

Appendix

Table 32: Mean utility difference from SharedTrunkNet, with paired 95% intervals. Positive values favor the L21 method.

$\lambda$	SharedTrunkNet	L21 kNN-100	L21 Family Adaptation
0	0.7916 [-1pt] [0.7882, 0.7952]	0.7854 [-1pt] [0.7818, 0.7891]	0.7921 [-1pt] [0.7887, 0.7958]
1	0.7887 [-1pt] [0.7853, 0.7925]	0.7805 [-1pt] [0.7769, 0.7843]	0.7889 [-1pt] [0.7854, 0.7926]
2	0.7855 [-1pt] [0.7820, 0.7891]	0.7771 [-1pt] [0.7734, 0.7809]	0.7856 [-1pt] [0.7822, 0.7893]
5	0.7754 [-1pt] [0.7719, 0.7790]	0.7678 [-1pt] [0.7642, 0.7716]	0.7759 [-1pt] [0.7724, 0.7796]
10	0.7585 [-1pt] [0.7551, 0.7622]	0.7587 [-1pt] [0.7549, 0.7626]	0.7661 [-1pt] [0.7624, 0.7698]
20	0.7207 [-1pt] [0.7171, 0.7246]	0.7439 [-1pt] [0.7401, 0.7478]	0.7523 [-1pt] [0.7486, 0.7560]

Appendix

Table 33: RouterBench: out-of-distribution utility at every penalty, with 95% intervals.

$\lambda$	SharedTrunkNet	L21 kNN-100	L21 Family Adaptation
0	0.8212 [-1pt] [0.8179, 0.8244]	0.8243 [-1pt] [0.8212, 0.8275]	0.8163 [-1pt] [0.8130, 0.8196]
0.5	0.8321 [-1pt] [0.8291, 0.8354]	0.8225 [-1pt] [0.8192, 0.8257]	0.8129 [-1pt] [0.8095, 0.8161]
1	0.8145 [-1pt] [0.8113, 0.8178]	0.8200 [-1pt] [0.8169, 0.8232]	0.8094 [-1pt] [0.8061, 0.8126]
2	0.8091 [-1pt] [0.8059, 0.8124]	0.8170 [-1pt] [0.8137, 0.8202]	0.8025 [-1pt] [0.7992, 0.8058]
5	0.8118 [-1pt] [0.8085, 0.8150]	0.8073 [-1pt] [0.8040, 0.8106]	0.8070 [-1pt] [0.8035, 0.8103]
10	0.8155 [-1pt] [0.8122, 0.8188]	0.8009 [-1pt] [0.7974, 0.8042]	0.8085 [-1pt] [0.8050, 0.8117]

Appendix

Table 34: SPROUT: out-of-distribution utility at every penalty, with 95% intervals.

Family / aggregation	Queries	GTE	L21	$\Delta$
ARC-Challenge	1,470	0.9080 [-1pt] [0.8979, 0.9177]	0.9126 [-1pt] [0.9015, 0.9232]	+0.0046 [-1pt] [-0.0021, 0.0107]
GSM8K	7,450	0.4125 [-1pt] [0.4088, 0.4162]	0.5723 [-1pt] [0.5690, 0.5755]	+0.1598 [-1pt] [0.1567, 0.1628]
HellaSwag	10,042	0.7220 [-1pt] [0.7155, 0.7288]	0.7263 [-1pt] [0.7194, 0.7330]	+0.0043 [-1pt] [0.0006, 0.0081]
MBPP	425	0.5316 [-1pt] [0.4922, 0.5700]	0.5730 [-1pt] [0.5323, 0.6112]	+0.0414 [-1pt] [0.0123, 0.0703]
MMLU	14,037	0.7253 [-1pt] [0.7192, 0.7316]	0.7287 [-1pt] [0.7223, 0.7355]	+0.0034 [-1pt] [0.0002, 0.0066]
MT-Bench	80	0.3995 [-1pt] [0.3170, 0.4805]	0.5922 [-1pt] [0.5212, 0.6589]	+0.1927 [-1pt] [0.1155, 0.2836]

Appendix

Table 35: RouterBench: kNN-100 utility by held-out family and under both averaging schemes, with 95% intervals.

\Delta

is L21 minus GTE.

Family / aggregation	Queries	GTE	L21	$\Delta$
GPQA	540	0.4037 [-1pt] [0.3692, 0.4384]	0.4715 [-1pt] [0.4399, 0.5038]	+0.0678 [-1pt] [0.0375, 0.0989]
MATH	9,884	0.7601 [-1pt] [0.7526, 0.7675]	0.7635 [-1pt] [0.7561, 0.7709]	+0.0034 [-1pt] [0.0003, 0.0067]
MMLU-Pro	11,786	0.6698 [-1pt] [0.6628, 0.6772]	0.6696 [-1pt] [0.6626, 0.6767]	-0.0002 [-1pt] [-0.0041, 0.0035]
MuSR	748	0.5500 [-1pt] [0.5216, 0.5785]	0.5149 [-1pt] [0.4840, 0.5460]	-0.0351 [-1pt] [-0.0550, -0.0153]
OpenHermes	19,340	0.8092 [-1pt] [0.8060, 0.8123]	0.8505 [-1pt] [0.8472, 0.8538]	+0.0414 [-1pt] [0.0391, 0.0437]
RAGBench	1,589	0.8539 [-1pt] [0.8435, 0.8646]	0.8831 [-1pt] [0.8732, 0.8926]	+0.0292 [-1pt] [0.0210, 0.0370]

Appendix

Table 36: SPROUT: kNN-100 utility by held-out family and under both averaging schemes, with 95% intervals.

\Delta

is L21 minus GTE.

	RouterBench			SPROUT
Layer	kNN-100	Validated kNN	Family Adaptation	kNN-100	Validated kNN	Family Adaptation
GTE	0.6615	0.6566	0.7006	0.7529	0.7534	0.7603
15	0.6933	0.6895	0.7157	0.7671	0.7644	0.7592
16	0.6865	0.6878	0.7153	0.7667	0.7647	0.7592
17	0.6890	0.6897	0.7165	0.7665	0.7645	0.7593
18	0.7013	0.6977	0.7170	0.7673	0.7645	0.7610

Appendix

Table 37: Held-out-family mean utility over all thirteen penalties.

	RouterBench			SPROUT
Layer	kNN-100	Validated kNN	Family Adaptation	kNN-100	Validated kNN	Family Adaptation
21	0.7352	0.7360	0.7217	0.8206	0.8212	0.8096
23	0.7356	0.7366	0.7218	0.8202	0.8209	0.8097
25	0.7354	0.7355	0.7219	0.8203	0.8211	0.8096

Appendix

Table 38: ID mean utility over five splits and thirteen penalties.

Benchmark	Predictor	Reward MSE	Reward MAE	Cost MSE	Cost MAE
RouterBench	GTE	0.2061 [-1pt] [0.2053, 0.2070]	0.4077 [-1pt] [0.4068, 0.4086]	2.5186 [-1pt] [2.4758, 2.5611]	0.5757 [-1pt] [0.5725, 0.5788]
RouterBench	L21	0.1780 [-1pt] [0.1770, 0.1791]	0.3383 [-1pt] [0.3370, 0.3395]	1.0323 [-1pt] [1.0077, 1.0580]	0.3185 [-1pt] [0.3160, 0.3208]
RouterBench	GTE, matching format	0.1902 [-1pt] [0.1894, 0.1909]	0.3852 [-1pt] [0.3844, 0.3860]	0.8648 [-1pt] [0.8402, 0.8895]	0.3406 [-1pt] [0.3382, 0.3428]
RouterBench	Training-model means	0.1899 [-1pt] [0.1893, 0.1905]	0.3943 [-1pt] [0.3937, 0.3949]	2.8637 [-1pt] [2.8255, 2.9016]	0.7833 [-1pt] [0.7806, 0.7863]
SPROUT	GTE	0.2073 [-1pt] [0.2064, 0.2082]	0.4031 [-1pt] [0.4020, 0.4042]	4.8932 [-1pt] [4.5794, 5.2115]	0.7142 [-1pt] [0.7086, 0.7199]
SPROUT	L21	0.1614 [-1pt] [0.1604, 0.1624]	0.3221 [-1pt] [0.3209, 0.3234]	5.0209 [-1pt] [4.7157, 5.3337]	0.7436 [-1pt] [0.7377, 0.7493]

Appendix

Table 39: Reward and cost prediction errors, with 95% intervals. Cost MSE and MAE are multiplied by

10^{6}

and

10^{3}

, respectively, for readability.

Target	Error	Centering	RouterBench	SPROUT
Reward	MSE	None	$+$ 13.64 [-1pt] [ $+$ 13.23, $+$ 14.05]	$+$ 22.15 [-1pt] [ $+$ 21.67, $+$ 22.64]
Reward	MAE	None	$+$ 17.03 [-1pt] [ $+$ 16.79, $+$ 17.27]	$+$ 20.07 [-1pt] [ $+$ 19.74, $+$ 20.41]
Reward	MSE	Across models	$+$ 5.84 [-1pt] [ $+$ 5.58, $+$ 6.10]	$+$ 2.43 [-1pt] [ $+$ 2.26, $+$ 2.59]
Reward	MAE	Across models	$+$ 6.25 [-1pt] [ $+$ 6.07, $+$ 6.44]	$+$ 4.01 [-1pt] [ $+$ 3.89, $+$ 4.14]
Cost	MSE	None	$+$ 59.01 [-1pt] [ $+$ 58.11, $+$ 59.95]	$-$ - 2.61 [-1pt] [ $-$ - 3.21, $-$ - 2.03]
Cost	MAE	None	$+$ 44.67 [-1pt] [ $+$ 44.19, $+$ 45.21]	$-$ - 4.11 [-1pt] [ $-$ - 4.42, $-$ - 3.78]

Appendix

Table 40: L21 error reduction relative to GTE (%), with paired 95% intervals. Negative values indicate higher error.

Reward estimate	Cost estimate	RouterBench	SPROUT
GTE	GTE	0.6615 [-1pt] [0.6583, 0.6651]	0.7529 [-1pt] [0.7498, 0.7559]
L21	GTE	0.6754 [-1pt] [0.6720, 0.6791]	0.7709 [-1pt] [0.7679, 0.7739]
GTE	L21	0.6965 [-1pt] [0.6934, 0.7000]	0.7560 [-1pt] [0.7529, 0.7590]
L21	L21	0.7007 [-1pt] [0.6972, 0.7043]	0.7731 [-1pt] [0.7701, 0.7760]
GTE with matching answer format		0.6974 [-1pt] [0.6942, 0.7010]	0.7584 [-1pt] [0.7553, 0.7614]
Training-model means		0.6618 [-1pt] [0.6586, 0.6652]	0.7575 [-1pt] [0.7544, 0.7604]

Appendix

Table 41: Mean out-of-distribution utility over thirteen penalties, with 95% intervals.

Comparison	RouterBench	SPROUT
Replace GTE reward estimate with L21	$+$ 0.0139 [-1pt] [ $+$ 0.0120, $+$ 0.0156]	$+$ 0.0180 [-1pt] [ $+$ 0.0163, $+$ 0.0198]
Replace GTE cost estimate with L21	$+$ 0.0350 [-1pt] [ $+$ 0.0342, $+$ 0.0357]	$+$ 0.0031 [-1pt] [ $+$ 0.0027, $+$ 0.0036]
Replace both GTE estimates with L21	$+$ 0.0391 [-1pt] [ $+$ 0.0371, $+$ 0.0410]	$+$ 0.0202 [-1pt] [ $+$ 0.0185, $+$ 0.0220]
GTE with matching format minus GTE	$+$ 0.0359 [-1pt] [ $+$ 0.0350, $+$ 0.0368]	$+$ 0.0055 [-1pt] [ $+$ 0.0042, $+$ 0.0069]
GTE with matching format minus L21	$-$ - 0.0032 [-1pt] [ $-$ - 0.0050, $-$ - 0.0015]	$-$ - 0.0147 [-1pt] [ $-$ - 0.0163, $-$ - 0.0131]

Appendix

Table 42: Mean utility differences, with paired 95% intervals. GTE and L21 alone denote ordinary kNN-100.

$\lambda$	GTE/GTE	L21/GTE	GTE/L21	L21/L21	GTE, matching format
0	0.7900 [-1pt] [0.7866, 0.7937]	0.7854 [-1pt] [0.7818, 0.7891]	0.7900 [-1pt] [0.7866, 0.7937]	0.7854 [-1pt] [0.7818, 0.7891]	0.7902 [-1pt] [0.7868, 0.7939]
1	0.7864 [-1pt] [0.7831, 0.7902]	0.7804 [-1pt] [0.7768, 0.7842]	0.7865 [-1pt] [0.7832, 0.7903]	0.7805 [-1pt] [0.7769, 0.7843]	0.7867 [-1pt] [0.7834, 0.7905]
2	0.7831 [-1pt] [0.7798, 0.7869]	0.7774 [-1pt] [0.7738, 0.7812]	0.7832 [-1pt] [0.7799, 0.7870]	0.7771 [-1pt] [0.7734, 0.7809]	0.7832 [-1pt] [0.7799, 0.7870]
5	0.7732 [-1pt] [0.7699, 0.7770]	0.7676 [-1pt] [0.7640, 0.7713]	0.7732 [-1pt] [0.7698, 0.7770]	0.7678 [-1pt] [0.7642, 0.7716]	0.7726 [-1pt] [0.7690, 0.7764]
10	0.7563 [-1pt] [0.7529, 0.7601]	0.7498 [-1pt] [0.7462, 0.7536]	0.7557 [-1pt] [0.7524, 0.7595]	0.7587 [-1pt] [0.7549, 0.7626]	0.7638 [-1pt] [0.7601, 0.7676]
20	0.7225 [-1pt] [0.7192, 0.7263]	0.7180 [-1pt] [0.7144, 0.7219]	0.7392 [-1pt] [0.7358, 0.7430]	0.7439 [-1pt] [0.7401, 0.7478]	0.7491 [-1pt] [0.7456, 0.7529]

Appendix

Table 43: RouterBench: utility at every penalty, with 95% intervals. The first four columns identify the reward/cost predictors; the last uses GTE neighbors matching the query answer format.

$\lambda$	GTE/GTE	L21/GTE	GTE/L21	L21/L21	GTE, matching format
0	0.8035 [-1pt] [0.8002, 0.8067]	0.8243 [-1pt] [0.8212, 0.8275]	0.8035 [-1pt] [0.8002, 0.8067]	0.8243 [-1pt] [0.8212, 0.8275]	0.7972 [-1pt] [0.7938, 0.8005]
0.5	0.8013 [-1pt] [0.7980, 0.8046]	0.8224 [-1pt] [0.8193, 0.8256]	0.8018 [-1pt] [0.7986, 0.8051]	0.8225 [-1pt] [0.8192, 0.8257]	0.7956 [-1pt] [0.7923, 0.7990]
1	0.7994 [-1pt] [0.7962, 0.8026]	0.8204 [-1pt] [0.8173, 0.8235]	0.7997 [-1pt] [0.7964, 0.8030]	0.8200 [-1pt] [0.8169, 0.8232]	0.7951 [-1pt] [0.7917, 0.7984]
2	0.7955 [-1pt] [0.7922, 0.7988]	0.8165 [-1pt] [0.8132, 0.8196]	0.7962 [-1pt] [0.7929, 0.7995]	0.8170 [-1pt] [0.8137, 0.8202]	0.7940 [-1pt] [0.7906, 0.7973]
5	0.7873 [-1pt] [0.7839, 0.7906]	0.8075 [-1pt] [0.8041, 0.8108]	0.7877 [-1pt] [0.7844, 0.7910]	0.8073 [-1pt] [0.8040, 0.8106]	0.7907 [-1pt] [0.7874, 0.7940]
10	0.7820 [-1pt] [0.7787, 0.7855]	0.8004 [-1pt] [0.7970, 0.8037]	0.7820 [-1pt] [0.7786, 0.7853]	0.8009 [-1pt] [0.7974, 0.8042]	0.7878 [-1pt] [0.7844, 0.7911]

Appendix

Table 44: SPROUT: utility at every penalty, with 95% intervals. The first four columns identify the reward/cost predictors; the last uses GTE neighbors matching the query answer format.

Explore similar work

SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing
Sep 28, 2026Vasilis Perifanis, Nikolaos Pavlidis, Symeon SymeonidisLLM Routing
Indigma Innovations · Democritus University of Thrace · Athena Research Center
Most of the LLM Routing Gap Is Task Type
Aug 24, 2026Janghoon LeeLLM Routing
Redrob
Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressions
Sep 28, 2026Yifan Lu, Qiyue Zhang, Haotian Shan +2LLM Inference Acceleration LLM Routing
Rice University

Sep 28, 2026J/K move · Enter open · S save

SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing

Vasilis Perifanis, Nikolaos Pavlidis, Symeon Symeonidis

Indigma Innovations · Democritus University of Thrace · Athena Research Center

LLM Routing

Open paper AskPDF arXiv

Large language model (LLM) routing aims to select the most suitable model for each incoming query. Most existing routers learn this decision directly from query embeddings, model representations, preference data, or clusters of similar examples. Such approaches can be effective, yet the representation used for routing rarely states what a query actually requires. We introduce SeLMRoute, a routing framework that separates the extraction of candidate-independent semantic evidence from the learning of candidate performance and the application of deployment objectives. A decision model first evaluates a set of interpretable questions about the query, such as its reasoning requirements and use of external knowledge, with each judgment retained as a probability distribution. The resulting probabilistic semantic state is used by a lightweight supervised router to estimate candidate model performance. Routing objectives are applied after performance estimation, which allows the same semantic state to support performance-oriented and cost-aware decisions. On the LLMRouterBench (15 datasets, 20 candidate models, 11,481 queries), SeLMRoute achieves an average accuracy of

72.08\% \pm 0.45

, while grouped five-fold out-of-fold evaluation reaches

72.64\%

, compared with

69.23\%

for the strongest fixed candidate. The representation achieves the highest mean performance among the evaluated semantic, dense, lexical, and domain-level representations. In a separate 13-model performance-cost setting, SeLMRoute improves performance in all five grouped splits, with a mean PerfGain of

2.66\%

. Our code is available at https://github.com/Indigma-Innovations/SeLMRoute.

Abstract

Explore similar work

SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing

Most of the LLM Routing Gap Is Task Type

Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressions