Evaluating Trajectory Features for Routing Final-Layer Attention

Yupeng Yao

cs.LGOct 7, 2026

Evaluating Trajectory Features for Routing Final-Layer Attention

Authors: Yupeng Yao

Abstract

Attention routing requires a signal that predicts the value of attention on the current prefix. We evaluate whether hidden-state extrapolation error, curvature and error change improve this prediction beyond uncertainty, one-step displacement, position and state projections. Paired executions of the final attention layer supply signed next-token loss differences in frozen SmolLM3-3B-Base and Qwen3.5-4B-Base checkpoints. Utility-supervised routers are tested on 100 held-out PG-19 books at an identical causal 20 percent invocation quota. None of six prespecified comparisons shows a positive gain after familywise correction. In Qwen3.5, a parameter-matched fixed-projection control lowers NLL by 0.00356 nats/token relative to the trajectory router (95 percent interval 0.00218 to 0.00487). Secondary results depend on the operation removed, feature location and scoring horizon; frozen thresholds also drift substantially at longer horizons. Actual selected-query execution yields small long-sequence latency reductions with increased NLL, while learned routers remain slower during cached continuation. The study identifies limits on the incremental value of these trajectory summaries and separates allocation quality from measured inference benefit.

Figures & tables

Figure 1: Paired final-attention utility and controlled routing. Both branches receive the same preceding-trunk output and retain their corresponding final feed-forward computation. Primary features use the bypass state. Every compared router receives the same signed utility supervision. The prefix-only quota uses frozen calibration thresholds and allocates 99 calls among 495 scored positions. Incoming-state routing is a separate secondary experiment.

Router	Inputs	Head	Projection
C48	$X$	289	0
J32	$X,T$	289	0
C_RP3	$X$ and three fixed state projections	289	0
S32	$X$ and learned rank-32 state projection	1,331	$32d$
S32_T	$X,T$ and learned rank-32 state projection	1,313	$32d$

Table 1: Primary feature comparisons.

X=(H_{t},M_{t},D_{t},P_{t})

and

T=(E_{t},C_{t},A_{t})

. The three comparisons per checkpoint are J32–C48, J32–C_RP3 and S32_T–S32. Fixed projections have no trainable weights.

Checkpoint	Candidate–control	Gain	95% interval	$P_{H}$
SmolLM3	J32–C48	0.437	$[-0.138,1.046]$	0.602
	J32–C_RP3	$-0.277$	$[-0.997,0.428]$	0.843
	S32_T–S32	$-0.335$	$[-0.968,0.295]$	0.843
Qwen3.5	J32–C48	0.856	$[-0.020,1.736]$	0.282
	J32–C_RP3	$-3.557$	$[-4.874,-2.182]$	0.00060
	S32_T–S32	0.426	$[-0.347,1.221]$	0.843

Table 2: All six primary contrasts. Gains and individual 95% whole-book intervals are in

10^{-3}

nats/token; positive values favor the candidate. Each estimate weights 100 books equally and averages three fixed router fits.

P_{H}

is Holm-adjusted over this complete family.

Figure 2: Signed utility of the final attention operation. Each empirical distribution uses all 99,000 scored PG-19 predictions. Blue compares full attention with bypass; orange compares full with local32 final attention. Negative utilities are retained. Display limits clip the outer 0.5% of pooled values per checkpoint; complete utilities are supplied in the review data. Earlier layers remain contextualized. Tokens describe the distributions; books are the inferential units.

Figure 3: Threshold drift and secondary feature comparisons. Top panels compare actual unforced invocation rates with their nominal calibration rates at 512 (solid) and 4,096 (dashed) predictions. Bottom panels show 20%-quota contrasts, including incoming-state additions (triangles), with individual unadjusted 95% book/article-bootstrap intervals. PG-19 has 100 core books and 99 eligible books in the overlapping horizon shift; WikiText-2 has 45 SmolLM3 and 46 Qwen3.5 test articles. Token NLL is not pooled across tokenizers.

Figure 4: Executed quality and latency. Columns are checkpoints; rows show scoring at 512 and 4,096 predictions and 64 cached forced-input computations after a 1,024-token prompt. Horizontal values divide native automatic full median latency by policy latency; values above one indicate faster execution. Intervals are descriptive, unadjusted 95% paired-bootstrap intervals from 30 blocks. Vertical values are executed policy NLL minus native full NLL. Scoring quality uses the complete locked roster; cached quality uses the 30 timed sources. Matched MATH full separates backend changes from query omission. Routing overhead is included; cached setup and the first prompt-to-continuation prediction are excluded.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: Primary contrasts with individual 95% intervals (thick) and simultaneous 99.167% Bonferroni intervals (thin). Positive gains favor the additional-trajectory arm. The six-contrast family uses 100 book clusters per checkpoint and three fixed fits. C48 and C_RP3 already contain one-step displacement.

Checkpoint	Contrast	Gain [95% interval]	Adjusted $P$
SmolLM3	J32–C48	0.437 [-0.138, 1.046]	0.6019
SmolLM3	J32–C_RP3	-0.277 [-0.997, 0.428]	0.8426
SmolLM3	S32_T–S32	-0.335 [-0.968, 0.295]	0.8426
Qwen3.5	J32–C48	0.856 [-0.020, 1.736]	0.2820
Qwen3.5	J32–C_RP3	-3.557 [-4.874, -2.182]	0.0006
Qwen3.5	S32_T–S32	0.426 [-0.347, 1.221]	0.8426

Appendix

Table A1: Six prospective PG-19 primary contrasts at exactly 20% calls. Gains and intervals are

10^{-3}

nats/token. Individual 95% intervals are descriptive; Holm-adjusted

P

values control the six-contrast family. The machine-readable source also includes 99.167% simultaneous intervals, all book values and each fitting seed.

Target	Router	NLL	Rate	Selected harm
bypass	C48	2.5649	0.2000	-0.0069
bypass	C_RP3	2.5642	0.2000	-0.0069
bypass	I_S35	2.5512	0.2000	-0.0090
bypass	I_ST32	2.5516	0.2000	-0.0088
bypass	I_T32	2.5677	0.2000	-0.0065
bypass	J32	2.5645	0.2000	-0.0069

Appendix

Table A2: Paired reference allocation at 20% known-horizon quota. NLL/rate/harm are token-weighted and averaged over three fits; intervals for contrasts below instead weight books/articles equally. These estimands coincide for balanced PG-19 chunks but can differ for WikiText-2. Rate is exact within rounding. Harm retains the selected negative-utility sign.

Target	Contrast	Gain [95% interval]
bypass	J32–C48	0.437 [-0.138, 1.046]
bypass	J32–C_RP3	-0.277 [-0.997, 0.428]
bypass	S32_T–S32	-0.335 [-0.968, 0.295]
bypass	J32–S32	-13.268 [-14.735, -11.872]
bypass	I_ST32–I_S35	-0.447 [-0.901, 0.027]
bypass	J32–J_sign	4.075 [3.148, 5.009]

Appendix

Table A3: Secondary paired feature/objective contrasts at 20% quota,

10^{-3}

nats/token. These are unadjusted 95% intervals; no additional confirmatory significance claim is made.

Target	Router	NLL	Rate	Selected harm
bypass	C48	2.1966	0.2000	-0.0068
bypass	C_RP3	2.1950	0.2000	-0.0067
bypass	I_S35	2.1922	0.2000	-0.0055
bypass	I_ST32	2.1916	0.2000	-0.0056
bypass	I_T32	2.1973	0.2000	-0.0053
bypass	J32	2.1968	0.2000	-0.0065

Appendix

Table A4: Paired reference allocation at 20% known-horizon quota. NLL/rate/harm are token-weighted and averaged over three fits; intervals for contrasts below instead weight books/articles equally. These estimands coincide for balanced PG-19 chunks but can differ for WikiText-2. Rate is exact within rounding. Harm retains the selected negative-utility sign.

Target	Contrast	Gain [95% interval]
bypass	J32–C48	-0.340 [-0.992, 0.280]
bypass	J32–C_RP3	-1.544 [-2.448, -0.639]
bypass	S32_T–S32	-0.149 [-0.555, 0.248]
bypass	J32–S32	-3.978 [-5.177, -2.866]
bypass	I_ST32–I_S35	0.469 [0.029, 0.911]
bypass	J32–J_sign	0.503 [-0.226, 1.239]

Appendix

Table A5: Secondary paired feature/objective contrasts at 20% quota,

10^{-3}

nats/token. These are unadjusted 95% intervals; no additional confirmatory significance claim is made.

Target	Router	NLL	Rate	Selected harm
bypass	C48	2.3921	0.2000	-0.0063
bypass	C_RP3	2.3920	0.2000	-0.0062
bypass	I_S35	2.3787	0.2000	-0.0084
bypass	I_ST32	2.3781	0.2000	-0.0085
bypass	I_T32	2.3929	0.2000	-0.0061
bypass	J32	2.3922	0.2000	-0.0063

Appendix

Table A6: Paired reference allocation at 20% known-horizon quota. NLL/rate/harm are token-weighted and averaged over three fits; intervals for contrasts below instead weight books/articles equally. These estimands coincide for balanced PG-19 chunks but can differ for WikiText-2. Rate is exact within rounding. Harm retains the selected negative-utility sign.

Target	Contrast	Gain [95% interval]
bypass	J32–C48	-0.067 [-0.199, 0.065]
bypass	J32–C_RP3	-0.201 [-0.370, -0.036]
bypass	S32_T–S32	-1.274 [-1.768, -0.847]
bypass	J32–S32	-11.151 [-12.275, -10.100]
bypass	I_ST32–I_S35	0.566 [0.308, 0.833]
bypass	J32–J_sign	0.132 [-0.374, 0.617]

Appendix

Table A7: Secondary paired feature/objective contrasts at 20% quota,

10^{-3}

nats/token. These are unadjusted 95% intervals; no additional confirmatory significance claim is made.

Target	Router	NLL	Rate	Selected harm
bypass	C48	2.6922	0.2000	-0.0281
bypass	C_RP3	2.6878	0.2000	-0.0272
bypass	I_S35	2.6834	0.2000	-0.0269
bypass	I_ST32	2.6833	0.2000	-0.0266
bypass	I_T32	2.6989	0.2000	-0.0243
bypass	J32	2.6914	0.2000	-0.0279

Appendix

Table A8: Paired reference allocation at 20% known-horizon quota. NLL/rate/harm are token-weighted and averaged over three fits; intervals for contrasts below instead weight books/articles equally. These estimands coincide for balanced PG-19 chunks but can differ for WikiText-2. Rate is exact within rounding. Harm retains the selected negative-utility sign.

Target	Contrast	Gain [95% interval]
bypass	J32–C48	0.856 [-0.020, 1.736]
bypass	J32–C_RP3	-3.557 [-4.874, -2.182]
bypass	S32_T–S32	0.426 [-0.347, 1.221]
bypass	J32–S32	-10.676 [-12.569, -8.871]
bypass	I_ST32–I_S35	0.087 [-0.649, 0.838]
bypass	J32–J_sign	28.125 [26.114, 30.133]

Appendix

Table A9: Secondary paired feature/objective contrasts at 20% quota,

10^{-3}

nats/token. These are unadjusted 95% intervals; no additional confirmatory significance claim is made.

Target	Router	NLL	Rate	Selected harm
bypass	C48	2.3575	0.2000	-0.0279
bypass	C_RP3	2.3513	0.2000	-0.0294
bypass	I_S35	2.3216	0.2000	-0.0309
bypass	I_ST32	2.3221	0.2000	-0.0304
bypass	I_T32	2.3735	0.2000	-0.0233
bypass	J32	2.3568	0.2000	-0.0285

Appendix

Table A10: Paired reference allocation at 20% known-horizon quota. NLL/rate/harm are token-weighted and averaged over three fits; intervals for contrasts below instead weight books/articles equally. These estimands coincide for balanced PG-19 chunks but can differ for WikiText-2. Rate is exact within rounding. Harm retains the selected negative-utility sign.

Target	Contrast	Gain [95% interval]
bypass	J32–C48	1.561 [-0.364, 3.533]
bypass	J32–C_RP3	-6.080 [-9.989, -2.365]
bypass	S32_T–S32	0.773 [-0.844, 2.213]
bypass	J32–S32	-36.719 [-42.281, -31.328]
bypass	I_ST32–I_S35	-0.470 [-1.959, 1.045]
bypass	J32–J_sign	56.417 [52.383, 60.394]

Appendix

Table A11: Secondary paired feature/objective contrasts at 20% quota,

10^{-3}

nats/token. These are unadjusted 95% intervals; no additional confirmatory significance claim is made.

Target	Router	NLL	Rate	Selected harm
bypass	C48	2.5045	0.2000	-0.0261
bypass	C_RP3	2.5009	0.2000	-0.0263
bypass	I_S35	2.4975	0.2000	-0.0258
bypass	I_ST32	2.4952	0.2000	-0.0258
bypass	I_T32	2.5111	0.2000	-0.0228
bypass	J32	2.5027	0.2000	-0.0263

Appendix

Table A12: Paired reference allocation at 20% known-horizon quota. NLL/rate/harm are token-weighted and averaged over three fits; intervals for contrasts below instead weight books/articles equally. These estimands coincide for balanced PG-19 chunks but can differ for WikiText-2. Rate is exact within rounding. Harm retains the selected negative-utility sign.

Target	Contrast	Gain [95% interval]
bypass	J32–C48	1.738 [1.356, 2.110]
bypass	J32–C_RP3	-1.794 [-2.342, -1.263]
bypass	S32_T–S32	-2.925 [-3.376, -2.490]
bypass	J32–S32	-9.207 [-10.166, -8.250]
bypass	I_ST32–I_S35	2.306 [1.715, 2.939]
bypass	J32–J_sign	14.397 [13.473, 15.336]

Appendix

Table A13: Secondary paired feature/objective contrasts at 20% quota,

10^{-3}

nats/token. These are unadjusted 95% intervals; no additional confirmatory significance claim is made.

Length	Retention policy	Accuracy (%)	NLL
256	full	100.00	0.0026
256	recent16	11.52	2.2821
256	recent_extended_B16	10.74	2.3735
256	random_B16	17.25	2.6083
256	error_B16	100.00	0.0026
256	displacement_B16	95.90	0.2079

Appendix

Table A14: Fresh-source retention performance at length 256, averaged over three fixed repaired checkpoints. Accuracy (%) and query NLL use the first 128 sequences. The nominal B32 eligibility-aware policy retains all eligible VALUE states and is nonbinding. Binding eligible budgets are secondary.

Length	Retention policy	Accuracy (%)	NLL
384	full	99.09	0.1443
384	recent16	12.70	2.1798
384	recent_extended_B16	10.74	2.2812
384	random_B16	17.25	2.3107
384	error_B16	99.09	0.1443
384	displacement_B16	93.62	0.4022

Appendix

Table A15: Fresh-source retention performance at length 384, averaged over three fixed repaired checkpoints. Accuracy (%) and query NLL use the first 128 sequences. The nominal B32 eligibility-aware policy retains all eligible VALUE states and is nonbinding. Binding eligible budgets are secondary.

Contrast	Endpoint	Gain [95% interval]	Adjusted $P$
L256 random	Accuracy	11.133 [7.552, 14.844]	0.0016
L256 random	NLL	-0.272 [-0.410, -0.139]	0.0016
L256 error	Accuracy	87.370 [85.156, 89.518]	0.0016
L256 error	NLL	2.422 [2.368, 2.479]	0.0016
L256 displacement	Accuracy	87.370 [85.156, 89.518]	0.0016
L256 displacement	NLL	2.422 [2.368, 2.479]	0.0016

Appendix

Table A16: The sixteen prespecified fresh-source contrasts. Accuracy gains are percentage points; NLL reductions are nats/query token. Positive favors the first policy named. Intervals resample 128 independent sequences, retaining paired checkpoint averages. Holm correction applies to all sixteen endpoints.

Model	Unit	Joint $-$ uncertainty	Both control $D$ and position
3.48M	byte	0.528 [0.271, 0.782]	0.049 [-0.217, 0.310]
20.08M	byte	0.537 [-0.165, 1.229]	0.234 [-0.244, 0.789]
3.48M, balanced	byte	0.161 [0.057, 0.265]	0.097 [-0.038, 0.227]
GPT-2 124M	BPE token	20.574 [16.974, 24.215]	7.987 [4.944, 10.846]

Appendix

Table A17: Prefix-only known-horizon quota policy at matched optional-branch call counts,

10^{-3}

nats per unit. Natural scored horizons round to 48/239 calls (20.08%); GPT-2 uses 99/495 (20.00%). Brackets condition on the fitted models and resample articles; this is an exploratory deadline heuristic, not a measured-cost or optimal-policy guarantee.

Configuration	Seed	Paired gain	95% article interval
3.48M	11	0.738	[0.301, 1.211]
3.48M	23	0.518	[0.197, 0.859]
3.48M	37	0.418	[0.023, 0.846]
20.08M	11	1.859	[0.956, 2.758]
20.08M	23	1.562	[0.527, 2.644]
20.08M	37	0.830	[-0.187, 1.977]

Appendix

Table A18: Individual-seed primary contrasts at exact 20% budgets,

10^{-3}

nats/byte. These remain visible alongside the fixed-seed mean intervals.

Configuration	Fast	Full correction	Threshold joint	Dense reference
Original 3.48M	1.997 $\pm$ 0.009	1.976 $\pm$ 0.008	1.981 $\pm$ 0.008	2.201 $\pm$ 0.051
20.08M	1.877 $\pm$ 0.009	1.834 $\pm$ 0.010	1.854 $\pm$ 0.009	1.949 $\pm$ 0.006
Balanced 3.48M	1.921 $\pm$ 0.001	1.910 $\pm$ 0.002	1.912 $\pm$ 0.002	1.911 $\pm$ 0.004

Appendix

Table A19: Finite-context test BPB, mean

\pm

sample SD across three seeds. Threshold joint uses a frozen validation threshold targeting 20%, with potentially different actual test rates; it is not the fixed-horizon quota policy. Dense references have the same parameter shapes and token/update exposure, but a different training objective and no extensive tuning. Amplitude sensitivity demonstrates why these are not evidence of superiority over an optimized dense model.

Configuration	$k=3$	$k=4$	$k=8$	$k=16$
Original 3.48M	10.891 $\pm$ 0.323	10.891 $\pm$ 0.061	10.662 $\pm$ 0.199	10.843 $\pm$ 0.367
20.08M	16.445 $\pm$ 0.643	15.962 $\pm$ 0.418	16.299 $\pm$ 0.149	16.696 $\pm$ 0.388
Balanced 3.48M	6.226 $\pm$ 0.469	6.262 $\pm$ 0.441	6.136 $\pm$ 0.528	6.239 $\pm$ 0.530

Appendix

Table A20: Trajectory-window sensitivity: signed utility captured by the joint router at an exact retrospective 20% query budget,

10^{-3}

nats/byte, mean

\pm

sample SD over three seeds. Each window used the same scored suffix and router recipe;

k=4

was the original setting. The values are reductions from the fast branch, not incremental gains over uncertainty. This exploratory diagnostic selects scores retrospectively and does not identify a universally optimal window.

Configuration	Length	Fast query accuracy %	Full query accuracy %
1k pilot	256	12.16 $\pm$ 1.02	11.95 $\pm$ 1.36
1k pilot	384	8.79 $\pm$ 0.74	8.20 $\pm$ 0.74
6k revision	256	13.51 $\pm$ 0.45	13.46 $\pm$ 0.57
6k revision	384	12.73 $\pm$ 1.80	12.70 $\pm$ 1.76
Support configuration	256	12.74 $\pm$ 0.59	100.00 $\pm$ 0.00
Support configuration	384	11.85 $\pm$ 0.31	99.25 $\pm$ 0.46

Appendix

Table A21: All synthetic training attempts, three seeds per configuration. Unsuccessful correction branches are retained and not used to infer retention quality. The support configuration changed both supervision and context eligibility, so it is not a one-factor causal ablation.

Figure A2: Outcome-triggered balanced-position sensitivity probe. All three seeds and the same gate/threshold distinction are shown. The reduced mean incremental gain is a material qualification of the original configuration, not an omitted failure.

Figure A3: All nine synthetic checkpoints and their recorded training configurations. Initial and longer attempts failed associative recall. The support configuration jointly changed causal reconstruction supervision and value eligibility and is explicitly exploratory. No unsuccessful seed or configuration is discarded.

Figure A4: Additional 20.08M implementation timing under shared graphics-device load. These observations retain unfavorable ratios and are not isolated-device or quality-matched deployment benchmarks.

Policy	Median	p95	Executed NLL	Native speed ratio [95%]
native_auto_full	1478.9	1515.9	2.5589	1.000 [1.000, 1.000]
matched_math_full	1462.0	1524.8	2.5589	1.012 [0.987, 1.021]
true_bypass	1475.5	1563.2	2.7064	1.002 [0.976, 1.017]
random_quota	1473.8	1572.2	2.6808	1.003 [0.986, 1.013]
I_T32	1514.2	1571.8	2.6759	0.977 [0.951, 0.987]
I_S35	1516.3	1556.8	2.6648	0.975 [0.958, 0.989]

Appendix

Table A22: Actual executed policies, fixed fitting seed 901. NLL uses the same 30 timing-sample blocks; it is not the complete-roster sequence-quality endpoint. Median and p95 are whole-region milliseconds. Ratio is native automatic full median divided by policy median; values above one favor the policy. Block intervals quantify repeated timing under recorded shared-desktop conditions, not unseen devices or free-generation performance.

Policy	Median	p95	Executed NLL	Native speed ratio [95%]
native_auto_full	1475.9	1547.0	2.5242	1.000 [1.000, 1.000]
matched_math_full	1454.5	1503.2	2.5242	1.015 [0.999, 1.025]
true_bypass	1470.4	1547.1	2.6980	1.004 [0.988, 1.024]
random_quota	1452.7	1536.3	2.6592	1.016 [1.003, 1.024]
I_T32	1497.9	1556.9	2.6605	0.985 [0.967, 0.995]
I_S35	1503.3	1592.0	2.6471	0.982 [0.966, 0.997]

Appendix

Table A23: Actual executed policies, fixed fitting seed 901. NLL uses the same 30 timing-sample blocks; it is not the complete-roster sequence-quality endpoint. Median and p95 are whole-region milliseconds. Ratio is native automatic full median divided by policy median; values above one favor the policy. Block intervals quantify repeated timing under recorded shared-desktop conditions, not unseen devices or free-generation performance.

Policy	Median	p95	Executed NLL	Native speed ratio [95%]
native_auto_full	192.4	210.5	2.4973	1.000 [1.000, 1.000]
matched_math_full	193.2	207.0	2.4973	0.996 [0.988, 1.006]
true_bypass	190.4	206.9	2.6602	1.010 [1.001, 1.019]
random_quota	191.0	217.2	2.6288	1.007 [0.999, 1.014]
I_T32	193.9	209.2	2.6248	0.992 [0.989, 1.000]
I_S35	191.2	211.2	2.6064	1.006 [0.994, 1.012]

Appendix

Table A24: Actual executed policies, fixed fitting seed 901. Scoring NLL uses the complete locked roster; latency uses the first 30 rows. Median and p95 are whole-region milliseconds. Ratio is native automatic full median divided by policy median; values above one favor the policy. Block intervals quantify repeated timing under recorded shared-desktop conditions, not unseen devices or free-generation performance.

Policy	Median	p95	Executed NLL	Native speed ratio [95%]
native_auto_full	975.8	1066.9	2.3863	1.000 [1.000, 1.000]
matched_math_full	996.2	1056.3	2.3863	0.980 [0.953, 1.017]
true_bypass	938.2	1016.7	2.5443	1.040 [1.035, 1.062]
random_quota	949.1	1038.9	2.5127	1.028 [0.988, 1.049]
I_T32	949.8	1074.7	2.5113	1.027 [1.020, 1.048]
I_S35	947.9	1002.1	2.4965	1.029 [1.024, 1.050]

Appendix

Table A25: Actual executed policies, fixed fitting seed 901. Scoring NLL uses the complete locked roster; latency uses the first 30 rows. Median and p95 are whole-region milliseconds. Ratio is native automatic full median divided by policy median; values above one favor the policy. Block intervals quantify repeated timing under recorded shared-desktop conditions, not unseen devices or free-generation performance.

Policy	Median	p95	Executed NLL	Native speed ratio [95%]
native_auto_full	96.2	105.1	2.5703	1.000 [1.000, 1.000]
matched_math_full	96.1	105.5	2.5703	1.001 [0.989, 1.017]
true_bypass	95.1	102.6	2.7364	1.012 [0.998, 1.028]
random_quota	95.9	105.2	2.7024	1.003 [0.986, 1.018]
I_T32	96.3	102.7	2.6991	1.000 [0.994, 1.014]
I_S35	96.4	104.9	2.6840	0.998 [0.992, 1.014]

Appendix

Table A26: Actual executed policies, fixed fitting seed 901. Scoring NLL uses the complete locked roster; latency uses the first 30 rows. Median and p95 are whole-region milliseconds. Ratio is native automatic full median divided by policy median; values above one favor the policy. Block intervals quantify repeated timing under recorded shared-desktop conditions, not unseen devices or free-generation performance.

Policy	Median	p95	Executed NLL	Native speed ratio [95%]
native_auto_full	1060.0	1100.4	2.5742	1.000 [1.000, 1.000]
matched_math_full	1060.8	1114.2	2.5742	0.999 [0.983, 1.014]
true_bypass	1072.2	1153.5	2.6103	0.989 [0.970, 1.002]
random_quota	1077.5	1159.3	2.6043	0.984 [0.969, 0.999]
I_T32	1114.6	1156.2	2.6080	0.951 [0.934, 0.965]
I_S35	1110.5	1147.4	2.5893	0.955 [0.937, 0.974]

Appendix

Table A27: Actual executed policies, fixed fitting seed 901. NLL uses the same 30 timing-sample blocks; it is not the complete-roster sequence-quality endpoint. Median and p95 are whole-region milliseconds. Ratio is native automatic full median divided by policy median; values above one favor the policy. Block intervals quantify repeated timing under recorded shared-desktop conditions, not unseen devices or free-generation performance.

Policy	Median	p95	Executed NLL	Native speed ratio [95%]
native_auto_full	1049.2	1136.4	2.4609	1.000 [1.000, 1.000]
matched_math_full	1047.3	1091.0	2.4609	1.002 [0.982, 1.029]
true_bypass	1066.9	1113.6	2.4961	0.983 [0.975, 1.016]
random_quota	1062.3	1114.9	2.4878	0.988 [0.973, 1.016]
I_T32	1105.1	1153.0	2.4884	0.949 [0.941, 0.977]
I_S35	1089.9	1138.3	2.4675	0.963 [0.956, 0.990]

Appendix

Table A28: Actual executed policies, fixed fitting seed 901. NLL uses the same 30 timing-sample blocks; it is not the complete-roster sequence-quality endpoint. Median and p95 are whole-region milliseconds. Ratio is native automatic full median divided by policy median; values above one favor the policy. Block intervals quantify repeated timing under recorded shared-desktop conditions, not unseen devices or free-generation performance.

Policy	Median	p95	Executed NLL	Native speed ratio [95%]
native_auto_full	134.6	146.2	2.4734	1.000 [1.000, 1.000]
matched_math_full	134.7	142.7	2.4734	0.999 [0.989, 1.016]
true_bypass	132.9	145.2	2.5041	1.013 [1.006, 1.028]
random_quota	134.2	141.2	2.4978	1.004 [1.000, 1.019]
I_T32	135.5	149.0	2.4947	0.993 [0.978, 1.010]
I_ST32	136.0	148.5	2.4808	0.990 [0.978, 1.011]

Appendix

Table A29: Actual executed policies, fixed fitting seed 901. Scoring NLL uses the complete locked roster; latency uses the first 30 rows. Median and p95 are whole-region milliseconds. Ratio is native automatic full median divided by policy median; values above one favor the policy. Block intervals quantify repeated timing under recorded shared-desktop conditions, not unseen devices or free-generation performance.

Policy	Median	p95	Executed NLL	Native speed ratio [95%]
native_auto_full	1162.5	1255.6	2.3695	1.000 [1.000, 1.000]
matched_math_full	1128.4	1171.6	2.3695	1.030 [1.002, 1.047]
true_bypass	1112.4	1158.4	2.3995	1.045 [1.015, 1.063]
random_quota	1120.4	1200.4	2.3935	1.038 [1.001, 1.057]
I_T32	1118.0	1225.2	2.3930	1.040 [0.984, 1.059]
I_S35	1114.9	1194.8	2.3787	1.043 [1.013, 1.060]

Appendix

Table A30: Actual executed policies, fixed fitting seed 901. Scoring NLL uses the complete locked roster; latency uses the first 30 rows. Median and p95 are whole-region milliseconds. Ratio is native automatic full median divided by policy median; values above one favor the policy. Block intervals quantify repeated timing under recorded shared-desktop conditions, not unseen devices or free-generation performance.

Policy	Median	p95	Executed NLL	Native speed ratio [95%]
native_auto_full	61.3	68.0	2.5433	1.000 [1.000, 1.000]
matched_math_full	61.4	68.7	2.5433	0.998 [0.961, 1.014]
true_bypass	60.9	69.1	2.5814	1.006 [0.991, 1.031]
random_quota	61.6	69.8	2.5739	0.994 [0.969, 1.006]
I_T32	62.5	70.6	2.5678	0.980 [0.957, 1.002]
I_S35	61.9	70.1	2.5506	0.989 [0.964, 1.006]

Appendix

Table A31: Actual executed policies, fixed fitting seed 901. Scoring NLL uses the complete locked roster; latency uses the first 30 rows. Median and p95 are whole-region milliseconds. Ratio is native automatic full median divided by policy median; values above one favor the policy. Block intervals quantify repeated timing under recorded shared-desktop conditions, not unseen devices or free-generation performance.

Model	Condition	Valid policies	Blocks	Failed policies
SmolLM3	scoring 512	10	30	None
SmolLM3	scoring 1024	9	30	I_S35
SmolLM3	scoring 4096	10	30	None
SmolLM3	cached 512	10	30	None
SmolLM3	cached 1024	10	30	None
Qwen3.5	scoring 512	10	30	None

Appendix

Table A32: Realized systems coverage. Scoring quality uses 200 core rows at 512 or 99 midpoint rows at longer lengths; cached quality uses the same 30 timed source rows. No projection from dense branch losses substitutes for executed policy quality.

Explore similar work

Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressions
Sep 28, 2026Yifan Lu, Qiyue Zhang, Haotian Shan +2Large Language Model Routing
Rice University
TriRoute: Unified Learned Routing for Joint Adaptive Attention, Experts, and KV-Cache Allocation
Jul 7, 2026Andrii Balashov, Olena PonomarovaKey-Value Cache Quantization Mixture-Of-Experts
Ukrainian State University of Science and Technologies
The Routing Plateau: Understanding the Accuracy Limits of LLM Routers
May 27, 2026Yifan Lu, Qiyue Zhang, Shenrun Zhang +4Large Language Model Routing Bottlenecks
Rice University · Amazon

Sep 28, 2026, cs.LGJ/K move · Enter open · S save

Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressions

Yifan Lu, Qiyue Zhang, Haotian Shan, Hanjie Chen+1

Rice University

Large Language Model Routing

Open paper AskPDF arXiv

Large Language Model (LLM) routers commonly rely on neural query embeddings, with larger encoders expected to better capture query intent and difficulty. Yet scaling Qwen2.5 encoders from 0.5B to 72B parameters brings little improvement in routing accuracy (Figure 1b), suggesting that small encoders may already capture the query properties needed for routing. We therefore investigate which properties matter and whether they can be extracted directly from text without a neural encoder. We introduce REGEXROUTE, a pipeline that uses sparse autoencoders (SAEs) to discover interpretable regular-expression (regex) features. Using unlabeled text, an LLM turns descriptions of grouped SAE latents into regex extractors and refines them to match latent activation patterns. These extractors supply numerical features to a lightweight routing head, eliminating neural encoding at inference (Figure 1a). Across four benchmarks, one fixed set of 128 features achieves 76.43% average routing accuracy, comparable to 76.41% for the strongest neural text encoder baseline, with much smaller latency and strong robustness. These findings establish explicit, interpretable text features as a practical basis for designing and understanding LLM routers.

Abstract

Explore similar work

Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressions

TriRoute: Unified Learned Routing for Joint Adaptive Attention, Experts, and KV-Cache Allocation

The Routing Plateau: Understanding the Accuracy Limits of LLM Routers