Evaluating Small Language Models for Front-Door Routing: A Harmonized Benchmark and Synthetic-Traffic Experiment

Warren Johnson; Charles Lee

cs.NIMar 26, 2026

Evaluating Small Language Models for Front-Door Routing: A Harmonized Benchmark and Synthetic-Traffic Experiment

Authors: Warren Johnson, Charles Lee

Organizations: Plexor Labs, Sammamish, WA, USA · Project Autobots, Seattle, WA, USA

Abstract

Selecting the appropriate model at inference time -- the routing problem -- requires jointly optimizing output quality, cost, latency, and governance constraints. Existing approaches delegate this decision to LLM-based classifiers or preference-trained routers that are themselves costly and high-latency, reducing a multi-objective optimization to single-dimensional quality prediction. We argue that small language models (SLMs, 1-4B parameters) have now achieved sufficient reasoning capability for sub-second, zero-marginal-cost, self-hosted task classification, potentially making the routing decision negligible in the inference budget. We test this thesis on a six-label taxonomy through two studies. Study 1 is a harmonized offline benchmark of Phi-3.5-mini, Qwen2.5-1.5B, and Qwen-2.5-3B on identical Azure T4 hardware, serving stack, quantization, and a fixed 60-case corpus. Qwen-2.5-3B achieves the best exact-match accuracy (0.783), the strongest latency-accuracy tradeoff, and the only nonzero accuracy on all six task families. Study 2 is a pre-registered four-arm randomized experiment under synthetic traffic with an effective sample size of 60 unique cases per arm, comparing Phi-4-mini, Qwen-2.5-3B, and DeepSeek-V3 against a no-routing control. DeepSeek-V3 attains the highest accuracy (0.830) but fails the pre-registered P95 latency gate (2,295 ms); Qwen-2.5-3B is Pareto-dominant among self-hosted models (0.793 accuracy, 988 ms median, $0 marginal cost). No model meets the standalone viability criterion (>=0.85 accuracy, <=2,000 ms P95). The cost and latency prerequisites for SLM-based routing are met; the accuracy gap of 6-8 percentage points and the untested question of whether correct classification translates to downstream output quality bound the remaining distance to production viability.

Figures & tables

Metric	Phi-3.5-mini	Qwen2.5-1.5B	Qwen-2.5-3B
Exact classification accuracy	$0.7167^{**}$	0.4000	$\mathbf{0.783}^{***}$
JSON parse rate	0.9833	0.9667	$\mathbf{1.0000}$
Median latency (ms)	5,772	$\mathbf{793}$	1,088
P95 latency (ms)	6,357	4,636	$\mathbf{1{,}554}$

Table 1: Harmonized 60-case benchmark metrics (Azure T4, vLLM 0.17.1, bitsandbytes 4-bit NF4, exclusive GPU, 2026-03-15/16). 3 3 3 Significance convention used throughout all tables: ∗

p<0.05

; ∗∗

p<0.01

; ∗∗∗

p<0.001

. Bold indicates best per metric. Pairwise McNemar tests: Phi vs. Qwen-1.5B

p=0.0009

; Phi vs. Qwen-3B

p=0.503

(ns); Qwen-1.5B vs. Qwen-3B

p<0.0001

Family	Phi-3.5-mini	Qwen2.5-1.5B	Qwen-2.5-3B	Best Model
code/complex	$\mathbf{1.00}$	0.50	0.70	Phi
code/simple	0.80	$\mathbf{0.90}$	0.70	Qwen-1.5B
CoT/complex	$\mathbf{1.00}$	0.00	0.90	Phi
CoT/simple	0.80	0.00	$\mathbf{0.90}$	Qwen-3B
hybrid/agentic	0.50	$\mathbf{1.00}$	0.90	Qwen-1.5B
hybrid/generative	0.20	0.00	$\mathbf{0.60}$	Qwen-3B

Table 2: Family-level exact-match accuracy (10 cases per family), harmonized benchmark.

Panel of Figure 1 — Figure 1: Accuracy–P95 latency scatterplot for all evaluated models. Both axes now use the pre-registered gating metrics: exact-match accuracy ( $y$ ) and P95 latency ( $x$ ). Filled markers: Study 1 harmonized benchmark (identical hardware). Open markers ( ${}^{\text{E}}$ ): Study 2 randomized experiment ( $n_{\text{eff}}\!=\!60$ ). The shaded green region marks the viable operating zone (accuracy $\geq 0.85$ , P95 $\leq 2{,}000$ ms). No model enters the viable region. DeepSeek-V3 approaches the accuracy threshold but exceeds the latency gate ( $P95=2{,}295$ ms). Qwen-2.5-3B is Pareto-dominant among self-hosted models.

Arm	Label	Model Backend	Hosting	Params
A	Control	None (pass-through)	N/A	—
B	Phi-4-mini	Phi-4-mini-instruct	Self-hosted vLLM (T4 GPU)	3.8B
C	Qwen-2.5-3B	Qwen-2.5-3B-Instruct	Self-hosted vLLM (T4 GPU)	3B
D	DeepSeek-V3	DeepSeek-V3 (API)	DeepSeek commercial API	671B MoE

Table 3: Four-arm experiment treatment definitions.

Scenario	Cost Winner	Quality Winner	Decision
SLM wins both	B or C	B or C	Deploy winning SLM
DeepSeek quality, SLM cost	B or C	D	SLM + DeepSeek fallback
DeepSeek wins both	D	D	DeepSeek primary; SLM fallback
All similar quality	Lowest	Comparable	Deploy cheapest
DeepSeek fails latency gate	Any	Any	Eliminate D; choose B vs. C

Table 4: Pre-registered decision matrix mapping experiment outcomes to deployment actions.

Arm	Model	Accuracy	Parse	Med. Lat	P95 Lat	Cost
A	Control	—	—	—	—	$0
B	Phi-4-mini	0.518	0.915	977 ms	1,541 ms	$0
C	Qwen-2.5-3B	$\mathbf{0.793}$	$\mathbf{1.000}$	$\mathbf{988}$ ms	$\mathbf{1{,}170}$ ms	$0
D	DeepSeek-V3	$\mathbf{0.830}$	$\mathbf{1.000}$	1,854 ms	2,295 ms	$0.034

Table 5: Study 2 results (

N=400

sessions per arm,

n_{\text{eff}}=60

for accuracy; corpus SHA dac3aac5 , 2026-03-16).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Artifact	Value
Models evaluated	Phi-3.5-mini, Qwen2.5-1.5B, Qwen-2.5-3B
Corpus SHA-256	dac3aac5…
GPU	NVIDIA Tesla T4 (16 GB GDDR6, exclusive)
VM class	Azure Standard_NC8as_T4_v3
Inference framework	vLLM 0.17.1
Quantization backend	bitsandbytes 4-bit NF4

Appendix

Table 6: Study 1 execution details.

Artifact	Value
Arms	A (control), B (Phi-4-mini), C (Qwen-2.5-3B), D (DeepSeek-V3)
Sample size	$N=400$ per arm ( $N_{\text{total}}=1{,}600$ )
Corpus SHA-256	dac3aac5…
Randomization	SHA-256( session_id ) $\bmod 4$
GPU (Arms B, C)	NVIDIA Tesla T4 (16 GB GDDR6), vLLM 0.17.1
Arm D	DeepSeek-V3 commercial API

Appendix

Table 7: Study 2 execution details.

Case ID	Ground Truth	Predicted	Error Corridor
PV2-001	CoT/simple	hybrid/generative	CoT $\to$ hybrid
PV2-011	code/simple	hybrid/generative	code $\to$ hybrid
PV2-013	code/simple	CoT/simple	code $\to$ CoT
PV2-020	code/simple	code/complex	complexity over-est.
PV2-021	CoT/complex	hybrid/generative	CoT $\to$ hybrid
PV2-033	code/complex	hybrid/agentic	code $\to$ hybrid

Appendix

Table 8: Complete per-case misclassifications for Qwen-2.5-3B on the 60-case benchmark (

13/60

errors, accuracy

=0.783

). Ground truth

\to

predicted label.

Artifact	File Path	Purpose
Experiment design	paper/main.tex (§ 4 )	Protocol specification
Analysis script	scripts/ab-test/rct_cross_arm_analysis.py	Endpoint analysis
Decision matrix	Table 4 (this paper)	Outcome $\to$ action map
Hypothesis specs	H5–H9 (§ 1.3 )	Testable predictions
Corpus (60-case)	data/progressive_test_cases_v2_60.jsonl	Evaluation corpus
Benchmark script	scripts/benchmark_openweights.py	Model evaluation driver

Appendix

Table 9: Pre-registered artifacts for Study 2, committed before experiment execution.

Explore similar work

Aug 7, 2026

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.

Tao Feng, Fangxu Yu, Haozhen Zhang +9

University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2

LLM Evaluation LLM Routing

Ask PDF

Sep 28, 2026

SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing

Large language model (LLM) routing aims to select the most suitable model for each incoming query. Most existing routers learn this decision directly from query embeddings, model representations, preference data, or clusters of similar examples. Such approaches can be effective, yet the representation used for routing rarely states what a query actually requires. We introduce SeLMRoute, a routing framework that separates the extraction of candidate-independent semantic evidence from the learning of candidate performance and the application of deployment objectives. A decision model first evaluates a set of interpretable questions about the query, such as its reasoning requirements and use of external knowledge, with each judgment retained as a probability distribution. The resulting probabilistic semantic state is used by a lightweight supervised router to estimate candidate model performance. Routing objectives are applied after performance estimation, which allows the same semantic state to support performance-oriented and cost-aware decisions. On the LLMRouterBench (15 datasets, 20 candidate models, 11,481 queries), SeLMRoute achieves an average accuracy of

72.08\% \pm 0.45

, while grouped five-fold out-of-fold evaluation reaches

72.64\%

, compared with

69.23\%

for the strongest fixed candidate. The representation achieves the highest mean performance among the evaluated semantic, dense, lexical, and domain-level representations. In a separate 13-model performance-cost setting, SeLMRoute improves performance in all five grouped splits, with a mean PerfGain of

2.66\%

. Our code is available at https://github.com/Indigma-Innovations/SeLMRoute.

Vasilis Perifanis, Nikolaos Pavlidis, Symeon Symeonidis

Indigma Innovations · Democritus University of Thrace · Athena Research Center

LLM Routing

Ask PDF

May 29, 2026

OrcaRouter: A Production-Oriented LLM Router with Hybrid Offline-Online Learning

The rapid development of large language models, each with distinct capabilities and inference costs, raises a practical deployment question: given an incoming request, which model should handle it? We present OrcaRouter, a production-oriented LLM router that combines a LinUCB-based contextual bandit over lexical and sentence-embedding features with a hybrid offline-online learning protocol. Offline, OrcaRouter obtains full-information feedback by evaluating each candidate model on a curated set of routing prompts, yielding a reward matrix used to fit one ridge regressor per arm. At deployment time, it initializes from these parameters and can optionally continue learning from bandit feedback, updating only the selected model's arm after observing its reward. At the time of our RouterArena submission (May 20, 2026), OrcaRouter-Adaptive ranked second on the public RouterArena leaderboard with an arena score of 72.08, achieving 75.54% accuracy at a cost of USD 1.00 per 1,000 queries.

Zhenghua Bao, Fengya Tian, Chris Zhang +3

Continuum AI

Model Selection LLM Routing LLM Request Scheduling

Ask PDF