vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models

Xunzhuo Liu; Huamin Chen; Samzong Lu; Yossi Ovadia; Guohong Wen; Hao Wu; Zhengda Tan; Jintao Zhang

Figure 1: Structural correspondence between Shannon’s communication system [ 35 ] and VSR’s routing pipeline. The dotted line marks the boundary between the information-theoretic regime (signal extraction maximizes mutual information with the routing outcome) and the Boolean-algebraic regime (the decision engine synthesizes routing policies from signal conditions via

\{\wedge,\vee,\neg\}

[ 34 ] ).

Figure 2: Entropy collapse during signal extraction (illustrative). Each additional signal reduces the routing uncertainty

H(M\mid\cdot\,)

from the uniform prior

\log_{2}K

bits until the decision formula

\phi(\mathbf{s})

yields a near-deterministic model selection. Bar heights are schematic rather than empirical measurements.

Figure 3: Three-layer architecture with closed-loop feedback. A deployment configuration

\Gamma

selects which signals, decisions, and plugins are active. Layer 1 extracts a signal vector

\mathbf{s}

from the request. Layer 2 evaluates Boolean decision formulas to select

d^{*}

. Layer 3 executes the per-decision plugin chain, selects a model from

d^{*}

’s candidate set, and routes to the provider endpoint. Response-side signals feed back to enable adaptive routing.

Figure 4: Signal extraction taxonomy and evaluation flow. An incoming request is evaluated in parallel against heuristic signals (sub-millisecond, deterministic) and learned signals (neural inference via LoRA classifiers). Only signal types referenced by configured decisions are computed (demand-driven evaluation). Results merge into the structured signal result

S(r)

.

Figure 5: Comparison of embedding strategies. Top : A Transformer embedding layer produces dense, opaque vectors optimized end-to-end. Bottom : The VSR signal extraction layer produces structured, interpretable signal vectors from heterogeneous extractors (heuristic, learned, LoRA-based), where each dimension has a human-readable semantic name. The dashed arrow marks the functional correspondence: both convert unstructured input into a representation for downstream computation, but the signal vector is interpretable by construction.

Figure 6: Rule-node expression trees at increasing depth. (a) A flat depth-1 tree: AND over three leaf conditions. (b) A NOR expression:

\textsc{not}(\textsc{or}(\text{cs},\text{math}))

, matching all non-STEM queries. (c) An XOR expression composed from AND, OR, and NOT primitives, routing requests that match exactly one of two signals. Leaf nodes (gray) reference signal conditions; composite nodes use AND ( blue ), OR ( green ), and NOT ( orange ).

Figure 7: Three-level correspondence between combinational logic circuits and the decision engine. Level 1 : A PLA with AND-plane and OR-plane corresponds to a flat (depth-1) decision formula. Level 2 : A general combinational circuit with arbitrarily nested AND, OR, and NOT gates corresponds to a recursive rule-node tree within a single decision. Level 3 : An array of circuits with a priority encoder corresponds to the full decision set with priority-ordered evaluation, realizing any routing policy.

Figure 8: Structural analogy between Mixture-of-Experts gating and the VSR decision engine. Left : In a sparse MoE layer, a learned gating network selects top-

k

experts via softmax. Right : In VSR, Boolean formulas over the signal vector act as symbolic expert gates, with priority ordering implementing deterministic early-exit selection. Dashed arrows mark the structural correspondence.

Figure 9: Transformer-style visualization of layered control depth in VSR. Signal extraction constructs a shared embedding

\mathbf{s}

; stacked decision layers apply Boolean gates

\phi_{\ell}(\mathbf{s})

that progressively reduce routing uncertainty

U_{\ell}

; selection resolves

(d^{*},m^{*})

via priority early-exit; plugin composition

\Psi_{d^{*}}

acts as an output-side projection head over request/response behavior.

Figure 11: Boolean expression AST for WHEN keyword("urgent") AND NOT domain("math") . Each leaf is a SignalRefExpr referencing a named signal; internal nodes are Boolean operators ( BoolAnd , BoolNot ). The compiler flattens this tree into a RuleCombination structure for the decision engine.

Figure 12: The DSL compilation pipeline. Source text is lexed, parsed into a resolved AST, and compiled to the internal RouterConfig . From this representation, three emitters produce deployment-specific formats (flat YAML, Kubernetes CRD, Helm values). The decompiler reverses the pipeline, reconstructing DSL source from an existing RouterConfig , enabling round-trip editing. The validator operates on the AST directly, producing three-level diagnostics.

Figure 13: Three-level validation architecture. The validator processes the resolved AST through progressively deeper analysis: syntactic errors (Level 1), reference resolution with fuzzy-matched QuickFix suggestions (Level 2), and semantic constraint checking (Level 3). Diagnostics at all levels are accumulated into a unified report.

Figure 14: Agent-based policy synthesis pipeline. A coding agent (LLM) translates natural-language routing specifications into DSL configurations, which are instantiated on the inference engine. Routing quality feedback closes the loop, enabling iterative policy refinement. The DSL’s functional completeness guarantees that any expressible routing policy is reachable by the synthesis process.

Figure 15: Signal-driven safety architecture. Safety classifiers (

\tau_{\text{jb}},\tau_{\text{pii}}

) run in the signal layer in parallel with intent signals, adding zero marginal latency. Decision rules compose safety signals with domain signals using AND/OR logic, enabling context-aware policies (e.g., strict jailbreak detection only for finance, relaxed PII policy for healthcare). High-priority safety decisions activate the fast_response plugin to return an immediate 200 OK without contacting any upstream model.

Figure 16: HaluGate three-stage gated pipeline. The Sentinel classifies queries on the request path; non-factual queries (40–60%) skip verification entirely (dashed). For factual queries, the Detector identifies hallucinated spans in the model response, and the Explainer provides NLI-based diagnostics per span.

Action	Semantics
block	Reject the response; return an error to the client. Appropriate for high-stakes factual applications.
header	Propagate detection metadata via HTTP headers, enabling downstream policy enforcement by the client or API gateway.
body	Prepend a warning to the response body, alerting users to potential inaccuracies.
none	Log detection results without modifying the response. Useful for monitoring and threshold calibration.

Table 1: HaluGate action policies upon hallucination detection

Figure 17: LoRA-based MoM inference architecture. The input query

\mathbf{x}

is processed by a single frozen base model (

W

). Each task-specific LoRA adapter pair (

A_{i}

,

B_{i}

) computes a low-rank perturbation

B_{i}A_{i}\mathbf{x}

. The base output and adapter perturbation are summed to produce task-specific classifications, enabling multi-task support with minimal memory overhead.

Model	Task	Training Data
mom-domain	Domain classification	MMLU categories
mom-pii	PII token classification	Presidio-annotated corpora
mom-jailbreak	Prompt injection detection	Adversarial prompt datasets
mom-sentinel	Fact-check gating	Factual vs. creative queries
mom-detector	Hallucination detection	Annotated LLM outputs
mom-explainer	NLI explanation	NLI benchmarks

Table 2: MoM model family. All models share a common base (ModernBERT [ 42 ] or mmBERT-32K) and are distributed as LoRA adapters.

Figure 18: Taxonomy of thirteen semantic model selection algorithms organized by selection mechanism. Families span from lightweight rating-based methods (Static, Elo) to learned approaches (RouterDC, Classical ML, RL), adaptive cascading (AutoMix), real-time latency tracking, and multi-round synthesis (ReMoM).

Figure 19: ReMoM execution flow with breadth schedule

\mathbf{b}=[4,2]

. Round 1 distributes 4 parallel calls across models (

m_{A}

,

m_{B}

,

m_{C}

); responses are (optionally compacted and) assembled into a synthesis prompt. Round 2 sends 2 parallel synthesis calls. A final round (

b_{3}=1

, auto-appended) produces the single output. Each synthesis prompt includes the original query and all previous-round responses as numbered references, delegating quality judgment to the synthesizing LLM.

Runtime	Target Hardware	Tasks	Framework
Candle	GPU (CUDA), CPU	Classification, LoRA, MLP	HF Candle [ 12 ]
Linfa	CPU only	KNN, KMeans, SVM	Linfa [ 21 ]
ONNX RT	CPU, GPU	Embeddings	ONNX Runtime [ 25 ]
NLP Binding	CPU only	BM25, N-gram matching	bm25 + ngrammatic (Rust)

Table 3: Inference runtime characteristics

Figure 20: Four-runtime ML inference architecture. The Go routing process links to four Rust shared libraries via CGo/C FFI. Each runtime is specialized for a different class of ML workload, avoiding Python overhead on the critical path.

Figure 21: Bidirectional request processing pipeline aligned with the three-layer architecture. Request path (top, left

\to

right): the signal layer parses and extracts

S(r)

, including safety signals (jailbreak, PII); the decision layer evaluates Boolean formulas to select

d^{*}

; the plugin chain executes pre-processing (fast response for safety enforcement, caching, RAG, memory), model selection, and endpoint routing. A cache hit or fast response short-circuits the pipeline (dashed). Response path (bottom, right

\to

left): the same three layers operate in the same order—the signal layer extracts response-side signals (token usage, HaluGate hallucination scores, streaming latency metrics); the decision layer evaluates quality gating and cache eligibility; plugins perform format translation, cache writes, and API translation.

Figure 22: Memory lifecycle. Write path : each conversation turn passes an entropy gate (discarding low-signal turns such as greetings), is sanitized and embedded as an episodic Q:/A: chunk, and stored directly—no LLM inference required. Every

s

turns an additional sliding-window chunk is stored. Read path : a heuristic gate skips retrieval for irrelevant queries; qualifying queries run hybrid search (vector + BM25 + n-gram) over the user’s store; retrieved memories pass through a ReflectionGate (safety blocklist, recency decay, deduplication, budget cap) before injection as a separate conversation message.

Figure 23: Hybrid search pipeline. Each candidate chunk is scored along three axes—vector cosine similarity, BM25 keyword relevance, and character

n

-gram Jaccard overlap—then combined via weighted fusion or Reciprocal Rank Fusion (RRF). Backends without native hybrid support fall back to a generic rerank path that fetches

4{\times}

top-

k

candidates from vector search before applying BM25 and

n

-gram scoring.

Figure 24: RAG backend architecture. The RAG plugin accesses vector stores through a common interface. Milvus and Llama Stack (with Milvus provider) support native hybrid search combining vector similarity with BM25 keyword matching via Reciprocal Rank Fusion. Other backends use vector-only retrieval with optional post-retrieval reranking.

Signal Type	Median	p99	Requires ML
Keyword	$<0.1$ ms	$<0.5$ ms	No
Context	$<0.1$ ms	$<0.5$ ms	No
Language	$<0.5$ ms	$<1$ ms	No
Authorization	$<0.1$ ms	$<0.5$ ms	No
Embedding	$15$ ms	$45$ ms	Yes
Domain	$60$ ms	$120$ ms	Yes

Table 4: Signal extraction latency by type

Seq. Length	CPU (ms)	GPU (ms)	Speedup
512	120	6.0	20 $\times$
1,024	263	7.7	34 $\times$
2,048	809	14.1	57 $\times$
4,096	2,664	57.6	46 $\times$
8,192	9,656	237	41 $\times$

Table 5: mmBERT-32K inference latency: CPU (FP32) vs. GPU SDPA (FP16), single session, batch size 1

Seq. Length	SDPA (ms)	FA (ms)	Speedup
512	19	19	1.0 $\times$
1,024	26	23	1.1 $\times$
2,048	51	32	1.6 $\times$
4,096	167	51	3.3 $\times$
8,192	OOM	105	—
16,384	OOM	259	—

Table 6: End-to-end classification latency: SDPA vs. CK Flash Attention (FP16, ROCm EP, 3 classifiers in parallel, concurrency 1, MI300X)

	$C{=}1$		$C{=}10$		$C{=}20$
Seq. Len.	med	p95	med	p95	med	p95
512	19	19	77	123	142	238
1,024	23	24	104	167	189	325
2,048	32	32	141	234	275	456
4,096	51	52	281	998	521	847
8,192	105	105	601	873	1,058	1,732

Table 7: CK Flash Attention latency under concurrent load (median / p95 in ms, MI300X)

Tasks ( $n$ )	Independent (MB)	LoRA (MB)
1	573	573
3	1,719	574
6	3,438	575

Table 8: Model memory: independent fine-tuned models vs. LoRA adapters (ModernBERT base, 150M params)

Scenario	Active Signals	Selection	Key Plugins
Privacy-regulated (healthcare)	authz, domain, language	Static (compliant models only)	Strict PII redaction, no caching, audit logging
Cost-optimized (developer tool)	complexity, embedding, keyword	AutoMix cascade	Aggressive semantic cache, header mutation for LoRA adapter
Multi-cloud enterprise	domain, modality, authz	Latency-aware	Multi-endpoint failover, provider auth factory, system prompt injection
Multi-turn assistant	embedding, feedback, preference	Elo with session pin	Responses API state, memory retrieval, RAG injection

Table 9: Composable signal orchestration across deployment scenarios. Each scenario activates a different subset of the thirteen signal types, selection algorithms, and plugin chains—using the same system binary and architecture.

Profile	Validated Behavior
Multi-endpoint	Multi-provider routing with weighted distribution and failover across heterogeneous backends
Multi-provider auth	Provider-specific auth injection (API key, OAuth2, cloud IAM) via authorization factory
AuthZ-RBAC	Role-based model access (admin/premium/free tiers) with authz signal
ML model selection	KNN, KMeans, SVM, MLP selection accuracy on held-out queries
Keyword routing	Keyword signal matching with AND/OR/NOR combinators
Embedding routing	Embedding similarity thresholds and confidence-based decision selection

Table 10: End-to-end test profiles

Figure 25: Unified MoM evaluation pipeline. Configuration and dataset sources feed into the data and model loaders, which populate the model registry. A process pool executor parallelizes inference across model variants, producing confusion matrices and aggregated metrics (weighted F1, per-class precision/recall) for both merged and LoRA models.

vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models

Abstract

Figures & tables

Explore similar work

SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference