cs.NIFeb 23, 2026
SavevLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models
Organizations: Steve
Abstract
As large language models (LLMs) diversify across modalities, capabilities, and cost profiles, the problem of intelligent request routing: selecting the right model for each query at inference time, has become a critical systems challenge. We present vLLM Semantic Router, a signal-driven decision routing framework for Mixture-of-Modality (MoM) model deployments. The architecture follows two complementary Shannon-inspired views. In the information-theoretic regime, signal extraction reduces the entropy of "which model?" by distilling routing-relevant information from raw queries. In the Boolean-algebraic regime, the decision engine composes functionally complete routing policies from signal conditions. The central innovation is composable signal orchestration: thirteen heterogeneous signal types, spanning sub-millisecond heuristics and neural classifiers for semantics, safety, and modality, are composed through configurable Boolean decision rules into deployment-specific routing policies, so that fundamentally different scenarios (multi-cloud enterprise, privacy-regulated, cost-optimized) are expressed as different configurations over the same architecture. Matched decisions drive semantic model routing via thirteen selection algorithms, while per-decision plugin chains enforce safety constraints including a three-stage HaluGate hallucination detection pipeline and a lightweight episodic memory system with ReflectionGate for personalized multi-turn context. A typed neural-symbolic DSL specifies these routing policies and compiles them to multiple deployment targets, enabling configuration-first adaptation without code changes. Together, these components show that composable signal orchestration enables a single framework to serve diverse deployment scenarios with differentiated cost, privacy, and safety policies.
Figures & tables
Figure 1: Structural correspondence between Shannon’s communication system [ 35 ] and VSR’s routing pipeline. The dotted line marks the boundary between the information-theoretic regime (signal extraction maximizes mutual information with the routing outcome) and the Boolean-algebraic regime (the decision engine synthesizes routing policies from signal conditions via [ 34 ] ).
Figure 2: Entropy collapse during signal extraction (illustrative). Each additional signal reduces the routing uncertainty from the uniform prior bits until the decision formula yields a near-deterministic model selection. Bar heights are schematic rather than empirical measurements.
Figure 3: Three-layer architecture with closed-loop feedback. A deployment configuration selects which signals, decisions, and plugins are active. Layer 1 extracts a signal vector from the request. Layer 2 evaluates Boolean decision formulas to select . Layer 3 executes the per-decision plugin chain, selects a model from ’s candidate set, and routes to the provider endpoint. Response-side signals feed back to enable adaptive routing.
Figure 4: Signal extraction taxonomy and evaluation flow. An incoming request is evaluated in parallel against heuristic signals (sub-millisecond, deterministic) and learned signals (neural inference via LoRA classifiers). Only signal types referenced by configured decisions are computed (demand-driven evaluation). Results merge into the structured signal result .
Figure 5: Comparison of embedding strategies. Top : A Transformer embedding layer produces dense, opaque vectors optimized end-to-end. Bottom : The VSR signal extraction layer produces structured, interpretable signal vectors from heterogeneous extractors (heuristic, learned, LoRA-based), where each dimension has a human-readable semantic name. The dashed arrow marks the functional correspondence: both convert unstructured input into a representation for downstream computation, but the signal vector is interpretable by construction.
Figure 6: Rule-node expression trees at increasing depth. (a) A flat depth-1 tree: AND over three leaf conditions. (b) A NOR expression: , matching all non-STEM queries. (c) An XOR expression composed from AND, OR, and NOT primitives, routing requests that match exactly one of two signals. Leaf nodes (gray) reference signal conditions; composite nodes use AND ( blue ), OR ( green ), and NOT ( orange ).
Figure 7: Three-level correspondence between combinational logic circuits and the decision engine. Level 1 : A PLA with AND-plane and OR-plane corresponds to a flat (depth-1) decision formula. Level 2 : A general combinational circuit with arbitrarily nested AND, OR, and NOT gates corresponds to a recursive rule-node tree within a single decision. Level 3 : An array of circuits with a priority encoder corresponds to the full decision set with priority-ordered evaluation, realizing any routing policy.
Figure 8: Structural analogy between Mixture-of-Experts gating and the VSR decision engine. Left : In a sparse MoE layer, a learned gating network selects top- experts via softmax. Right : In VSR, Boolean formulas over the signal vector act as symbolic expert gates, with priority ordering implementing deterministic early-exit selection. Dashed arrows mark the structural correspondence.
Figure 9: Transformer-style visualization of layered control depth in VSR. Signal extraction constructs a shared embedding ; stacked decision layers apply Boolean gates that progressively reduce routing uncertainty ; selection resolves via priority early-exit; plugin composition acts as an output-side projection head over request/response behavior.
Figure 11: Boolean expression AST for WHEN keyword("urgent") AND NOT domain("math") . Each leaf is a SignalRefExpr referencing a named signal; internal nodes are Boolean operators ( BoolAnd , BoolNot ). The compiler flattens this tree into a RuleCombination structure for the decision engine.
Figure 12: The DSL compilation pipeline. Source text is lexed, parsed into a resolved AST, and compiled to the internal RouterConfig . From this representation, three emitters produce deployment-specific formats (flat YAML, Kubernetes CRD, Helm values). The decompiler reverses the pipeline, reconstructing DSL source from an existing RouterConfig , enabling round-trip editing. The validator operates on the AST directly, producing three-level diagnostics.
Figure 13: Three-level validation architecture. The validator processes the resolved AST through progressively deeper analysis: syntactic errors (Level 1), reference resolution with fuzzy-matched QuickFix suggestions (Level 2), and semantic constraint checking (Level 3). Diagnostics at all levels are accumulated into a unified report.
Figure 14: Agent-based policy synthesis pipeline. A coding agent (LLM) translates natural-language routing specifications into DSL configurations, which are instantiated on the inference engine. Routing quality feedback closes the loop, enabling iterative policy refinement. The DSL’s functional completeness guarantees that any expressible routing policy is reachable by the synthesis process.
Figure 15: Signal-driven safety architecture. Safety classifiers ( ) run in the signal layer in parallel with intent signals, adding zero marginal latency. Decision rules compose safety signals with domain signals using AND/OR logic, enabling context-aware policies (e.g., strict jailbreak detection only for finance, relaxed PII policy for healthcare). High-priority safety decisions activate the fast_response plugin to return an immediate 200 OK without contacting any upstream model.
Figure 16: HaluGate three-stage gated pipeline. The Sentinel classifies queries on the request path; non-factual queries (40–60%) skip verification entirely (dashed). For factual queries, the Detector identifies hallucinated spans in the model response, and the Explainer provides NLI-based diagnostics per span.
| Action | Semantics |
|---|---|
| block | Reject the response; return an error to the client. Appropriate for high-stakes factual applications. |
| header | Propagate detection metadata via HTTP headers, enabling downstream policy enforcement by the client or API gateway. |
| body | Prepend a warning to the response body, alerting users to potential inaccuracies. |
| none | Log detection results without modifying the response. Useful for monitoring and threshold calibration. |
Table 1: HaluGate action policies upon hallucination detection
Figure 17: LoRA-based MoM inference architecture. The input query is processed by a single frozen base model ( ). Each task-specific LoRA adapter pair ( , ) computes a low-rank perturbation . The base output and adapter perturbation are summed to produce task-specific classifications, enabling multi-task support with minimal memory overhead.
| Model | Task | Training Data |
|---|---|---|
| mom-domain | Domain classification | MMLU categories |
| mom-pii | PII token classification | Presidio-annotated corpora |
| mom-jailbreak | Prompt injection detection | Adversarial prompt datasets |
| mom-sentinel | Fact-check gating | Factual vs. creative queries |
| mom-detector | Hallucination detection | Annotated LLM outputs |
| mom-explainer | NLI explanation | NLI benchmarks |
Table 2: MoM model family. All models share a common base (ModernBERT [ 42 ] or mmBERT-32K) and are distributed as LoRA adapters.
Figure 18: Taxonomy of thirteen semantic model selection algorithms organized by selection mechanism. Families span from lightweight rating-based methods (Static, Elo) to learned approaches (RouterDC, Classical ML, RL), adaptive cascading (AutoMix), real-time latency tracking, and multi-round synthesis (ReMoM).
Figure 19: ReMoM execution flow with breadth schedule . Round 1 distributes 4 parallel calls across models ( , , ); responses are (optionally compacted and) assembled into a synthesis prompt. Round 2 sends 2 parallel synthesis calls. A final round ( , auto-appended) produces the single output. Each synthesis prompt includes the original query and all previous-round responses as numbered references, delegating quality judgment to the synthesizing LLM.
| Runtime | Target Hardware | Tasks | Framework |
|---|---|---|---|
| Candle | GPU (CUDA), CPU | Classification, LoRA, MLP | HF Candle [ 12 ] |
| Linfa | CPU only | KNN, KMeans, SVM | Linfa [ 21 ] |
| ONNX RT | CPU, GPU | Embeddings | ONNX Runtime [ 25 ] |
| NLP Binding | CPU only | BM25, N-gram matching | bm25 + ngrammatic (Rust) |
Table 3: Inference runtime characteristics
Figure 20: Four-runtime ML inference architecture. The Go routing process links to four Rust shared libraries via CGo/C FFI. Each runtime is specialized for a different class of ML workload, avoiding Python overhead on the critical path.
Figure 21: Bidirectional request processing pipeline aligned with the three-layer architecture. Request path (top, left right): the signal layer parses and extracts , including safety signals (jailbreak, PII); the decision layer evaluates Boolean formulas to select ; the plugin chain executes pre-processing (fast response for safety enforcement, caching, RAG, memory), model selection, and endpoint routing. A cache hit or fast response short-circuits the pipeline (dashed). Response path (bottom, right left): the same three layers operate in the same order—the signal layer extracts response-side signals (token usage, HaluGate hallucination scores, streaming latency metrics); the decision layer evaluates quality gating and cache eligibility; plugins perform format translation, cache writes, and API translation.
Figure 22: Memory lifecycle. Write path : each conversation turn passes an entropy gate (discarding low-signal turns such as greetings), is sanitized and embedded as an episodic Q:/A: chunk, and stored directly—no LLM inference required. Every turns an additional sliding-window chunk is stored. Read path : a heuristic gate skips retrieval for irrelevant queries; qualifying queries run hybrid search (vector + BM25 + n-gram) over the user’s store; retrieved memories pass through a ReflectionGate (safety blocklist, recency decay, deduplication, budget cap) before injection as a separate conversation message.
Figure 23: Hybrid search pipeline. Each candidate chunk is scored along three axes—vector cosine similarity, BM25 keyword relevance, and character -gram Jaccard overlap—then combined via weighted fusion or Reciprocal Rank Fusion (RRF). Backends without native hybrid support fall back to a generic rerank path that fetches top- candidates from vector search before applying BM25 and -gram scoring.
Figure 24: RAG backend architecture. The RAG plugin accesses vector stores through a common interface. Milvus and Llama Stack (with Milvus provider) support native hybrid search combining vector similarity with BM25 keyword matching via Reciprocal Rank Fusion. Other backends use vector-only retrieval with optional post-retrieval reranking.
| Signal Type | Median | p99 | Requires ML |
|---|---|---|---|
| Keyword | ms | ms | No |
| Context | ms | ms | No |
| Language | ms | ms | No |
| Authorization | ms | ms | No |
| Embedding | ms | ms | Yes |
| Domain | ms | ms | Yes |
Table 4: Signal extraction latency by type
| Seq. Length | CPU (ms) | GPU (ms) | Speedup |
|---|---|---|---|
| 512 | 120 | 6.0 | 20 |
| 1,024 | 263 | 7.7 | 34 |
| 2,048 | 809 | 14.1 | 57 |
| 4,096 | 2,664 | 57.6 | 46 |
| 8,192 | 9,656 | 237 | 41 |
Table 5: mmBERT-32K inference latency: CPU (FP32) vs. GPU SDPA (FP16), single session, batch size 1
| Seq. Length | SDPA (ms) | FA (ms) | Speedup |
|---|---|---|---|
| 512 | 19 | 19 | 1.0 |
| 1,024 | 26 | 23 | 1.1 |
| 2,048 | 51 | 32 | 1.6 |
| 4,096 | 167 | 51 | 3.3 |
| 8,192 | OOM | 105 | — |
| 16,384 | OOM | 259 | — |
Table 6: End-to-end classification latency: SDPA vs. CK Flash Attention (FP16, ROCm EP, 3 classifiers in parallel, concurrency 1, MI300X)
| Seq. Len. | med | p95 | med | p95 | med | p95 |
|---|---|---|---|---|---|---|
| 512 | 19 | 19 | 77 | 123 | 142 | 238 |
| 1,024 | 23 | 24 | 104 | 167 | 189 | 325 |
| 2,048 | 32 | 32 | 141 | 234 | 275 | 456 |
| 4,096 | 51 | 52 | 281 | 998 | 521 | 847 |
| 8,192 | 105 | 105 | 601 | 873 | 1,058 | 1,732 |
Table 7: CK Flash Attention latency under concurrent load (median / p95 in ms, MI300X)
| Tasks ( ) | Independent (MB) | LoRA (MB) |
|---|---|---|
| 1 | 573 | 573 |
| 3 | 1,719 | 574 |
| 6 | 3,438 | 575 |
Table 8: Model memory: independent fine-tuned models vs. LoRA adapters (ModernBERT base, 150M params)
| Scenario | Active Signals | Selection | Key Plugins |
|---|---|---|---|
| Privacy-regulated (healthcare) | authz, domain, language | Static (compliant models only) | Strict PII redaction, no caching, audit logging |
| Cost-optimized (developer tool) | complexity, embedding, keyword | AutoMix cascade | Aggressive semantic cache, header mutation for LoRA adapter |
| Multi-cloud enterprise | domain, modality, authz | Latency-aware | Multi-endpoint failover, provider auth factory, system prompt injection |
| Multi-turn assistant | embedding, feedback, preference | Elo with session pin | Responses API state, memory retrieval, RAG injection |
Table 9: Composable signal orchestration across deployment scenarios. Each scenario activates a different subset of the thirteen signal types, selection algorithms, and plugin chains—using the same system binary and architecture.
| Profile | Validated Behavior |
|---|---|
| Multi-endpoint | Multi-provider routing with weighted distribution and failover across heterogeneous backends |
| Multi-provider auth | Provider-specific auth injection (API key, OAuth2, cloud IAM) via authorization factory |
| AuthZ-RBAC | Role-based model access (admin/premium/free tiers) with authz signal |
| ML model selection | KNN, KMeans, SVM, MLP selection accuracy on held-out queries |
| Keyword routing | Keyword signal matching with AND/OR/NOR combinators |
| Embedding routing | Embedding similarity thresholds and confidence-based decision selection |
Table 10: End-to-end test profiles
Figure 25: Unified MoM evaluation pipeline. Configuration and dataset sources feed into the data and model loaders, which populate the model registry. A process pool executor parallelizes inference across model variants, producing confusion matrices and aggregated metrics (weighted F1, per-class precision/recall) for both merged and LoRA models.