cs.LGMar 22, 2026

The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project

Authors: Huamin Chen, Xunzhuo Liu, Bowei He, Fuyuan Lyu, Yankai Chen, Xue Liu, Yuhan Liu, Junchen Jiang

Organizations: vLLM Semantic Router Project · MBZUAI · McGill University · Mila · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Over the past year, the vLLM Semantic Router project has released a series of work spanning: (1) core routing mechanisms -- signal-driven routing, context-length pool routing, router performance engineering, policy conflict detection, low-latency embedding models, category-aware semantic caching, user-feedback-driven routing adaptation, hallucination detection, and hierarchical content-safety classification for privacy and jailbreak protection; (2) fleet optimization -- fleet provisioning and energy-efficiency analysis; (3) agentic and multimodal routing -- multimodal agent routing, tool selection, CUA security, and multi-turn context memory and safety; (4) governance and standards -- inference routing protocols and multi-provider API extensions. Each paper tackled a specific problem in LLM inference, but the problems are not independent; for example, fleet provisioning depends on the routing policy, which depends on the workload mix, shifting as organizations adopt agentic and multimodal workloads. This paper distills those results into the Workload-Router-Pool (WRP) architecture, a three-dimensional framework for LLM inference optimization. Workload characterizes what the fleet serves (chat vs. agent, single-turn vs. multi-turn, warm vs. cold, prefill-heavy vs. decode-heavy). Router determines how each request is dispatched (static semantic rules, online bandit adaptation, RL-based model selection, quality-aware cascading). Pool defines where inference runs (homogeneous vs. heterogeneous GPU, disaggregated prefill/decode, KV-cache topology). We map our prior work onto a 3x3 WRP interaction matrix, identify which cells we have covered and which remain open, and propose twenty-one concrete research directions at the intersections, each grounded in our prior measurements, tiered by maturity from engineering-ready to open research.

Figures & tables

Figure 1: The Workload–Router–Pool (WRP) architecture. Three dimensions interact through dashed bidirectional arrows. Safety & Privacy is a cross-cutting layer: content gating and hazard classification shape all three dimensions. All are further constrained by the optimization objectives at the bottom. Our prior work addresses one or more cells in this framework.
Short nameTitle / venueWRP
PoolRouting [ 12 ]Token-Budget-Aware Pool Routing (arXiv)∙\bullet∙\bullet∙\bullet
FleetOpt [ 13 ]Analytical Fleet Provisioning with C&R (arXiv)∙\bullet∙\bullet∙\bullet
1/W Law [ 15 ]Context-Length Routing and Energy Efficiency (arXiv)∙\bullet∙\bullet∙\bullet
FeedbackDet [ 7 ]mmBERT-32K Feedback Detector (HuggingFace)∙\bullet∙\bullet 
AVR [ 16 ]Adaptive VLM Routing for CUAs (arXiv)∙\bullet∙\bullet 
MPExt [ 21 ]Multi-Provider Extensions for Agentic AI (IETF)∙\bullet∙\bullet 
Table 1: Publications from the vLLM Semantic Router project that form the foundation of the WRP vision. Each row indicates which WRP dimension(s) the paper primarily addresses. W =Workload, R =Router, P =Pool.
 WorkloadRouterPool
WorkloadCDF archetypes [ 13 ] ; traces [ 23 , 26 ]AVR [ 16 ] (modality); VCD [ 18 ] (safety); SafetyL1/L2 [ 10 , 11 ] (content safety); FeedbackDet [ 7 ] (chat adapt.); HaluGate [ 8 ] (factual valid.); sparse—see Opportunities 1 and 4FleetOpt [ 13 ] (CDF →\to sizing); 1/W Law [ 15 ] (energy); sparse—see Opportunity 16
RouterFleetOpt [ 13 ] (compression changes CDF); WhenToReason [ 5 ]vLLM-SR [ 1 ] (signals); ProbPol [ 2 ] (conflicts); FastRouter [ 3 ] (latency); mmBERT-Embed [ 4 ] (embeddings); OATS [ 17 ] (tools); SemCache [ 6 ] (caching); HaluGate [ 8 ] (validation)FleetOpt [ 13 ] (co-design γ\gamma ); sparse—see Opportunities 11 and 15
PoolFleetSim [ 14 ] (topology →\to cost); 1/W [ 15 ] (topology →\to energy)sparse—see Opportunity 12FleetSim [ 14 ] (A10G/A100/H100); Mélange [ 63 ] ; DistServe [ 44 ]
Table 2: WRP interaction matrix. Each cell lists our publications that address the corresponding cross-dimensional interaction, plus key external references. Sparse cells indicate open opportunities.
#OpportunityTierPrimary barrier
2HaluGate model reputation routingEngineeringSample size per (model, domain) cell
5Token-budget enforcement for agentsEngineeringDomain-specific budget model calibration
8Follow-the-sun pool rebalancingEngineeringSliding-window CDF estimation + FleetOpt
12KV-cache retention directivesEngineeringRetention-directive API integration
15Governance-as-codeEngineeringExtend DSL with WRP constraints
19Request-level RBAC enforcementEngineeringGateway hook + JWT integration
Table 3: Maturity and feasibility of the twenty-one proposed opportunities. Tier indicates the primary barrier: engineering (building blocks exist, integration needed), research (open technical questions).
Figure 2: Knob-interaction dependencies across WRP opportunities. Adjusting one knob (e.g., model selection) cascades through the workload CDF into pool-layer parameters ( τ∗\tau^{*} , cascade threshold), requiring coordinated adaptation. The green arrow from model selection to token budget captures the memory-management feedback loop: poor model choices cause failures that pollute session context, driving token-budget explosion. Bidirectional arrow indicates the γ\gamma – τ∗\tau^{*} coupling from Opportunity 17 .
#OpportunityMonitor (observable)Knob (control)Evaluate (metric)
1Session-length RLSession trajectories (model, tool, outcome per turn)Model + tool selection policyMedian session length (turns, tokens)
2HaluGate reputationPer-response hallucination verdictsPer-(model, domain) routing weightsHallucination rate per domain
3Cumulative riskPer-turn SafetyL1 confidence scoresSafety escalation thresholdFPR vs. attack catch rate
4Failure-class routingMulti-signal outcome per (model, domain, failure class)Failure-class routing weightsOracle-utility capture rate
5Token budgetPer-session token counter + failure-driven context bloatBudget threshold + domain-specific compression policyToken waste from failed sessions
17Joint tool–model optimizationPer-(domain, turn, tool, model) outcomes + transition graphTool subset + resolving model + transition priors per dispatchAggregate session cost ×\times task-completion rate
Table 4: MAPE-K role of each opportunity: what the adaptation loop monitors, which knob it adjusts, and how it evaluates improvement.
More in this topic

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 349 papers

Explore similar work

CardsList
Mar 13, 2026

98×\times Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router

System-level routers that intercept LLM requests for safety classification, domain routing, and PII detection must be both fast and operationally lightweight: they should add minimal latency to every request, yet not require a dedicated GPU -- an expensive resource better used for LLM inference itself. When the router co-locates on the same GPU as vLLM serving instances, standard attention's O(n2)O(n^2) memory makes long-context classification (8K--32K tokens) impossible: at 8K tokens, three concurrent classifiers need ∼{\sim}4.5,GB for attention masks alone, far exceeding the memory left by vLLM. We present three staged optimizations for the vLLM Semantic Router, benchmarked on AMD Instinct MI300X, that solve both the latency and the memory problem. Stage~1: a custom CK Flash Attention operator for ONNX Runtime on ROCm reduces attention memory from O(n2)O(n^2) to O(n)O(n) and end-to-end (E2E) latency from 4{,}918,ms to 127,ms (\textbf{38.7×\times}), enabling 8K--32K tokens where SDPA OOMs. Stage~2: classical NLP prompt compression (TextRank, position weighting, TF-IDF, and novelty scoring) reduces all inputs to ∼{\sim}512 tokens without neural inference, capping both latency and GPU memory at a constant regardless of original prompt length (E2E 127→\to62,ms, \textbf{2.0×\times}). Stage~3: near-streaming body processing with adaptive chunking and zero-copy JSON eliminates serialization overhead (E2E 62→\to50,ms, \textbf{1.2×\times}). Cumulatively: \textbf{98×\times} improvement (4{,}918,ms to 50,ms), 16K-token routing in 108,ms, and a total router GPU footprint under 800,MB -- small enough to share a GPU with LLM serving and removing the need for a dedicated accelerator. Stage1 targets AMD ROCm (NVIDIA GPUs already have FlashAttention via cuDNN); Stages2 and~3 are hardware-agnostic.

Xunzhuo Liu, Bowei He, Xue Liu +3

vLLM Semantic Router Project · MBZUAI · McGill University +2

LLM RoutingLLM ServingLLM Inference AccelerationAttention Mechanisms
AskPDF
Feb 23, 2026

vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models

As large language models (LLMs) diversify across modalities, capabilities, and cost profiles, the problem of intelligent request routing: selecting the right model for each query at inference time, has become a critical systems challenge. We present vLLM Semantic Router, a signal-driven decision routing framework for Mixture-of-Modality (MoM) model deployments. The architecture follows two complementary Shannon-inspired views. In the information-theoretic regime, signal extraction reduces the entropy of "which model?" by distilling routing-relevant information from raw queries. In the Boolean-algebraic regime, the decision engine composes functionally complete routing policies from signal conditions. The central innovation is composable signal orchestration: thirteen heterogeneous signal types, spanning sub-millisecond heuristics and neural classifiers for semantics, safety, and modality, are composed through configurable Boolean decision rules into deployment-specific routing policies, so that fundamentally different scenarios (multi-cloud enterprise, privacy-regulated, cost-optimized) are expressed as different configurations over the same architecture. Matched decisions drive semantic model routing via thirteen selection algorithms, while per-decision plugin chains enforce safety constraints including a three-stage HaluGate hallucination detection pipeline and a lightweight episodic memory system with ReflectionGate for personalized multi-turn context. A typed neural-symbolic DSL specifies these routing policies and compiles them to multiple deployment targets, enabling configuration-first adaptation without code changes. Together, these components show that composable signal orchestration enables a single framework to serve diverse deployment scenarios with differentiated cost, privacy, and safety policies.

Xunzhuo Liu, Huamin Chen, Samzong Lu +30

Steve

Multimodal Model RoutingLLM RoutingAdaptive Model RoutingLLM Serving
AskPDF
Oct 2, 2026

Dynamic LLM Routers are Often Misguided

Dynamic LLM routers promise to cut inference costs by sending each query to the cheapest model that can answer it correctly. We analyze six commercial routers across 14 settings on a diverse benchmark spanning eight task categories, finding that none of them outperforms a router that randomly selects between two well-chosen models at matched cost. Some underperform by more than 10 percentage points. We trace this gap to four patterns prevalent across routers: difficulty blindness, length reversal, semantic matching, and roster suboptimality. We show that the first three are what the standard objective rewards: cost-accuracy Pareto efficiency on realized costs favors escalating moderately hard queries over the hardest ones, shorter queries over longer ones, and routing by a query's source over its difficulty. We also argue that the two assumptions that would justify large rosters, model granularity and model specialization, do not hold empirically. We propose an alternative evaluation methodology that does not reward these patterns, and as a proof of concept, we design a simple two-model router that avoids all four. Nevertheless, its gain over random routing is limited, because a well-chosen roster leaves little to route.

Sam Wang, Julia White, Sahibzada Allahyar +3

Fastino Labs

LLM RoutingCost-Aware InferenceLLM Evaluation
AskPDF