98$\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router

Xunzhuo Liu; Bowei He; Xue Liu; Andy Luo; Haichen Zhang; Huamin Chen

cs.CLMar 13, 2026

98 $\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router

Authors: Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo, Haichen Zhang, Huamin Chen

Organizations: vLLM Semantic Router Project · MBZUAI · McGill University · AMD ·

Abstract

System-level routers that intercept LLM requests for safety classification, domain routing, and PII detection must be both fast and operationally lightweight: they should add minimal latency to every request, yet not require a dedicated GPU -- an expensive resource better used for LLM inference itself. When the router co-locates on the same GPU as vLLM serving instances, standard attention's $O(n^2)$ memory makes long-context classification (8K--32K tokens) impossible: at 8K tokens, three concurrent classifiers need ${\sim}$ 4.5,GB for attention masks alone, far exceeding the memory left by vLLM. We present three staged optimizations for the vLLM Semantic Router, benchmarked on AMD Instinct MI300X, that solve both the latency and the memory problem. Stage~1: a custom CK Flash Attention operator for ONNX Runtime on ROCm reduces attention memory from $O(n^2)$ to $O(n)$ and end-to-end (E2E) latency from 4{,}918,ms to 127,ms (\textbf{38.7 $\times$ }), enabling 8K--32K tokens where SDPA OOMs. Stage~2: classical NLP prompt compression (TextRank, position weighting, TF-IDF, and novelty scoring) reduces all inputs to ${\sim}$ 512 tokens without neural inference, capping both latency and GPU memory at a constant regardless of original prompt length (E2E 127 $\to$ 62,ms, \textbf{2.0 $\times$ }). Stage~3: near-streaming body processing with adaptive chunking and zero-copy JSON eliminates serialization overhead (E2E 62 $\to$ 50,ms, \textbf{1.2 $\times$ }). Cumulatively: \textbf{98 $\times$ } improvement (4{,}918,ms to 50,ms), 16K-token routing in 108,ms, and a total router GPU footprint under 800,MB -- small enough to share a GPU with LLM serving and removing the need for a dedicated accelerator. Stage~~1 targets AMD ROCm (NVIDIA GPUs already have FlashAttention via cuDNN); Stages~~2 and~3 are hardware-agnostic.

Figures & tables

Method	Requires Neural Inf.	Compression Granularity	Attention Preserved?	Compression Latency
LLMLingua [ 9 ]	LLaMa-7B	Token	Yes ∗	${\sim}$ 2–5 s
LLMLingua-2 [ 10 ]	XLM-R (560M)	Token	Yes ∗	${\sim}$ 0.3–1 s
Selective Context [ 11 ]	GPT-2/LLaMa	Sentence/phrase	Partial	${\sim}$ 1–3 s
RECOMP [ 12 ]	Trained encoder	Abstractive	No (rewritten)	${\sim}$ 0.5–2 s
Ours	None	Sentence	Yes (U-shaped + novelty)	${\sim}$ 19 ms

TABLE I: Comparison of prompt compression approaches. Compression latency is for a 16K-token input. Our method is the only one requiring zero neural inference, keeping compression overhead to

{\sim}

19 ms at 16K tokens vs. seconds for neural methods. †

Panel of Figure 2 — Fig. 1: Co-located deployment: the router’s ONNX classifiers share the MI300X GPU with vLLM inference, consuming only ${\sim}$ 0.8 GB vs. vLLM’s ${\sim}$ 190 GB—no dedicated accelerator needed.

Panel of Figure 3 — Fig. 2: Near-streaming body handler state machine. The first chunk determines the path: specified models bypass body inspection entirely; “auto” requests accumulate for classification with incremental preprocessing.

Strategy	Inspect?	Mutate?	Overhead	Copies
Pure passthrough	No	No	${\sim}$ 0	0
Header-only routing	No	Headers	${\sim}$ 0	0
Full buffering	Yes	Yes	$O(n)$	2–3
Ours	Adaptive	Adaptive	$O(1)$ / $O(n)$ ∗	0/1 ∗

TABLE II: Body-processing strategies for LLM request proxies. Our near-streaming approach achieves passthrough-level latency when the model is specified, retaining inspection capability when needed.

Component	Version / Spec
GPU	AMD Instinct MI300X, 192 GB HBM3
GPU arch	gfx942 (CDNA 3)
ROCm	7.0
ONNX Runtime	1.22.1 (ROCm wheel)
CK Flash Attn lib	libort_ck_flash_attn.so (192 KB)
Rust ort crate	2.0.0-rc.10 (load-dynamic)

TABLE III: Testbed configuration.

End-to-end routing latency (ONNX GPU+FA vs ONNX CPU)
Tokens	Signal	GPU (ms)	CPU (ms)	Speedup
${\sim}$ 500	(all)	22	803	36.5 $\times$
${\sim}$ 2,000	(all)	31	1,773	57.2 $\times$
${\sim}$ 8,000	(all)	127	4,918	38.7 $\times$
Per-signal extraction latency
${\sim}$ 500	Jailbreak	14.2	779.8	54.9 $\times$

TABLE IV: GPU acceleration on MI300X: E2E routing latency (top) and per-signal extraction latency (bottom). Candle CPU at 8K truncates to 512 tokens (1,818 ms E2E).

N{=}10

per run; see § V-A .

Seq Length	SDPA (ms)	FA (ms)	Speedup
512	19	19	1.0 $\times$
1,024	26	23	1.1 $\times$
2,048	51	32	1.6 $\times$
4,096	167	51	3.3 $\times$
8,192	OOM	105	—
16,384	OOM	259	—

TABLE V: SDPA vs CK Flash Attention (ms), concurrency 1. SDPA OOMs at 8K+ with 3 concurrent sessions.

Fig. 3: Attention memory per classifier session vs. sequence length. SDPA exceeds the ${\sim}$ 718 MB available (dashed line) at ${\sim}$ 6K tokens. FA stays below but grows linearly. With compression, all inputs are reduced to 512 tokens, yielding a constant footprint regardless of original prompt length.

Tokens	Signal	No Compression	Compressed
${\sim}$ 500	Jailbreak	10.1 ms	9.3 ms
${\sim}$ 2,000	Jailbreak	18.0 ms	11.1 ms
${\sim}$ 8,000	Jailbreak	45.3 ms	10.5 ms
${\sim}$ 16,000	Jailbreak	126.6 ms	10.4 ms
${\sim}$ 16,000	Domain	85.3 ms	7.1 ms
${\sim}$ 16,000	PII	84.6 ms	6.2 ms

TABLE VI: Signal extraction latency with compression (STREAMED, GPU+FA). 16K

\to

512 tokens yields 12

\times

jailbreak speedup with identical results.

Input Tokens	Output Tokens	Ratio	Latency (ms)
${\sim}$ 2,000	510	25.1%	2
${\sim}$ 4,000	511	12.7%	4
${\sim}$ 8,000	512	6.4%	9
${\sim}$ 16,000	512	3.2%	19

TABLE VII: Prompt compression ratio and latency by input size (

N{=}96

per size, max_tokens

=

512). Compression latency is CPU-only (no GPU inference).

Configuration	Tokens	E2E (ms)	Stage Speedup	Cumulative
ONNX CPU, BUFFERED (baseline)	8K	4,918	—	1.0 $\times$
Candle CPU, BUFFERED	8K	1,818	2.7 $\times$	2.7 $\times$
ONNX GPU (SDPA), BUFFERED	8K	OOM	—	—
Stage 1: GPU + CK Flash Attention
ONNX GPU (FA), BUFFERED	8K	127	38.7 $\times$	38.7 $\times$
Stage 2: + Prompt Compression (8K $\to$ ${\sim}$ 512 tokens)

TABLE VIII: Staged acceleration at 8K tokens. Each stage builds on the previous; per-stage and cumulative speedups are measured E2E through Envoy (

N{=}10

per run, 2–3 runs; § V-A ). The 16K row shows scaling beyond the CPU-operable regime.

Tokens	BUFFERED (ms)	STREAMED (ms)	Reduction
${\sim}$ 500	12	11	8.3%
${\sim}$ 2,000	15	14	6.7%
${\sim}$ 8,000	30	20	33.3%
${\sim}$ 16,000	61	31	49.2%

TABLE IX: Domain-only E2E latency: BUFFERED vs. STREAMED with prompt compression and GPU+FA (

N{=}15

per size, single classifier). Reduction is relative to BUFFERED at the same token count.

Fig. 4: E2E latency comparison (log scale). GPU+FA with streaming and compression achieves 50 ms at 8K tokens. At 16K+, only GPU+FA operates (108 ms at 16K; Table VIII ).

Raw FA classifier latency (median ms, no compression)
Original Tokens	Mode	C=1	C=10	C=20
512	FA only	19	77	142
2,048	FA only	32	141	275
8,192	FA only	105	601	1,058
16,384	FA only	259	1,567	3,089
32,768	FA only	756	5,406	9,872

TABLE X: Classifier latency under concurrent load on MI300X: raw FA (top) vs. effective with compression + streaming (bottom). With compression all inputs reduce to

{\sim}

512 tokens. Bottom-panel C=1 values at 512–16K are measured; 32K C=1 is estimated. C

>

1 values are derived from C=1 overhead plus the 512-token FA concurrency profile.

Domain classification (correct domain, $n{=}384$ )
Metric	Original	Compressed
Overall	53.1%	61.2%
PII detection (when PII signal embedded, $n{=}288$ )
PII detected	78.5%	92.4%
Jailbreak detection (when signal embedded, $n{=}288$ )
Jailbreak detected	70.8%	56.6%

TABLE XI: Classification accuracy: original (uncompressed) vs. compressed prompts classified by the production mmBERT-32K models (384 test cases, 8 domains

\times

4 sizes

\times

12 signal combos, MI300X GPU with FA).

Configuration	Latency (ms)	req/s
ONNX CPU, 8K (baseline)	4,918	0.2
GPU+FA, 500 tok	22	45.5
GPU+FA, 8K tok	127	7.9
GPU+FA+comp+stream, 500 tok	17	58.8
GPU+FA+comp+stream, 8K tok	50	20.0
GPU+FA+comp+stream, 16K tok	108	9.3

TABLE XII: Single-stream throughput (C=1) derived from E2E latency.

Router	Model	GPU Mem.	Ded. GPU?
RouteLLM (MF)	Mat. factorization	0 (CPU)	No
RouteLLM (BERT)	0.3B classifier	${\sim}$ 0.6 GB	Yes
RouteLLM (Causal)	8B LLM	${\sim}$ 16 GB	Yes
NVIDIA Blueprint	BERT/CLIP + Triton	${\sim}$ 1–2 GB	Yes
R2-Router	Reasoning LLM	${\sim}$ 16+ GB	Yes
Ours	3 $\times$ mmBERT-32K	${\sim}$ 0.8 GB	No

TABLE XIII: GPU resource requirements of LLM routing systems. Our router is the only one designed to co-locate on the serving GPU.

Router Config.	GPU Mem.	GPU	$/yr/node
SDPA, 8K, C=1	${\sim}$ 5 GB	L4	$3,500
SDPA, 16K, C=1	${\sim}$ 19 GB	A10G	$8,800
SDPA, 32K, C=1	${\sim}$ 73 GB	A100 80G	$10,100
RouteLLM (Causal)	${\sim}$ 16 GB	A10G	$8,800
R2-Router	${\sim}$ 16+ GB	A10G	$8,800
Ours (co-located)	0.8 GB	None	$0

TABLE XIV: Annual cost of a dedicated router GPU per serving node, based on cloud on-demand pricing. Our router adds $0 by co-locating on the existing serving GPU.

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 351 papers

Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference
Mar 13, 2026Huamin Chen, Xunzhuo Liu, Junchen Jiang +2LLM Inference Efficiency LLM Routing
vLLM Semantic Router Project · Tensormesh Inc / UChicago · MBZUAI / McGill University
LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
Aug 7, 2026Tao Feng, Fangxu Yu, Haozhen Zhang +9LLM Evaluation LLM Routing
University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2
Dynamic LLM Routers are Often Misguided
Oct 2, 2026Sam Wang, Julia White, Sahibzada Allahyar +3LLM Routing Cost-Aware Inference
Fastino Labs

Mar 13, 2026J/K move · Enter open · S save

Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference

Huamin Chen, Xunzhuo Liu, Junchen Jiang, Bowei He+1

vLLM Semantic Router Project · Tensormesh Inc / UChicago · MBZUAI / McGill University

LLM Inference Efficiency LLM Routing Cost-Aware Inference LLM Request Scheduling

Open paper AskPDF arXiv

Production vLLM fleets provision every instance for worst-case context length, wasting 4-8x concurrency on the 80-95% of requests that are short and simultaneously triggering KV-cache failures -- OOM crashes, preemption storms, and request rejections. Both problems share a single root cause: configuration-traffic mismatch. We propose token-budget-aware pool routing: estimate each request's total token budget using a self-calibrating per-category bytes-per-token ratio, then dispatch it to one of two vLLM pools -- a high-throughput short pool or a high-capacity long pool -- each right-sized for its workload class. The ratio is learned online via exponential moving average from usage.prompt_tokens feedback, requiring no tokenizer. A closed-form cost model, savings = alpha * (1 - 1/rho), predicts fleet-level GPU savings from two observable quantities: the short-traffic fraction alpha and the throughput gain ratio rho. On traces from the Azure LLM Inference Dataset and LMSYS-Chat-1M serving Llama-3-70B on A100 GPUs, token-budget routing reduces GPU instances by 17-39% ($1.2-2.0M/yr at 1,000 req/s), with savings verified by a self-contained discrete-event simulator. A case study projecting Qwen3-235B-A22B on AMD MI300X at 10,000 req/s shows $15.4M/yr in savings. The algorithm adds O(1) dispatch overhead, self-calibrates across content types without a tokenizer, and composes with PagedAttention, continuous batching, and prefill-decode disaggregation.

Abstract

LLM Routing

Explore similar work

Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

Dynamic LLM Routers are Often Misguided