VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling
Organizations: Sprinklr · Microsoft
Abstract
Interactive LLM services must meet latency service-level objectives (SLOs) for the time to first token (TTFT) and the time between tokens (TBT), as streaming and soft real-time applications require predictable token delivery. We identify decode jitters in both aggregated and disaggregated serving. A jitter delays every request in the batch, and long responses span hundreds of steps, so even rare overruns cause many SLO violations. In aggregated serving, SLO-aware schedulers set each step's prefill budget until its predicted latency just meets the TBT target, leaving no room for mispredictions or host stalls. Disaggregation meets average TBT targets at a cost in capacity, yet the steps that admit new requests still run late. We present \sysname, a scheduler that calibrates how much prefill work each decode step carries using the measured tail of its own iteration latency. The SLO determines how rare a late step must be, and \sysname leaves enough headroom in each iteration to prevent overruns and meet the target decode latency. This headroom slows prefill, so \sysname serves the shortest waiting prompt first. When TTFT deadlines grow with prompt length, the long prompts it delays have the most time to spare. Under disaggregation, it spaces admissions so that no request waits on more slow admission steps than its P99 target allows. On three attention families and varying workloads, \sysname sustains 2.4--2.9 the SLO goodput of the strongest baseline on Qwen3-8B and DeepSeek-V4-Flash, and at least 4.4 on DeepSeek-V2-Lite. With prefill and decode disaggregated on eight GPUs, \sysname meets the SLO for more than twice as many requests as state-of-the-art baselines.
Figures & tables
| System | Token budget | Prompt order |
|---|---|---|
| vLLM | fixed (8192 tokens) | FCFS |
| Sarathi | fixed, profiled for the SLO | FCFS |
| SPF | fixed | shortest prompt first |
| Medha | up to , no margin | least slack |
| Niyama | up to slack, p99 margin † | deadline + work left |
| VelServe | up to minus measured tail | shortest left ‡ |
| System | Qwen3-8B | DS-V2-Lite | DS-V4-Flash |
|---|---|---|---|
| Niyama | 1.71 / 2.15 | 1.00 / 1.11 | 8.37 / 12.40 |
| Medha | 0.00 / 1.36 | 0.00 / 1.60 | 0.00 / 3.97 |
| VelServe | 0.00 / 0.00 | 0.00 / 0.01 | 0.00 / 0.02 |
| Rate (req/s) | ||||
|---|---|---|---|---|
| Scheduler | Margin | 0.6 | 1.0 | 1.4 |
| VelServe | online ( 48 ms) | 99.3 | 99.6 | 96.4 |
| frozen offline (9.4 ms) | 66.4 | 47.6 | 35.6 | |
| frozen, other workload (29.3 ms) | 99.3 | 83.1 | 91.8 | |
| Medha | none | 100.0 | 57.6 | 25.7 |
| 29.3 ms, other workload | 100.0 | 87.4 | 51.7 | |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| GPU | 1 NVIDIA H100 NVL (94 GB HBM3, 132 SMs, 400 W cap) |
| Software | vLLM 0.30.0 (V1), CUDA 13.0, PyTorch 2.13; CUDA graphs up to 2304-token iterations, asynchronous scheduling, 256 running requests, 90% of memory for weights and KV |
| Models (attention; layers; weights; KV per token) | |
| GQA | Qwen3-8B; 36; real weights, YaRN-extended to 128k; 144 KB |
| MLA | DeepSeek-V2-Lite; 27; real weights; 30 KB |
| CSA/HCA | DeepSeek-V4-Flash; 43; real weights of routed experts 0–63 of 256 per layer, router sliced to them (the per-GPU expert footprint of a 4-way expert-parallel shard); 10 KB |
| Workloads | Mixed : Poisson arrivals; prompts 128–2k tokens (75%), 8k–32k (20%) and 64k–120k (5%); outputs 64–512 tokens. Mooncake-shaped : Poisson arrivals; prompt and output lengths drawn from the Mooncake conversation trace [ 28 ] . Mooncake trace (CSA only): 150 consecutive requests of the same trace, replayed and time-scaled. Multi-turn : sessions of 3–6 turns over a 16k–96k-token document, plus short background requests. Agent trace (CSA): Mooncake tool-agent requests with the prefixes they share. Prefix caching is on for the last two only |
| Model | TBT SLO | vLLM | Sarathi | SPF | Niyama | Medha | VelServe | best |
|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | P99 TBT | 0.1 | 0.1 | 0.10 | 0.1 | 0.65 | 1.54 | 2.4 |
| (GQA) | Worst gap | 0.1 | 0.1 | 0.10 | 0.1 | 0.18 | 1.50 | 8.4 |
| DeepSeek-V2-Lite | P99 TBT | 0.125 | 0.14 | 0.19 | 0.125 | 0.67 | 3.00 | 4.4 |
| (MLA) | Worst gap | 0.125 | 0.14 | 0.19 | 0.125 | 0.22 | 0.48 | 2.2 |
| DeepSeek-V4-Flash | P99 TBT | 0.1 | 0.1 | 0.1 | 0.13 | 0.44 | 1.29 | 2.9 |
| (CSA/HCA) | Worst gap | 0.1 | 0.1 | 0.1 | 0.1 | 0.44 | 1.29 | 2.9 |
| Model | System | 1 s | 2 s | 5 s | 10 s | 30 s |
|---|---|---|---|---|---|---|
| Qwen3-8B | vLLM | 0.1 | 0.1 | 0.1 | 0.10 | 0.10 |
| Sarathi | 0.1 | 0.10 | 0.10 | 0.16 | 0.80 | |
| SPF | 0.10 | 0.15 | 0.48 | 1.02 | 1.03 | |
| Niyama | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | |
| Medha | 0.18 | 0.18 | 0.18 | 0.18 | 0.18 | |
| VelServe | 1.08 | 1.08 | 1.08 | 1.08 | 1.08 |
| Workload | Model | vLLM | Sarathi | SPF | Niyama | Medha | VelServe | best |
|---|---|---|---|---|---|---|---|---|
| Mixed | Qwen3-8B | 0.10 | 0.1 | 0.10 | 0.83 | 1.06 | 2.00 | 1.8 |
| V2-Lite | 0.50 | 0.14 | 0.19 | 1.54 | 1.15 | 3.00 | 1.9 | |
| V4-Flash | 0.1 | 0.1 | 0.1 | 0.40 | 0.46 | 1.29 | 2.8 | |
| Mooncake-sh. | Qwen3-8B | 0.44 | 0.41 | 0.43 | 0.64 | 0.71 | 1.12 | 1.6 |
| V2-Lite | 0.75 | 0.41 | 0.41 | 2.05 | 1.33 | 3.09 | 1.5 | |
| V4-Flash | 0.14 | 0.14 | 0.14 | 0.50 | 0.17 | 0.57 | 1.1 |
| Workload | Model | req/s | vLLM | Sarathi | SPF | Niyama | Medha | VelServe |
|---|---|---|---|---|---|---|---|---|
| Mixed | Qwen3-8B | 0.4 | 3.2 / 21 (85) | 3.6 / 22 (76) | 3.3 / 22 (80) | 3.2 / 20 (99) | 3.3 / 21 (100) | 3.3 / 21 (100) |
| Mixed | Qwen3-8B | 0.8 | 12.3 / 35 (55) | 20.4 / 39 (27) | 11.3 / 42 (51) | 11.3 / 35 (97) | 13.5 / 39 (96) | 13.0 / 38 (98) |
| Mixed | V2-Lite | 0.5 | 1.9 / 10 (90) | 5.8 / 23 (71) | 4.7 / 23 (82) | 1.9 / 11 (100) | 3.1 / 20 (100) | 2.2 / 13 (100) |
| Mixed | V2-Lite | 1 | 2.5 / 12 (89) | 10.6 / 28 (47) | 8.4 / 27 (68) | 2.9 / 16 (100) | 4.8 / 22 (97) | 3.7 / 19 (100) |
| Mixed | V4-Flash | 0.2 | 6.1 / 24 (75) | 10.9 / 31 (67) | 9.2 / 30 (78) | 4.0 / 8 (93) | 4.7 / 10 (100) | 4.6 / 9 (100) |
| Mixed | V4-Flash | 0.4 | 4.7 / 15 (80) | 12.2 / 35 (62) | 8.9 / 35 (73) | 6.3 / 31 (90) | 8.5 / 35 (100) | 7.9 / 36 (95) |
| Planned | Over | Residual (ms) | |||
|---|---|---|---|---|---|
| Model | System | (%) | (%) | median | p99 |
| Qwen3-8B | Niyama | 1.71 | 2.15 | +1.1 | 16 |
| Medha | 0.00 | 1.36 | +4.9 | 20 | |
| VelServe | 0.00 | 0.00 | +1.0 | 16 | |
| V2-Lite | Niyama | 1.00 | 1.11 | +2.4 | 125 |
| Medha | 0.00 | 1.60 | -1.2 | 25 | |
| Rate (req/s) | 0.6 | 0.8 | 1.0 | 1.2 | 1.4 | 1.6 |
| Qwen3-8B | ||||||
| VelServe | 99.3 | 98.4 | 99.6 | 97.5 | 96.4 | 87.3 |
| margin frozen offline (9.4 ms) | 66.4 | – | 47.6 | – | 35.6 | – |
| margin frozen, from Mooncake-shaped lengths (29.3 ms) | 99.3 | – | 83.1 | – | 91.8 | – |
| Medha | 100.0 | – | 57.6 | – | 25.7 | – |
| + 29.3 ms margin (from Mooncake-shaped lengths) | 100.0 | – | 87.4 | – | 51.7 | – |
| Rate (req/s) | 0.4 | 0.6 | 0.8 | 1.0 | 1.4 | 1.8 |
|---|---|---|---|---|---|---|
| VelServe | 93.7 | 89.4 | 79.8 | 76.1 | 68.0 | 60.2 |
| VelServe + promotion | 94.5 | 85.1 | 79.5 | 72.8 | 64.8 | 57.1 |
| TTFT (s) | Slowdown | Timed | TTFT | TBT p99 | SLO | |||
|---|---|---|---|---|---|---|---|---|
| Class | System | p50 | p99 | p50 | out (%) | met (%) | (ms) | met (%) |
| Short ( 8k) | vLLM | 30.6 | 107 | 706.6 | 0.0 | 19.8 | 1200 | 14.3 |
| Niyama | 0.3 | 1.2 | 7.1 | 0.0 | 97.0 | 454 | 11.8 | |
| VelServe | 0.2 | 0.4 | 5.3 | 0.0 | 99.8 | 126 | 99.8 | |
| Medium (8–32k) | vLLM | 38.2 | 115 | 34.3 | 0.0 | 17.6 | 1216 | 10.5 |
| Niyama | 1.8 | 214 | 1.7 | 0.4 | 90.5 | 459 | 8.8 | |
| Baselines | VelServe | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | SLO | vLLM | Niyama | Medha | SGL-CA | blind | CA | CA-tok. | best |
| Qwen3-8B | Strict | 0.036 | 0.036 | 0.055 | 0.047 | 0.060 | 0.184 | 0.161 | 3.3 |
| (GQA) | Relaxed | 0.036 | 0.037 | 0.084 | 0.047 | 0.093 | 0.184 | 0.161 | 2.2 |
| V2-Lite | Strict | 0.051 | 0.048 | 0.105 | 0.053 | 0.093 | 0.276 | 0.070 | 2.6 |
| (MLA) | Relaxed | 0.052 | 0.051 | 0.111 | 0.053 | 0.150 | 0.382 | 0.070 | 3.4 |
| V4-Flash | Strict | 0.03 | 0.03 | 0.062 | 0.037 | 0.130 | 0.112 | n/a | 1.8 |
| Attainment (%) | TTFT (s) | TTLT (s) | TPOT (ms) | Gaps | |||||||||
| System | SLO | TBT | TTFT | P50 | P90 | P99 | P50 | P90 | P99 | P50 | P90 | P99 | (%) |
| 20 minutes, 1,200 requests | |||||||||||||
| vLLM | 15.6 | 19.6 | 74.6 | 2.38 | 7.35 | 17.57 | 14.6 | 25.5 | 44.4 | 33.7 | 40.7 | 60.9 | 3.25 |
| Niyama | 14.8 | 18.6 | 78.9 | 2.30 | 5.62 | 18.57 | 14.6 | 25.5 | 43.7 | 34.2 | 41.5 | 64.4 | 3.36 |
| VelServe | 35.1 | 45.3 | 75.1 | 3.25 | 7.10 | 19.83 | 14.2 | 24.2 | 42.4 | 30.4 | 36.1 | 40.5 | 1.57 |
| vs. vLLM | +125% | +131% | +1% | +37% | 3% | +13% | 2% | 5% | 5% | 10% | 11% | 34% | 52% |
| Parameter | Value here | Set by |
|---|---|---|
| Epoch period | 4 s | median of (2.15 s), rounded up on a s grid |
| Cap on | 4 s | hand; did not bind |
| Gate off | join ms | join cost measured online (181–183 ms) |
| Opening window | 0.15 s | fixed |
| Urgent sub-epoch | 0.5 s | fixed |
| Urgent slack | 0.25 s (rescue: 0.05 s) | fixed |
| req/s | System | 150 | 200 | 300 | Misses | worst TBT p99 (ms) |
|---|---|---|---|---|---|---|
| 0.2 | Sarathi-Serve, official | 0.96 | 0.96 | 0.96 | 1/0/0 | 40 |
| Sarathi, ours | 1.00 | 1.00 | 1.00 | 0/0/0 | 31 | |
| Niyama, official (tuned) | 0.69 | 0.88 | 1.00 | 0/8/0 | 273 | |
| Niyama, ours | 0.81 | 0.81 | 0.85 | 0/5/0 | 494 | |
| 0.6 | Sarathi-Serve, official | 0.56 | 0.56 | 0.56 | 27/0/0 | 144 |
| Sarathi, ours | 0.77 | 0.77 | 0.77 | 14/0/0 | 85 |
LLM Serving
Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.
See all 249 papers