A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines
Organizations: Thompson Rivers University, Kamloops, British Columbia, Canada
Abstract
Multi-agent LLM applications repeat most of their prompt tokens across requests, and serving engines reuse the cached state of shared prefixes. vLLM and SGLang reuse it at different granularities, and benchmarks of their default deployments report large performance gaps, usually credited to the cache. Such comparisons rarely control for other configuration differences. One of them, whether a long prompt is prefilled in chunks, is set by each engine's defaults from the hardware and the release, and it confounds the comparison. We built a reproducible testbed on Kubernetes and a seeded multi-agent workload generator, and compared the two engines under vendor defaults and with chunked prefill matched. SGLang's configuration is the same in both arms, which makes it an internal control across independently provisioned deployments. Under defaults, vLLM's median time to first token is up to 64x SGLang's. Matching one setting removes 78% of the capacity gap where that gap is largest and 30 to 67% of the latency gap. Near saturation the remaining latency gap reflects queue growth. A fluid queue model driven only by a 4.4% throughput shortfall reproduces it, and shows that such a gap scales with the length of the benchmark run. What survives matching is a tail-latency advantage of 2.4 to 6.4x at equal throughput, consistent with cache-aware ordering of waiting requests. We derive six recommendations for comparing serving engines.
Figures & tables
| vLLM 0.6.3 | SGLang 0.4.1 | |
| Prefix cache | PagedAttention with APC | RadixAttention |
| Match unit | 16-token block | token |
| Memory budget | gpu_memory_utilization 0.88 | mem_fraction_static 0.85 |
| Attention backend | engine default | FlashInfer |
| Order of waiting requests | arrival order | longest prefix match |
| Chunked prefill | ||
| Sweep | Varied | Held fixed | Duration per run |
|---|---|---|---|
| Rate | , | 900 s if , else 600 s | |
| Sharing | , | 2,400 s | |
| Prompt length | ; scaled with | 300, 400, 900 s | |
| Matched | rate 0.12 and 0.28; | as in the defaults arm | as in the defaults arm |
| Requests per run | Requests | Lowest | |||||
|---|---|---|---|---|---|---|---|
| Sweep | Runs | Mean | Median | Std. dev. | Range | in total | success (%) |
| Rate sweep | 30 | 886 | 774 | 418 | 276–1,674 | 26,580 | 99.9 |
| Sharing sweep | 30 | 170 | 144 | 42 | 138–228 | 5,100 | 100.0 |
| Prompt-length sweep | 18 | 919 | 594 | 708 | 240–1,926 | 16,548 | 99.8 |
| Matched arm | 21 | 717 | 720 | 294 | 240–1,104 | 15,060 | 100.0 |
| Total | 99 | 63,288 | 99.8 | ||||
| Condition | Metric | First deployment | Second deployment | Change |
|---|---|---|---|---|
| Rate 0.12 | TTFT p50 | 0.468 s | 0.458 s | |
| Rate 0.12 | TTFT p95 | 1.311 s | 1.241 s | |
| Rate 0.12 | E2E p50 | 13.75 s | 13.65 s | |
| Rate 0.28 | TTFT p50 | 1.089 s | 0.961 s | |
| Rate 0.28 | TTFT p95 | 17.80 s | 16.72 s | |
| Rate 0.28 | E2E p50 | 21.17 s | 20.80 s |
| Condition | Metric | Defaults | Matched | Removed |
|---|---|---|---|---|
| Rate 0.12 | TTFT p50 | 1.09 | 0.95 ∗ | — |
| Rate 0.12 | TTFT p95 | 8.10 ∗ | 2.44 ∗ | 57% |
| Rate 0.28 | TTFT p50 | 36.4 ∗ | 12.3 ∗ | 30% |
| Rate 0.28 | TTFT p95 | 4.16 ∗ | 1.70 | 63% |
| 8,192 tokens | TTFT p50 | 64.1 ∗ | 3.94 | 67% |
| 8,192 tokens | TTFT p95 | 27.0 ∗ | 6.43 ∗ | 43% |
| vLLM | SGLang | ||||
|---|---|---|---|---|---|
| Condition | Configuration | Tail ratio | Tail ratio | ||
| Rate 0.12 | defaults | 24.0 | +1.29 | 2.8 | +0.93 |
| matched | 6.6 | +0.79 | 2.7 | +0.91 | |
| Rate 0.28 | defaults | 1.7 | 2.54 | 16.4 | +0.71 |
| matched | 2.2 | 1.16 | 17.7 | +0.80 | |
| 8,192 tokens | defaults | 2.2 | 1.93 | 4.5 | +1.40 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Image tag | Manifest digest |
|---|---|
| vllm/vllm-openai:v0.6.3 | sha256:b8374cee0a1acaec8b64525ff77560f3 |
| 0443f67bd0fc1956a3529504a89f823b | |
| lmsysorg/sglang:v0.4.1-cu121 | sha256:50e7c37bda650334e505eb776e97ea92 |
| 0de8dda4fd94811a877330fa684a5a37 |
LLM Serving
Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.
See all 247 papers