Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs
Organizations: Independent researcher, United States
Abstract
A team serving a 9B-20B open-weight language model on vLLM or SGLang must choose a quantization recipe, and needs two numbers: the speedup over BF16 and the accuracy given up. We assemble every public measurement of these numbers that we could verify and find that the record supplies them only in pieces. Only two checkpoints, both INT4 builds of Gemma 3 12B, have both a vLLM timing and an accuracy score, and that score covers a single MMLU subject, one that the AWQ build was calibrated on. The batch-1 timings predate current engine releases, AMD and Intel Gaudi have no timing in this band at all, and FP4 has none against BF16. The timings that do exist follow one rule. Decoding a small batch is limited by memory bandwidth, so 4-bit weights roughly double speed (2.12x for Qwen2.5-14B on an A100), but the gain vanishes once a large batch makes the GPU compute-bound (1.02x for Gemma 3 12B at batch size 256), where FP8 is the recipe that can still help. Many published accuracy deltas, including every HumanEval delta, lie within benchmark sampling noise. We close with a seven-item reporting checklist for future measurements.
Figures & tables
| Recipe | Speed or latency | Speedup vs BF16 | Memory † |
|---|---|---|---|
| vLLM 0.6.3, one A100 80GB, Qwen2.5-14B-Instruct, batch size 1, 2048 output tokens; | |||
| tokens/s at input length 1 / 30,720 ( Qwen Team, 2024a ) | |||
| BF16 (baseline) | 46.30 / 37.18 | 1.00 / 1.00 | 28.08 GB |
| GPTQ-Int8 (W8A16) | 70.40 / 49.23 | 1.52 / 1.32 | 16.11 GB |
| AWQ (W4A16) | 92.66 / 59.01 | 2.00 / 1.59 | 9.79 GB |
| GPTQ-Int4 (W4A16) | 98.02 / 60.91 | 2.12 / 1.64 | 9.94 GB |
| Recipe | MMLU | GSM8K | HumanEval | C-Eval |
|---|---|---|---|---|
| Qwen3-14B, AngelSlim, lm-eval, engine n.r. ( Tencent AngelSlim Project Contributors, 2025 ) | ||||
| BF16 | 78.90 | 88.40 | 55.49 | 83.06 |
| FP8-static | ||||
| FP8-dynamic | ||||
| INT8-dynamic (W8A8) | ||||
| GPTQ-INT4 (W4A16) | ||||
| Question | What the record shows | Confidence |
|---|---|---|
| Speed at batch size 1 | W4A16 is – faster than BF16; FP8 ties AWQ on the H20 | Moderate: 3 sources, 3 GPUs, 2024–25 engines |
| Speed at large batch | W4A16 gains nothing at batch 256; FP8 untested | Low: 1 model, 1 source |
| Speed on long prompts | FP8 beats AWQ ( vs ) | Low: 1 model, 1 GPU (H20) |
| 8-bit accuracy | Within 0.77 MMLU points of BF16 | Moderate–high: 5 models, 2 publishers |
| INT4 accuracy | 0.43–1.43 MMLU points; up to about 4 GSM8K points | Moderate: 3 models, 2 publishers |
| gpt-oss-20b engine | vLLM SGLang at defaults | Low: 1 source, untuned |
LLM Quantization
Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.
See all 434 papers