The Noise Floor of a Quantization Comparison: Engine, restart, batch and kernel drift, measured against 8-bit and 4-bit effects in two decision models
Organizations: Level5
Abstract
A quantized model is usually judged by comparing it with its BF16 original, but that comparison changes more than the weights. It often changes the serving engine, it always runs on a different server instance, it may change how requests are batched, and on some stacks the library selects a different kernel depending only on which packages import. We measure each of these sources on the same 2,400 public decisions for two of Cloudflare's decision models, Clef-flash (9B) and Clef (27B), on an NVIDIA GB10, and state every quantization effect as the excess over them. Sources that are not quantization move between 0 and 0.92% of decisions. llama.cpp restarts are bit-identical on every record for both models. vLLM restarts flip 0, 5 and 20 decisions at BF16, FP8 and NVFP4 on Clef-flash and 4, 0 and 22 on Clef; a third instance of Clef's quantized builds (exploratory) leaves FP8 at 0 and puts NVFP4 at 4 to 22, depending on which two instances are compared. An engine change at BF16 flips 6 to 16, a batch-size change 8 to 15, and the Gated DeltaNet kernel transformers selects unasked 10 to 14. Quantization inside one engine moves more: for Clef-flash and Clef, 0.88% and 1.04% at Q8_0, 2.12% and 1.62% at FP8, 5.79% and 3.50% at Q4_K_M, and 5.79% and 5.67% at NVFP4. On both models, every 4-bit interval excludes every non-quantization source. Two of our five pre-registered predictions failed, and the same two fail when the tests are rerun on the 27B: FP8 is not inside restart noise, and changing engine moves fewer decisions than 8-bit quantization does. Accuracy hides most of this. Over the full 31,275 public questions, Clef's Q4_K_M build moves accuracy by a net six questions (-0.02 points [-0.20, +0.16]) while 436 of its decisions get worse (exploratory), and NVFP4 costs Clef-flash 0.95 points and Clef 0.65. We release the code, quantization recipes, per-record outputs and the pre-registration.
Figures & tables
| Engine | Version | Used for | Batching during quality runs |
|---|---|---|---|
| transformers (Cloudflare's load_release_model ) | 5.16.1 | ref , ref-r2 , ref-fla | batch 1 ( ref , ref-fla ); batch 8 ( ref-r2 ) |
| vLLM (Kwon et al., 2023) with a Clef decision plugin | 0.29.0+5013de39 | bf16-srv , fp8 , nvfp4 and their restarts | max-num-seqs 16, 16 requests in flight |
| llama.cpp decision server | b11671 (ba3c05f7, includes PR #29831) | gguf-bf16 , q8_0 , q4_k_m and the BF16 restart | single sequence |
| id | Format | Engine | Role |
|---|---|---|---|
| ref | BF16 | transformers, PyTorch GDN fallback | ground truth |
| bf16-srv | BF16 | vLLM | engine baseline for FP8 and NVFP4 |
| fp8 | FP8 W8A8 (llm-compressor) | vLLM | 8-bit, Blackwell-native |
| nvfp4 | NVFP4 W4A4 (llm-compressor) | vLLM | 4-bit, Blackwell-native |
| gguf-bf16 | GGUF BF16 | llama.cpp | engine baseline for GGUF |
| q8_0 | GGUF Q8_0 | llama.cpp | the 8-bit most local users run |
| Model | ref | gguf-bf16 | fp8 | q8_0 | nvfp4 | q4_k_m |
|---|---|---|---|---|---|---|
| Clef-flash | 19.06 | 18.16 | 12.15 | 9.66 | 9.13 | 5.72 |
| Clef | 54.97 | 54.06 | 30.65 | 28.73 | 20.00 | 16.59 |
| Source | Pair | Clef-flash (9B) | Clef (27B) |
|---|---|---|---|
| Not quantization | |||
| Restart, llama.cpp BF16 | gguf-bf16 vs -r2 | 0, 0.00% [exact 0.00, 0.15], 100.0% | 0, 0.00% [exact 0.00, 0.15], 100.0% |
| Restart, vLLM BF16 | bf16-srv vs -r2 | 0, 0.00% [exact 0.00, 0.15], 93.7% | 4, 0.17% [0.04, 0.33], 6.7% |
| Restart, vLLM FP8 | fp8 vs -r2 | 5, 0.21% [0.04, 0.42], 15.0% | 0, 0.00% [exact 0.00, 0.15], 95.3% |
| Restart, vLLM NVFP4 | nvfp4 vs -r2 | 20, 0.83% [0.50, 1.21], 76.8% | 22, 0.92% [0.54, 1.33], 87.0% |
| Engine at BF16, transformers to llama.cpp | ref vs gguf-bf16 | 6, 0.25% [0.08, 0.46], 0.0% | 16, 0.67% [0.33, 1.00], 0.0% |
| Prediction | Clef-flash: test (difference of flip rates, pp [95% CI]) and outcome | Clef, exploratory | |
|---|---|---|---|
| N1 | FP8's flip rate against bf16-srv is not significantly larger than bf16-srv against its restart | +2.12 [+1.54, +2.71]. Not supported | +1.46 [+0.96, +1.96]. Not supported |
| N2 | NVFP4 and Q4_K_M exceed every non-quantization source, with intervals that exclude them | 16 differences, weakest +4.96 [+4.00, +5.96]; all exclude 0, also with Bonferroni (k = 16). Supported | 16 differences, weakest +2.58 [+1.79, +3.46]; all exclude 0, also with Bonferroni. Supported |
| N3 | Changing engine at BF16 moves at least as many decisions as 8-bit quantization within one engine | -1.79, -0.54, -1.88 and -0.62; every interval below 0. Not supported | -1.08, -0.50, -0.96 and -0.37; three intervals below 0, the fourth [-0.79, 0.00]. Not supported |
| N4 | Kernel selection flips a nonzero number of decisions, within the range of vLLM restart drift | 10 flips, against a restart range of 0 to 20. Supported | 14 flips, against a restart range of 0 to 22. Supported |
| N5 | llama.cpp at BF16 is bit-identical across restarts | 2,400 of 2,400 records. Supported | 2,400 of 2,400 records. Supported |
| Pair of instances | FP8: flips [95% CI], bit-identical | NVFP4: flips [95% CI], bit-identical | NVFP4: mean TVD |
|---|---|---|---|
| first vs second | 0, 0.00% [exact 0.00, 0.15], 95.3% | 22, 0.92% [0.54, 1.33], 87.0% | 0.00710 |
| first vs third | 0, 0.00% [exact 0.00, 0.15], 95.5% | 22, 0.92% [0.58, 1.33], 78.6% | 0.00754 |
| second vs third | 0, 0.00% [exact 0.00, 0.15], 96.5% | 4, 0.17% [0.04, 0.33], 86.8% | 0.00141 |
| Config | Accuracy | accuracy [95% CI] | macro-F1 [95% CI] | Agreement with ref | Brier | ECE |
|---|---|---|---|---|---|---|
| ref | 82.2% | 0.2638 | 0.0355 | |||
| bf16-srv | 82.2% | -0.01 [-0.06, +0.04] | -0.02 [-0.10, +0.06] | 99.8% | 0.2641 | 0.0355 |
| gguf-bf16 | 82.2% | -0.01 [-0.05, +0.04] | -0.00 [-0.08, +0.08] | 99.8% | 0.2641 | 0.0359 |
| q8_0 | 82.2% | -0.00 [-0.08, +0.08] | -0.09 [-0.22, +0.04] | 99.4% | 0.2643 | 0.0358 |
| fp8 | 82.0% | -0.20 [-0.32, -0.09] | -0.27 [-0.46, -0.08] | 98.8% | 0.2658 | 0.0370 |
| q4_k_m | 81.9% | -0.36 [-0.56, -0.18] | -0.64 [-0.95, -0.32] | 96.5% | 0.2684 | 0.0343 |
| Config | Accuracy | accuracy [95% CI] | macro-F1 [95% CI] | Agreement with ref | Brier | ECE |
|---|---|---|---|---|---|---|
| ref | 83.7% | 0.2344 | 0.0252 | |||
| bf16-srv | 83.7% | -0.01 [-0.08, +0.06] | +0.01 [-0.10, +0.12] | 99.5% | 0.2343 | 0.0251 |
| gguf-bf16 | 83.7% | -0.02 [-0.10, +0.05] | -0.02 [-0.12, +0.09] | 99.5% | 0.2344 | 0.0255 |
| q8_0 | 83.6% | -0.07 [-0.17, +0.02] | -0.10 [-0.25, +0.05] | 99.2% | 0.2347 | 0.0255 |
| fp8 | 83.6% | -0.11 [-0.23, +0.02] | -0.10 [-0.29, +0.09] | 98.3% | 0.2353 | 0.0261 |
| q4_k_m | 83.7% | -0.02 [-0.20, +0.16] | +0.05 [-0.23, +0.32] | 96.8% | 0.2372 | 0.0226 |
| Model | Build | Flips | Got worse | Got better | Wrong to wrong | Net change |
|---|---|---|---|---|---|---|
| Clef-flash | Q8_0 | 183 | 83 | 82 | 18 | -1 |
| Clef-flash | FP8 | 374 | 193 | 129 | 52 | -64 |
| Clef-flash | Q4_K_M | 1,108 | 527 | 413 | 168 | -114 |
| Clef-flash | NVFP4 | 1,426 | 769 | 472 | 185 | -297 |
| Clef | Q8_0 | 264 | 126 | 104 | 34 | -22 |
| Clef | FP8 | 520 | 241 | 208 | 71 | -33 |
| Unit | Model | Floor used (flips) | Q8_0 | FP8 | Q4_K_M | NVFP4 |
|---|---|---|---|---|---|---|
| 2,400 records | Clef-flash | vLLM NVFP4 restart (20) | 1.05 | 2.55 | 6.95 | 6.95 |
| 2,400 records | Clef | vLLM NVFP4 restart (22) | 1.14 | 1.77 | 3.82 | 6.18 |
| 31,275 questions | Clef-flash | engine change to vLLM (75) | 2.44 | 4.99 | 14.77 | 19.01 |
| 31,275 questions | Clef | engine change to llama.cpp (156) | 1.69 | 3.33 | 6.41 | 9.63 |
| Build | Highest ref confidence among its flips |
|---|---|
| Q8_0 | 0.663 |
| FP8 | 0.867 |
| Q4_K_M | 0.941 |
| NVFP4 | 0.969 |
| Hypothesis | Outcome | |
|---|---|---|
| H1 | 8-bit is indistinguishable from BF16 within the noise floor | Not supported: FP8 and Q8_0 both exceed their restart floor |
| H2 | 4-bit loses accuracy, more on Clef-flash than Clef in absolute flips | Mixed (see below) |
| H3 | Flips concentrate where ref confidence is low | Supported |
| H4 | Calibration (ECE/Brier) degrades before accuracy does | Not supported |
| H5 | Out-of-scope detection degrades more than in-scope classification | Mixed: FP8 only. On CLINC150 its out-of-scope recall falls 0.66 points [-1.33, -0.11] while in-scope accuracy holds. The 4-bit builds flip 2.0 to 2.6% of out-of-scope items against 0.2% of in-scope items, but in both directions, so recall does not fall measurably |
| H6 | Drift grows with input length, so real-world data drifts more than benchmarks | Mixed: grows with length for FP8 and both 4-bit builds (longest input quartile minus shortest: +0.35 points [+0.02, +0.69] for FP8, +1.15 and +1.20 for Q4_K_M and NVFP4); real-world data drifts more for all four builds, but low confidence explains that better than length does |
| Task | ref accuracy | Q8_0 | FP8 | Q4_K_M | NVFP4 |
|---|---|---|---|---|---|
| ANLI R1 | 67.3% | -0.20 [-0.80, +0.40] | -0.30 [-1.40, +0.70] | -1.70 [-3.40, -0.10] | -4.50 [-6.60, -2.60] |
| ANLI R2 | 58.3% | -0.30 [-1.10, +0.40] | -0.50 [-1.60, +0.60] | -1.90 [-3.90, +0.00] | -2.20 [-4.10, -0.20] |
| ANLI R3 | 51.6% | -1.08 [-1.83, -0.42] | -1.00 [-2.00, -0.08] | -1.67 [-3.50, +0.08] | -3.00 [-5.00, -1.08] |
| ARC-Challenge | 98.5% | +0.00 [-0.26, +0.26] | +0.17 [-0.17, +0.51] | -0.09 [-0.43, +0.26] | -0.60 [-1.11, -0.09] |
| BANKING77 | 95.4% | +0.00 [-0.10, +0.10] | -0.10 [-0.23, +0.03] | -0.06 [-0.32, +0.19] | -0.10 [-0.45, +0.26] |
| BoolQ | 91.1% | +0.03 [-0.12, +0.18] | -0.18 [-0.52, +0.15] | -0.49 [-0.95, +0.03] | -0.49 [-1.07, +0.06] |
| Task | ref accuracy | Q8_0 | FP8 | Q4_K_M | NVFP4 |
|---|---|---|---|---|---|
| ANLI R1 | 70.8% | -0.60 [-1.60, +0.30] | +0.00 [-1.10, +1.10] | +0.00 [-1.40, +1.50] | -2.60 [-4.50, -0.80] |
| ANLI R2 | 64.1% | -0.30 [-1.20, +0.60] | +0.40 [-0.80, +1.60] | +0.50 [-1.20, +2.10] | -2.30 [-4.30, -0.40] |
| ANLI R3 | 56.6% | +0.17 [-0.58, +1.00] | -0.50 [-1.67, +0.67] | -0.33 [-1.92, +1.25] | -0.33 [-2.25, +1.67] |
| ARC-Challenge | 98.0% | +0.00 [-0.26, +0.26] | -0.17 [-0.43, +0.00] | +0.09 [-0.34, +0.51] | -0.68 [-1.28, -0.17] |
| BANKING77 | 94.3% | -0.16 [-0.39, +0.03] | -0.16 [-0.39, +0.06] | -0.06 [-0.45, +0.29] | -0.19 [-0.58, +0.19] |
| BoolQ | 91.2% | +0.00 [-0.18, +0.18] | +0.06 [-0.21, +0.31] | +0.03 [-0.34, +0.40] | -0.18 [-0.64, +0.31] |
| Config | Engine | Clef-flash mean / p50 / p95 ms | Clef-flash public mean ms | Clef-flash peak GB | Clef mean / p50 / p95 ms | Clef public mean ms | Clef peak GB | Clef / Clef-flash (mean) |
|---|---|---|---|---|---|---|---|---|
| ref | transformers | 354 / 172 / 1,577 | 381 | 19.97 | 1,162 / 594 / 5,005 | 1,243 | 56.27 | 3.28x |
| bf16-srv | vLLM | 184 / 105 / 704 | 196 | 19.93 | 596 / 348 / 2,170 | 635 | 55.90 | 3.24x |
| fp8 | vLLM | 123 / 68 / 507 | 131 | 13.06 | 381 / 198 / 1,776 | 411 | 32.26 | 3.11x |
| nvfp4 | vLLM | 88 / 50 / 352 | 93 | 10.00 | 270 / 148 / 1,048 | 284 | 21.75 | 3.08x |
| gguf-bf16 | llama.cpp | 236 / 125 / 975 | 251 | 20.54 | 727 / 406 / 2,863 | 772 | 57.36 | 3.09x |
| q8_0 | llama.cpp | 255 / 127 / 1,053 | 267 | 12.95 | 799 / 457 / 3,259 | 846 | 33.16 | 3.13x |
| Config | Clef-flash s per record | Clef s per record |
|---|---|---|
| ref (transformers, batch 1) | 0.472 | 1.384 |
| ref-r2 (transformers, batch 8) | 0.664 | 1.887 |
| bf16-srv | 0.195 | 0.634 |
| fp8 | 0.264 | 1.489 |
| nvfp4 | 0.128 | 0.526 |
| gguf-bf16 | 0.296 | 0.877 |
LLM Quantization
Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.
See all 434 papers