Beyond Bitwidth: Deployment-Constrained Mixed-Precision Quantization for Multilingual LLM Inference on NVIDIA Blackwell
Organizations: ThakiCloud
Abstract
Low-bit LLM quantization is useful only if a model preserves service-relevant accuracy, executes through supported GPU kernels, and meets measured memory and latency limits. We study Qwen3-8B on NVIDIA B200 with vLLM, including INT2/INT3, GPTQ, NVFP4, and mixed-precision configurations. We verify native packed execution and precision on all 144 fused linear modules and evaluate Korean, English, numerical, and reasoning quality under preregistered gates. Packed INT2 and INT3 execute, but all seven tested INT2 grids degenerate on generation. Isolated module-restoration effects are strongly non-additive; Korean and reasoning sensitivity rankings disagree, and a promising INT4/INT8 development candidate fails independent final evaluation. A configuration with 16-bit activations, 98 NVFP4 weight-only modules, 46 INT8 modules, and an INT8 output head passes all gates on the untouched FINAL-81 set (5,168 items): 5.917 GiB of weight-loading memory (limit 5.98), Korean teacher-correct retention 0.904 (gate 0.90), and TPOT ratio 0.939 relative to BF16. It improves paired accuracy over NVFP4 W4A4 by +1.4 percentage points (Bonferroni-adjusted interval [+0.2, +2.6]), compared with +3.4 points on the selection set. This comparison changes both activation precision and INT8 protection; exploratory analyses suggest that the activation change contributes substantially, but the isolated allocation effect is not established. Final checkpoints were not retained, and bitwise rebuild reproducibility remains unverified.
Figures & tables
| Family | Weights | Activations | Kernel |
|---|---|---|---|
| BF16 | BF16 | BF16 | reference |
| NVFP4 W4A4 (n4) | NVFP4 | FP4 | FlashInferCuteDslNvFp4 |
| NVFP4 W4A16 | NVFP4 | 16-bit | MarlinNvFp4 |
| GPTQ W8A16 / W4A16 | INT8 / INT4 | 16-bit | Marlin (uint8b128 / uint4b8) |
| GPTQ W3A16 | INT3 | 16-bit | Humming (uint3b4) |
| GPTQ W2A16 | INT2 | 16-bit | Humming (uint2b2) |
| Arm | Mem | BPW | Tier | TPOT | |
|---|---|---|---|---|---|
| GPTQ W4 | 5.68 | 5.93 | 0.898 | 0.832 | 0.846 |
| GPTQ W3 | 4.86 | 5.08 | 0.748 | 0.575 | 0.857 |
| RTN W3 | 4.86 | 5.08 | 0.361 | 0.467 | 0.857 |
| GPTQ Attn W2 + MLP W4 (h2) | 5.33 | 5.56 | 0.521 | 0.459 | 0.864 |
| GPTQ Attn W4 + MLP W2 (h1) | 4.42 | 4.60 | 0.120 | 0.138 | 0.829 |
| GPTQ W2 | 4.07 | 4.23 | 0.115 | 0.158 | 0.837 |
| Grid | Retention | MC | Gen. | Loop | Trunc. |
|---|---|---|---|---|---|
| 4-level balanced, MSE | 0.243 | 0.454 | 0.000 | 1.00 | 1.00 |
| GPTQ asym. INT2 | 0.209 | 0.392 | 0.000 | 1.00 | 1.00 |
| asym. INT2, MSE | 0.203 | 0.379 | 0.000 | 1.00 | 1.00 |
| GPTQ sym. INT2 | 0.163 | 0.304 | 0.000 | 1.00 | 1.00 |
| sym. INT2, min-max | 0.158 | 0.296 | 0.000 | 1.00 | 1.00 |
| sym. INT2, MSE | 0.151 | 0.283 | 0.000 | 1.00 | 1.00 |
| Arm | Mem | Tier dev | Tier fin | ||||
|---|---|---|---|---|---|---|---|
| NVFP4 W4A4 (n4) | 5.977 | 0.894 | 0.879 | 0.838 | 0.805 | 3.9 | 3.9 |
| GPTQ W4 (w4) | 5.684 | 0.909 | 0.900 | 0.864 | 0.828 | 8.7 | +2.6 |
| W4 + W8 units + W8 head | 5.980 | 0.926 | 0.891 | 0.908 | 0.877 | 5.1 | 9.0 |
| GPTQ W3 (w3) | 4.864 | 0.753 | 0.667 | 0.634 | 0.578 | 23.8 | 17.7 |
| INT4/INT3 mix (D80) | 5.191 | 0.856 | 0.804 | 0.783 | 0.730 | 0.9 | 5.4 |
| Base (track) | single / joint | |
|---|---|---|
| A: W4 body | 0.018 | 7.57 |
| A: NVFP4A16 body | 0.256 | 5.39 |
| B: W3, cal0 | 0.046 | 7.46 |
| B: W3, cal25 | 0.152 | 2.42 |
| B: W3, cal50 | 0.045 | 4.50 |
| Arm | Mem | BPW | Tier | acc | fmt | TPOT | Fails | |||
|---|---|---|---|---|---|---|---|---|---|---|
| Mixed NVFP4A16 + W8 (A81) | 5.917 | 6.17 | 0.931 | 0.904 | 0.3 | 0.9 | 0.1 | +0.14 | 0.939 | 0 |
| NVFP4 W4A4 (n4) | 5.977 | 6.25 | 0.887 | 0.821 | 1.7 | 1.3 | 3.9 | 0.18 | 0.660 | 3 |
| INT4/INT3 mix (B81) | 5.153 | 5.35 | 0.810 | 0.700 | 6.7 | 12.2 | 3.8 | +1.23 | 0.863 | 5 |
| GPTQ W3 (w3) | 4.864 | 5.08 | 0.748 | 0.643 | 10.9 | 20.8 | 8.3 | +2.92 | 0.866 | 6 |
| Axis | BF16 | A81 | n4 | B81 | w3 | |
|---|---|---|---|---|---|---|
| Korean | 1,565 | 51.7 | 51.3 | 50.2 | 45.9 | 44.3 |
| English | 1,043 | 67.8 | 68.2 | 66.9 | 64.1 | 62.1 |
| Numerical | 1,320 | 90.5 | 89.5 | 89.2 | 78.3 | 69.7 |
| Reasoning | 870 | 73.9 | 73.8 | 70.0 | 70.1 | 65.6 |
| Comparison (B A) | Set | Diff (pp) | CI (pp) | |
|---|---|---|---|---|
| NVFP4A16 n4 | DEV-80 | 1.96 | +2.8 | [+1.6, +4.0] |
| +W8 units NVFP4A16 | DEV-80 | 1.96 | 0.3 | [ 1.3, +0.8] |
| W4+W8 head W4 | DEV-80 | 1.96 | 0.6 | [ 1.7, +0.6] |
| +W8 units W4+W8 head | DEV-80 | 1.96 | +1.6 | [+0.4, +2.8] |
| A81 n4 | SEL-81 | 2.24 | +3.4 | [+2.3, +4.6] |
| A81-W4 w4 | SEL-81 | 2.24 | 0.5 | [ 1.7, +0.6] |
| Arm | Mem | TTFT 50 | TTFT 95 | TPOT | tok/s | TPOT ratio |
|---|---|---|---|---|---|---|
| BF16 teacher | 15.27 | 15.3 | 20.3 | 3.915 | 255.4 | 1.000 |
| Mixed NVFP4A16 + W8 (A81) | 5.92 | 15.4 | 16.2 | 3.676 | 272.0 | 0.939 |
| NVFP4 W4A4 (n4) | 5.98 | 42.3 | 44.0 | 2.584 | 387.0 | 0.660 |
| INT4/INT3 mix (B81) | 5.15 | 16.8 | 17.5 | 3.379 | 295.9 | 0.863 |
| GPTQ W3 (w3) | 4.86 | 16.3 | 16.9 | 3.389 | 295.0 | 0.866 |
Mixed-Precision Quantization
Mixed-precision quantization assigns different numeric precisions across a model to balance accuracy with memory, speed, energy, and hardware constraints.
See all 189 papers