BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design
Organizations: University of Edinburgh · Tencent Hunyuan
Abstract
The key-value (KV) cache dominates memory capacity and bandwidth in long-context LLM serving. While 4-bit microscaling formats (NVFP4, MXFP4) shrink the cache and run attention on FP4 Tensor Cores, a fundamental gap remains between algorithmic quantization and system-aware acceleration: existing FP4 KV systems use vanilla quantization that causes severe accuracy drops, whereas accuracy-oriented 4-bit algorithms rely on vector codebooks whose lookup overhead degrades decoding throughput. We present BitDecoding-FP4, an algorithm--system co-design for accurate and efficient native FP4 KV decoding. We mitigate three error sources: Dual-Residual Query Quantization splits queries into two FP4 tensors, cutting query logit error from 60% to 1%; Underflow-Aware Probability Scaling prevents E4M3 scales from rounding long-context probabilities to zero; and Outlier-Aware Key Quantization combines static smoothing, Hadamard rotation, and scale search to suppress key outliers under token-wise scaling. At the system level, we remove the softmax bottleneck via transposed softmax with fused probability quantization and stacked query MMA on paged KV caches. Across dense and MoE models with GQA and MLA, BitDecoding-FP4 stays within 0.5 points of BF16 KV accuracy with NVFP4 (1 point with MXFP4), and tracks BF16 KV in RL training rollouts. In end-to-end serving on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B, it delivers 2.4-3.0 higher peak serving throughput than BF16 KV. The implementation is available at https://github.com/OpenBitSys/BitDecoding-FP4.
Figures & tables
| Input: | Query , , block size . |
|---|---|
| Require: | Hadamard matrix , key mean , constants , , and . |
| Output: | Attention output . |
| Long context | Math | Code | Knowl. | Latency @256K | |||||||
| Model | Method | LB-Pro | MRCR | RULER | AIME24 | AIME25 | LCB v6 | GPQA | Avg. | TPOT | Speedup |
| BF16 KV | 53.27 | 38.82 | 96.28 | 83.44 | 81.15 | 53.86 | 66.98 | 67.69 | 7.74 | 1.00 | |
| FP8 KV | 52.60 | 37.67 | 94.42 | 82.29 | 78.75 | 50.86 | 66.86 | 66.21 | 5.43 | 1.43 | |
| TurboQuant-4bit-nc | 49.78 | 32.91 | 93.15 | 77.40 | 72.29 | 47.43 | 64.96 | 62.56 | 94.46 | 0.08 | |
| SAW-INT4 | 49.92 | 31.57 | 91.19 | 79.69 | 75.63 | 52.86 | 64.02 | 63.55 | – ∗ | – | |
| TRT-LLM-NVFP4 | 44.89 | 28.84 | 80.04 | 76.56 | 72.29 | 48.71 | 63.01 | 59.19 | 4.63 | 1.67 | |
| Method | LB-Pro | MRCR | AIME25 | KV (GiB) |
|---|---|---|---|---|
| GLM-4.7 (355B, GQA), KV at 200K | ||||
| BF16 KV | 67.40 | 61.85 | 93.23 | 71.88 |
| FP8 KV | 65.87 | 61.27 | 92.40 | 35.94 |
| TRT-LLM-NVFP4 | 62.28 | 56.29 | 91.88 | 20.21 |
| BitDecoding-NVFP4 | 66.74 | 60.83 | 93.75 | 20.27 |
| Hy4-preview (770B, MLA), KV at 1M | ||||
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Configuration | KL | Top-1 mis. (%) | MRCR |
|---|---|---|---|
| BF16 KV | 0.29 | 0.32 | 38.82 |
| BitDecoding-NVFP4 | 3.83 | 1.00 | 36.68 |
| BitDecoding-MXFP4 | 10.27 | 1.56 | 35.61 |
| TRT-LLM-NVFP4 | 39.62 | 3.59 | 28.84 |
| Calibration | Tokens | Time | MRCR | AIME24 |
|---|---|---|---|---|
| No smoothing | – | – | 11.79 (+215%) | 8.11 (+76%) |
| 8 seeds (ours) | 133K | 13 | 3.74 | 4.62 |
| 8 other seeds | 131K | 12 | 3.73 ( 0.3%) | 4.70 (+1.7%) |
| 10 retrieval | 1.3M | 43 | 4.52 (+21%) | 5.27 (+14%) |
| 40 mixed | 5.2M | 173 | 4.35 (+16%) | 4.88 (+5.5%) |
| Subsets at code 10 (%) | |||||
| Calibration source | Code | 8 | 16 | 32 | |
| UUID key retrieval | 10 | 100 | 100 | 100 | 100 |
| Marker retrieval | 10 | 100 | 100 | 100 | 100 |
| Code and math filler | 10 | 100 | 100 | 100 | 100 |
| Key smoothing (ours) | 10 | 100 | 100 | 100 | 100 |
| Reasoning traces | 10 | 97.9 | 84.8 | 70.2 | 100 |
| Query format | Total | Query term | Share |
|---|---|---|---|
| BF16 | 19.32 | 0.00 | – |
| NVFP4 | 46.17 | 27.50 | 60% |
| + smoothing Q (token mean) | 33.03 | 13.95 | 42% |
| + smoothing Q (GQA mean) | 24.57 | 4.89 | 20% |
| + dual residual (ours) | 19.55 | 0.23 | 1.2% |
| scaling | KL | Top-1 | MRCR | RULER | LB-Pro |
|---|---|---|---|---|---|
| BF16 KV | – | – | 36.08 | 90.25 | 55.79 |
| Standard | 9.31 | 98.50 | 35.53 | 82.00 | 54.99 |
| Two-level | 6.06 | 98.75 | 35.51 | 88.00 | 55.12 |
| Ours (Eq. 4 ) | 6.47 | 98.73 | 35.51 | 88.75 | 55.02 |
| Configuration | MRCR | AIME24 |
|---|---|---|
| NVFP4 full recipe | 3.83 | 4.70 |
| key smoothing | 11.79 (+208%) | 8.11 (+72%) |
| Hadamard rotation | 6.68 (+74%) | 6.84 (+45%) |
| block-scale search | 4.70 (+23%) | 5.58 (+19%) |
| MXFP4 full recipe | 10.27 | 11.75 |
| key smoothing | 50.73 (+394%) | 37.76 (+221%) |
| Input: | Content query , positional query , latent cache , positional cache , block size . |
|---|---|
| Require: | Projections , Hadamard matrix , latent mean , and . |
| Output: | Attention output . |
LLM Inference Acceleration
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
See all 1,211 papers