The key-value (KV) cache dominates memory capacity and bandwidth in long-context LLM serving. While 4-bit microscaling formats (NVFP4, MXFP4) shrink the cache and run attention on FP4 Tensor Cores, a fundamental gap remains between algorithmic quantization and system-aware acceleration: existing FP4 KV systems use vanilla quantization that causes severe accuracy drops, whereas accuracy-oriented 4-bit algorithms rely on vector codebooks whose lookup overhead degrades decoding throughput. We present BitDecoding-FP4, an algorithm--system co-design for accurate and efficient native FP4 KV decoding. We mitigate three error sources: Dual-Residual Query Quantization splits queries into two FP4 tensors, cutting query logit error from 60% to 1%; Underflow-Aware Probability Scaling prevents E4M3 scales from rounding long-context probabilities to zero; and Outlier-Aware Key Quantization combines static smoothing, Hadamard rotation, and scale search to suppress key outliers under token-wise scaling. At the system level, we remove the softmax bottleneck via transposed softmax with fused probability quantization and stacked query MMA on paged KV caches. Across dense and MoE models with GQA and MLA, BitDecoding-FP4 stays within 0.5 points of BF16 KV accuracy with NVFP4 (1 point with MXFP4), and tracks BF16 KV in RL training rollouts. In end-to-end serving on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B, it delivers 2.4-3.0× higher peak serving throughput than BF16 KV. The implementation is available at https://github.com/OpenBitSys/BitDecoding-FP4.
Figures & tables
Figure 1: Overview of BitDecoding-FP4 (left), achieving higher accuracy and serving throughput when running Qwen3-4B (right). See Table 1 for the average accuracy and Figure 10(b) for the per-GPU serving throughput on B300 at 128K context.
Figure 2: Dequantization overhead of FlashInfer-XQA ( Ye et al., 2025 ) kernel with NVFP4 KV Cache.
Input:
Query q∈Rd , K,V∈RN×d , block size Bk .
Require:
Hadamard matrix H , key mean μk , constants c , τ , and γ=log2e/d .
Output:
Attention output o .
Algorithm 1 Overview of the BitDecoding-FP4 algorithm.
Figure 3: Mean squared logit error for each query format, split into key and query terms. Dual-Residual Query Quantization reduces the query quantization error to 1% of the total.
Figure 4: (a) E4M3 scaling underflow on probability blocks in long-context decoding over the empirical block distribution. (b) Relative block quantization error (RMSE) across max(pb) .
Figure 5: (a) Roofline of decode attention on B300 (TC: Tensor Core): the conventional softmax caps every KV precision. (b) Conventional softmax: threads are assigned to head rows, so most sit idle. (c) Ours: threads are assigned to keys, so all threads do useful work, reductions run in parallel via butterfly shuffles, and the FP4 block scales come for free from the butterfly.
Figure 6: Stacked Query MMA with In-Softmax Joining. (1) Before the KV loop, q(2) is stacked into the zero padding of the q(1) tile. (2) Each KV block issues one FP4 MMA for both terms. (3) The softmax joins the two halves of Sj⊤ in FP32.
Figure 7: BitDecoding-FP4 in vLLM ( ri : batch row, si : BF16 slot). (1) The control kernel assigns each row a slot and splits its sequence. (2) The decode kernel reads the FP4 pages and the BF16 slot. (3) A full recent page is promoted into its FP4 block.
Long context
Math
Code
Knowl.
Latency @256K
Model
Method
LB-Pro
MRCR
RULER
AIME24
AIME25
LCB v6
GPQA
Avg.
TPOT
Speedup
BF16 KV
53.27
38.82
96.28
83.44
81.15
53.86
66.98
67.69
7.74
1.00 ×
FP8 KV
52.60
37.67
94.42
82.29
78.75
50.86
66.86
66.21
5.43
1.43 ×
TurboQuant-4bit-nc
49.78
32.91
93.15
77.40
72.29
47.43
64.96
62.56
94.46
0.08 ×
SAW-INT4
49.92
31.57
91.19
79.69
75.63
52.86
64.02
63.55
– ∗
–
TRT-LLM-NVFP4
44.89
28.84
80.04
76.56
72.29
48.71
63.01
59.19
4.63
1.67 ×
Table 1: Accuracy (%) and end-to-end decoding latency of KV cache quantization methods; BitDecoding-NVFP4 stays within 0.5 average points of BF16 KV (BitDecoding-MXFP4 within 1 point) and outperforms FP8 and prior 4-bit methods in both average accuracy and decoding speed. LB-Pro denotes LongBench Pro, TPOT is the per-request time per output token (ms) at 256K context, and Speedup is relative to BF16 KV.
Method
LB-Pro
MRCR
AIME25
KV (GiB)
GLM-4.7 (355B, GQA), KV at 200K
BF16 KV
67.40
61.85
93.23
71.88
FP8 KV
65.87
61.27
92.40
35.94
TRT-LLM-NVFP4
62.28
56.29
91.88
20.21
BitDecoding-NVFP4
66.74
60.83
93.75
20.27
Hy4-preview (770B, MLA), KV at 1M
Table 2: Accuracy (%) and KV cache memory footprint on large FP8-weight models with GQA and sparse MLA attention; BitDecoding-NVFP4 stays within 0.4 average points of BF16 KV while shrinking the KV cache by up to 3.5 × .
Figure 9: Decode attention kernel speedup over BF16 KV at 128K context across batch sizes (BitDecoding denotes the NVFP4 integration of Du et al. (2026) ); BitDecoding-NVFP4 consistently outperforms all baselines on both GPUs.
Figure 10: End-to-end serving throughput on Qwen3-4B and Qwen3-30B-A3B across 1, 8, 16, 32, and 64 concurrent requests (up and to the right is better). When concurrent requests exceed KV cache capacity, excess requests are queued or preempted. BitDecoding-NVFP4 achieves the best latency–throughput trade-off across both workloads.
Figure 11: Accuracy ablation on Qwen3-4B (MXFP4 omits P scaling since E8M0 scales do not underflow).
Figure 12: Decode kernel latency ablation at batch 64 and 128K context on B300.
Figure 13: Quantization and metadata overhead of VLLM on Qwen3-4B. Both Kitty and BitDecoding-NVFP4 manage BF16 sink tokens.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
KL ↓
Top-1 mis. (%) ↓
MRCR ↑
BF16 KV
0.29
0.32
38.82
BitDecoding-NVFP4
3.83
1.00
36.68
BitDecoding-MXFP4
10.27
1.56
35.61
TRT-LLM-NVFP4
39.62
3.59
28.84
Appendix
Table 3: Teacher-forced KL tracks end-to-end accuracy on Qwen3-4B: the lower a configuration’s KL and top-1 mismatch on the MRCR trajectories, the higher its MRCR accuracy. KL is in units of 10−3 nats; top-1 mismatch is the fraction of steps where the FP4 argmax token differs from BF16 KV’s, and the BF16 KV row reruns BF16 KV against its own reference.
Calibration
Tokens
Time
MRCR
AIME24
No smoothing
–
–
11.79 (+215%)
8.11 (+76%)
8 seeds (ours)
133K
13
3.74
4.62
8 other seeds
131K
12
3.73 ( − 0.3%)
4.70 (+1.7%)
10 retrieval
1.3M
43
4.52 (+21%)
5.27 (+14%)
40 mixed
5.2M
173
4.35 (+16%)
4.88 (+5.5%)
Appendix
Table 4: Key smoothing calibration on Qwen3-4B. Time is the seconds on one B300 GPU to produce the calibration text and measure μk with the serving engine. MRCR and AIME24 are the teacher-forced KL from BF16 KV ( ×10−3 , Appendix A ), with the change relative to our calibration in parentheses.
Subsets at code 10 (%)
Calibration source
Code
N=1
8
16
32
UUID key retrieval
10
100
100
100
100
Marker retrieval
10
100
100
100
100
Code and math filler
10
100
100
100
100
Key smoothing (ours)
10
100
100
100
100
Reasoning traces
10
97.9
84.8
70.2
100
Appendix
Table 5: Threshold code selected by each calibration source on Qwen3-4B-Thinking-2507; code 10 is the production τ . The last four columns give the percentage of 1,000 random subsets of N samples that select code 10.
Query format
Total ↓
Query term ↓
Share
BF16
19.32
0.00
–
NVFP4
46.17
27.50
60%
+ smoothing Q (token mean)
33.03
13.95
42%
+ smoothing Q (GQA mean)
24.57
4.89
20%
+ dual residual (ours)
19.55
0.23
1.2%
Appendix
Table 6: Mean squared logit error ( ×10−2 ) of each query format against NVFP4 keys on Qwen3-4B. Dual-residual query quantization achieves the lowest error.
P scaling
KL ↓
Top-1 ↑
MRCR
RULER
LB-Pro
BF16 KV
–
–
36.08
90.25
55.79
Standard max/6
9.31
98.50
35.53
82.00
54.99
Two-level
6.06
98.75
35.51
88.00
55.12
Ours (Eq. 4 )
6.47
98.73
35.51
88.75
55.02
Appendix
Table 7: Comparison of probability scaling rules on Qwen3-4B. Underflow-aware scaling matches the accuracy of two-level scaling without per-tile rescaling overhead.
Configuration
MRCR
AIME24
NVFP4 full recipe
3.83
4.70
− key smoothing
11.79 (+208%)
8.11 (+72%)
− Hadamard rotation
6.68 (+74%)
6.84 (+45%)
− block-scale search
4.70 (+23%)
5.58 (+19%)
MXFP4 full recipe
10.27
11.75
− key smoothing
50.73 (+394%)
37.76 (+221%)
Appendix
Table 8: Leave-one-out KL ablation of Outlier-Aware Key Quantization on Qwen3-4B-Thinking-2507. Each row removes one part from the full recipe; values are the teacher-forced KL from BF16 KV ( ×10−3 , Appendix A ). Since KL tracks end-to-end accuracy (Table 3 ), every part contributes to accuracy.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.