Modern large language models (LLMs) increasingly support longer contexts and serve more concurrent requests, increasing both sequence lengths and batch sizes. Both trends increase KV-cache size and memory traffic during inference. KV-cache quantization reduces these costs, with recent approaches using codebook-based vector quantization to achieve extremely low bit widths. However, hardware constraints limit codebook size, and most methods reconstruct the cache in higher-precision floating-point formats for attention, limiting the computational benefits of quantization. To address these limitations, we introduce MiSTiQ, a drop-in replacement for codebook-based vector quantizers that combines trellis-coded storage with microscaling (MX) formats for low-precision computation. We combine trellis quantization with beam search to encode the KV cache over large reconstruction spaces without large codebooks. To support MX computation, we derive MX-Hash functions that generate reconstruction values directly in the target MX element format. These functions use the scale-conditioned marginal distribution of block-normalized Gaussian inputs. On Llama3 and Qwen3, MiSTiQ achieves lower WikiText perplexity and higher GSM8K accuracy than the evaluated baselines across several block-scale formats at approximately 1.25 effective bits per element. With NVFP4, it also achieves the highest LongBench average on both models.
Figures & tables
Figure 1: MicroScaling trellis quantization overview. (a) Inputs are scaled blockwise before beam search on the bitshift trellis. (b) Trellis-state bits are mixed and mapped through the scale-dependent inverse CDF to form MX-Gaussian reconstruction values. The distribution diagrams are schematic.
Figure 2: MiSTiQ parameter analysis with standard Gaussian inputs , consistent with the MX Gaussian hash derivation. (a) Vector length: beam search and Viterbi. (b) Trellis length: both methods with bstart=8 (S8) or bstart=0 (S0). (c) MX block size with fixed UE8M0 scales, T=1024 , V=8 , and beam widths 128 and 1024, at approximately matched effective rates. The ordinate is RMSE⋅2beff ; beff counts payload, scale, and start-state bits per scalar.
Figure 3: Comparison between the standard Gaussian distribution and the MX-Gaussian marginal distribution for a block size of 32 with E2M1 scaling.
Table 1: WikiText-2 perplexity, mean LongBench score, and GSM8K accuracy (%) for MX KV-cache quantization. Effective bits include payload and stored quantization metadata. Blue marks the best quantized result, including ties, per model and metric within each rate and format group.
Figure 5
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Sampled relative inverse-CDF error at s=2 with linear interpolation; zero errors are omitted.
Block-scale format
Variant
Effective Bit
Llama-3.1-8B-Instruct
Qwen3-8B
Perplexity ↓
Perplexity ↓
—
BF16
16
7.170
9.750
UE8M0
MiSTiQ
1.25
11.957
15.072
MiSTiQ efficient hash
1.25
11.980
14.831
UE5M3
MiSTiQ
1.25
11.630
14.657
MiSTiQ efficient hash
1.25
11.712
14.537
Appendix
Table 3: Perplexity of MiSTiQ with and without efficient hashing. Bold marks the best result between the two implementations for each model and scale format. BF16 is excluded from this comparison. UE8M0 uses initial scale selection and UE5M3 uses joint optimization.
Block-scale format
Variant
Effective Bit
Llama-3.1-8B-Instruct
Qwen3-8B
—
BF16
16.00
7.170
9.750
UE8M0
VecInfer
1.00
693.850
409.782
NSNQuant
0.99
52.257
80.740
MiSTiQ (Ours)
0.99
38.315
57.616
MiSTiQ (Ours)
0.93
57.510
79.069
UE5M3
VecInfer
1.00
694.108
324.561
Appendix
Table 4: Below-1-bit MX KV-cache quantization perplexity. Effective Bit denotes mean effective bits per element using dynamic storage accounting. Within each block-scale format and model, dark blue cells mark the lowest perplexity and light blue cells mark the second-lowest perplexity among the displayed quantized methods.
Block-scale format
Variant
Effective Bit
Llama-3.1-8B-Instruct
Qwen3-8B
Perplexity ↓
LongBench ↑
GSM8K ↑
Perplexity ↓
LongBench ↑
GSM8K ↑
UE8M0
VecInfer
1.25
258.713
1.20
1.97
35.042
6.48
2.20
VecInfer + MiSTiQ (Ours)
1.25
188.996
2.11
1.97
18.967
11.46
3.41
UE5M3
VecInfer
1.25
232.335
1.22
1.90
30.791
11.43
2.50
VecInfer + MiSTiQ (Ours)
1.25
69.729
3.96
1.97
15.494
19.32
9.40
Appendix
Table 5: VecInfer with and without MiSTiQ at 1.25 effective bits per element. Blue marks the best result per model, scale format, and metric; both UE8M0 Llama GSM8K results tie at the displayed precision.
Block-scale format
Variant
Effective Bit
GovReport
Qasper
QMSum
SAMSum
MultiNews
TriviaQA
TREC
—
BF16
16.00
34.52
44.68
25.20
43.92
27.02
91.65
72.50
UE8M0
VecInfer
2.25
29.27
34.39
22.03
39.01
26.27
81.72
63.00
NSNQuant
2.24
33.70
42.96
24.86
43.81
26.40
91.12
67.50
MiSTiQ (Ours)
2.25
34.54
43.91
24.95
43.37
26.67
89.23
70.50
UE5M3
VecInfer
2.25
30.41
39.93
22.42
38.64
26.41
79.27
68.00
NSNQuant
2.24
33.30
45.38
25.33
43.83
26.57
88.93
70.00
Appendix
Table 6: LongBench task scores for Llama-3.1-8B-Instruct ( ↑ ). Bold marks the best per scale and bit section, excluding historical BF16.
Block-scale format
Variant
Effective Bit
GovReport
Qasper
QMSum
SAMSum
MultiNews
TriviaQA
TREC
—
BF16
16.00
27.86
12.03
18.06
44.54
22.77
90.71
71.50
UE8M0
VecInfer
2.25
26.08
12.88
18.92
39.30
21.86
78.53
69.50
NSNQuant
2.24
26.62
11.62
17.64
43.24
22.58
88.55
72.50
MiSTiQ (Ours)
2.25
27.41
11.42
17.85
42.35
22.62
89.56
72.00
UE5M3
VecInfer
2.25
26.34
12.54
18.44
39.09
21.84
81.07
71.00
NSNQuant
2.24
27.10
11.33
17.80
44.36
22.75
89.81
75.00
Appendix
Table 7: LongBench task scores for Qwen3-8B ( ↑ ). Bold marks the best per scale and bit section, excluding historical BF16. UE8M0/UE5M3 VecInfer uses MX-unaware quantization; NVFP4 retains the original setup. All MiSTiQ rows use beam 128.
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory +2