Modern large language models (LLMs) increasingly support longer contexts and serve more concurrent requests, increasing both sequence lengths and batch sizes. Both trends increase KV-cache size and memory traffic during inference. KV-cache quantization reduces these costs, with recent approaches using codebook-based vector quantization to achieve extremely low bit widths. However, hardware constraints limit codebook size, and most methods reconstruct the cache in higher-precision floating-point formats for attention, limiting the computational benefits of quantization. To address these limitations, we introduce MiSTiQ, a drop-in replacement for codebook-based vector quantizers that combines trellis-coded storage with microscaling (MX) formats for low-precision computation. We combine trellis quantization with beam search to encode the KV cache over large reconstruction spaces without large codebooks. To support MX computation, we derive MX-Hash functions that generate reconstruction values directly in the target MX element format. These functions use the scale-conditioned marginal distribution of block-normalized Gaussian inputs. On Llama3 and Qwen3, MiSTiQ achieves lower WikiText perplexity and higher GSM8K accuracy than the evaluated baselines across several block-scale formats at approximately 1.25 effective bits per element. With NVFP4, it also achieves the highest LongBench average on both models.
Figures & tables
Figure 1: MicroScaling trellis quantization overview. (a) Inputs are scaled blockwise before beam search on the bitshift trellis. (b) Trellis-state bits are mixed and mapped through the scale-dependent inverse CDF to form MX-Gaussian reconstruction values. The distribution diagrams are schematic.
Figure 2: MiSTiQ parameter analysis with standard Gaussian inputs , consistent with the MX Gaussian hash derivation. (a) Vector length: beam search and Viterbi. (b) Trellis length: both methods with bstart=8 (S8) or bstart=0 (S0). (c) MX block size with fixed UE8M0 scales, T=1024 , V=8 , and beam widths 128 and 1024, at approximately matched effective rates. The ordinate is RMSE⋅2beff ; beff counts payload, scale, and start-state bits per scalar.
Figure 3: Comparison between the standard Gaussian distribution and the MX-Gaussian marginal distribution for a block size of 32 with E2M1 scaling.
Table 1: WikiText-2 perplexity, mean LongBench score, and GSM8K accuracy (%) for MX KV-cache quantization. Effective bits include payload and stored quantization metadata. Blue marks the best quantized result, including ties, per model and metric within each rate and format group.
Figure 5
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Sampled relative inverse-CDF error at s=2 with linear interpolation; zero errors are omitted.
Block-scale format
Variant
Effective Bit
Llama-3.1-8B-Instruct
Qwen3-8B
Perplexity ↓
Perplexity ↓
—
BF16
16
7.170
9.750
UE8M0
MiSTiQ
1.25
11.957
15.072
MiSTiQ efficient hash
1.25
11.980
14.831
UE5M3
MiSTiQ
1.25
11.630
14.657
MiSTiQ efficient hash
1.25
11.712
14.537
Appendix
Table 3: Perplexity of MiSTiQ with and without efficient hashing. Bold marks the best result between the two implementations for each model and scale format. BF16 is excluded from this comparison. UE8M0 uses initial scale selection and UE5M3 uses joint optimization.
Block-scale format
Variant
Effective Bit
Llama-3.1-8B-Instruct
Qwen3-8B
—
BF16
16.00
7.170
9.750
UE8M0
VecInfer
1.00
693.850
409.782
NSNQuant
0.99
52.257
80.740
MiSTiQ (Ours)
0.99
38.315
57.616
MiSTiQ (Ours)
0.93
57.510
79.069
UE5M3
VecInfer
1.00
694.108
324.561
Appendix
Table 4: Below-1-bit MX KV-cache quantization perplexity. Effective Bit denotes mean effective bits per element using dynamic storage accounting. Within each block-scale format and model, dark blue cells mark the lowest perplexity and light blue cells mark the second-lowest perplexity among the displayed quantized methods.
Block-scale format
Variant
Effective Bit
Llama-3.1-8B-Instruct
Qwen3-8B
Perplexity ↓
LongBench ↑
GSM8K ↑
Perplexity ↓
LongBench ↑
GSM8K ↑
UE8M0
VecInfer
1.25
258.713
1.20
1.97
35.042
6.48
2.20
VecInfer + MiSTiQ (Ours)
1.25
188.996
2.11
1.97
18.967
11.46
3.41
UE5M3
VecInfer
1.25
232.335
1.22
1.90
30.791
11.43
2.50
VecInfer + MiSTiQ (Ours)
1.25
69.729
3.96
1.97
15.494
19.32
9.40
Appendix
Table 5: VecInfer with and without MiSTiQ at 1.25 effective bits per element. Blue marks the best result per model, scale format, and metric; both UE8M0 Llama GSM8K results tie at the displayed precision.
Block-scale format
Variant
Effective Bit
GovReport
Qasper
QMSum
SAMSum
MultiNews
TriviaQA
TREC
—
BF16
16.00
34.52
44.68
25.20
43.92
27.02
91.65
72.50
UE8M0
VecInfer
2.25
29.27
34.39
22.03
39.01
26.27
81.72
63.00
NSNQuant
2.24
33.70
42.96
24.86
43.81
26.40
91.12
67.50
MiSTiQ (Ours)
2.25
34.54
43.91
24.95
43.37
26.67
89.23
70.50
UE5M3
VecInfer
2.25
30.41
39.93
22.42
38.64
26.41
79.27
68.00
NSNQuant
2.24
33.30
45.38
25.33
43.83
26.57
88.93
70.00
Appendix
Table 6: LongBench task scores for Llama-3.1-8B-Instruct ( ↑ ). Bold marks the best per scale and bit section, excluding historical BF16.
Block-scale format
Variant
Effective Bit
GovReport
Qasper
QMSum
SAMSum
MultiNews
TriviaQA
TREC
—
BF16
16.00
27.86
12.03
18.06
44.54
22.77
90.71
71.50
UE8M0
VecInfer
2.25
26.08
12.88
18.92
39.30
21.86
78.53
69.50
NSNQuant
2.24
26.62
11.62
17.64
43.24
22.58
88.55
72.50
MiSTiQ (Ours)
2.25
27.41
11.42
17.85
42.35
22.62
89.56
72.00
UE5M3
VecInfer
2.25
26.34
12.54
18.44
39.09
21.84
81.07
71.00
NSNQuant
2.24
27.10
11.33
17.80
44.36
22.75
89.81
75.00
Appendix
Table 7: LongBench task scores for Qwen3-8B ( ↑ ). Bold marks the best per scale and bit section, excluding historical BF16. UE8M0/UE5M3 VecInfer uses MX-unaware quantization; NVFP4 retains the original setup. All MiSTiQ rows use beam 128.
The growing context length of Large Language Models (LLMs) enlarges the Key-Value (KV) cache, limiting deployment in resource-limited environments. Prior training-free approaches for KV cache compression typically rely on low-rank approximation or scalar quantization, which fail to simultaneously achieve high compression ratios and high reconstruction fidelity. We propose VQKV, a novel, training-free method introducing vector quantization (VQ) to obtain highly compressed KV representations while preserving high model fidelity, allowing for the representation of thousands of floating-point values with just a few integer indices. As a result, VQKV achieves an 82.8% compression ratio on LLaMA3.1-8B while retaining 98.6% of the baseline performance on LongBench and enabling 4.3x longer generation length on the same memory footprint.
Yixuan Wang, Qingyu Shi, Jiayu Zhou +3
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory +2
When transformer-based language models are deployed for text generation, most of the inference time is spent in the decoding stage, where output tokens are generated sequentially. Reducing the hardware cost of each decoding step is therefore critical for efficient long-context generation. A major bottleneck is the key-value (KV) cache, whose size grows with sequence length and often dominates the model's memory footprint. Prior work has proposed quantization methods to compress the KV cache while minimizing its loss of precision. We present InnerQ, a hardware-aware KV cache quantization scheme that reduces decode latency without compromising evaluation performance. InnerQ performs group-wise quantization by grouping cache matrices along their inner dimension. This grouping strategy aligns dequantization with vector-matrix multiplication and increases data reuse across GPU compute units. As a result, InnerQ reduces memory access and accelerates dequantization, achieving an average 1.3× speedup over prior KV cache quantization methods and 2.7× over the non-quantized baseline. To maintain fidelity under aggressive compression, InnerQ incorporates three techniques: (i) hybrid quantization, which chooses symmetric or asymmetric quantization for each group based on local statistics; (ii) high-precision windows for both recent tokens and attention sink tokens to mitigate outlier leakage; and (iii) per-channel normalization of the key cache, computed once during prefill and folded into the model parameters to eliminate runtime overhead. Beyond reducing latency, experiments on Llama and Mistral models show that InnerQ also improves few-shot evaluation scores relative to prior KV cache quantization methods.
Sayed Mohammadreza Tayaranian Hosseini, Amir Ardakani, Warren J. Gross
Department of Electrical and Computer Engineering McGill University Montreal, QC, Canada
The key-value (KV) cache is the dominant memory bottleneck of long-context large language model (LLM) inference, growing linearly with context length. We show that uniform KV quantization on a fractional-bit grid does not degrade gracefully: under a prespecified multi-seed statistical protocol, Llama-3.1-8B-Instruct with an affine quantizer is statistically indistinguishable from FP16 KV down to 2.322 code bits/value and collapses at 2.0 bits - a quality cliff in (2.0, 2.322] that reappears in generation-time quantization and multi-turn dialogue and transfers to Mistral-7B. The cliff reframes importance-aware mixed precision: above it, eight model-internal importance indicators are statistically interchangeable, so the benefit of mixing is grid interpolation, reaching average precisions uniform quantization cannot realize. SemKV preserves every token, ranks tokens by a model-internal score, and assigns two adjacent above-cliff precisions, achieving a measured 6.0x storage reduction with no statistically detectable quality difference from full KV (n=900, three seeds), and outperforming FP16 token pruning granted a 1.5x larger memory budget. Replacing the affine base with a distortion-optimized quantizer (TurboQuant-MSE) lowers the cliff in every protocol tested, raising the no-detectable-loss operating point to 7.9x. The recipe: measure the cliff for the target deployment setting, then interpolate above it.
Daeha Lee, Do-Hyung Kim, Jae-Hong Kim
Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of Korea