cs.LGNov 16, 2026

MiSTiQ: MicroScaling Trellis Quantization for Extreme KV Cache Compression

Authors: Ofir Gordon, Lior Dikstein, Paul Balanca, Idan Achituve, Douglas Orr, Hai Victor Habi

Organizations: Arm AI Research (AAIR), Arm Ltd · Graphcore Research

Abstract

Modern large language models (LLMs) increasingly support longer contexts and serve more concurrent requests, increasing both sequence lengths and batch sizes. Both trends increase KV-cache size and memory traffic during inference. KV-cache quantization reduces these costs, with recent approaches using codebook-based vector quantization to achieve extremely low bit widths. However, hardware constraints limit codebook size, and most methods reconstruct the cache in higher-precision floating-point formats for attention, limiting the computational benefits of quantization. To address these limitations, we introduce MiSTiQ, a drop-in replacement for codebook-based vector quantizers that combines trellis-coded storage with microscaling (MX) formats for low-precision computation. We combine trellis quantization with beam search to encode the KV cache over large reconstruction spaces without large codebooks. To support MX computation, we derive MX-Hash functions that generate reconstruction values directly in the target MX element format. These functions use the scale-conditioned marginal distribution of block-normalized Gaussian inputs. On Llama3 and Qwen3, MiSTiQ achieves lower WikiText perplexity and higher GSM8K accuracy than the evaluated baselines across several block-scale formats at approximately 1.25 effective bits per element. With NVFP4, it also achieves the highest LongBench average on both models.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

    Mar 17, 2026Yixuan Wang, Qingyu Shi, Jiayu Zhou +3KV-Cache CompressionKV-Cache Quantization

  2. InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models

    Feb 26, 2026Sayed Mohammadreza Tayaranian Hosseini, Amir Ardakani, Warren J. GrossHardware-Aware QuantizationKV-Cache Quantization

  3. SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference

    Aug 28, 2026Daeha Lee, Do-Hyung Kim, Jae-Hong KimKV-Cache QuantizationMixed-Precision Quantization