Rotation-Based Quantization
Overview
Rotation-based quantization studies how orthogonal transforms can make weights, activations, embeddings, and attention key-value caches easier to encode at low bit widths. Methods range from fixed randomized Hadamard transforms to learned or covariance-aware rotations, often paired with scalar or vector codebooks, clipping, and mixed precision. They are evaluated using language-model perplexity and task accuracy, retrieval quality, long-context performance, and inference memory and speed; similar techniques also appear in vision, diffusion, and multimodal models. A central challenge is choosing rotations that suit the quantizer’s grouping and number format while preserving useful structure, without adding costly calibration or runtime overhead.
Latest papers
57 papers5 papers in the last four weeks, down 37% on the four weeks before. 0.0% of all new papers.