Rotation-Based Quantization

Overview

Rotation-based quantization studies how orthogonal transforms can make weights, activations, embeddings, and attention key-value caches easier to encode at low bit widths. Methods range from fixed randomized Hadamard transforms to learned or covariance-aware rotations, often paired with scalar or vector codebooks, clipping, and mixed precision. They are evaluated using language-model perplexity and task accuracy, retrieval quality, long-context performance, and inference memory and speed; similar techniques also appear in vision, diffusion, and multimodal models. A central challenge is choosing rotations that suit the quantizer’s grouping and number format while preserving useful structure, without adding costly calibration or runtime overhead.

Latest papers

CardsList

57 papers5 papers in the last four weeks, down 37% on the four weeks before. 0.0% of all new papers.

  1. SignRot: LLM Quantization with Massive Outlier-Aware Sign-Adjusted Rotation

    Nov 12, 2026Jaehun Gim, Jaewoo Kim, Gyuwan Kim +1LLM QuantizationNeural Network Quantization

  2. Rotation Is Token-Blind: A Structural Criterion for Post-Training Quantization of Feed-Forward Geometry Transformers

    Nov 5, 2026Huanfei He, Bohao Zhang, Jiaxin Zhang +4Post-Training QuantizationNeural Network Quantization

  3. FlagRot: Selective Rotation for Low-Bit KV Caches

    Nov 2, 2026Kunming Shao, Xijie Huang, Jierun Chen +7KV-Cache QuantizationRotation-Based Quantization

  4. RoLaQ: Joint Rotation and Range Calibration for 2-Bit KV Caches in Multi-Head Latent Attention

    Oct 14, 2026Shiyu Ji, Peng Ling, Chen Lin +4KV-Cache QuantizationLow-Bit Quantization

  5. SpectrumQuant: Spectrum Alignment for LLM Quantization

    Oct 14, 2026Dwaipayan Haldar, Indranil Patra, Azhar Yousuf +1LLM QuantizationRotation-Based Quantization

  6. ApexQuant: Data-Free Elastic Quantization by Residual Re-Isotropization

    Oct 6, 2026Aksel Fristrup, Sumit Pandey, Ankit KariryaaLLM QuantizationRotation-Based Quantization

  7. ThinQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs

    Sep 28, 2026Mehdi Makni, Ryan Lucas, Rahul MazumderLLM QuantizationRotation-Based Quantization

  8. TORQUE: Optimizing What (not) to Quantize Before and After Rotation

    Sep 28, 2026Ran Ben Basat, Michael Mitzenmacher, Shay VargaftikRotation-Based QuantizationVector Quantization

  9. SubRot: Signed Gradient Subspace Calibration for VLM Rotation Quantization

    Sep 28, 2026Zhenhao Shang, Haizhao Jing, Haokui Zhang +4VLM QuantizationRotation-Based Quantization

  10. PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers

    Sep 26, 2026Yanlong Chen, Yining Chen, Song Zhang +2LLM QuantizationRotation-Based Quantization

  11. A Note on Scaling in Randomly Rotated Quantization and Its Connection to the CDEF +1 Pythagorean Relation

    Sep 8, 2026Uri ErezRotation-Based Quantization

  12. RSLM: Training-Free Vector Quantization for Approximate Nearest Neighbor Search

    Aug 31, 2026Rastislav Lenhardt, Teodora Dobos, Thomas Vecchiato +2Rotation-Based QuantizationVector Quantization

  13. RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

    Aug 8, 2026Anthony. Lui, Mohamed. Elsaied, N. P. SavaniExpert OffloadingMixture-of-Experts Language Models

  14. Beyond Rotations: AuroOFT for Expressive Quantized Orthogonal Fine-Tuning

    Aug 5, 2026Yue Han, Dianlin WangLLM QuantizationFine-Tuning

  15. Output-Aware Rotation for INT2 KV-Cache Quantization

    Aug 3, 2026Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong +4LLM InferenceKV Caching

  16. LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference

    Jul 30, 2026Sangjin Kim, Yuseon Choi, Jungjun Oh +2LLM QuantizationAI Accelerator Inference

  17. GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

    Jul 30, 2026Sangjin Kim, Yuseon Choi, Byeongcheol Kim +2AI Accelerator InferenceRotation-Based Quantization

  18. GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries

    Jul 22, 2026Miguel P. Bento, João F. SeabraLLM QuantizationRotation-Based Quantization

  19. KronQ: LLM Quantization via Kronecker-Factored Hessian

    Jul 8, 2026Donghyun Lee, Yuhang Li, Ruokai Yin +1Mixed-Precision QuantizationLLM Quantization

  20. OrbitQuant: Data-Agnostic Quantization for Image and Video Diffusion Transformers

    Jul 2, 2026Donghyun Lee, Jitesh Chavan, Duy Nguyen +5Video Diffusion ModelsDiffusion Transformer

  21. RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation

    Jul 1, 2026Yaofu Liu, Wanli Lan, Jinxi Li +2Mixed-Precision QuantizationRotary Positional Embeddings

  22. Fast-TurboQuant: A Multiplier-Free Online Vector Quantization Approach

    Jun 19, 2026Pedro M. R. Pereira, Felipe A. P. de Figueiredo, Rausley A. A. de SouzaLLM QuantizationRotation-Based Quantization

  23. Ablation, Statistical Inference, and Validation for KV-Cache Compression

    Jun 14, 2026Paolo D'Alberto, Ashish Siarasao, Elliott Delaye +1KV CachingRotation-Based Quantization