Rotation-Based Quantization

Ask about this topic

Overview

Rotation-based quantization studies how orthogonal transforms can make weights, activations, embeddings, and attention key-value caches easier to encode at low bit widths. Methods range from fixed randomized Hadamard transforms to learned or covariance-aware rotations, often paired with scalar or vector codebooks, clipping, and mixed precision. They are evaluated using language-model perplexity and task accuracy, retrieval quality, long-context performance, and inference memory and speed; similar techniques also appear in vision, diffusion, and multimodal models. A central challenge is choosing rotations that suit the quantizer’s grouping and number format while preserving useful structure, without adding costly calibration or runtime overhead.

Latest papers

CardsList

53 papers8 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

  1. LIRQuant: An Efficient Locally Isolated Rotation Quantization Method for Large Language Models

    Oct 19, 2026Desheng Xu, Hanglin Li, Tianchen Song +2LLM QuantizationPost-Training Quantization

    Li Auto Inc.

  2. RoLaQ: Joint Rotation and Range Calibration for 2-Bit KV Caches in Multi-Head Latent Attention

    Oct 14, 2026Shiyu Ji, Peng Ling, Chen Lin +4KV-Cache QuantizationLow-Bit Quantization

    Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China · Tsinghua University, China · Baidu Inc., China

  3. SpectrumQuant: Spectrum Alignment for LLM Quantization

    Oct 14, 2026Dwaipayan Haldar, Indranil Patra, Azhar Yousuf +1LLM QuantizationRotation-Based Quantization

    Indian Institute of Science Bengaluru-560012, Karnataka, India

  4. ApexQuant: Data-Free Elastic Quantization by Residual Re-Isotropization

    Oct 6, 2026Aksel Fristrup, Sumit Pandey, Ankit KariryaaLLM QuantizationRotation-Based Quantization

    Henselian (henselian.com) · Department of Computer Science University of Copenhagen Denmark · Towards Deep Learning

  5. RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers

    Oct 3, 2026Mengyuan Fan, Bokai Huang, JiaMing Pan +4Low-Bit QuantizationRotation-Based Quantization

    Peking University

  6. ThinQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs

    Sep 28, 2026Mehdi Makni, Ryan Lucas, Rahul MazumderLLM QuantizationRotation-Based Quantization

    Operations Research Center Massachusetts Institute of Technology

  7. TORQUE: Optimizing What (not) to Quantize Before and After Rotation

    Sep 28, 2026Ran Ben Basat, Michael Mitzenmacher, Shay VargaftikRotation-Based QuantizationVector Quantization

    University College London · Harvard University · VMware Research by Broadcom

  8. SubRot: Signed Gradient Subspace Calibration for VLM Rotation Quantization

    Sep 28, 2026Zhenhao Shang, Haizhao Jing, Haokui Zhang +4VLM QuantizationRotation-Based Quantization

    Northwest Polytechnical University · Nanjing University of Science and Technology · Intellifusion +1

  9. PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers

    Sep 26, 2026Yanlong Chen, Yining Chen, Song Zhang +2LLM QuantizationRotation-Based Quantization

    Nanyang Technological University, Singapore · Nanjing University of Posts and Telecommunications, Nanjing, China · Nanjing University, Nanjing, China +1

  10. Understanding LLM Quantization through Activation-Guided Compensation and Orthogonal Residuals

    Sep 18, 2026Yamato Narita, Issei SatoPost-Training QuantizationNeural Network Quantization

    Department of Computer Science, The University of Tokyo

  11. A Note on Scaling in Randomly Rotated Quantization and Its Connection to the CDEF +1 Pythagorean Relation

    Sep 8, 2026Uri ErezRotation-Based Quantization

    School of Electrical Engineering Tel Aviv University Tel Aviv, Israel

  12. RSLM: Training-Free Vector Quantization for Approximate Nearest Neighbor Search

    Aug 31, 2026Rastislav Lenhardt, Teodora Dobos, Thomas Vecchiato +2Rotation-Based QuantizationVector Quantization

    Google Zurich, Switzerland · Technical University of Munich Munich, Germany · University of Copenhagen Copenhagen, Denmark +1

  13. Transforms for LLM Quantization: The Great Inversion and Format Co-Design

    Aug 25, 2026Ehsan JokarLLM QuantizationNeural Network Quantization

  14. AudioTQ: A Data-Oblivious 6-Bit CPU Audio Codec via Randomized Hadamard Rotation and Lloyd-Max Quantization

    Aug 15, 2026Sahil GangurdeRotation-Based QuantizationLow-Bit Quantization

  15. When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation

    Aug 13, 2026Shuhan Wang, Yilin Luo, Nan Xu +1LLM Quantization4-Bit Quantization

  16. RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

    Aug 8, 2026Anthony. Lui, Mohamed. Elsaied, N. P. SavaniExpert OffloadingMixture-of-Experts Language Models

    Cognizant AI & Analytics, London, UB8 3PH, UK · Southern Methodist University (SMU), Engineering Management, Lyle School of Engineering, Dallas, TX, USA · Cognizant AI Lab, Responsible AI Office, San Francisco, USA +1

  17. Beyond Rotations: AuroOFT for Expressive Quantized Orthogonal Fine-Tuning

    Aug 5, 2026Yue Han, Dianlin WangLLM QuantizationFine-Tuning

    College of Systems Engineering, National University of Defense Technology · College of Computer Science and Technology, National University of Defense Technology

  18. Output-Aware Rotation for INT2 KV-Cache Quantization

    Aug 3, 2026Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong +4LLM InferenceKV Caching

    University of Southern California · Seoul National University · Inha University

  19. LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference

    Jul 30, 2026Sangjin Kim, Yuseon Choi, Jungjun Oh +2LLM QuantizationAI Accelerator Inference

    PIM Semiconductor Design Research Center (AI-PIM), Daejeon 34141, South Korea

  20. GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

    Jul 30, 2026Sangjin Kim, Yuseon Choi, Byeongcheol Kim +2AI Accelerator InferenceRotation-Based Quantization

    School of Electrical Engineering, Korea Advanced Institue of Science and Technology (KAIST)

  21. GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries

    Jul 22, 2026Miguel P. Bento, João F. SeabraLLM QuantizationRotation-Based Quantization

    Independent Researcher · Lisbon, Portugal

  22. KronQ: LLM Quantization via Kronecker-Factored Hessian

    Jul 8, 2026Donghyun Lee, Yuhang Li, Ruokai Yin +1Mixed-Precision QuantizationLLM Quantization

    University of Southern California · Yale University

  23. OrbitQuant: Data-Agnostic Quantization for Image and Video Diffusion Transformers

    Jul 2, 2026Donghyun Lee, Jitesh Chavan, Duy Nguyen +5Video Diffusion ModelsDiffusion Transformer

    Cantina Labs · University of Southern California · *Work done during an internship at Cantina Labs. +1

  24. RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation

    Jul 1, 2026Yaofu Liu, Wanli Lan, Jinxi Li +2Mixed-Precision QuantizationRotary Positional Embeddings

    The Hong Kong University of Science and Technology, Hong Kong SAR, China · Independent Researcher, China

  25. Fast-TurboQuant: A Multiplier-Free Online Vector Quantization Approach

    Jun 19, 2026Pedro M. R. Pereira, Felipe A. P. de Figueiredo, Rausley A. A. de SouzaLLM QuantizationRotation-Based Quantization

    National Institute of Telecommunications (Inatel), Santa Rita do Sapucaí, MG 37536-001, Brazil

  26. Ablation, Statistical Inference, and Validation for KV-Cache Compression

    Jun 14, 2026Paolo D'Alberto, Ashish Siarasao, Elliott Delaye +1KV CachingRotation-Based Quantization

    Advanced Micro Devices, Inc.

  27. Trainable Smooth-Rotation Transforms with Learned Channel Scales for LLM Quantization

    Jun 7, 2026Patrik Czakó, Gábor Kertész, Sándor SzénásiLLM Quantization4-Bit Quantization

    Doctoral School of Applied Informatics and Applied Mathematics, Obuda University Budapest, Hungary · John von Neumann Faculty of Informatics, Obuda University Budapest, Hungary

  28. Qift: Shift-Friendly No-Zero W2 Post-Training Quantization for Rotated W2A4/KV4 LLM Inference

    Jun 1, 2026Chi-Wei Huang, Chia-Chi TsaiLLM QuantizationRotation-Based Quantization

    National Cheng Kung University

  29. HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization

    May 28, 2026Artur Zagitov, Gleb Molodtsov, Aleksandr BeznosikovLLM QuantizationRotation-Based Quantization

    1MIRAI · 2BRAIn Lab · 3Innopolis University

  30. Ω-QVLA: Robust Quantization for Vision-Language-Action Models via Composite Rotation and Per-step Scaling

    May 27, 2026Xinyu Wang, Mingze Li, Sicheng Lyu +6Post-Training QuantizationEfficient VLA Model Inference

    McGill University · Université de Montréal · Mila – Quebec AI Institute +2