cs.ITNov 1, 2026

How Much Does Random Rotation Help GPTQ? Sharp Worst-Case Rate Gaps in the High-Rate Regime

Authors: Runguo Li

Organizations: University of Illinois Urbana-Champaign (UIUC)

Abstract

GPTQ is Babai's nearest-plane algorithm on the lattice of the Cholesky factor of the layer's input second moment ΣΣ. In the high-rate, entropy-coded regime it needs 12log⁡2(\AM/\GM)\tfrac12\log_2(\AM/\GM) of its Cholesky pivots more bits than WaterSIC, which is within 0.2550.255 bit of the rate-distortion limit for every ΣΣ; random rotations close most of this gap in practice, and Ordentlich and Polyanskiy asked for the worst-case gap between rotated GPTQ and WaterSIC. We show that after a Haar rotation the kk-th pivot converges almost surely, uniformly in k≥δnk\geδn and over spectra νν of bounded condition number, to 1/Sν(−k/n)1/S_ν(-k/n), a classical formula from the theory of Lyapunov exponents, and use it to answer the question. Over spectra with condition number κκ the worst rotated gap is 12log⁡2κ−log⁡2ln⁡κ+12−1ln⁡2+o(1)\tfrac12\log_2κ-\log_2\lnκ+\tfrac12-\tfrac1{\ln2}+o(1) bits, attained only by two-point spectra, just 12log⁡2ln⁡κ+O(1)\tfrac12\log_2\lnκ+O(1) bits below the worst case without rotation, so rotation cannot replace waterfilling in general. It nearly can on real layers: rotation cuts the cost of an ηη-fraction of outlier eigenvalues of size μμ from about ημημ to η2μ/2η^2μ/2 to leading order, but cannot remove spectral spread on a logarithmic scale; for non-Gaussian weights it also adds a weight-side rate cost, which we characterize exactly under a standard high-rate assumption. On 9696 Hessians of Qwen3 and Llama-3.1 the rotated gap has median 0.040.04 bit and maximum 0.360.36 bit, against 22--55 bits in the worst case for the same condition numbers, and the limit matches it to within 0.0090.009 bit.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. High-Rate Quantized Matrix Multiplication II

    May 13, 2026Or Ordentlich, Yury PolyanskiyLLM QuantizationRate-Distortion Theory

  2. WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization

    Mar 5, 2026Egor Lifar, Semyon Savkin, Or Ordentlich +1Neural Network QuantizationLLM Quantization

  3. GPTQ-2D: Cubic-Time Two-Sided Adaptive Rounding

    Jul 29, 2026Jiale Chen, Torsten Hoefler, Dan AlistarhPost-Training Quantization