TORQUE: Optimizing What (not) to Quantize Before and After Rotation

Ran Ben Basat; Michael Mitzenmacher; Shay Vargaftik

cs.LGSep 28, 2026

TORQUE: Optimizing What (not) to Quantize Before and After Rotation

Authors: Ran Ben Basat, Michael Mitzenmacher, Shay Vargaftik

Organizations: University College London · Harvard University · VMware Research by Broadcom

Abstract

Uniform random rotations are an effective preprocessing step for quantization: they make normalized coordinate distributions approximately Gaussian, enabling the use of codebooks optimized offline. We introduce TORQUE, a framework that improves on previous quantization works that use random rotations by jointly optimizing how many and which coordinates to preserve at high precision both before and after rotation, under a fixed overall expected bit budget. Intuitively, before rotation, preserving large input coordinates at high precision can reduce overall error by preventing the rotation from spreading their values across many coordinates. Likewise, after rotation, preserving a small fraction of the largest-magnitude coordinates at high precision allows the remaining values to be quantized more accurately using codebooks optimized offline for the resulting truncated Gaussian distribution. We derive a quantization error upper bound and prove that top- $k$ pre-rotation retention minimizes it for each $k$ . This reduces the search over coordinate subsets to an optimization over $k$ , enabling a fast optimizer that uses offline codebooks and parallel parameter selection for practical implementation. We demonstrate an improved tradeoff between reconstruction accuracy and storage cost through numerical evaluation under the Gaussian model and experiments on nearest-neighbor retrieval, KV-cache compression, and activation compression.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of the TORQUE framework. Given an input vector $x\in\mathbb{R}^{d}$ and a total budget of $db$ bits, the joint optimization selects the number $k$ of largest-magnitude coordinates to retain at high precision before rotation, the retention threshold $c$ after rotation, and the bit budget $s$ with a matching offline precomputed codebook $Q_{c,s}$ for quantizing the remaining entries.

Figure 2: Qwen2.5-0.5B TORQUE improvement: (a) activations; (b) key caches; (c) value caches.

Figure 3: GloVe retrieval: (a) NMSE reduction, (b) change in recall, and (c) Recall@1.

Panel of Figure 4 — Figure 4: CUDA encoding and decoding time on an NVIDIA RTX 5090 for $d=256$ , batch size 128, and a target coordinate budget of 4.5 bits. Pre, Post, and Joint denote TORQUE retention before rotation, after rotation, and at both stages, respectively.

Panel of Figure 5 — Figure 5: NMSE reductions from applying TORQUE .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Biased reconstruction: NMSE reductions for activations, key caches, and value caches. The KV-cache results use retention only after rotation.

Figure 7: GIST retrieval: Recall@1, recall change, and NMSE reduction.

Figure 8: Biased retrieval on GloVe and GIST: Recall@1, recall change, and NMSE reduction.

Figure 9: Native CPU timing: Encoding and decoding at four bits and eight bits.

Figure 10: Apple GPU timing, with the same layout as Figure 9 .

Panel of Figure 11 — Figure 11: Codebooks numerically optimized for mean squared distance with $G\sim N(0,I_{2})$ and $p=0.05$ . The shaded union of disks and the dashed centered disk each contain $95\%$ of the Gaussian probability. Crosses mark all four or eight codewords. Blocks outside the shaded region are retained.

Explore similar work

Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven
May 7, 2026Ran Ben-Basat, William Kuszmaul, Michael Mitzenmacher +2Randomized Hadamard Transforms Residual Vector Quantization
UCL and Broadcom · Carnegie Mellon University · Harvard University +2
Output-Aware Rotation for INT2 KV-Cache Quantization
Aug 3, 2026Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong +4Key-Value Cache Quantization Kv-Cache Management
University of Southern California · Seoul National University · Inha University
OCTOPUS: Optimized KV Cache for Transformers via Octahedral Parametrization Under optimal Squared error quantization
May 20, 2026Mark Boss, Vikram Voleti, Simon Donné +1Key-Value Cache Quantization Key-Value Cache Compression
Stability AI

May 7, 2026, cs.LGJ/K move · Enter open · S save

Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven

Ran Ben-Basat, William Kuszmaul, Michael Mitzenmacher, Amit Portnoy+1

UCL and Broadcom · Carnegie Mellon University · Harvard University+2

Randomized Hadamard Transforms Residual Vector Quantization Quantized Random

Open paper AskPDF arXiv

Uniform random rotations (URRs) are a common preprocessing step in modern quantization approaches used for gradient compression, inference acceleration, KV-cache compression, model weight quantization, and approximate nearest-neighbor search in vector databases. In practice, URRs are often replaced by randomized Hadamard transforms (RHTs), which preserve orthogonality while admitting fast implementations. The remaining issue is the performance for worst-case inputs. With a URR, each coordinate is individually distributed as a shifted beta distribution, which converges to a Gaussian distribution in high dimensions. Generally, one RHT is not suitable in the worst case, as individual coordinates can be far from these distributions. We show that after composing two RHTs on any

d

-sized input vector, the marginal distribution of every fixed coordinate of the normalized rotated vector is within

O(d^{-1/2})

of a standard Gaussian both in Kolmogorov distance and in

1

-Wasserstein distance. We then plug these bounds into the analyses of modern compression schemes, namely DRIVE and QUIC-FL, and show that two RHTs achieve performance that asymptotically matches URRs. However, we show that two RHTs may not be sufficient for Vector Quantization (VQ), which often requires weak correlation across fixed-size blocks of coordinates (as opposed to only marginal distribution convergence for single coordinates). We prove that a composition of three RHTs leads to decaying coordinate covariance. This ensures that any fixed, bounded, multi-dimensional VQ codebook optimized for URRs has the same expected error when using three RHTs, up to an additive term that vanishes with the dimension. Finally, because practical inputs are rarely adversarial, we propose a linear-time

{O}(d)

check on the input's moments to dynamically adapt the number of RHTs used at runtime to improve performance.

Abstract

Explore similar work

Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven

Output-Aware Rotation for INT2 KV-Cache Quantization

OCTOPUS: Optimized KV Cache for Transformers via Octahedral Parametrization Under optimal Squared error quantization