Transforms for LLM Quantization: The Great Inversion and Format Co-Design

Ehsan Jokar

cs.LGAug 25, 2026

Transforms for LLM Quantization: The Great Inversion and Format Co-Design

Authors: Ehsan Jokar

Abstract

Most competitive 4-bit LLM research pipelines now open the same way: apply a linear, function-preserving transform (rotation, scaling, permutation, non-orthogonal affine) so the outlier mass sits more favorably against the group scales, and only then round. Yet we are aware of no survey dedicated to this transform stage, and its literature is quietly re-deriving an older theory. We identify and formalize the principle that organizes it, the Great Inversion: allocation-flexible coding rewards energy concentration, whereas the grouped shared-scale quantization a deployed matrix instruction performs rewards within-group flattening. Classical transform coding (1963: decorrelate, allocate bits, quantize) spends different bits per coordinate at a fixed total rate; for a Gaussian source at high rate the Karhunen-Loeve transform's concentration minimizes distortion. A deployed operand tile instead carries one absolute-maximum scale per group and equal bits everywhere, with no allocation; on a uniform grid that objective rewards flattening, approached by Hadamard incoherence. We prove that opposition under within-group majorization: the prescriptions point in opposite directions, each backed by a proof against its own objective, and for a generic spectrum no optimality guarantee transfers. A second axis is the number format: the non-uniform FP4 grid makes flattening buy less, MXFP4's power-of-two block scale still rewards a rotation confined to that block, and NVFP4's mantissa-carrying scale largely removes that pull, so the target pole depends jointly on allocation regime and format. We survey 200 works to a June 2026 cutoff; classify 43 transform methods by structure, data-awareness, searched-versus-constructed, and runtime cost; record, where reported, how they compose with GPTQ rounding; distill a first-choice guide by deployment regime; and close with the open problems it exposes.

Figures & tables

Panel of Figure 1 — Figure 1: The low-bit post-training-quantization pipeline the surveyed methods assemble, as a two-lane swimlane. Top (offline, built once): calibrate activation statistics $\to$ construct a function-preserving transform $T$ ( Section 6 ) $\to$ fix the target grid/format ( Section 8 ) and error-feedback- round the transformed weights onto it ( Section 7 ) $\to$ pack static low-bit weights. Bottom (online, per token): the deployed datapath ( Section 10 ). The transform straddles the deployment seam (dashed line, Section 2.8 ): its weight side $WT^{-1}$ folds into the packed weights offline ( Definition 2.1 ), while of its two sides only the activation side $XT^{\!\top}$ crosses it (dashed arrow), to an online kernel unless it too can be absorbed ( Theorems 2.17 and 2.2 ). Representative methods are named per stage.

Figure 2: The outlier tax and the $6$ dB/bit crest-factor law ( Corollary 2.7 ). Left: one synthetic $16$ -value group (fifteen $\mathcal{N}(0,1)$ draws plus one injected outlier) quantized on a shared $4$ -bit AbsMax grid; each true value (dot, above the line) is joined to the grid level it rounds to (bold tick, on the line), so the slant of the join is its quantization error and the bold ticks are the levels actually used. The bulk values (near the origin) snap to a handful of levels ( $\mathrm{CF}=3.7$ , $\mathrm{SQNR}=17.2$ dB); after a per-group Hadamard the grid tightens over the now-flat group ( $\mathrm{CF}=1.9$ , $\mathrm{SQNR}=21.6$ dB). Right: the analytic law $\mathrm{SQNR}_{\mathrm{dB}}=6.02\,b+4.77-20\log_{10}\mathrm{CF}$ for $b\in\{3,4,8\}$ . The two marked points are the measured INT4 groups from the left panel; they both sit just below the analytic $b=4$ curve (the flattened group further below): the $2^{b-1}-1\approx 2^{b-1}$ constant simplification shifts the exact curve down by $\approx 1.2$ dB at $b=4$ , and a sub-dB Bennett-model residual of either sign accounts for the rest.

Figure 3: The Great Inversion ( Theorem 4.3 ): one majorization axis, two opposed costs, computed by us on the real

H_{X}

of the layer-

17

q

-projection of TinyLlama-1.1B over

3072

tokens. (a) A single fixed-energy vector morphed from internally flat to concentrated (the measured

H_{X}

eigenvalue spectrum, whose single largest coordinate already holds a quarter of the energy), drawn as Lorenz curves: height at coordinate-fraction

x

is the energy share of the

x

largest coordinates, so the flat pole is the diagonal and concentration bows the curve toward the top-left. (b) Where three energy-preserving transforms actually land on the two costs of Theorem 4.3 , each normalized by its own reference value: the shared-scale AbsMax cost

\sum_{g}M_{g}^{2}

at group size

G=128

, as a multiple of the flat floor

E/G

( Proposition 4.2 ), which is a lower bound neither reading attains here, against the coding cost

D_{\mathrm{vr}}=\mathrm{GM}(v)

, as a multiple of the optimum the KLT does attain ( Theorem 3.3 ). Every transform appears twice on the shared-scale axis: the second-moment profile cost

D_{\mathrm{fr}}=\sum_{g}\max_{k\in g}v_{k}

(filled) and the realized per-token cost an AbsMax kernel pays,

\mathbb{E}_{t}\sum_{g}\max_{k\in g}(Tx_{t})_{k}^{2}

(open), joined by the inflation between them, which Jensen’s inequality keeps positive: a mean of group maxima dominates the maximum of the per-coordinate means. That inflation is largest for the Hadamard (

3.1\times

, its pair sitting at

2.6\times

and

8.0\times

the

E/G

floor, against

1.3\times

for Identity and

1.05\times

for the KLT), so the surrogate flatters the flattening pole most; Lemma 2.14 bounds inflations of this kind in amplitude, this ratio being in its square. The KLT sits exactly on the coding-optimal wall and the randomized Hadamard nearest the flat floor on either reading, opposite ends of the majorization path (gray dashed, drawn in profile space). The plotted path is a global majorization chain, along which the two costs do move oppositely; Theorem 4.3 ’s step-by-step guarantee is the within-group one, and Identity , though majorized by the KLT profile, still carries the higher shared-scale cost. The both-optimal corner (

\star

) is unreachable for this measured spectrum ( Remark 4.4 ). Each point is placed in its transform’s own coordinate order, for the KLT the conventional decreasing-eigenvalue one. The three-transform ordering is the same on both readings, and unchanged at the other group sizes we ran,

G\in\{32,64\}

. This is one layer of one

1.1

B model (the base checkpoint TinyLlama-1.1B-intermediate-step-1431k-3T ,

3072

tokens (six windows of

512

, capture seed fixed) from the wikitext-2-raw-v1 train split, activations cached in fp16 , figure arithmetic in fp64 , the Hadamard sign seed fixed), so it illustrates the two costs rather than measuring their magnitude at deployment scale, and whether the gap widens with model size we do not test.

#	Result (what is proven)	Regime	Objective it optimizes	Reference
1	For a Gaussian source with $\Sigma\succ 0$ at high rate, the KLT with optimal (logarithmic) bit allocation, every coordinate receiving positive rate, is MSE-optimal at a fixed total rate among orthogonal transforms ; the optimal transform concentrates energy	Allocation-flexible $\to$ concentrate	MSE at fixed total rate (per-coefficient allocation)	Huang & Schultheiss (1963) ; Goyal (2001) ; Theorem 3.3
2	Coding gain (the arithmetic-over-geometric-mean measure of the transformed variances; the cited source uses the geometric-over-geometric variant) is maximized by maximally unequal variances; when all coefficients share one scale-invariant quantizer family, a KLT is optimal among orthogonal transforms for any bit allocation	Allocation-flexible $\to$ concentrate	Coding gain (distortion reduction at fixed rate)	Goyal (2001)
3	Nested-lattice quantization is asymptotically optimal for the matrix-multiplication distortion (i.i.d. Gaussian matrices)	Lattice / rate–distortion	Inner-product MSE at fixed rate	Ordentlich & Polyanskiy (2024)
4	$\mu$ - incoherence processing bounds the proxy quantization error; a random rotation achieves low coherence with high probability	Shared-scale $\to$ flatten	Layer-output MSE under uniform quant	Chee et al. (2023)
5	A randomized Hadamard with a dithered Lloyd/Gaussian-companded scalar quantizer attains the same leading MSE constant that Zandieh et al. (2025) proved for a fully random rotation, up to an $o(1)$ vanishing as the bit-width grows	Shared-scale $\to$ flatten (companded grid)	Per-vector $\ell_{2}$ MSE (worst-case unit vector)	Feng et al. (2026)
6	Block-Hadamard rotation has non-asymptotic bounds on the outlier suppression it can achieve	Shared-scale $\to$ flatten	Post-rotation within-group $\ell_{\infty}$ , set by how evenly $\ell_{1}$ mass is spread across blocks	Sanjeet et al. (2026)

Table 1: The master optimality ledger. Rigorous results exist on both sides of the inversion ( Theorem 4.3 ), each under its own stated assumptions and against its own functional , the point being that none transfers across the boundary. Rows 1–2 prove optimality of concentration for allocation-flexible coding (row 3 is the nested-lattice matrix-multiplication optimum); rows 4 and 6 establish achievability and guarantees for flattening on a shared-scale uniform grid, and row 5 the same on a shared-scale Lloyd/Gaussian-companded codebook, and row 10 proves model-optimality (under an explicit stochastic-noise model), while rows 7–9 delimit flattening’s benefit, which is format-dependent and, in row 9, not yet explained by any evaluated surrogate; rows 11–12 and 14 are regime-neutral machinery, and row 13 bridges the two regimes.

Method	Fam.	D	L	Cost	Transform, in one line
CLE ( Nagel et al., 2019 )	Diag	N	N	fold	Data-free pairwise channel rescale equalizing adjacent-layer weight ranges via ReLU scale-equivariance.
Outlier Supp. ( Wei et al., 2022 )	Diag	N	N	fold	Fold LayerNorm’s per-channel $\gamma$ (the outlier amplifier) into the next weight, with a compensating parameter on the residual branch (data-free); token-wise clipping is a separate calibrated add-on.
SmoothQuant ( Xiao et al., 2023 )	Diag	Y	N	fold	$s_{j}=\max\|X_{j}\|^{\alpha}/\max\|W_{j}\|^{1-\alpha}$ migrates activation outliers into weights.
AWQ ( Lin et al., 2024b )	Diag	Y	Y	fold	Per-channel scale $s_{X}^{\alpha}$ protecting the channels salient by activation magnitude; $\alpha$ grid-searched on output MSE.
Outlier Supp.+ ( Wei et al., 2023 )	Diag	Y	Y	fold	Per-channel shift (center) then scale (outlier threshold grid-searched); both migrated into the next weight and bias.
OmniQuant ( Shao et al., 2024 )	Diag	Y	Y	fold	Gradient-learned per-channel scale and shift (LET) plus learnable weight clipping (LWC).

Table 2: The transform methods surveyed, along the classification axes of Section 6 . The complete cross-role index of all cited works is Table 18 . Fam. : Diag (diagonal scaling), Perm (permutation), FixO (fixed orthogonal), LrnO (learned orthogonal, read broadly to include data-fitted rotations obtained in closed form), Aff (non-orthogonal affine), and, within the sixth family, Seq (sequence-axis) and Fr (frame); hybrids are listed under their primary lever, and a starred code is glossed in that method’s own description. D : data-aware. L : the transform’s parameters are obtained by searching over candidate transforms against a quantization objective (gradient descent; an objective-driven search as in AWQ’s exponent grid or PQF’s search over orderings; an alternating solve as in Z-Fold’s ALS or DFRot’s and ConQuR’s Procrustes iterations). They are graded N where the transform is instead given by a construction , for instance: a data-free map (QuaRot’s Hadamard, GSR, Kashin, the folded LayerNorm gain of Outlier Suppression), a formula in calibration statistics (SmoothQuant’s balanced exponent, WUSH’s Cholesky and SVD, ResQ’s PCA), or an ordering or selection read off a clustering or ranking of channel or token statistics (RPTQ, Atom, DuQuant, PrefixQuant); a closed-form rule propagated along a chain of layers (CLE) counts as a construction too, even where it iterates internally. Cost : deployment cost of the transform, counted on the activation side, the weight-side companion always folding offline ( Definition 2.1 ): fold (the activation side is absorbed too, nothing at inference), hybrid (the main map folds but an auxiliary stage stays online), online (an inference-time kernel applies it on every token), none ∗ (the map is undone before storage; a residual index survives only under group-wise scales). Constructions are taken from the source papers; two of the 43 (GPTQ act-order, QuIP#) are rows here by structure but belong primarily to the rounding-and-codebook story of Section 7 , where Table 18 indexes them by that primary role.

Figure 5: The three transform families reshaping a real activation distribution , our own computation on the $q$ -projection input of layer 17 of the base TinyLlama-1.1B-intermediate-step-1431k-3T checkpoint over $3072$ wikitext-2-raw-v1 train tokens, the activations cached in fp16 and the figure arithmetic in fp64 : an illustrative single layer, not a benchmark. Each panel sorts a per-coordinate statistic before and after the transform. Migration (after SmoothQuant, Xiao et al., 2023 ) applies SmoothQuant’s per-channel scale ( 11 ) at $\alpha=\tfrac{1}{2}$ , i.e. $s_{j}=\max|X_{j}|^{1/2}\max|W_{j}|^{-1/2}$ , pulling $X$ down and $W$ up until the two operands coincide at their geometric mean: it relocates crest between the operands rather than reducing the joint per-channel burden ( Section 6.1 , for that invariance and for what it can still equalize across a group), hence no crest number is quoted for it. Rotation (after QuaRot, Ashkboos et al., 2024c ; a randomized Hadamard $R=H\operatorname{diag}(\pm 1)$ ) genuinely collapses the crest factor $\mathrm{CF}$ ( 4 ) of a single operand from $28$ to $5$ , stably across Hadamard sign seeds. Whitening (an activation-side stand-in for the bilateral WUSH map, Chen et al., 2026a , which a single-operand curve cannot fully depict) drives the per-coordinate standard deviations toward a common value (spread, meaning their maximum over their median, $9.0\!\to\!1.2$ , the panel using the regularized inverse square root $(\Sigma+\lambda I)^{-1/2}$ with $\lambda=0.02\,\operatorname{tr}(\Sigma)/d$ , so the residual is that ridge rather than a limit of whitening: exact whitening drives the spread to $1$ by construction here (those tokens against $d{=}2048$ leave $\Sigma$ full rank), and the ridge is what a deployed data-aware map would need on a finite calibration sample); the panel shows the whitening factor alone, with WUSH’s Hadamard omitted, though under WUSH’s own model the two are jointly essential ( Theorem 4.9 ); whitening’s advantage is spectral and bilateral rather than a further cut to the activation crest.

Method	Type	Optimizes	Relation to transform
WaterSIC ( Lifar et al., 2026 )	grid/allocation (variable-rate)	geometric-mean (det $\Sigma_{X}$ ) distortion; basis-free scalar-INT optimum under entropy coding (rates reported as entropy, not log-cardinality)	near-substitute for a rotation at high rate (waterfills grid spacings instead); the worst-case equivalence is left open and the overlap shrinks at $2$ – $4$ bits
FLUTE ( Guo et al., 2024 )	kernel	throughput: LUT dequant keeping pace with tensor cores	none directly; it serves the codebook stage downstream of the transform, and only the scalar lookup-table codebooks, vector and lattice ones being future work in the source
GPTQ ( Frantar et al., 2023 )	rounder	quadratic layer loss; fixed-order blocked error feedback	applies no basis-changing transform; rounds on the given basis (composes; at high rate it is a rotation and per-coordinate allocation that are the near-substitutes, not the rounder, Section 7.2 )
GPFQ ( Lybrand & Saab, 2021 )	rounder	layer output error under the mismatched input pair; greedy path-following	composes; corrects the committed previous-layer input mismatch, adjusts no future weights
LDLQ (QuIP) ( Chee et al., 2023 )	rounder	quadratic proxy loss $\propto\operatorname{tr}(D)$ ; proved optimal, worst and average case, among rounders whose linear feedback depends on $H$ and not on $W$ , when rounding to integers	composes; QuIP pairs it with incoherence processing, which supplies nearly all of the 2-bit gain ( Section 7.2 )
OBC ( Frantar et al., 2022 )	rounder	quadratic layer loss; exact greedy per-row order with optimal compensation	rounds on transform’s basis (composes)

Table 3: Rounding, codebook, allocation, and kernel methods ( Section 7 ), by type and by how each stands to the transform: composing with it, substituting for it, or enabled by it.

Method	Target format	Transform used	Scale treatment
HiFloat4 ( Luo et al., 2026 )	HiFloat4 (E1M2 elements)	none (format definition)	E6M2 per-block base scale (8 b) $+$ two levels of 1-bit micro-exponents (8-way, 16-way); 32 b metadata per G=64 block
QuEST ( Panferov et al., 2025 )	INT or FP4 (grid-agnostic; QAT), INT the main setting	Hadamard (Gaussianize)	single MSE-optimal clip threshold
INT-FP flip (high-rate) ( Ordentlich & Polyanskiy, 2026a )	INT vs FP, 4–8 bit ( $+$ block-scaled)	random rotation (analyzed, not proposed)	real scales; high-rate rate–distortion (INT $\sim$ absmax/ $\ell_{\infty}$ , FP $\sim$ joint concentration $\Delta_{\mathrm{FP}}$ )
Block Rotation ( Shao et al., 2025b )	MXFP4	block-confined Hadamard (block $=G=32$ ), its sole proposed transform, a learned block rotation being evaluated as a variant, applied on top of GPTQ	E8M0 power-of-two
MR-GPTQ ( Egiazarian et al., 2026 )	MXFP4 $+$ NVFP4	fused block-diagonal Hadamard (tunable block $k$ )	alternating per-tensor $+$ per-group MSE fit, reported for NVFP4 without rotations; a static value for MXFP4 with rotations; separately, a proposed E8M0 exponent grid fitted to the tensor’s scale range
Quartet ( Castro et al., 2025 )	MXFP4	block-diagonal Hadamard (block=G=32)	E8M0 power-of-two

Table 4: Number formats and transform–format co-design methods ( Section 8 ).

Element grid	no rotation	$+$ Hadamard	gain
INT4 (uniform)	$13.1$	$19.3$	$+6.2$
FP4 (E2M1)	$16.6$	$19.4$	$+2.8$

Table 5: Format-dependent rotation benefit (real activations). Mean SQNR in dB (our computation) of a 4-bit AbsMax quantizer on

12{,}288

real

128

-channel activation groups (

768

wikitext-2-raw-v1 train tokens

\times\,16

groups, from the mechanism figures’ cache; base TinyLlama-1.1B-intermediate-step-1431k-3T , layer 17

q

-projection input, activations cached in fp16 , arithmetic in fp64 ), with and without a per-group randomized Hadamard, its sign draw seeded per token and the reported gains stable across seeds. The FP4 grid’s non-uniform spacing already accommodates the outliers, so it starts

3.5

dB ahead and has less left to win; rotated, the two converge to within

0.1

dB. Both grids still improve here. The outright sign reversal, in which a rotation becomes harmful for FP, is the separate finding of Ordentlich & Polyanskiy (2026a) on real Llama-3 weights under their joint energy-concentration (

\Delta_{\mathrm{FP}}

) analysis; this measurement substantiates only the weaker statement, that the FP grid benefits far less than INT, not that reversal.

Method	Tensor / site	Lever	Key idea
OSCAR ( Zhou et al., 2026 )	KV-keys $+$ values	rotation	keys rotated by the query covariance $Q^{\!\top}Q$ , values by the score-weighted covariance ( $+$ Hadamard); minimizes the downstream attention ( $QK^{\!\top}$ / $PV$ ) distortion, not cache reconstruction
KIVI ( Liu et al., 2024c )	KV-keys/KV-values	none	per-channel keys, per-token values from an element-distribution study; tuning-free, FP16 residual window of recent tokens, fused 2-bit kernel
KVQuant ( Hooper et al., 2024 )	KV-keys/KV-values	none	per-channel keys, per-token values, pre-RoPE, Fisher-weighted codebook + dense-and-sparse
RotateKV ( Su et al., 2025 )	KV-keys/KV-values	rotation	Walsh–Hadamard + calibrated channel permutation, grouped heads on keys, pre-RoPE
PolarQuant (Han et al.) ( Han et al., 2025 )	KV-keys/KV-values	reparam.	random-rotation preconditioning makes recursive-polar angles data-independent; stores no per-block scale or zero-point; small per-level angle codebooks; $4.2\times$ KV compression
PolarQuant (Wu et al.) ( Wu et al., 2025 )	KV-keys	reparam.	2D sub-vectors (radius $+$ angle) exploiting RoPE’s paired rotations; $QK^{\!\top}$ precomputed into a decode-time lookup table

Table 6: Transforms applied beyond the weight matrix ( Section 9 ): the KV-cache, the attention matmuls, state-space models, mixture-of-experts, diffusion transformers, and training. Lever names what the method actually does to the tensor: rotation (orthogonal change of basis, with the axis given where it is not the feature axis), scaling (multiplicative per-channel), additive (a function-preserving shift), reparam. (a nonlinear re-coordinatization), routing or sampling (bits or rows chosen rather than reshaped), additive-lowrank (an off-precision correction term), none (the method changes granularity, axis or calibration only, with none (calib.) marking a calibration-side fix).

Method	Role	What it provides	Rate
CCQ ( Zhou et al., 2025 )	fixed-rate coder	$\sim$ 2-bit (2.06 bpw) lookup-free convolutional code, bit-shift decode (no codebook lookup)	fixed-rate
HadaCore ( Agarwal et al., 2024 )	kernel	online Walsh–Hadamard rotation for incoherence processing	fixed-rate
QUIK ( Ashkboos et al., 2024b )	kernel	mixed-precision W4A4 GEMM; $\sim$ 256 outlier columns kept in FP16, summed into the INT4 GEMM epilogue (no extra round-trip)	fixed-rate
DFloat11 ( Zhang et al., 2025f )	lossless-coder	BF16 exponent field, GPU decode at transformer-block granularity, $\sim$ 30% weight-memory cut	variable-length
NeuZip ( Hao et al., 2024 )	lossless-coder	BF16 exponent field, GPU pre-GEMM decode	variable-length
ZipNN ( Hershcovitch et al., 2025 )	lossless-coder	BF16 exponent field for storage/network transfer	variable-length

Table 7: Systems: fused kernels, serving stacks, and lossless/variable-length weight coders ( Section 10 ).

Accelerator (arch.)	Low-precision matmul	MX/NVFP4	Bearing on the dequantization cost
A100 (Ampere, ’20)	INT4 , INT8 , FP16 / BF16	–	INT4 GEMM is tensor-core-native ( $4\times$ FP16 ), but W4A4’s per-group scales still rescale INT32 partial sums on the CUDA cores.
H100/H800 (Hopper, ’22)	FP8 , INT8 , FP16 / BF16	–	INT4 dropped from the tensor core: it compiles to CUDA-core IMAD ; low-bit compute shifts onto the FP8 / INT8 path.
B200/GB200 (Blackwell, ’24)	FP4 , FP6 , FP8 , INT8 , FP16 / BF16	NVFP4 , MXFP4/6/8	The tensor core applies the block scale in hardware, so the FP4 block-scale “dequantization” is not a separate software step.
MI300X (CDNA3, ’23)	FP8 , INT8 , FP16 / BF16	–	FP8 / INT8 matrix-core path; no native FP4 / FP6 .
MI355X (CDNA4, ’25)	FP4 , FP6 , FP8 , INT8 , FP16 / BF16	MXFP4/6/8	First AMD data-center FP4 ; the MX block scale is handled in the matrix core.

Table 8: Low-precision matmul support across GPU generations, and its bearing on the dequantization cost. The datapath the transforms must survive is a moving target: integer INT4 was a first-class tensor-core type on Ampere, silently lost its tensor-core path on Hopper, though Ada kept it, and has been superseded on Blackwell and CDNA4 by hardware-native 4-bit floating-point with microscaling, where the tensor core applies the per-block scale itself. Sources: NVIDIA (2020) ; NVIDIA (2022) ; NVIDIA (2024) ; NVIDIA (2025d) ; AMD (2023) ; AMD (2025) ; Luo et al. (2024) , the NVIDIA block-scale instruction semantics being documented in the PTX ISA reference.

Lane	W / A	Where the per-scale-group multiply lands, and representative kernels
W4A16	INT4 / FP16	Weight-only: an INT4 $\to$ FP16 upconversion that a well-pipelined kernel hides behind the weight loads it saves at the batch sizes decode runs at (Marlin holds near- $4\times$ to batch $16$ – $32$ ; Machete), or eliminates outright by tabulating partial products (LUT-GEMM); at larger batch the GEMM turns compute-bound and the memory saving has nothing left to hide behind.
W4A8	INT4 / INT8	INT4 weights decoded to INT8 (register-level, or the scale folded offline), then INT8 $\times$ INT8 on the tensor cores: no partial-sum rescale (QServe/QoQ, QQQ, LiquidGEMM).
W4A4 INT , coarse	INT4 / INT4	Any scale constant along $K$ (per-tensor, per-token activation, or per-output-channel weight) $\Rightarrow$ one rescale of the INT32 accumulator in the epilogue (cheap); viable once a strong transform flattens the tensor (QuaRot) or the outliers are pulled to FP16 (QUIK), with MergeQuant folding the scale away and PrefixQuant making it static.
W4A4 INT , per-group	INT4 / INT4	Per-group scale varies along $K$ $\Rightarrow$ partial sums rescaled in the main loop on the CUDA cores, the upper end of the $20$ – $90\%$ dequantization tax QServe measures across INT4 methods (Atom fuses it into the pipeline; COMET, APEX4 attack it directly).
W4A4 FP4 , native	FP4 / FP4	Per-block ( $16$ or $32$ ) scale applied inside the tensor core by a block-scaled MMA: no software step (Blackwell tcgen05.mma ; CDNA4 documents the capability without naming the instruction).

Table 9: The four-bit lanes, compared by where the per-scale-group multiply (the “dequantization”) lands. The cost of low-bit inference is set less by the nominal bit-width than by how many operands are quantized and at what granularity the shared scale varies, because a scale that varies along the matmul’s contraction (

K

) axis cannot be folded into a single output scale and must be applied mid-reduction. Coarse (per-tensor/channel) scales rescale once in the epilogue; per-group scales rescale in the main loop on the CUDA cores; and native FP4 moves the per-block multiply into the tensor core itself.

Transform family	Representative	W4A4 PPL
Diagonal (rebalance)	SmoothQuant †	83.12
Permutation (regroup)	Atom	6.12
Fixed orthogonal (Hadamard)	QuaRot	6.19
Learned orthogonal	SpinQuant	5.95
Non-orthogonal affine	FlatQuant †	5.78
Sequence & frame	PrefixQuant	5.93

Table 10: The six transform families across two W4A4 protocols. A representative method from each family of the taxonomy ( Figure 4 ) on Llama-2-7B WikiText-2 perplexity (

\downarrow

). Rows marked

\dagger

are from Sun et al. (2025) (Table 1); the rest from Chen et al. (2026c) (Table 2; W4A4KV4 , context 2048, per-token-dynamic activations except Atom, which is group-wise at

g{=}128

in both operands and adds mixed precision). The two protocols agree to within

1.5\%

on the methods they share, QuaRot (

6.10^{\dagger}

6.19

) and SpinQuant (

5.96^{\dagger}

5.95

), so the rows can be read together for the family-level contrast they are used for here, though the two protocols are not identical and differ in calibration corpus, and SpinQuant is the one row the source grays as trained on the evaluation corpus, so the

0.01

inversion there is not independent evidence for the stitch. Every value verbatim. The

\dagger

rows come from that source’s own rounder blocks (SmoothQuant is reported with RTN only, FlatQuant with GPTQ); the Chen et al. (2026c) rows carry that paper’s Atom and QuaRot entries from Lin et al. (2025) and its own reproductions for the rest. The diagonal-versus-rotation verdict is not a rounder artifact: under matched RTN in the same source it is SmoothQuant

83.12

against QuaRot

8.56

( Section 11.1 ). Pure diagonal scaling alone fails; every other family, at its best, lands within

\sim\!0.7

perplexity of FP16 . (Permutation and sequence/frame maps usually compose with the others rather than run alone; Atom, e.g., composes its permutation with mixed precision.) Best low-bit entry in bold .

Figure 6: Activation quantization ( W4A4 , INT ): the transform ladder. All numbers from Sun et al. (2025) (Tables 1–2). (a) Llama-2-7B WikiText-2 perplexity (log scale) climbing the degrees-of-freedom ladder, diagonal (SmoothQuant) $\to$ fixed rotation (QuaRot) $\to$ learned rotation (SpinQuant) $\to$ learned affine (FlatQuant), each with the RTN and GPTQ rounder where reported. (b) The same ordering among the three transform rungs (the diagonal rung is not reported here) in zero-shot accuracy across five models (GPTQ rounder), against each model’s FP16 reference (dashed, as in (a)).

WikiText-2 perplexity $\downarrow$
Method (family)	L2-7B	L2-13B	L2-70B	L3-8B	L3-70B
FP16 (reference)	5.47	4.88	3.32	6.14	2.86
SmoothQuant (diagonal)	83.12	35.88	26.01	210.19	9.60
QuaRot (fixed rotation)	6.10	5.40	3.79	8.16	6.60
SpinQuant (learned rotation)	5.96	5.24	3.70	7.39	6.21
FlatQuant (learned affine)	5.78	5.11	3.54	6.90	3.77

Table 11: Activation quantization ( W4A4 , INT ): the transform ladder, quantified. All values transcribed verbatim from Sun et al. (2025) (Tables 1–2): one consistent protocol (4-bit weights, activations and KV cache, integer grid, GPTQ rounder for the rotation/affine methods; SmoothQuant uses RTN). WikiText-2 perplexity (

\downarrow

) and zero-shot average accuracy (

\uparrow

, %). Reading down each block climbs the degrees-of-freedom ladder of Section 6 (diagonal

\to

fixed rotation

\to

learned rotation

\to

learned affine); the best transform recovers to within

2.4

accuracy points of the FP16 reference, and

0.22

–

0.91

perplexity of it. Best low-bit entry per column in bold .

Bits	Method	L2-7B	L2-13B	L2-70B
16	FP16 (reference)	5.47	4.88	3.32
4	AWQ	6.15	5.12	–
	OmniQuant	5.74	5.02	3.47
	QuIP#	5.56	4.95	3.38
3	AWQ	24.0	10.5	–
	OmniQuant	6.58	5.58	3.92

Table 12: Weight-only quantization: the low-bit regime is where the codebook and fine-tuning pay off, on top of an RHT-plus-scalar-LDLQ baseline doing most of the work. WikiText-2 perplexity (

\downarrow

), transcribed verbatim from Tseng et al. (2024a) (Table 2; Llama-2, context length 2048, activations FP16 ). At 4 bits every method is near FP16 ; as the budget drops the scalar methods (AWQ, OmniQuant) diverge, while QuIP#’s incoherence rotation plus lattice codebook ( Section 7.3 ), together with the method’s inter-layer fine-tuning, stays close; at 2 bits it stays within

1.2

perplexity of FP16 , close to 3-bit OmniQuant on one fewer bit, where the (per-channel) scalar baselines diverge, a raw gap a finer group size would narrow but the codebook’s space-filling advantage sustains. “–” = not reported. Best per column within each bit-width block, excluding the FP16 reference, in bold .

Format	Method	MMLU-CoT	GSM8K	HellaSw.	WinoG.	Avg.
BF16	(reference)	72.76	85.06	80.01	77.90	78.93
NVFP4	RTN	68.26	78.39	78.15	74.11	74.73
	RTN $+$ block-H	67.41	78.01	77.31	73.48	74.05
	QuaRot (global)	66.50	77.40	77.25	75.14	74.10
	GPTQ	68.85	81.25	78.26	74.51	75.72
	MR-GPTQ	69.12	80.80	78.17	75.24	75.84

Table 13: The format flip: on FP4 grids the near-essential-for- INT4 rotation reverses sign. W4A4 accuracy (%,

\uparrow

) on Llama-3.1-8B-Instruct: seven of the nine method rows of Chen et al. (2026a) ’s Table 2, transcribed verbatim ( Avg. as printed by the source, which can differ from the mean of its four task columns by up to

0.05

). The two omitted rows read SmoothQuant

75.70

70.30

and RTN-WUSH

75.28

73.21

( NVFP4 / MXFP4 averages); the second is that source’s own headline for pairing its transform with plain RTN, and the first is the source’s SmoothQuant baseline, notable for how much better a pure diagonal scaling fares here than the same family fares in the INT4 literature ( Table 10 ), though on a different model, metric and scale granularity, so that contrast is suggestive rather than a matched instance of the flip. Row labels are shortened from the source (RTN

=

RTN-I, RTN

+

block-H

=

RTN-H, GPTQ

=

GPTQ-I, MR-GPTQ

=

GPTQ-H, WUSH

=

GPTQ-WUSH). Two floating-point formats: NVFP4 (block 16, FP8 scale) and MXFP4 (block 32, power-of-two E8M0 scale). Rows are grouped by rounder , QuaRot’s taken from the setup paper below and the one row whose rounder no source states placed last, so each transform reads against the no-transform baseline at matched rounding ( Section 12.1 ): a Hadamard confined to the scale block helps MXFP4 under both rounders (

69.32\!\to\!70.45

with RTN,

70.62\!\to\!73.65

with GPTQ) and is neutral-to-mildly-harmful on NVFP4 (

74.73\!\to\!74.05

75.72\!\to\!75.84

). The global Hadamard (QuaRot) that is near-essential on the integer grid ( Table 11 ) instead craters MXFP4 ,

6.4

points below plain per-block RTN and

7.7

below GPTQ. The AbsMax surrogate does not account for that contrast between a block-confined and a global rotation: it puts both families at the same ideal floor, and neither map’s actual residual was computed on the published models; Section 8.2 states what is and is not explained. The source states that for its block-transform rows “the transform block size is always the same as the quantization group size,” a condition it notes is “not applicable to SmoothQuant, QuaRot, and SpinQuant” (hence global); it does not state the rounder used for the SpinQuant row; for QuaRot, the setup paper whose protocol it adopts specifies round-to-nearest after the rotation ( Egiazarian et al., 2026 ) . The adaptive whitening transform (WUSH, Theorem 4.9 ) reports the best average in both formats, though Chen et al. (2026a) describe their NVFP4 improvements as often within run-to-run variability, on a different model and benchmark suite, so the format-level contrast rather than the within-format ranking is what these averages settle. Best per column within each format block , excluding the BF16 reference and the two source rows omitted above, in bold .

Method	KV4	KV3	KV2
FP16 (reference)	5.68
INT (naive, per-token)	5.98	10.87	11779
Atom	5.77	6.17	37.37
FlexGen	5.73	5.93	11.09
KVQuant	5.72	5.87	7.23
KVQuant $+1\%$ outliers	5.69	5.75	6.01

Table 14: KV-cache quantization: axis choice and outlier retention. WikiText-2 perplexity (

\downarrow

) on the first-generation LLaMA-7B with only the KV cache quantized (weights and activations FP16 ), transcribed verbatim from Hooper et al. (2024) (arXiv v6, Table 1). The sub-4-bit Atom and FlexGen entries are Hooper et al. (2024) ’s re-implementations of those 4-bit methods extended to 3 and 2 bits; the Atom and FlexGen rows quantize per-token post-RoPE with grouped uniform scales, so their edge over the naive row is finer granularity, not axis choice. At 4-bit every scheme is near FP16 ; at 2-bit a naive per-token integer scale explodes, because the keys carry per-channel outliers that RoPE mixes across positions ( Section 9 ), whereas KVQuant’s pre-RoPE per-channel key layout, non-uniform codebook, attention-sink-aware quantization (the first token kept in FP16 ), and

1\%

full-precision outlier retention hold within about a third of a perplexity point. The source’s groupless NormalFloat rows (nf4

5.87

, nf3

7.33

, nf2

3210

) are omitted for space. Best low-bit entry per column in bold .

System	Regime	Reported gain (metric)	Baseline
QuaRot ( Ashkboos et al., 2024c )	W4A4	$2.16$ – $3.33\times$ prefill; ${\geq}3.6\times$ memory †	FP16
FlatQuant ( Sun et al., 2025 )	W4A4	up to $2.30\times$ prefill; $1.76\times$ decode ‡	FP16
QUIK ( Ashkboos et al., 2024b )	W4A4 §	up to $3.4\times$ end-to-end	FP16
Atom ( Zhao et al., 2024 )	W4A4	up to $7.7\times$ throughput (fixed latency)	FP16
QServe ( Lin et al., 2025 )	W4A8KV4	$1.2$ – $3.5\times$ throughput	TensorRT-LLM
HadaCore ( Agarwal et al., 2024 )	Hadamard kernel	$1.1$ – $1.4\times$ A100; $1.0$ – $1.3\times$ H100 ¶	prior FWHT kernel

Table 15: Reported efficiency of low-bit inference systems. Each factor is transcribed verbatim from the cited paper; the metrics and baselines differ (throughput vs. latency; FP16 vs. a production TensorRT-LLM server vs. a prior kernel), so the numbers state each system’s own claim and are not a controlled cross-comparison. ¶ HadaCore’s peaks are

3.5\times

(A100) and

3.6\times

(H100). § QUIK’s

3.4\times

keeps the down-projections at 8 bits. ‡ FlatQuant’s figures are at batch 64. ∥ FLUTE’s

2

–

4\times

is at batch sizes below

32

and group size

128

. † QuaRot’s factors are kernel-level: the prefill figures are upper ends of a batch sweep and both are measured on a single transformer block rather than the full model. The direction is nonetheless uniform: low-bit inference buys a multiplicative speed and memory gain, and the transforms and codebooks that enable it are themselves fast. Throughput measured at a fixed latency target (Atom) yields larger factors than a latency speed-up.

Scenario	First choice	Rationale / governing result
Weight-only W4A16 , INT	Per-channel scaling + error-feedback rounding; codebook (NF4) if a LUT kernel is available	Activations stay high-precision, so the weight distribution alone matters; AWQ’s scaling is reported orthogonal to GPTQ and composes with it ( Lin et al., 2024b ) ; NF4 matches the codebook to the Gaussianized weight distribution, while QuIP#’s lattice codebook adds the space-filling gain of Theorem 3.2 ( Section 7.3 ). A rotation typically helps little at 4-bit weight-only, though it becomes valuable at 2–3 bits ( Table 12 ).
W4A4 , INT	Randomized Hadamard rotation (fixed, i.e. data-free) $\to$ learned rotation $\to$ learned affine (FlatQuant); + GPTQ	The activation crest factor is the wall; incoherence is near-optimal at zero calibration ( Proposition 2.15 ), while a data-aware whitening is near-optimal for the integer grid within a $d^{o(1)}$ factor, $d$ the transform block size, on Gaussian or Laplacian data (a factor $d$ for arbitrary distributions; WUSH, Theorem 4.9 ), worth adding where a fused kernel exists. This is the regime the rotations were built for ( Figure 6 ; six families, Table 10 ).
W4A4 , MXFP4 (power-of-two E8M0 block scale)	Block-sized (group-matched) Hadamard for the scale ; avoid a global rotation; add a WUSH-style data-aware block whitening where a fused kernel exists	The FP element grid already handles per-coordinate range, so a rotation buys less, and on an ideal grid it can cost by destroying a favorable weight–activation anti-alignment ( Section 8.2 ); the coarse power-of-two E8M0 block scale then re-imposes an $\ell_{\infty}$ -like penalty at the block maximum, so a rotation helps again provided its support stays near that block (MR-GPTQ, whose block is a tunable power of two, Table 13 ; Quartet fixes its block to the microscaling group at training time). The source reports a single static grid value sufficing once a block rotation is in place, so per-layer scale re-optimization is not the lever here.
W4A4 , NVFP4 (mantissa-carrying FP8 block scale)	Little or no plain rotation; a data-aware block whitening (WUSH) where a fused kernel exists; per-block scale fitting; consider a normalized architecture if training	The mantissa-carrying FP8 scale removes most of the block-level penalty ( Table 13 ); normalized architectures show the rotation can sometimes be dropped entirely ( Section 8 ). A data-aware whitening still leads: WUSH, exactly optimal for the floating-point AbsMax model ( Theorem 4.9 ), holds the best average on both FP4 formats in Table 13 , by $0.70$ points on MXFP4 and $0.26$ here, the source describing its NVFP4 gains as often within run-to-run variability.
KV-cache, low-bit	Per-channel keys / per-token values, pre-RoPE; at 2-bit either add a codebook with outlier retention or switch to a per-token rotation	Keys carry per-channel outliers and RoPE destroys the structure ( Section 9 ); axis choice suffices at 3–4 bit (KVQuant, Table 14 ). At 2-bit the two routes diverge: keep the per-channel axis and add a non-uniform codebook with outlier retention (KVQuant) or a residual window (KIVI), or go per-token under a pre-RoPE grouped-head Hadamard (RotateKV), whose rotation replaces the per-channel axis rather than composing with it.
Attention matmuls	Mean-subtraction smoothing of K; exploit softmax boundedness for P; two-level scale for P in FP4	The probability matrix is bounded and needs a scale trick, not a transform; K has a shared-bias outlier (SageAttention, Section 9 ).

Table 16: A first-recommendation guide indexed by deployment scenario. Entries name the transform family and representative methods to try first, with the governing result. The unifying rule is the inversion of Section 4 , refined by the format flip of Section 8 .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$d$	ambient dimension: the number of coordinates a transform acts on ( $d_{\mathrm{in}},d_{\mathrm{out}}$ for a layer’s input and output). Wherever the WUSH guarantee is stated ( $d^{o(1)}$ or factor- $d$ near-optimality, Theorem 4.9 ), $d$ is instead the transform’s block dimension, which Chen et al. (2026a) set equal to $G$ ; in the W $b$ A $c$ KV $d$ shorthand it is a bit-width
$W\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}$	a layer’s weight matrix
$X\in\mathbb{R}^{N\times d_{\mathrm{in}}}$	the layer’s input activations ( $N$ tokens as rows)
$\bar{X}$	the normalized stream: the output of whichever RMSNorm opens the sub-block under discussion, that norm taking the layer input in attention and the post-attention residual in the MLP ( Section 2.1 )
$Y=XW^{\top}$	the layer output the quantized layer must preserve
$T\in\mathrm{GL}(d)$	a function-preserving transform; $Y=(XT^{\top})(WT^{-1})^{\top}$

Appendix

Table 17: Notation used throughout the survey.

Work	Section	Role
Classical transform coding and quantization ( Section 3 )
Ahmed et al. (DCT) ( Ahmed et al., 1974 )	3	The discrete cosine transform; the fixed, data-independent transform that approximates the KLT for correlated sources.
Ben-Basat et al. (RHT proven) ( Ben-Basat et al., 2026 )	2	Composing two randomized Hadamards puts every marginal within $O(d^{-1/2})$ of a Gaussian, and three makes the coordinate covariance decay, so Haar-designed codebooks transfer to Hadamards.
Bennett (1948) ( Bennett, 1948 )	2	Bennett’s integral and the additive-white-noise high-resolution model behind the 6 dB/bit law.
Choi Hessian-weighted ( Choi et al., 2017 )	3	Diagonal-Hessian-weighted Lloyd–Max quantization; the CNN-era sibling of GPTQ’s second-order objective, not its parent.
Conway–Sloane lattices ( Conway & Sloane, 1982a )	3	Fast closest-point decoding algorithms for the classical lattices; this is what makes a high-dimensional codebook usable at all.

Appendix

Table 18: Complete index of the 200 cited works, grouped by role. Each is classified once; methods also appear in the per-domain tables of the corresponding section.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 386 papers

A Comprehensive Evaluation on Quantization Techniques for Large Language Models
Jul 23, 2025Yutong Liu, Cairong Zhao, Guosheng HuLLM Quantization Post-Training Quantization
School of Computer Science and Technology, Tongji University, Shanghai 201804, China · School of Engineering Math and Technology, University of Bristol, BS8 1QU, UK
FPTQuant: Function-Preserving Transforms for LLM Quantization
Jun 5, 2025Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1LLM Quantization LLM Inference Acceleration
Qualcomm AI Research, Amsterdam, Netherlands
SpectrumQuant: Spectrum Alignment for LLM Quantization
Oct 14, 2026Dwaipayan Haldar, Indranil Patra, Azhar Yousuf +1LLM Quantization Rotation-Based Quantization
Indian Institute of Science Bengaluru-560012, Karnataka, India

Jul 23, 2025J/K move · Enter open · S save

A Comprehensive Evaluation on Quantization Techniques for Large Language Models

Yutong Liu, Cairong Zhao, Guosheng Hu

School of Computer Science and Technology, Tongji University, Shanghai 201804, China · School of Engineering Math and Technology, University of Bristol, BS8 1QU, UK

LLM Quantization Post-Training Quantization Low-Bit Quantization 4-Bit Quantization

Open paper AskPDF arXiv

For large language models (LLMs), post-training quantization (PTQ) significantly reduces memory and computational overhead. Despite rapid advancements, existing papers often lack experiments on the same ground. In addition, since quantization solutions typically contain multiple components, it is important to analyze the connections among numerous existing methods to gain an in-depth understanding of this field. To bridge these gaps, we conduct an up-to-date and extensive review of the state-of-the-art methods and perform comprehensive evaluations of low-bit quantization on representative text-only LLMs on the same ground to ensure fair comparisons. To our knowledge, this fair and extensive investigation remains critically important yet significantly underexplored in the research community. To better understand the connections, we first decouple published quantization into two steps: pre-quantization transformation to reduce outliers, and quantization error mitigation to offset performance loss. Second, we extensively evaluate the impact of quantization settings, including granularity and symmetry. Third, we analyze the performance of the latest MXFP4 and NVFP4 data formats. Our experimental results first demonstrate that optimizing rotation and scaling yields the best pre-quantization transformation, while combining low-rank compensation with GPTQ occasionally outperforms using GPTQ alone. Second, we verify that finer granularity improves performance at the cost of storage and asymmetric quantization benefits activations significantly more than weights. Third, we find that the format and precision of scaling factors have a significant impact on performance; notably, rotation-based transformation is effective for INT4 but demonstrates little improvement for MXFP4 and NVFP4, inspiring further investigation in this field.

Abstract

LLM Quantization

Explore similar work

A Comprehensive Evaluation on Quantization Techniques for Large Language Models

FPTQuant: Function-Preserving Transforms for LLM Quantization

SpectrumQuant: Spectrum Alignment for LLM Quantization