Perplexity Cost Understates What Activation Quantisation Breaks

Anish Sathyanarayanan

cs.LGSep 19, 2026

Perplexity Cost Understates What Activation Quantisation Breaks

Authors: Anish Sathyanarayanan

Organizations: BITS Pilani, K K Birla Goa Campus

Abstract

Activation quantisation is usually evaluated with an aggregate metric, perplexity, averaged over every token a model predicts. We ask whether that average identifies which computations a quantiser damages. Perplexity turns out to be a reliable aggregate signal: across 12 models from four families and 780 within-model comparisons, the arm perplexity prefers also retains more induction and more retrieval in all but 2.1 and 4.0 percent of cases respectively. But where perplexity has risen by only a factor of 1.2 to 1.5, induction still keeps 0.959 of its intact accuracy while retrieval has already fallen to 0.554, a gap the aggregate number does not surface. This gap has structure, not just size: a matched Gaussian-noise control of the same per-channel magnitude leaves it largely intact, and randomising only the sign of the quantisation error, every magnitude held fixed, is nearly as harmless, so magnitude alone does not explain the damage. Quantising in a rotated basis, which changes coordinate alignment without changing error magnitude, restores induction from 0.001 to 0.980 at three average bits per token in a single-block intervention, though retrieval recovers less completely at the same setting (0.694); end-to-end at four average bits, induction reaches 0.968 and retrieval 0.534. The pattern holds on two further models up to 32B parameters and, in the deployed configurations we tested, under AWQ once activations are pushed to 4 bits. A perplexity target bounds the average cost of a transformation applied to the activation; it does not, by itself, show which computations survived.

Figures & tables

Perturbation	Induction	At distance	Retrieval
Quantised, 4 bits	0.681	0.662	0.272
Quantised, 3 bits	0.001	0.001	0.000
Gaussian noise, matched, 3 bits	0.681	0.698	0.219
Signs randomised, 3 bits	0.695	0.677	0.188
Rotated basis, 3 bits	0.980	0.981	0.694

Table 1: Fraction of accuracy retained on each probe, median over model families at three relative depths. Intact accuracy is 0.954–0.987 for the copying probes and lower for retrieval.

Perplexity, relative to intact	Settings	Induction	Retrieval
1 to 1.2	95	0.999	0.991
1.2 to 1.5	22	0.959	0.554
1.5 to 2	26	0.929	0.502
2 to 3	17	0.689	0.246
3 to 5	15	0.377	0.080
5 to 10	19	0.172	0.019

Table 2: Probe accuracy by perplexity band, relative to the unquantised model. Pooled over the three deployable arms, all models, depths, and bitwidths.

Precision	Average bits	Perplexity	Induction	Retrieval
Uniform	2.00	30,429.6	0.000	0.000
Uniform	3.00	1,310.6	0.001	0.000
Uniform	4.00	55.1	0.681	0.272
Uniform	6.00	25.5	0.998	0.960
Uniform	8.00	22.0	1.000	0.986
Protected, Fisher	3.00	324.3	0.007	0.000

Table 3: Perplexity and probe accuracy at matched average bitwidth, median over model families at three relative depths. Full-precision perplexity is 21.6.

Precision	Average bits	Perplexity	Induction	Retrieval
Uniform	4.00	20,442.3	0.003	0.000
Uniform	6.00	38.4	0.905	0.597
Rotated	4.00	35.7	0.968	0.534
Rotated	6.00	22.8	0.999	0.972

Table 4: End-to-end quantisation, matched average bitwidth. Same arms and metrics as Table 3 .

Figure 1: Every arm from Sections 6 – 8.1 on one axis, pythia-1.4b. A weight and an activation quantiser are not on the same bit axis, but are on the same perplexity axis; comparisons across that boundary are not matched-budget (Section 11 ). Deployed quantisers sit between our uniform and rotated arms; QuaRot’s Hadamard rotation reaches the best tradeoff of anything plotted.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Arm	Target	Bits	Config	Perplexity	Induction	Retrieval
pythia-1.4b, full precision: perplexity 19.7, induction 0.955, retrieval 0.281
Uniform (activation)	activation	4	W4A4	31,098.5	0.005	0.000
Rotated (activation)	activation	4	W4A4	31.8	0.842	0.191
GPTQ	weight	3	W3A16	22.8	0.916	0.262
GPTQ	weight	4	W4A16	20.1	0.948	0.273
GPTQ	weight	4	W4A4	1,037.7	0.019	0.000

Appendix

Table 5: Full deployed-quantiser sweep. Target separates arms that perturb weights from our own arms that perturb the residual stream; comparisons across that boundary are not matched-budget (§ 11 ). GPTQ’s W4A4 perplexity, two orders of magnitude worse than AWQ’s, follows from its weight-reconstruction calibration objective and is not a claim about its intended weight-only use. pythia-1.4b’s SmoothQuant and AWQ W4A16 rows are independently measured and coincide almost exactly (verified against the per-layer raw output); at this mild, weight-only setting the two methods barely perturb this model differently. The Uniform (activation) and Rotated (activation) rows use the end-to-end variant of Section 8 (every block from the intervened one onward), not the single-block setting the rest of the paper reports by default, since GPTQ/AWQ/SmoothQuant also quantise the whole model; this is why they read far worse than the single-block figures elsewhere.

Figure 2: Standard benchmarks under each deployed-quantiser arm, pythia-1.4b (referenced from Section 6 ). GPTQ’s W4A4 collapse on LAMBADA (a next-token-prediction task that needs the passage’s context, like this paper’s retrieval probe) matches the mechanistic damage; the four multiple-choice commonsense benchmarks move little under any arm, GPTQ W4A4 included.

	Per token		Per tensor
Bits	Dense	Hadamard	Dense	Hadamard
8	0.980	0.979	0.967	0.979
6	0.979	0.979	0.649	0.955
4	0.974	0.978	0.000	0.002
3	0.965	0.968	0.000	0.000
2	0.051	0.291	0.000	0.000

Appendix

Table 6: The dense-versus-Hadamard comparison of Section 8.1 , both granularities in full.

Model	Baseline (ms)	+rotation (ms)	+rotation+quantise, 4-bit (ms)
Pythia-1.4b	0.83	1.46 (+75%)	1.45 (+74%)
GPT-2	0.32	0.52 (+65%)	0.52 (+65%)
Gemma-2-2b	1.60	2.43 (+52%)	2.41 (+51%)
Qwen3-1.7B	1.12	1.77 (+58%)	1.76 (+58%)

Appendix

Table 7: Measured latency: baseline, the rotation alone, and the rotation with 4-bit quantisation (Section 8.1 ). The quantisation step adds little next to the rotation matmul on every model. We time the rotation as a dense

d\times d

matmul, the way every rotation arm in this paper applies it, not as QuaRot’s fused fast-Hadamard-transform kernel, which a production deployment would use and which would cost less.

Figure 3: Left: Table 6 , QuaRot’s Hadamard construction against the diagnostic dense rotation, median induction over four models at three depths. Right: Table 7 , measured forward-pass latency added by the rotation, with and without the 4-bit quantisation step, on the same four models.

Perturbation	8 bits	6 bits	4 bits	3 bits	2 bits
Uniform	0.999	0.997	0.940	0.007	0.000
Gaussian noise, matched	1.000	0.999	0.981	0.878	0.671
Rotated	1.000	1.000	1.005	0.993	0.251

Appendix

Table 8: Qwen3-8B (instruct), full bitwidth sweep, matching the schema of Table 22 .

Perturbation	8 bits	6 bits	4 bits	3 bits	2 bits
Uniform	1.000	0.999	1.000	0.983	0.086
Gaussian noise, matched	1.000	1.000	0.998	0.993	0.868
Rotated	1.000	1.000	1.001	0.996	0.219

Appendix

Table 9: Qwen2.5-32B, full bitwidth sweep, matching the schema of Table 22 . The collapse the four-model set shows at three to four bits does not appear here until two bits, where the same quantisation/noise/rotation ordering returns.

Figure 4: Tables 8 and 9 plotted on the perplexity axis instead of the bitwidth axis, all five bitwidths for both arms and both models. Rotation Pareto-dominates uniform quantisation throughout: at every perplexity the dense rotation reaches, it retains induction the uniform arm does not reach until a substantially higher perplexity.

Model	Context	Induction	At-distance	Retrieval
gemma-2-2b	128	0.644 $\pm$ 0.009	0.612 $\pm$ 0.005	0.265 $\pm$ 0.059
gemma-2-2b	512	0.847 $\pm$ 0.003	0.877 $\pm$ 0.002	0.520 $\pm$ 0.144
gemma-2-2b	1024	0.912 $\pm$ 0.001	0.924 $\pm$ 0.001	0.424 $\pm$ 0.157
Qwen3-8B-Base	128	0.974 $\pm$ 0.002	0.966 $\pm$ 0.002	0.967 $\pm$ 0.028
Qwen3-8B-Base	512	0.974 $\pm$ 0.001	0.965 $\pm$ 0.001	0.546 $\pm$ 0.098
Qwen3-8B-Base	1024	0.959 $\pm$ 0.001	0.942 $\pm$ 0.001	0.474 $\pm$ 0.188

Appendix

Table 10: Retained accuracy against context length, uniform per-token quantisation at 4 bits, single-block setting, mean

\pm

standard deviation over 5 seeds, median over 3 relative depths. context_length scales span, filler, and retrieval-pair counts proportionally from their 128-token defaults, so each column rescales the whole regime rather than stretching one probe over a longer distance.

Figure 5: Point estimate and 95% CI, cluster bootstrap over models, for every row of the six tables above. Most intervals are tight relative to the effect sizes the main text reports; the widest interval in the set is at the transition this paper’s central claim depends on: quantised induction at 3 bits (top left panel), the regime in which a point estimate is least trustworthy and the one every number in Section 4.1 comes from. The ordering between arms holds in every resample even where the absolute level does not.

Substitute for the error	8 bits	6 bits	4 bits	3 bits	2 bits
Gaussian, second moment matched	0.631	0.632	0.173	0.020	0.001
Signs randomised, magnitudes exact	0.651	0.646	0.164	0.017	0.001
Errors moved to other tokens	0.473	0.730	0.201	0.034	0.004
Quantised in a rotated basis	0.064	0.188	0.047	0.011	0.033

Appendix

Table 11: Share of quantisation’s excess perplexity reproduced by each substitute, under per-token scaling. Median over four families at three relative depths.

Figure 6: Capability retained as the fraction of resampled signs rises from 0 to 100 percent (solid lines, markers), against the matched-Gaussian control at the same bitwidths and models (dashed). Both probes recover smoothly and monotonically, meeting the Gaussian reference almost exactly at 100 percent.

Signs randomised	Induction	Induction (Gaussian)	Retrieval	Retrieval (Gaussian)
0%	0.731	0.946	0.266	0.412
25%	0.846	0.946	0.288	0.412
50%	0.889	0.946	0.298	0.412
75%	0.931	0.946	0.313	0.412
100%	0.949	0.946	0.400	0.412

Appendix

Table 12: Capability retained as the fraction of resampled signs rises from 0 (the original quantisation error) to 100 percent (full sign randomisation), against the matched-Gaussian reference at the same fixed model, layer, and bitwidth (constant across the dial by construction, since only the quantisation arm’s sign fraction varies); this single-setting reference is not the pooled Gaussian-noise median of Table 22 , which aggregates over every family and bitwidth.

Model	$d_{\mathrm{model}}$	Width	Expansion	Active	FVU
pythia-70m	512	4096	8	64	0.143
pythia-160m	768	6144	8	64	0.165
pythia-410m	1024	8192	8	64	0.174
pythia-1.4b	2048	16384	8	64	0.222
pythia-2.8b	2560	20480	8	64	0.243
pythia-6.9b	4096	32768	8	64	0.264

Appendix

Table 13: The dictionaries trained for this study, one row per Pythia model, median over the depths measured. Width is held at eight times the residual width and sparsity at 64 active features, so the comparison across scale is not also a comparison across dictionary capacity. Reconstruction error grows with model size at fixed expansion.

Figure 7: Left: difference in CKA score between the two arms against how much more damage quantisation does. Points above the dashed line are settings where the more damaging arm scores as the more similar one (Section 5.1 ). Right: share of damage falling on the most active one percent of features, against the share of activation those features carry. Quantisation sits on the diagonal, matched noise below it.

Share of damage on the top	1%	3%	5%	10%
Quantisation	5.6	11.5	15.7	25.6
Matched noise	3.2	7.1	10.5	17.6
Activation reference	5.7	11.6	16.7	26.1

Appendix

Table 14: Share of total damage carried by the most affected features, against the share of activation those features carry (Section 5.1 ). Quantisation concentrates roughly 5.8 percent of its damage in the top one percent, close to the activation share of 5.7 percent; noise of identical size concentrates far less. Median over four families.

Bits	Quantisation	Matched noise
8	+0.0005	+0.0007
6	+0.0009	+0.0052
4	-0.0231	+0.0219
3	-0.0397	+0.0242
2	-0.0450	+0.0250

Appendix

Table 15: Signed shift of feature activations, as a fraction of their full-precision scale (Section 5.2 ). Quantisation moves features down and noise of identical per-channel size moves them up. The split appears at the same bitwidth as the perplexity gap. Median over four families.

	Restore			Zero
Features	Most damaged	Random	Least damaged	Most damaged
8	0.940	0.998	1.000	1.30
16	0.941	0.996	1.000	1.50
32	0.870	0.991	1.000	1.66
64	0.737	0.981	1.000	1.72
128	0.567	0.964	0.999	1.73

Appendix

Table 16: Causal interventions (Section 5.2 ), perplexity relative to the quantised model. Values below one are a recovery, above one a further loss. Restoring the most damaged features dominates restoring as many random or least damaged ones at every size, and zeroing them is harmful throughout. Median over four families.

Figure 8: Table 16 plotted across every intervention size measured: perplexity relative to the quantised model as the number of intervened features grows, for restoration of the most damaged, random, and least damaged sets, and for zeroing the most damaged set.

Scale shared over	8 bits	6 bits	4 bits	3 bits	2 bits
One channel	1.00	1.00	1.00	1.01	16.87
One token	1.00	1.00	2.25	37.36	299.92
Whole tensor	31.27	230.54	655.59	856.30	728.48

Appendix

Table 17: Structure premium: quantised perplexity divided by the perplexity of Gaussian noise of identical per-channel mean squared error. A value of one means the error behaves exactly like noise of its size. Median over 12 models and three relative depths.

Model	8 bits	6 bits	4 bits	3 bits	2 bits
Pythia 70M	1.00	1.00	1.01	6.60	30.5
GPT-2 124M	1.01	1.01	1.19	18.9	93.9
Pythia 160M	1.01	1.03	1.44	6.48	20.4
Pythia 410M	1.01	0.99	1.18	29.7	97.5
Qwen3 1.7B	0.99	1.00	1.72	21.6	771.3
Pythia 1.4B	1.02	1.07	51.6	153.9	297.7

Appendix

Table 18: Structure premium per model under per-token scaling, the granularity deployed quantisers use. Median over three relative depths. Every model is indistinguishable from noise at eight and six bits. Where it departs below that varies with both size and architecture.

Figure 9: Structure premium against bitwidth (Table 17 , main text). A value of one, marked by the dashed line, means quantisation and matched noise do equal harm. Curves separate by how widely the quantiser shares a scale, not by bitwidth.

	Sensitivity vs magnitude			Top 1% Fisher, over uniform
Family	Median corr.	Range	Overlap	Shallow	Middle	Deep
GPT-2	-0.040	-0.61 to +0.18	0.320	20.5	18.0	4.6
Gemma-2	+0.100	-0.49 to +0.68	0.328	2.1	3.4	5.5
Pythia	-0.115	-0.46 to +0.38	0.203	1.5	2.6	10.6
Qwen3	-0.196	-0.84 to +0.48	0.359	3.2	3.1	6.2

Appendix

Table 19: Left: loss sensitivity against activation magnitude, by family. Rank correlation is Spearman’s over residual coordinates, and overlap is the share of members common to the 64 most sensitive coordinates and the 64 largest by mean absolute activation. Right: sensitivity mass in the most sensitive one percent of coordinates, as a multiple of what a flat spectrum would place there, at relative depth below 0.2, between 0.4 and 0.6, and above 0.8.

Figure 10: Left: rank correlation between sensitivity and activation magnitude against relative depth. Right: sensitivity mass in the most sensitive one percent of coordinates, as a multiple of a flat spectrum. Only the models swept at every depth are drawn.

	Per tensor			Per token
Average bits	Uniform	Protected	Ratio	Ratio
2	75,912	12,923	5.87	6.27
3	87,789	9,077	9.67	1.90
4	56,719	2,478	22.89	0.16
6	10,541	504	20.91	0.37
8	2,608	294	8.88	0.07

Appendix

Table 20: Precision that protects the most sensitive directions, against uniform precision at the same average bitwidth. Perplexities are for per-tensor scaling; the final column gives the same ratio under per-token scaling. Median over 12 models and three relative depths. Protection is worth an order of magnitude where the error is organised and is a liability where it is already close to noise.

Figure 11: Perplexity against average bitwidth under uniform precision and under precision protecting the most sensitive directions.

Average bits	Fisher-selected	Random	Uniform reference
3.00	0.007	0.001	0.001 at 3
4.00	0.109	0.098	0.681 at 4
5.00	0.643	0.591	0.998 at 6
6.00	0.916	0.893	0.998 at 6

Appendix

Table 21: Induction retained when a subspace is held at eight bits and the remainder at two, against uniform precision costing no less on average. Fisher selects the most sensitive directions; random selects as many at random. Median over four families.

Perturbation	8 bits	6 bits	4 bits	3 bits	2 bits
Quantised, one scale per token	1.000	0.998	0.681	0.001	0.000
Gaussian noise, matched	1.000	0.998	0.936	0.681	0.535
Signs randomised	1.000	0.998	0.943	0.695	0.493
Quantised, one scale per tensor	0.000	0.000	0.000	0.000	0.000
Gaussian noise, matched	0.684	0.500	0.450	0.453	0.443

Appendix

Table 22: Fraction of induction accuracy retained under each perturbation, median over four families at three relative depths. The intact models score between 0.954 and 0.987, and the chance floor is zero. Rows within a block carry the same per-channel error magnitude as the quantised row above them.

Figure 12: Table 3 (main text) on a perplexity axis. Subspace-selection (Protected) tracks uniform quantisation and then collapses with it; rotation (Rotated) reaches a materially better tradeoff at every average bitwidth from four down to two.

Perturbation	8 bits	6 bits	4 bits	3 bits	2 bits
Rotated basis, one scale per token	1.000	1.000	0.997	0.980	0.051
Rotated basis, one scale per tensor	0.997	0.680	0.000	0.000	0.000

Appendix

Table 23: Induction retained when the same quantiser is applied in a randomly rotated frame. Compare the unrotated rows of Table 22 . Median over four families at three relative depths.

Family	Settings	Median	Range	Median	Range
		Induction		Retrieval
pythia-1.4b	5	0.943	0.899 to 0.969	0.681	0.464 to 0.870
Qwen3-1.7B-Base	3	0.984	0.480 to 0.985	0.720	0.034 to 1.025
gemma-2-2b	5	0.960	0.711 to 0.987	0.551	0.374 to 1.000
gpt2	9	0.969	0.842 to 0.993	0.372	0.230 to 0.655

Appendix

Table 24: The headline band of Table 2 , perplexity between 1.2 and 1.5 times intact, resolved by family. Ranges are over the settings in the band, which differ in depth, bitwidth, arm and whether one block or every block is quantised, so a range is the spread of a heterogeneous set rather than an interval on a repeated measurement. Every family has retrieval below induction. Qwen3-1.7B-Base’s retrieval range reaches above 1.0 (retention exceeding the intact model); at 256 positions this reflects probe measurement noise, not a real gain from quantisation.

Precision	Average bits	Perplexity	Induction	Retrieval
Uniform	2.00	199,138.2	0.000	0.000
Uniform	3.00	453,736.7	0.000	0.000
Uniform	4.00	20,442.3	0.003	0.000
Uniform	6.00	38.4	0.905	0.597
Uniform	8.00	25.1	0.999	0.965
Matched noise	2.00	7,209,360.4	0.000	0.000

Appendix

Table 25: Every block from the intervened one onward quantised, at every average bitwidth measured, median over four families. Full precision is 21.6. This resolves Section 8 , which reads the four and six bit rows.

	Intact accuracy		Induction retained
Model	Induction	Retrieval	Quant 4b	Quant 3b	Rotated 3b
Qwen3-1.7B-Base	0.976	0.461	0.745	0.001	0.992
gemma-2-2b	0.983	0.418	0.651	0.001	0.988
gpt2	0.987	0.441	0.926	0.042	0.864
pythia-1.4b	0.954	0.270	0.014	0.000	0.932

Appendix

Table 26: Per model rather than pooled, so that heterogeneity between families is visible. Intact accuracy is the unquantised model. The last three columns are the fraction of induction retained. Pythia-1.4B is the outlier: it has already lost induction at four bits where the others retain most of it, and it is the model whose median pulls the pooled four bit figure down.

Activation Quantization

Activation quantization reduces the precision of neural-network activations to lower inference and training memory use and accelerate computation.

See all 120 papers

Perplexity Can Miss SAE Feature Damage Under Quantization
Jun 2, 2026Evan DuanLLM Quantization LLM Interpretability
University of Michigan
The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
Jul 9, 2026Baha Rababah, Cuneyt Gurcan Akcora, Carson K. LeungLLM Evaluation Quantization Sensitivity Analysis
Department of Computer Science, University of Manitoba, Canada · AI Initiative, University of Central Florida, USA
The Undetected Damage of Quantization on Retrieval and How to Fix It
Sep 21, 2026Luca Zhou, Alessandro Zirilli, Daniele Solombrino +2Mixed-Precision Quantization Information Retrieval
Sapienza University of Rome · Not Diamond · Paradigma

Jun 2, 2026J/K move · Enter open · S save

Perplexity Can Miss SAE Feature Damage Under Quantization

Evan Duan

University of Michigan

LLM Quantization LLM Interpretability Sparse Autoencoders Post-Training Quantization

Open paper AskPDF arXiv

Quantization is a standard path to deploying large language models, and quantized models are typically judged acceptable when perplexity or downstream accuracy remains close to the full-precision original. But behavioral parity need not imply feature fidelity: the sparse-autoencoder (SAE) features used to interpret a full-precision model may change after weight rounding. We test this directly by using a frozen SAE as a fixed measurement basis, encoding full-precision and round-to-nearest (RTN) quantized activations on identical tokens, and measuring per-feature survival by Pearson correlation across bit-widths from INT8 to INT4 on Pythia-70M and Gemma-2-2B. Our central finding is that perplexity can miss feature damage: on Gemma-2-2B, INT7 improves perplexity while degrading 18.7% of active SAE features, and under sliding-window evaluation INT6 also improves perplexity while only 51.3% of active features survive. Feature survival is graded rather than cliff-like, with 62.4% of active Pythia features and 51.3% of active Gemma features surviving at INT6; most non-surviving features are blurred rather than fully damaged. Survival is also predictable from full-precision feature statistics alone, with cross-validated AUC 0.92--0.97 and peak activation as the strongest marginal predictor. Finally, RTN quantization and matched-perplexity magnitude pruning damage strongly overlapping feature sets, with Jaccard overlap 0.79--0.86 and damage-score Spearman correlation 0.98. These results show that behavioral metrics alone are insufficient evidence that full-precision interpretability findings transfer to quantized models, motivating feature-level audits of compression.

Abstract

Activation Quantization

Explore similar work

Perplexity Can Miss SAE Feature Damage Under Quantization

The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs

The Undetected Damage of Quantization on Retrieval and How to Fix It