Activation quantisation is usually evaluated with an aggregate metric, perplexity, averaged over every token a model predicts. We ask whether that average identifies which computations a quantiser damages. Perplexity turns out to be a reliable aggregate signal: across 12 models from four families and 780 within-model comparisons, the arm perplexity prefers also retains more induction and more retrieval in all but 2.1 and 4.0 percent of cases respectively. But where perplexity has risen by only a factor of 1.2 to 1.5, induction still keeps 0.959 of its intact accuracy while retrieval has already fallen to 0.554, a gap the aggregate number does not surface. This gap has structure, not just size: a matched Gaussian-noise control of the same per-channel magnitude leaves it largely intact, and randomising only the sign of the quantisation error, every magnitude held fixed, is nearly as harmless, so magnitude alone does not explain the damage. Quantising in a rotated basis, which changes coordinate alignment without changing error magnitude, restores induction from 0.001 to 0.980 at three average bits per token in a single-block intervention, though retrieval recovers less completely at the same setting (0.694); end-to-end at four average bits, induction reaches 0.968 and retrieval 0.534. The pattern holds on two further models up to 32B parameters and, in the deployed configurations we tested, under AWQ once activations are pushed to 4 bits. A perplexity target bounds the average cost of a transformation applied to the activation; it does not, by itself, show which computations survived.
Figures & tables
Perturbation
Induction
At distance
Retrieval
Quantised, 4 bits
0.681
0.662
0.272
Quantised, 3 bits
0.001
0.001
0.000
Gaussian noise, matched, 3 bits
0.681
0.698
0.219
Signs randomised, 3 bits
0.695
0.677
0.188
Rotated basis, 3 bits
0.980
0.981
0.694
Table 1: Fraction of accuracy retained on each probe, median over model families at three relative depths. Intact accuracy is 0.954–0.987 for the copying probes and lower for retrieval.
Perplexity, relative to intact
Settings
Induction
Retrieval
1 to 1.2
95
0.999
0.991
1.2 to 1.5
22
0.959
0.554
1.5 to 2
26
0.929
0.502
2 to 3
17
0.689
0.246
3 to 5
15
0.377
0.080
5 to 10
19
0.172
0.019
Table 2: Probe accuracy by perplexity band, relative to the unquantised model. Pooled over the three deployable arms, all models, depths, and bitwidths.
Precision
Average bits
Perplexity
Induction
Retrieval
Uniform
2.00
30,429.6
0.000
0.000
Uniform
3.00
1,310.6
0.001
0.000
Uniform
4.00
55.1
0.681
0.272
Uniform
6.00
25.5
0.998
0.960
Uniform
8.00
22.0
1.000
0.986
Protected, Fisher
3.00
324.3
0.007
0.000
Table 3: Perplexity and probe accuracy at matched average bitwidth, median over model families at three relative depths. Full-precision perplexity is 21.6.
Precision
Average bits
Perplexity
Induction
Retrieval
Uniform
4.00
20,442.3
0.003
0.000
Uniform
6.00
38.4
0.905
0.597
Rotated
4.00
35.7
0.968
0.534
Rotated
6.00
22.8
0.999
0.972
Table 4: End-to-end quantisation, matched average bitwidth. Same arms and metrics as Table 3 .
Figure 1: Every arm from Sections 6 – 8.1 on one axis, pythia-1.4b. A weight and an activation quantiser are not on the same bit axis, but are on the same perplexity axis; comparisons across that boundary are not matched-budget (Section 11 ). Deployed quantisers sit between our uniform and rotated arms; QuaRot’s Hadamard rotation reaches the best tradeoff of anything plotted.
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
Arm
Target
Bits
Config
Perplexity
Induction
Retrieval
pythia-1.4b, full precision: perplexity 19.7, induction 0.955, retrieval 0.281
Uniform (activation)
activation
4
W4A4
31,098.5
0.005
0.000
Rotated (activation)
activation
4
W4A4
31.8
0.842
0.191
GPTQ
weight
3
W3A16
22.8
0.916
0.262
GPTQ
weight
4
W4A16
20.1
0.948
0.273
GPTQ
weight
4
W4A4
1,037.7
0.019
0.000
Appendix
Table 5: Full deployed-quantiser sweep. Target separates arms that perturb weights from our own arms that perturb the residual stream; comparisons across that boundary are not matched-budget (§ 11 ). GPTQ’s W4A4 perplexity, two orders of magnitude worse than AWQ’s, follows from its weight-reconstruction calibration objective and is not a claim about its intended weight-only use. pythia-1.4b’s SmoothQuant and AWQ W4A16 rows are independently measured and coincide almost exactly (verified against the per-layer raw output); at this mild, weight-only setting the two methods barely perturb this model differently. The Uniform (activation) and Rotated (activation) rows use the end-to-end variant of Section 8 (every block from the intervened one onward), not the single-block setting the rest of the paper reports by default, since GPTQ/AWQ/SmoothQuant also quantise the whole model; this is why they read far worse than the single-block figures elsewhere.
Figure 2: Standard benchmarks under each deployed-quantiser arm, pythia-1.4b (referenced from Section 6 ). GPTQ’s W4A4 collapse on LAMBADA (a next-token-prediction task that needs the passage’s context, like this paper’s retrieval probe) matches the mechanistic damage; the four multiple-choice commonsense benchmarks move little under any arm, GPTQ W4A4 included.
Per token
Per tensor
Bits
Dense
Hadamard
Dense
Hadamard
8
0.980
0.979
0.967
0.979
6
0.979
0.979
0.649
0.955
4
0.974
0.978
0.000
0.002
3
0.965
0.968
0.000
0.000
2
0.051
0.291
0.000
0.000
Appendix
Table 6: The dense-versus-Hadamard comparison of Section 8.1 , both granularities in full.
Model
Baseline (ms)
+rotation (ms)
+rotation+quantise, 4-bit (ms)
Pythia-1.4b
0.83
1.46 (+75%)
1.45 (+74%)
GPT-2
0.32
0.52 (+65%)
0.52 (+65%)
Gemma-2-2b
1.60
2.43 (+52%)
2.41 (+51%)
Qwen3-1.7B
1.12
1.77 (+58%)
1.76 (+58%)
Appendix
Table 7: Measured latency: baseline, the rotation alone, and the rotation with 4-bit quantisation (Section 8.1 ). The quantisation step adds little next to the rotation matmul on every model. We time the rotation as a dense d×d matmul, the way every rotation arm in this paper applies it, not as QuaRot’s fused fast-Hadamard-transform kernel, which a production deployment would use and which would cost less.
Figure 3: Left: Table 6 , QuaRot’s Hadamard construction against the diagnostic dense rotation, median induction over four models at three depths. Right: Table 7 , measured forward-pass latency added by the rotation, with and without the 4-bit quantisation step, on the same four models.
Perturbation
8 bits
6 bits
4 bits
3 bits
2 bits
Uniform
0.999
0.997
0.940
0.007
0.000
Gaussian noise, matched
1.000
0.999
0.981
0.878
0.671
Rotated
1.000
1.000
1.005
0.993
0.251
Appendix
Table 8: Qwen3-8B (instruct), full bitwidth sweep, matching the schema of Table 22 .
Perturbation
8 bits
6 bits
4 bits
3 bits
2 bits
Uniform
1.000
0.999
1.000
0.983
0.086
Gaussian noise, matched
1.000
1.000
0.998
0.993
0.868
Rotated
1.000
1.000
1.001
0.996
0.219
Appendix
Table 9: Qwen2.5-32B, full bitwidth sweep, matching the schema of Table 22 . The collapse the four-model set shows at three to four bits does not appear here until two bits, where the same quantisation/noise/rotation ordering returns.
Figure 4: Tables 8 and 9 plotted on the perplexity axis instead of the bitwidth axis, all five bitwidths for both arms and both models. Rotation Pareto-dominates uniform quantisation throughout: at every perplexity the dense rotation reaches, it retains induction the uniform arm does not reach until a substantially higher perplexity.
Model
Context
Induction
At-distance
Retrieval
gemma-2-2b
128
0.644 ± 0.009
0.612 ± 0.005
0.265 ± 0.059
gemma-2-2b
512
0.847 ± 0.003
0.877 ± 0.002
0.520 ± 0.144
gemma-2-2b
1024
0.912 ± 0.001
0.924 ± 0.001
0.424 ± 0.157
Qwen3-8B-Base
128
0.974 ± 0.002
0.966 ± 0.002
0.967 ± 0.028
Qwen3-8B-Base
512
0.974 ± 0.001
0.965 ± 0.001
0.546 ± 0.098
Qwen3-8B-Base
1024
0.959 ± 0.001
0.942 ± 0.001
0.474 ± 0.188
Appendix
Table 10: Retained accuracy against context length, uniform per-token quantisation at 4 bits, single-block setting, mean ± standard deviation over 5 seeds, median over 3 relative depths. context_length scales span, filler, and retrieval-pair counts proportionally from their 128-token defaults, so each column rescales the whole regime rather than stretching one probe over a longer distance.
Figure 5: Point estimate and 95% CI, cluster bootstrap over models, for every row of the six tables above. Most intervals are tight relative to the effect sizes the main text reports; the widest interval in the set is at the transition this paper’s central claim depends on: quantised induction at 3 bits (top left panel), the regime in which a point estimate is least trustworthy and the one every number in Section 4.1 comes from. The ordering between arms holds in every resample even where the absolute level does not.
Substitute for the error
8 bits
6 bits
4 bits
3 bits
2 bits
Gaussian, second moment matched
0.631
0.632
0.173
0.020
0.001
Signs randomised, magnitudes exact
0.651
0.646
0.164
0.017
0.001
Errors moved to other tokens
0.473
0.730
0.201
0.034
0.004
Quantised in a rotated basis
0.064
0.188
0.047
0.011
0.033
Appendix
Table 11: Share of quantisation’s excess perplexity reproduced by each substitute, under per-token scaling. Median over four families at three relative depths.
Figure 6: Capability retained as the fraction of resampled signs rises from 0 to 100 percent (solid lines, markers), against the matched-Gaussian control at the same bitwidths and models (dashed). Both probes recover smoothly and monotonically, meeting the Gaussian reference almost exactly at 100 percent.
Signs randomised
Induction
Induction (Gaussian)
Retrieval
Retrieval (Gaussian)
0%
0.731
0.946
0.266
0.412
25%
0.846
0.946
0.288
0.412
50%
0.889
0.946
0.298
0.412
75%
0.931
0.946
0.313
0.412
100%
0.949
0.946
0.400
0.412
Appendix
Table 12: Capability retained as the fraction of resampled signs rises from 0 (the original quantisation error) to 100 percent (full sign randomisation), against the matched-Gaussian reference at the same fixed model, layer, and bitwidth (constant across the dial by construction, since only the quantisation arm’s sign fraction varies); this single-setting reference is not the pooled Gaussian-noise median of Table 22 , which aggregates over every family and bitwidth.
Model
dmodel
Width
Expansion
Active
FVU
pythia-70m
512
4096
8
64
0.143
pythia-160m
768
6144
8
64
0.165
pythia-410m
1024
8192
8
64
0.174
pythia-1.4b
2048
16384
8
64
0.222
pythia-2.8b
2560
20480
8
64
0.243
pythia-6.9b
4096
32768
8
64
0.264
Appendix
Table 13: The dictionaries trained for this study, one row per Pythia model, median over the depths measured. Width is held at eight times the residual width and sparsity at 64 active features, so the comparison across scale is not also a comparison across dictionary capacity. Reconstruction error grows with model size at fixed expansion.
Figure 7: Left: difference in CKA score between the two arms against how much more damage quantisation does. Points above the dashed line are settings where the more damaging arm scores as the more similar one (Section 5.1 ). Right: share of damage falling on the most active one percent of features, against the share of activation those features carry. Quantisation sits on the diagonal, matched noise below it.
Share of damage on the top
1%
3%
5%
10%
Quantisation
5.6
11.5
15.7
25.6
Matched noise
3.2
7.1
10.5
17.6
Activation reference
5.7
11.6
16.7
26.1
Appendix
Table 14: Share of total damage carried by the most affected features, against the share of activation those features carry (Section 5.1 ). Quantisation concentrates roughly 5.8 percent of its damage in the top one percent, close to the activation share of 5.7 percent; noise of identical size concentrates far less. Median over four families.
Bits
Quantisation
Matched noise
8
+0.0005
+0.0007
6
+0.0009
+0.0052
4
-0.0231
+0.0219
3
-0.0397
+0.0242
2
-0.0450
+0.0250
Appendix
Table 15: Signed shift of feature activations, as a fraction of their full-precision scale (Section 5.2 ). Quantisation moves features down and noise of identical per-channel size moves them up. The split appears at the same bitwidth as the perplexity gap. Median over four families.
Restore
Zero
Features
Most damaged
Random
Least damaged
Most damaged
8
0.940
0.998
1.000
1.30
16
0.941
0.996
1.000
1.50
32
0.870
0.991
1.000
1.66
64
0.737
0.981
1.000
1.72
128
0.567
0.964
0.999
1.73
Appendix
Table 16: Causal interventions (Section 5.2 ), perplexity relative to the quantised model. Values below one are a recovery, above one a further loss. Restoring the most damaged features dominates restoring as many random or least damaged ones at every size, and zeroing them is harmful throughout. Median over four families.
Figure 8: Table 16 plotted across every intervention size measured: perplexity relative to the quantised model as the number of intervened features grows, for restoration of the most damaged, random, and least damaged sets, and for zeroing the most damaged set.
Scale shared over
8 bits
6 bits
4 bits
3 bits
2 bits
One channel
1.00
1.00
1.00
1.01
16.87
One token
1.00
1.00
2.25
37.36
299.92
Whole tensor
31.27
230.54
655.59
856.30
728.48
Appendix
Table 17: Structure premium: quantised perplexity divided by the perplexity of Gaussian noise of identical per-channel mean squared error. A value of one means the error behaves exactly like noise of its size. Median over 12 models and three relative depths.
Model
8 bits
6 bits
4 bits
3 bits
2 bits
Pythia 70M
1.00
1.00
1.01
6.60
30.5
GPT-2 124M
1.01
1.01
1.19
18.9
93.9
Pythia 160M
1.01
1.03
1.44
6.48
20.4
Pythia 410M
1.01
0.99
1.18
29.7
97.5
Qwen3 1.7B
0.99
1.00
1.72
21.6
771.3
Pythia 1.4B
1.02
1.07
51.6
153.9
297.7
Appendix
Table 18: Structure premium per model under per-token scaling, the granularity deployed quantisers use. Median over three relative depths. Every model is indistinguishable from noise at eight and six bits. Where it departs below that varies with both size and architecture.
Figure 9: Structure premium against bitwidth (Table 17 , main text). A value of one, marked by the dashed line, means quantisation and matched noise do equal harm. Curves separate by how widely the quantiser shares a scale, not by bitwidth.
Sensitivity vs magnitude
Top 1% Fisher, over uniform
Family
Median corr.
Range
Overlap
Shallow
Middle
Deep
GPT-2
-0.040
-0.61 to +0.18
0.320
20.5
18.0
4.6
Gemma-2
+0.100
-0.49 to +0.68
0.328
2.1
3.4
5.5
Pythia
-0.115
-0.46 to +0.38
0.203
1.5
2.6
10.6
Qwen3
-0.196
-0.84 to +0.48
0.359
3.2
3.1
6.2
Appendix
Table 19: Left: loss sensitivity against activation magnitude, by family. Rank correlation is Spearman’s over residual coordinates, and overlap is the share of members common to the 64 most sensitive coordinates and the 64 largest by mean absolute activation. Right: sensitivity mass in the most sensitive one percent of coordinates, as a multiple of what a flat spectrum would place there, at relative depth below 0.2, between 0.4 and 0.6, and above 0.8.
Figure 10: Left: rank correlation between sensitivity and activation magnitude against relative depth. Right: sensitivity mass in the most sensitive one percent of coordinates, as a multiple of a flat spectrum. Only the models swept at every depth are drawn.
Per tensor
Per token
Average bits
Uniform
Protected
Ratio
Ratio
2
75,912
12,923
5.87
6.27
3
87,789
9,077
9.67
1.90
4
56,719
2,478
22.89
0.16
6
10,541
504
20.91
0.37
8
2,608
294
8.88
0.07
Appendix
Table 20: Precision that protects the most sensitive directions, against uniform precision at the same average bitwidth. Perplexities are for per-tensor scaling; the final column gives the same ratio under per-token scaling. Median over 12 models and three relative depths. Protection is worth an order of magnitude where the error is organised and is a liability where it is already close to noise.
Figure 11: Perplexity against average bitwidth under uniform precision and under precision protecting the most sensitive directions.
Average bits
Fisher-selected
Random
Uniform reference
3.00
0.007
0.001
0.001 at 3
4.00
0.109
0.098
0.681 at 4
5.00
0.643
0.591
0.998 at 6
6.00
0.916
0.893
0.998 at 6
Appendix
Table 21: Induction retained when a subspace is held at eight bits and the remainder at two, against uniform precision costing no less on average. Fisher selects the most sensitive directions; random selects as many at random. Median over four families.
Perturbation
8 bits
6 bits
4 bits
3 bits
2 bits
Quantised, one scale per token
1.000
0.998
0.681
0.001
0.000
Gaussian noise, matched
1.000
0.998
0.936
0.681
0.535
Signs randomised
1.000
0.998
0.943
0.695
0.493
Quantised, one scale per tensor
0.000
0.000
0.000
0.000
0.000
Gaussian noise, matched
0.684
0.500
0.450
0.453
0.443
Appendix
Table 22: Fraction of induction accuracy retained under each perturbation, median over four families at three relative depths. The intact models score between 0.954 and 0.987, and the chance floor is zero. Rows within a block carry the same per-channel error magnitude as the quantised row above them.
Figure 12: Table 3 (main text) on a perplexity axis. Subspace-selection (Protected) tracks uniform quantisation and then collapses with it; rotation (Rotated) reaches a materially better tradeoff at every average bitwidth from four down to two.
Perturbation
8 bits
6 bits
4 bits
3 bits
2 bits
Rotated basis, one scale per token
1.000
1.000
0.997
0.980
0.051
Rotated basis, one scale per tensor
0.997
0.680
0.000
0.000
0.000
Appendix
Table 23: Induction retained when the same quantiser is applied in a randomly rotated frame. Compare the unrotated rows of Table 22 . Median over four families at three relative depths.
Induction
Retrieval
Family
Settings
Median
Range
Median
Range
pythia-1.4b
5
0.943
0.899 to 0.969
0.681
0.464 to 0.870
Qwen3-1.7B-Base
3
0.984
0.480 to 0.985
0.720
0.034 to 1.025
gemma-2-2b
5
0.960
0.711 to 0.987
0.551
0.374 to 1.000
gpt2
9
0.969
0.842 to 0.993
0.372
0.230 to 0.655
Appendix
Table 24: The headline band of Table 2 , perplexity between 1.2 and 1.5 times intact, resolved by family. Ranges are over the settings in the band, which differ in depth, bitwidth, arm and whether one block or every block is quantised, so a range is the spread of a heterogeneous set rather than an interval on a repeated measurement. Every family has retrieval below induction. Qwen3-1.7B-Base’s retrieval range reaches above 1.0 (retention exceeding the intact model); at 256 positions this reflects probe measurement noise, not a real gain from quantisation.
Precision
Average bits
Perplexity
Induction
Retrieval
Uniform
2.00
199,138.2
0.000
0.000
Uniform
3.00
453,736.7
0.000
0.000
Uniform
4.00
20,442.3
0.003
0.000
Uniform
6.00
38.4
0.905
0.597
Uniform
8.00
25.1
0.999
0.965
Matched noise
2.00
7,209,360.4
0.000
0.000
Appendix
Table 25: Every block from the intervened one onward quantised, at every average bitwidth measured, median over four families. Full precision is 21.6. This resolves Section 8 , which reads the four and six bit rows.
Intact accuracy
Induction retained
Model
Induction
Retrieval
Quant 4b
Quant 3b
Rotated 3b
Qwen3-1.7B-Base
0.976
0.461
0.745
0.001
0.992
gemma-2-2b
0.983
0.418
0.651
0.001
0.988
gpt2
0.987
0.441
0.926
0.042
0.864
pythia-1.4b
0.954
0.270
0.014
0.000
0.932
Appendix
Table 26: Per model rather than pooled, so that heterogeneity between families is visible. Intact accuracy is the unquantised model. The last three columns are the fraction of induction retained. Pythia-1.4B is the outlier: it has already lost induction at four bits where the others retain most of it, and it is the model whose median pulls the pooled four bit figure down.
Quantization is a standard path to deploying large language models, and quantized models are typically judged acceptable when perplexity or downstream accuracy remains close to the full-precision original. But behavioral parity need not imply feature fidelity: the sparse-autoencoder (SAE) features used to interpret a full-precision model may change after weight rounding. We test this directly by using a frozen SAE as a fixed measurement basis, encoding full-precision and round-to-nearest (RTN) quantized activations on identical tokens, and measuring per-feature survival by Pearson correlation across bit-widths from INT8 to INT4 on Pythia-70M and Gemma-2-2B. Our central finding is that perplexity can miss feature damage: on Gemma-2-2B, INT7 improves perplexity while degrading 18.7% of active SAE features, and under sliding-window evaluation INT6 also improves perplexity while only 51.3% of active features survive. Feature survival is graded rather than cliff-like, with 62.4% of active Pythia features and 51.3% of active Gemma features surviving at INT6; most non-surviving features are blurred rather than fully damaged. Survival is also predictable from full-precision feature statistics alone, with cross-validated AUC 0.92--0.97 and peak activation as the strongest marginal predictor. Finally, RTN quantization and matched-perplexity magnitude pruning damage strongly overlapping feature sets, with Jaccard overlap 0.79--0.86 and damage-score Spearman correlation 0.98. These results show that behavioral metrics alone are insufficient evidence that full-precision interpretability findings transfer to quantized models, motivating feature-level audits of compression.
Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity. We show that these metrics fail to capture behavioral changes induced by quantization. We introduce correctness agreement, a decision-level metric that measures overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy. Across multiple models and quantization schemes from 8-bit to 2-bit, we find that behavioral divergence emerges under moderate quantization even when task performance appears preserved. To explain this effect, we analyze quantization as a structural operator on attention weights and quantify layer-wise distortions using statistical and distributional measures. Our results reveal non-linear breakpoints at low bit-widths and show that query and key projections are consistently more sensitive than value and output projections. These findings expose an illusion of equivalence between base and quantized models and motivate behavioral evaluation beyond conventional performance metrics.
Baha Rababah, Cuneyt Gurcan Akcora, Carson K. Leung
Department of Computer Science, University of Manitoba, Canada · AI Initiative, University of Central Florida, USA
We show that a quantized model that keeps its classification accuracy still changes 14 to 46% of its top-1 retrieval results, and that aggregate ranking metrics reveal only part of this damage. We tie this failure to the gap between the two highest model scores and use that gap to decide when to trust a quantized answer and where additional precision should be spent. We show that the top-1 result is guaranteed to survive quantization only when this gap exceeds twice the largest rounding error. In classification, the scores are logits, and training compares the correct class against every other class, which encourages this gap. In retrieval, the scores are query-document similarities, and training compares each positive only against sampled negatives, so nothing separates the top-1 item from the second. This gap can be measured without labels. Before deployment, it predicts which models will break under quantization, and at deployment time it tells, per input, whether the quantized answer still matches the full-precision answer. Most classification inputs have a gap wide enough to trust the quantized answer, but few retrieval queries do. That gap motivates a different fix in each task. In retrieval, spending extra bit-width on the layers whose quantization moves the gap most recovers up to three-quarters of an extra bit's benefit for half its cost. In classification, routing the few low-gap inputs to full precision recovers most of the lost accuracy at a fraction of the cost.