Organizations: Indraprastha Institute of Information Technology, Delhi · TurboML · Independent Researcher · Microsoft · Nanyang Technological University · Indian Institute of Technology, Kanpur · Singapore Institute of Technology
Quantization and pruning reduce the cost of deploying language models, but their effects are usually reported as benchmark scores. Such scores record only the loss that remains at the output. Inside the model, compression damages some components more than others, and the rest of the network can partly compensate. The observed loss therefore reflects both the damage and this compensation, which benchmark scores cannot separate. How the two combine, and whether their balance changes with model size, remains unclear. We study this compensation for in-context retrieval, a capability carried by a small, identifiable set of attention heads. To separate the two, the expected loss from ablating these heads is decomposed exactly into the part restored by the rest of the network, attributed to the components responsible, and the part that reaches the output. This decomposition is then compared across model sizes. For damage of comparable size, smaller models restore most of the loss, whereas larger models retain most of it, because the compensating responses offset less of it as models grow. Compressing the heads produces the same pattern. Compression that is nearly harmless in small models can therefore be costly in large ones.
Figures & tables
Figure 1: Repair decomposition in GPT-2 XL as the top- k retrieval heads are ablated. The gap between the no-response baseline and the measured output is filled by copy-suppression release, the net backup response, and final normalization. Values are percentages of the clean logit difference.
Figure 2: Repair under 4-bit weight quantization of the top- k retrieval heads in Pythia-410M, using the deliberately poor calibration described in Appendix A . The shaded gap between the no-response baseline and the measured output is total repair, R+B+N . Values are percentages of the clean logit difference.
Figure 3: Repair decomposition at the two ends of the Pythia suite, Pythia-410M (left) and Pythia-12B (right), as the top- k retrieval heads are ablated. The gap between the no-response baseline and the measured output is filled by copy-suppression release, the net backup response, and final normalization, which together account for it to within 10−3 logits. At 410M the response offsets most of the removed contribution; at 12B it offsets a much smaller share. Values are percentages of each model’s clean logit difference.
Figure 4
Figure 6: Mean clean-run answer–distractor logit difference across layer depth, with the intermediate residual normalized separately at each layer, as a percentage of each model’s final-layer value (dashed line at 100%). Several models show an intermediate peak above the final value; the 6.9B and 12B curves reach their maxima at the final layer.
Figure 7: Net attention response after comparable retrieval-head damage. Gray bars show the deficit left after copy-suppression release, D−R , and blue bars show the net attention response. Bar heights are percentages of D ; labels below the models give the percentage of D−R restored. Error bars show 95% bootstrap intervals.
Output harm H
Model
Deficit D
Natural
Backup held clean
Both held clean
410M
0.797
−0.067
0.078
0.728
1B
0.795
−0.226
−0.125
0.815
1.4B
0.756
−0.027
0.354
0.464
2.8B
0.888
0.476
1.316
1.770
6.9B
0.829
0.561
0.675
0.905
Table 1: Output harm after retrieval-head damage with natural responses, selected backup heads held at their clean outputs, or both backup and reference copy-suppression heads held clean. Values are means in logits; negative H indicates an answer margin above its clean value. D gives the natural-condition deficit; at 1.4B, both prevention conditions reduce it to 0.707 logits.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Acc.
Logit diff
∣R∣
∣CS∣
410M
0.99
4.31
98
2
1B
1.00
5.05
27
2
1.4B
1.00
4.21
105
2
2.8B
1.00
4.30
283
2
6.9B
1.00
3.95
266
2
12B
1.00
4.32
389
2
Appendix
Table 2: Clean-run statistics per model on the exploratory prompts: pairwise IOI accuracy, clean logit difference, and the sizes of the retrieval ( R ) and reference copy-suppression ( CS ) sets.
Figure 8: Repair decomposition under retrieval-head ablation in Pythia-410M and Pythia-12B (top row), and OPT-125M and OPT-6.7B (bottom row). The smaller model is on the left in each row. The dashed curve is the no-response baseline, mclean−D ; the solid curve is the measured output margin. Their difference is total repair, decomposed into copy-suppression release, the backup response, and final normalization. Values are percentages of each model’s clean logit difference. Both OPT models retain substantial repair across the displayed sweeps. The panels have different ranges of k and do not match damage across models.
Figure 9: Repair decomposition in Llama 3.1 8B as the top- k retrieval heads are ablated. The gap between the no-response baseline and the measured output is filled by copy-suppression release, the net backup response, and final normalization. Values are percentages of the clean logit difference.
Figure 10: Response to the first ten retrieval-head ablations in Pythia-410M. Top: surviving retrieval contribution. Middle: net and summed positive backup responses. Bottom: copy-suppression strength after full ablation and a path cut; zero indicates complete release. Values are percentages of the clean logit difference, using the clean final-normalization scale.
Figure 11: Mean change in direct contribution per selected head across Pythia, in logits under clean final normalization. Backup-head responses are positive under full retrieval-head ablation and near-zero or slightly negative when their direct retrieval input is removed (edge cut). Copy-suppression heads increase their contribution under the cut. Removing the direct input alone therefore does not reproduce the backup response to full ablation.
Figure 12: Deficit remaining after copy-suppression release at D=0.5 logits in OPT-125M, 1.3B, 2.7B, and 6.7B. The dashed line marks the initial deficit; the dark blue curve shows the remainder D−R . Shading indicates release, with a remainder above the dashed line indicating negative release. The light blue curve shows output harm plus the backup response, H+B=D−R−N . Both curves are percentages of D ; neither is final output harm alone. The S annotations give standing suppression as a percentage of each model’s clean logit difference.
Figure 13: Total repair at three discovery-calibrated ablation strengths. Each point reports the achieved held-out deficit D ; error bars are 95% paired template-cluster bootstrap intervals for total repair D−H . Crosses mark conditions outside the ±10% tolerance around the discovery target and are not connected as a matched-damage series. The middle target supplies the primary response-channel and response-prevention comparisons in the main text.
Figure 14: Full response-channel decomposition at the primary ablation condition used in Figure 7 . Left: copy-suppression release and the net attention response. Centre: MLP and final normalization responses. Right: fractional repair, (D−H)/D . All values are percentages of the deficit D . Error bars are 95% paired template-cluster bootstrap intervals over the held-out prompts. The attention and MLP responses together form the broader backup term B . Achieved deficits, including the 1.4B condition outside the discovery tolerance, are reported in Table 3 .
Figure 15: Extra output harm when selected responses to retrieval-head damage are prevented. Ablations produce deficits of about 0.8 logits on 2,000 held-out prompts, as in Figure 7 . Selected heads are held at their clean output at the answer position while subsequent computation remains live. Extra harm is the increase in H relative to the natural response. Backup receivers are selected to cover 90% of the positive attention response on discovery prompts; low-response heads provide a control. The functional-CS curve uses a separate head selection from the paper-reference CS set. Error bars are 95% paired template-cluster bootstrap intervals. The 1.4B backup and joint clamps also change the removed contribution by about −0.05 logits; backup receiver counts and condition-specific deficits are given in Appendix Table 3 .
Figure 16: Absolute output harm under natural response and response prevention at 410M, 1.4B, and 12B. Panels show the natural response, CS held clean, backup receivers held clean, and both held clean. Harm is shown as a percentage of the deficit D ; the dashed line marks H=D . Natural deficits are approximately 0.80, 0.76, and 0.85 logits, respectively. At 1.4B, the backup and joint clamps reduce the measured deficit to about 0.71 logits. Error bars are 95% paired template-cluster bootstrap intervals. Table 1 gives natural, backup, and joint-condition harm across all six models.
Figure 17: Total repair in the original ten-head ablation comparison: copy-suppression release, backup response, and normalization summed. Repair is shown as a percentage of each model’s deficit, which differs across model sizes. It falls from about 80% at 410M to about 29% at 12B in this exploratory evaluation.
Figure 18: Backup response in the original ten-head ablation comparison on exploratory prompts. Values are in logits. The removed contribution varies from approximately 1.3 to 3.7 logits across models. This fixed-head-count view is separate from the held-out comparable-damage evaluation in Figure 7 .
Figure 19: Repair under structural pruning of half the eligible heads across the Pythia suite. Red shows unlucky masks with high overlap with the top ten retrieval heads; gray shows random comparison masks. Lines are means over three seeds, and shaded bands span their ranges. Repair is D−H , the measured output minus the no-response baseline; negative values indicate harm beyond the removed direct contribution. Unlucky-mask repair decreases while remaining positive, whereas random-mask repair is non-monotonic.
Figure 20: Output harm as a fraction of the achieved deficit under 4-bit retrieval-head weight quantization in Pythia (left) and OPT (right). For each model, the number of quantized heads is chosen to give a deficit closest to 0.5 logits; Pythia-1.4B and 2.8B use their largest-deficit settings because they do not reach this target. Bars and labels show H/D , and the dashed line marks no net repair, H=D . The selected settings need not have identical deficits.
Figure 21: Co-development of intermediate excess and standing suppression on clean runs across saved training checkpoints. Red circles, left axis: the maximum of the mean answer–distractor logit-difference trajectory minus that checkpoint’s own final-layer value, using the layer-renormalized logit lens. Blue squares, right axis: standing suppression of the copy-suppression heads identified at that checkpoint. Both quantities are percentages of the final model’s clean logit difference. Shading gives 95% paired template-cluster bootstrap intervals for suppression; the excess is a mean-curve statistic without an interval. Hollow markers indicate pairwise IOI accuracy below 0.6. Axis scales are shared across models but differ between the two quantities. Only the four models with usable intermediate checkpoint histories are shown.
Figure 22: Revision from the intermediate peak to the final value of the mean layer-renormalized answer–distractor trajectory in four Pythia models. Bars show component writes under fixed clean-final normalization and the change in the difference between readout conventions; the latter is distinct from the repair term N . Diamonds show observed revision with 95% paired template-cluster bootstrap intervals, reselecting the peak in each replicate. Values are percentages of each model’s final answer–distractor logit difference. Pythia-6.9B and 12B are omitted because their mean trajectories peak at the final layer.
Figure 23: Mean clean-run answer scores with normalization recomputed at each layer, as a percentage of each model’s final score. The score is the correct-answer logit minus the mean logit of all other vocabulary entries, including special and padded entries. Dashed lines mark the final value (100%); shading gives 95% paired template-cluster bootstrap intervals. These mean trajectories peak at the final layer; individual prompts may still exhibit intermediate declines.
Deploying small language models on resource-constrained hardware requires compression, yet practitioners lack systematic guidance on how quantization, pruning, and post-compression recovery interact when combined. We present a factorial ablation study evaluating 72 compression configurations across four SLM architectures (2.7B-7B parameters), extended with a 48-configuration ordering analysis, benchmark accuracy validation on 30 compressed models, and robustness checks including multi-seed validation and full-corpus perplexity evaluation. INT8 quantization is effectively lossless (<0.5% accuracy degradation), while NF4 quantization combined with L1 magnitude pruning causes model collapse in all 16 tested combinations. LoRA-based fine-tuning reduces degraded perplexity by up to 97% after aggressive pruning; an ablation against standard fine-tuning without a teacher model shows the recovery is driven by LoRA adaptation itself rather than the distillation signal. Compression ordering shows no significant effect among viable configurations (ANOVA p = 0.96) but determines viability for NF4+pruning combinations. INT8 quantization alone provides 40-45% memory reduction with no quality loss. On-device CPU deployment via llama.cpp confirms these findings transfer off the GPU: 4-bit quantization runs 46-54% faster than 8-bit for a perplexity difference under 0.5 points, while un-structured pruning yields no CPU latency or memory benefit, consistent with its dense storage format.
Model compression makes large language models deployable, but today's evaluation tells an incomplete story. Standard practice measures retained task performance and stops there. This misses a critical fact: compression perturbs the distributed knowledge embedded across billions of weights, and task scores alone cannot reveal which underlying abilities survive and which quietly collapse. To solve this question, we introduce ACE-Compression, an ability-centric evaluation framework. Instead of asking only "does the compressed model still answer correctly?", we ask "which of its core abilities remain intact?" Our framework probes four deployment-facing dimensions: procedural reasoning and constraint execution, knowledge grounding and truthfulness, social cognition and affective understanding, and persona fidelity and stylistic consistency. Across more than 35 compressed checkpoints from six open-weight LLMs-covering quantization and pruning-we find that compression redistributes ability profiles. Coarse, local, and surface-level signals often persist, while fine-grained, long-tail, and context-dependent abilities become fragile. Quantization consistently outperforms pruning in retention, but the practical meaning of that gap depends entirely on which ability an application requires. Based on our findings, deployers can choose compression methods by the ability profile their application needs, not just by a leaderboard average. And algorithm designers gain diagnostic signals to build compression methods that preserve what matters. We believe that ACE-Compression turns compression from a black-box accuracy drop into an interpretable intervention-one we can measure, understand, and ultimately improve.
Jiahui Ning, Liang Chen, Yuchen Liang +2
City University of Hong Kong · Tongji University · Rensselaer Polytechnic Institute +1
Compression can preserve a language model's average accuracy while selectively degrading individual capabilities. Whether these losses follow method-specific patterns or a transferable order is unclear. We applied seven perturbation and compression methods to eight instruction-tuned models, scoring 16 tasks in four answer formats against preregistered predictions. Short-answer capabilities failed in nearly the same order under four random perturbations with different mechanisms (median correlation 0.90) and, within each model, under pruning and quantization (0.875). Intact answer margin, answer frequency and answer format predicted this order. Fragility profiles measured under random perturbation predicted which tasks and facts compression removes first and, after four-task calibration, held-out task accuracy (median absolute error 0.11-0.12, against 0.20-0.21 for uniform decline); profiles borrowed from other model families did almost as well. The order did not transfer across models under GPTQ quantization or hold for long-form tasks. Within short answers, what compression removes is predictable before deployment.
Zhixiang Wang, Ulas Bagci
Department of Radiology, Feinberg School of Medicine, Northwestern University, Chicago, IL, USA