Compression can preserve a language model's average accuracy while selectively degrading individual capabilities. Whether these losses follow method-specific patterns or a transferable order is unclear. We applied seven perturbation and compression methods to eight instruction-tuned models, scoring 16 tasks in four answer formats against preregistered predictions. Short-answer capabilities failed in nearly the same order under four random perturbations with different mechanisms (median correlation 0.90) and, within each model, under pruning and quantization (0.875). Intact answer margin, answer frequency and answer format predicted this order. Fragility profiles measured under random perturbation predicted which tasks and facts compression removes first and, after four-task calibration, held-out task accuracy (median absolute error 0.11-0.12, against 0.20-0.21 for uniform decline); profiles borrowed from other model families did almost as well. The order did not transfer across models under GPTQ quantization or hold for long-form tasks. Within short answers, what compression removes is predictable before deployment.
Figures & tables
Figure 1: Study design and main findings. a , Seven damages at rising dose: four random damages (Gaussian noise, weight zeroing, neuron ablation, layer removal) and three compression methods (Wanda, SparseGPT, GPTQ), in eight models; each of 16 tasks is asked in four formats ( choose , answer , json , twice ). Top, the answer of Qwen2.5-7B to one item as the noise dose rises. b , Every damage removes the same capabilities first, each at its own dose. c , Three separable axes: how securely the task is held, how familiar the answer is and how much of it must be produced. Because the effects add, the failure front is a plane, and more damage moves it towards the robust corner; the fact axis was measured within recall tasks only. d , Measured once under random damage, in the model itself or in other models, the profile predicts what compression removes first and how much accuracy it leaves. Panels b–d are schematics.
Figure 2: Damage sets the rate, not the order. a , Accuracy left in 16 tasks ( answer , chance-corrected) at the dose that halves each model’s overall ability, mean of eight models, for each damage; tasks from most robust to most fragile; grey band, range over damages; ρ , Spearman correlation with the mean order. b , Relative score of every measure of Qwen2.5-7B against nmh(d) from one joint fit for all damages, with e−x ( R2=0.78 ; eight models 0.72–0.82; exploratory). c , As in a , averaged over damages, for four sizes of Qwen2.5; band, range over sizes; ρ against the mean over sizes.
Figure 3: The three axes add. a , Fragility (fold change of the exponent, averaged over models and random damages) on three faces of the task × fact × format space: (1) task × format, 16 tasks; (2) task × fact, quarters of answer frequency; (3) fact × format, capitals and SciQ. † , measurable in fewer than four models. b , How far each axis moves a capability: share of ability kept over all damage, first against last quarter of each axis; numbers, median gap over eight models; dots, models (exploratory). c , Task plus format effects against observed log exponents (Qwen2.5-7B, noise, R2=0.95 ); right, R2 in all 26 random-damage cells (median 0.89). d , Tasks the intact model holds securely fail later (Spearman 0.58–0.73 in every model). e , Partial correlation of capital survival with answer frequency (country held constant) and with country frequency (capital held constant). f , Items a damaged model fails in twice but answers in answer , and the reverse.
Level
Predicted
From
Result
Against
Cells
1
Order of capabilities under a held-out random damage
task and format effects of the other three random damages
r=0.851
task effects alone 0.597; format effects alone 0.512
better in 20 of 20
2
Order of capabilities under Wanda, SparseGPT and GPTQ
task and format effects of the four random damages
r=0.807
task effects alone 0.525; format effects alone 0.575
better in 18 of 19
3
Which facts compression removes (capitals, SciQ)
survival of each fact under the four random damages
Table 1: Five levels of prediction, from the order of capabilities to the accuracy they keep. Levels 1–3 predict an order and use no data from the predicted damage. Levels 4 and 5 predict accuracy and take one strength per compression condition from four fixed calibration tasks of the compressed model; every measure of the other twelve tasks is predicted with all its formats held out. Values are medians over cells (model and damage, or model, task and compression method). As a reference for levels 4 and 5, the accuracy of the same measure under two runs of the same random damage at the same dose differs by a median of 0.117 (exploratory).
Figure 4: Measured once, the axes predict unseen damage. a – d , Qwen2.5-7B. a , Log exponents of neuron ablation predicted from the other three random damages. b , Wanda exponents predicted from all four random damages. c , Survival of each capital under Wanda against its survival under random damage. d , Accuracy (fraction correct, not corrected for chance) after Wanda pruning to 63% sparsity, in answer , for the twelve tasks held out from calibration. Bars and the numbers at their base, measured accuracy; markers and the numbers beside them, predictions from the model’s own random-damage profile (circles), from the profile transferred from other model families, without the model’s own profile (diamonds) and from a uniform decline (dashes). The strength of the damage was estimated on the four calibration tasks (WinoGrande, OpenBookQA, PIQA, copying a word); numbers in the key, mean absolute error over the twelve tasks. e , Median error over all models for each compression method, for the uniform decline (grey), the transferred profile (purple) and the model’s own profile (dark); dotted line, the difference between two seeds of the same random damage (exploratory). All predictions computed in preregistered analyses; summary statistics over models in the main text.
Study
Damage
Published ranking
On a common footing
Reverses
Men et al. 7
layer removal
summarization (generated) collapses in the 7B models
on the same items, chain of thought is more robust than direct answers
Shrinks or vanishes
Table 2: Published rankings of tasks by fragility, before and after chance correction or a same-format comparison. Numbers are from our reading of the original papers; values read from figures are approximate. “Corrected” means rescaled so that chance is zero. All 20 audited studies are listed in Supplementary Table 1.
Model
Noise
Zeroing
Neuron
Layers
Wanda
SparseGPT
GPTQ
Joint (separate)
Qwen2.5-1.5B
41, 0.95
41, 0.94
41, 0.94
33, 0.84
39, 0.85
38, 0.78
32, 0.85
0.82 (0.85)
Qwen2.5-7B
54, 0.94
55, 0.92
56, 0.93
52, 0.86
47, 0.90
50, 0.89
43, 0.76
0.78 (0.90)
Qwen2.5-14B
59, 0.92
61, 0.87
61, 0.89
61, 0.74
55, 0.88
55, 0.84
54, 0.70
0.74 (0.87)
Qwen2.5-32B
61, 0.91
62, 0.91
63, 0.91
64, 0.81
58, 0.91
56, 0.90
46, 0.91
0.74 (0.91)
Llama-3.2-3B
44, 0.90
44, 0.88
45, 0.90
41, 0.79
43, 0.84
40, 0.77
37, 0.77
0.72 (0.84)
Mistral-7B
43, 0.93
44, 0.89
45, 0.92
43, 0.77
39, 0.83
38, 0.90
21, 0.93
0.77 (0.90)
Extended Data Table 1: Fit of the rank-one model. For each model and damage, the number of measures fitted (of 64) and the share of the variance of −logcm(d) explained by equation ( 1 ) within the fitting window. Last column, the joint fit of all seven damages with one exponent per measure (exploratory), and in brackets the median over damages of the separate fits. –, not fitted (fewer than 12 measures or fewer than three doses).
Task
Code
Source
Bare answer
k
Items
Add 2
ARITH1
generated
number
4
300
Add 3
ARITH2
generated
number
4
300
Add 4
ARITH3
generated
number
4
300
Capitals
FACT
generated (78 countries)
city name
4
300
Key lookup
RETR
generated
three-digit code
4
300
Copy word
COPY
generated
nonsense word
4
300
Extended Data Table 2: Tasks and calibrated doses. a , The 16 tasks: source, what the bare answer is, number of candidates in choose and number of evaluation items. Chance in answer , json and twice is 1/k for tasks with listed options or yes/no answers and 0 otherwise. b , Calibrated doses: range over models of the weakest and strongest dose, and number of conditions per model in the main analyses (doses × seeds; random damages with three seeds); with controls and extensions, more than 1,100 conditions were evaluated. Noise, standard deviation relative to each matrix; zeroing and neuron ablation, fraction removed; layers, number removed; Wanda and SparseGPT, sparsity; GPTQ, index of the configuration, ordered from 4 bits with groups of 128 to 2 bits per channel.
Extended Data Fig. 1: Agreement in detail. Points are Pearson correlations of task effects between two cells; bars, medians; colours, damages. a , Between two random damages within a model (35 pairs; median 0.904, 95% interval 0.81–0.95; every model ≥0.778 ; preregistered); diamonds, two seeds of one damage. b , Between models under the same damage (preregistered): random damages 0.832–0.913 (69 pairs); Wanda 0.924, SparseGPT 0.927, GPTQ 0.095 (six pairs; the test fails because of GPTQ). Dotted line, threshold. c , Reliability: odd versus even items (0.985), reproduction of an earlier batch (0.933), two seeds (0.883), two damages from single seeds (0.779) and from seed-averaged effects (0.904); right, agreement corrected for seed noise (0.922 and 0.944; exploratory). d , Compression against the same model’s random-damage reference (18 cells, median 0.875; preregistered); right, without the three most robust tasks and the most fragile task (0.727) against shuffled task labels (grey, 1,000 permutations; 95th percentile 0.184). e , Task subsets and mild damage (first three doses; exploratory): within-model (diamonds) and across-model (points) agreement; bars, 95% intervals. f , Item-level order (preregistered, partial): agreement of item survival between two damages against two seeds of one damage, per model; ratio 0.90 (0.84–1.12), both agreements 0.17–0.33.
Extended Data Fig. 2: Long-form tasks do not keep their position across damages (preregistered, failed). GSM8K with chain of thought, code (HumanEval and MBPP, unit tests), IFEval and function calling (BFCL), 200 items each, in four models under Gaussian noise and Wanda pruning. Colours, tasks; shapes, models. a , Percentile position of each task among all measures of its model (0, most robust) under noise ( x ) and Wanda ( y ); grey band, ±25 points; open symbols, pairs outside the band. Ten of 16 pairs (62.5%) lie within the band, short of the required 75%; the median correlation of log exponents between the two damages (0.778) met its criterion. GSM8K became more fragile under Wanda in three models; IFEval stayed between the 9th and 24th percentile everywhere. b , Intact accuracy; dotted line, the 40% usability threshold. c , Exponent of each task under noise and Wanda (log scale; lines join one model). d , Accuracy left at the dose that halves each model’s overall ability, with the mean of five short-answer tasks for reference (exploratory).
Extended Data Fig. 3: The task and format axes in detail. a , Two kinds of fact recall: rank of each task’s intact margin (left; 1, largest) and damage-averaged task effect (right; positive, more fragile) in each model. Capitals (dark circles) have one of the four largest margins and are among the most robust tasks in every model; MMLU knowledge questions (diamonds) have median margins and are more fragile than the average task. b , Fragility of each format relative to choose in every cell (lines; log scale). Medians 1.5, 2.5 and 8.3 under random damage (26 cells; order holds in 25) and 1.4, 2.1 and 6.5 under compression (21 cells; 19). c , Function calling (preregistered; four models, noise and Wanda): share of damage-induced errors that are format failures (68.4%), share of these with the right function and arguments (77.4%), and strict over content-only exponent (2.3). d , How twice answers fail at intermediate damage (exploratory): template kept with wrong parts 0.54, answer present twice but template broken 0.15, answer written once 0.13, no answer 0.12; 83% of twice errors violate the template, 79% of these contain the correct answer. e , Strict over content-only loss for each format under each compression method (exploratory). f , Errors drift towards familiar answers (exploratory): wrong capitals more frequent than the correct one (62–100%); choice errors on the most frequent distractor (48–62%; chance 33%); more common yes/no answer in intact and damaged models.
Extended Data Fig. 4: The fact axis per model: rare answers fail first. Survival, mean correctness over all damaged conditions for items the intact model answered correctly; frequency, log10(count+1) in Dolma. Dots, models; bars, medians. a , Survival of each capital against country–capital co-occurrence in Gemma-2-9B (Spearman 0.50). *, Bulgaria–Sofia and Italy–Rome returned a count of 0 under the default approximation of the index and are drawn left of the axis break; re-queried, they have 541,362 and 896,384 co-occurrences, and the capitals results become stronger (Supplementary Note 4). b , Spearman correlation of frequency with survival for capitals (median 0.42), SciQ (0.38) and the key-lookup control (0.08); filled, answer (preregistered); open, choose (exploratory). c , The same under each damage alone and under the three compression methods pooled (preregistered); grey band, key-lookup control (0.01–0.08). d , Partial correlations holding answer length or the item margin constant (0.26–0.55), and for capitals, capital frequency with the country held constant (0.48–0.70) and the reverse ( −0.22 to 0.18 ).
Extended Data Fig. 5: Prediction of accuracy under compression, every model. a , Predicted against observed accuracy of every measure of the twelve tasks held out from calibration, in every informative compression condition, all models pooled, for each method; the strength of each condition was estimated on four fixed calibration tasks. Top, the model’s own random-damage profile (dark); bottom, the profile transferred from other model families (purple); grey, uniform decline. Numbers, median absolute error over models. b , Error of each model for six predictions under the same calibration: uniform decline, intact accuracy, format effects alone, intact margins, the profile transferred from other model families, and the model’s own profile; bars, medians. Predictions that borrow from other models exclude the target model’s family. c , Median error under three calibration schemes: leaving out one measure at a time, four fixed calibration tasks with whole tasks held out, and the same excluding the target’s family for predictions that borrow from other models. All panels from preregistered analyses.
Extended Data Fig. 6: Confidence and label-free signals. a , Qwen2.5-7B under noise: accuracy and mean confidence over 15 usable choose measures, and the area under the curve (AUC) with which confidence separates right from wrong answers, against dose; small dots, seeds; lines, means (exploratory). b , AUC against accuracy for all damaged (circles) and intact (squares) cells of eight models, in accuracy bands of 0.1; numbers at the top, damaged minus intact; bars, 95% intervals (exploratory). c , Held-out prediction of content loss from three label-free signals, the rise in twice template failures, in perplexity on a probe text and in KL divergence from the intact model, each fitted on seven models and tested on the eighth ( n=547 conditions; R2 0.383, −0.880 and −0.153 ; preregistered, failed).
Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.
Model compression makes large language models deployable, but today's evaluation tells an incomplete story. Standard practice measures retained task performance and stops there. This misses a critical fact: compression perturbs the distributed knowledge embedded across billions of weights, and task scores alone cannot reveal which underlying abilities survive and which quietly collapse. To solve this question, we introduce ACE-Compression, an ability-centric evaluation framework. Instead of asking only "does the compressed model still answer correctly?", we ask "which of its core abilities remain intact?" Our framework probes four deployment-facing dimensions: procedural reasoning and constraint execution, knowledge grounding and truthfulness, social cognition and affective understanding, and persona fidelity and stylistic consistency. Across more than 35 compressed checkpoints from six open-weight LLMs-covering quantization and pruning-we find that compression redistributes ability profiles. Coarse, local, and surface-level signals often persist, while fine-grained, long-tail, and context-dependent abilities become fragile. Quantization consistently outperforms pruning in retention, but the practical meaning of that gap depends entirely on which ability an application requires. Based on our findings, deployers can choose compression methods by the ability profile their application needs, not just by a leaderboard average. And algorithm designers gain diagnostic signals to build compression methods that preserve what matters. We believe that ACE-Compression turns compression from a black-box accuracy drop into an interpretable intervention-one we can measure, understand, and ultimately improve.
Jiahui Ning, Liang Chen, Yuchen Liang +2
City University of Hong Kong · Tongji University · Rensselaer Polytechnic Institute +1
Deploying small language models on resource-constrained hardware requires compression, yet practitioners lack systematic guidance on how quantization, pruning, and post-compression recovery interact when combined. We present a factorial ablation study evaluating 72 compression configurations across four SLM architectures (2.7B-7B parameters), extended with a 48-configuration ordering analysis, benchmark accuracy validation on 30 compressed models, and robustness checks including multi-seed validation and full-corpus perplexity evaluation. INT8 quantization is effectively lossless (<0.5% accuracy degradation), while NF4 quantization combined with L1 magnitude pruning causes model collapse in all 16 tested combinations. LoRA-based fine-tuning reduces degraded perplexity by up to 97% after aggressive pruning; an ablation against standard fine-tuning without a teacher model shows the recovery is driven by LoRA adaptation itself rather than the distillation signal. Compression ordering shows no significant effect among viable configurations (ANOVA p = 0.96) but determines viability for NF4+pruning combinations. INT8 quantization alone provides 40-45% memory reduction with no quality loss. On-device CPU deployment via llama.cpp confirms these findings transfer off the GPU: 4-bit quantization runs 46-54% faster than 8-bit for a perplexity difference under 0.5 points, while un-structured pruning yields no CPU latency or memory benefit, consistent with its dense storage format.
Model compression techniques such as quantization and pruning are widely used to reduce the deployment cost of large language models (LLMs), with existing evaluations focusing almost exclusively on accuracy preservation. However, in safety-critical applications, a model's ability to reliably quantify its own uncertainty is equally important. We ask: does compression preserve this ability? To answer this question, we benchmark 12 LLMs under various compression configurations across five NLP tasks, using conformal prediction to provide a rigorous, distribution-free measure of uncertainty. Our experiments reveal that: (I) compression frequently decouples accuracy from uncertainty; (II) larger models absorb compression-induced uncertainty far more effectively than smaller ones; and (III) uncertainty inflation is often threshold-like rather than gradual. These results suggest that accuracy-only evaluation is insufficient for assessing the deployment readiness of compressed LLMs, and that uncertainty-aware benchmarking should be a standard component of model compression pipelines.
Yujia Tong, Yuxi Wang, Yunyang Wan +3
Wuhan University of Technology · Nanyang Technological University