ACE-Compression: An Ability-Centric Evaluation of Compressed Large Language Models

Jiahui Ning; Liang Chen; Yuchen Liang; Yi Zhang; Fenglei Fan

cs.AIOct 20, 2026

ACE-Compression: An Ability-Centric Evaluation of Compressed Large Language Models

Authors: Jiahui Ning, Liang Chen, Yuchen Liang, Yi Zhang, Fenglei Fan

Organizations: City University of Hong Kong · Tongji University · Rensselaer Polytechnic Institute · Sichuan University

Abstract

Model compression makes large language models deployable, but today's evaluation tells an incomplete story. Standard practice measures retained task performance and stops there. This misses a critical fact: compression perturbs the distributed knowledge embedded across billions of weights, and task scores alone cannot reveal which underlying abilities survive and which quietly collapse. To solve this question, we introduce ACE-Compression, an ability-centric evaluation framework. Instead of asking only "does the compressed model still answer correctly?", we ask "which of its core abilities remain intact?" Our framework probes four deployment-facing dimensions: procedural reasoning and constraint execution, knowledge grounding and truthfulness, social cognition and affective understanding, and persona fidelity and stylistic consistency. Across more than 35 compressed checkpoints from six open-weight LLMs-covering quantization and pruning-we find that compression redistributes ability profiles. Coarse, local, and surface-level signals often persist, while fine-grained, long-tail, and context-dependent abilities become fragile. Quantization consistently outperforms pruning in retention, but the practical meaning of that gap depends entirely on which ability an application requires. Based on our findings, deployers can choose compression methods by the ability profile their application needs, not just by a leaderboard average. And algorithm designers gain diagnostic signals to build compression methods that preserve what matters. We believe that ACE-Compression turns compression from a black-box accuracy drop into an interpretable intervention-one we can measure, understand, and ultimately improve.

Figures & tables

Figure 1: Motivating scenario for ability-centric compression evaluation. A single role-grounded request requires the LLM to interact with abilities including persona grounding, clue extraction, response planning, knowledge grounding, contextual reasoning, and trustworthiness.

Figure 2: Evaluation design. We compare quantization methods (BnB-4bit, AWQ-INT4, GPTQ-INT4) and pruning methods (2:4 structured sparsity and unstructured sparsity) across six base models. Ten benchmarks are mapped to four ability dimensions so that pre/post-compression changes can be read as procedural, knowledge, social, or persona profile shifts.

Dimension	Benchmark Main Diagnostic Readout
Procedural reasoning and constraint execution	IFEval ( Zhou et al., 2023 ) Prompt-/instruction-level strict and loose compliance GSM8K ( Cobbe et al., 2021 ) Strict versus flexible answer matching HumanEval ( Chen et al., 2021 ) pass@k and sampling recoverability
Knowledge grounding and truthfulness	TruthfulQA ( Lin et al., 2022 ) MC1/MC2 and generation-side BLEURT behavior C-Eval ( Huang et al., 2023 ) Accuracy retention RoleEval ( Shen et al., 2023 ) Role-grounded knowledge retention over shot and language slices
Social cognition and affective understanding	ToMBench ( Chen et al., 2024 ) ToM profiles across task, ability, and subability views EQ-Bench ( Paech, 2023 ) Benchmark score and parseability as auxiliary affective evidence
Persona fidelity and stylistic consistency	CharacterEval ( Tu et al., 2024 ) role-play quality across grouped persona metrics PsychoBench ( Huang et al., 2024 ) psychometric drift and 16P dominant-type stability

Table 1: Ability dimensions, benchmarks, and native readouts in ACE-Compression. We preserve benchmark-native views when a single aggregate score would hide diagnostically important distinctions.

Figure 3: Procedural reasoning and constraint execution. (a) IFEval instruction-level strict retention minus prompt-level strict retention, diagnosing local constraint recognition versus full-response coordination. (b) GSM8K flexible-match retention. (c) GSM8K flexible-versus-strict retention gap shift, diagnosing answer-form brittleness. (d) HumanEval greedy pass@ $1$ retention by base model and compression method. (e) HumanEval pass@ $100$ minus pass@ $1$ retention, diagnosing sampling-based recoverability; hatched cells mark unavailable combinations.

Figure 4: Knowledge grounding and truthfulness. (a) TruthfulQA generation-side BLEURT-accuracy retention. (b) TruthfulQA generation-selection divergence, computed as generation-side BLEURT-accuracy retention minus MC1 retention. (c) Generic knowledge versus role-grounded knowledge retention, using C-Eval and RoleEval-ZH.

Figure 5: Social cognition and affective understanding. (a) ToMBench ability retention by base model, with one radar plot per FP16 parent and compression-method traces relative to that parent. (b) Most compression-sensitive ToMBench subabilities, averaged by method and displayed as retention percentages; abbreviated labels denote subabilities such as location false beliefs, irony/sarcasm, second-order beliefs, intentions explanations, and belief-action links. Compression primarily affects hidden-state, belief-, intention-, and ambiguity-centered reasoning.

Figure 6: Persona fidelity and stylistic consistency. (a) PsychoBench latent drift fingerprint from each FP16 parent; light segments summarize the quantization mean and darker extensions indicate additional sparse-setting drift. (b) Representative psychometric scales grouped by DTDD, BFI, ECR-R, LMS, and Empathy, with the line marking mean normalized drift by method. (c) 16P stability: stacked bars indicate axis-level drift/risk and the red line labels dominant-type stability. The panels connect visible persona flattening to latent psychometric and identity-profile movement.

Figure 7: Overall findings of ACE-Compression. Bar length indicates the mean FP16-relative retention $R$ for each ability layer under quantization or pruning.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model Group	Variant	Prompt-level Strict Loose		Instruction-level Strict Loose
Qwen2.5-7B	FP16	0.7172 $\pm$ 0.0194	0.7394 $\pm$ 0.0189	0.7926	0.8106
	BnB-4bit	0.7209 $\pm$ 0.0193	0.7449 $\pm$ 0.0188	0.7962	0.8177
	AWQ	0.7098 $\pm$ 0.0195	0.7246 $\pm$ 0.0192	0.7866	0.8046
	GPTQ-INT4	0.7079 $\pm$ 0.0196	0.7264 $\pm$ 0.0192	0.7950	0.8118
	2:4-Sparse	0.4547 $\pm$ 0.0214	0.4824 $\pm$ 0.0215	0.5863	0.6139
	Unstructured-Sparse	0.6359 $\pm$ 0.0207	0.6765 $\pm$ 0.0201	0.7302	0.7650

Appendix

Table 3: Full IFEval results for the procedural reasoning and constraint execution dimension. Prompt-level and instruction-level scores are reported under strict and loose evaluation, separating whole-response coordination from local instruction recognition.

Model Group	Variant	Strict Match	Flexible Match
Qwen2.5-7B	FP16	0.3351 $\pm$ 0.0130	0.7597 $\pm$ 0.0118
	BnB-4bit	0.2434 $\pm$ 0.0118	0.7377 $\pm$ 0.0121
	AWQ	0.1971 $\pm$ 0.0110	0.7202 $\pm$ 0.0124
	GPTQ-INT4	0.1691 $\pm$ 0.0103	0.7225 $\pm$ 0.0123
	2:4-Sparse	0.3995 $\pm$ 0.0135	0.5019 $\pm$ 0.0138
	Unstructured-Sparse	0.4367 $\pm$ 0.0137	0.7036 $\pm$ 0.0126

Appendix

Table 4: Full GSM8K results for the procedural reasoning and constraint execution dimension. Strict and flexible matching are reported together to distinguish canonical answer realization from recoverable numeric correctness.

Model Group	Variant	Greedy $p@1$	Sampling $p@1$ ( $n=20$ )	Sampling $p@10$ ( $n=20$ )
Qwen2.5-7B	FP16	0.7927 $\pm$ 0.0318	0.7034 $\pm$ 0.0315	0.8172 $\pm$ 0.0294
	BnB-4bit	0.7683 $\pm$ 0.0330	0.7034 $\pm$ 0.0315	0.8172 $\pm$ 0.0294
	AWQ	0.7744 $\pm$ 0.0327	0.6424 $\pm$ 0.0326	0.7837 $\pm$ 0.0312
	GPTQ-INT4	0.7683 $\pm$ 0.0330	0.6598 $\pm$ 0.0320	0.7966 $\pm$ 0.0306
	2:4-Sparse	0.3354 $\pm$ 0.0370	0.1692 $\pm$ 0.0268	0.2636 $\pm$ 0.0329
	Unstructured-Sparse	0.6890 $\pm$ 0.0363	0.4936 $\pm$ 0.0339	0.6757 $\pm$ 0.0349

Appendix

Table 5: HumanEval results for the procedural reasoning and constraint execution dimension under greedy decoding and low-budget sampling. Greedy pass@1 captures first-sample executable consistency, while n=20 sampling reports short-budget recoverability.

Model Group	Variant	Sampling $p@1$ ( $n=100$ )	Sampling $p@10$ ( $n=100$ )	Sampling $p@100$ ( $n=100$ )
Qwen2.5-7B	FP16	0.7021 $\pm$ 0.0315	0.8173 $\pm$ 0.0283	0.8598 $\pm$ 0.0272
	BnB-4bit	0.7029 $\pm$ 0.0315	0.8157 $\pm$ 0.0283	0.8598 $\pm$ 0.0272
	AWQ	0.6453 $\pm$ 0.0325	0.7791 $\pm$ 0.0306	0.8293 $\pm$ 0.0295
	GPTQ-INT4	0.6549 $\pm$ 0.0321	0.7920 $\pm$ 0.0305	0.8476 $\pm$ 0.0282
	2:4-Sparse	0.1745 $\pm$ 0.0268	0.2665 $\pm$ 0.0322	0.3780 $\pm$ 0.0380
	Unstructured-Sparse	0.4926 $\pm$ 0.0335	0.6767 $\pm$ 0.0340	0.7622 $\pm$ 0.0333

Appendix

Table 6: HumanEval high-budget sampling results for the procedural reasoning and constraint execution dimension. The n=100 pass@k views audit whether correct programs remain recoverable beyond the first decoded solution.

Model Group	Variant	0-shot MC1 MC2 Gen-BLEURT-max			3-shot MC1 MC2
Qwen2.5-7B	FP16	0.4737 $\pm$ 0.0175	0.6306 $\pm$ 0.0160	-0.6324 $\pm$ 0.0135	0.4712 $\pm$ 0.0044	0.6362 $\pm$ 0.0017
	BnB-4bit	0.4663 $\pm$ 0.0175	0.6229 $\pm$ 0.0160	-0.6418 $\pm$ 0.0134	0.4716 $\pm$ 0.0043	0.6317 $\pm$ 0.0021
	AWQ	0.4480 $\pm$ 0.0174	0.6149 $\pm$ 0.0160	-0.6234 $\pm$ 0.0140	0.4598 $\pm$ 0.0062	0.6294 $\pm$ 0.0004
	GPTQ-INT4	0.4688 $\pm$ 0.0175	0.6309 $\pm$ 0.0160	-0.6224 $\pm$ 0.0134	0.4639 $\pm$ 0.0056	0.6271 $\pm$ 0.0042
	2:4-Sparse	0.4284 $\pm$ 0.0173	0.5913 $\pm$ 0.0162	-0.6436 $\pm$ 0.0128	0.4186 $\pm$ 0.0032	0.5873 $\pm$ 0.0049
	Unstructured-Sparse	0.4296 $\pm$ 0.0173	0.5931 $\pm$ 0.0161	-0.5756 $\pm$ 0.0133	0.4353 $\pm$ 0.0049	0.6014 $\pm$ 0.0028

Appendix

Table 7: Selected TruthfulQA results for the knowledge grounding and truthfulness dimension. MC1/MC2 report selection-side truthfulness, while Gen-BLEURT-max reports generation-side reference compatibility; the 3-shot columns audit selection behavior under few-shot context.

Model Group	Variant	Valid Acc.	Test Acc.
Qwen2.5-7B	FP16	0.7964 $\pm$ 0.0105	0.7931 $\pm$ 0.0035
	BnB-4bit	0.7868 $\pm$ 0.0108	0.7787 $\pm$ 0.0036
	AWQ	0.7868 $\pm$ 0.0108	0.7804 $\pm$ 0.0036
	GPTQ-INT4	0.7779 $\pm$ 0.0109	0.7785 $\pm$ 0.0036
	2:4-Sparse	0.4985 $\pm$ 0.0133	0.4736 $\pm$ 0.0044
	Unstructured-Sparse	0.6746 $\pm$ 0.0124	0.6674 $\pm$ 0.0041

Appendix

Table 8: C-Eval accuracy results for the knowledge grounding and truthfulness dimension. Validation and test splits are reported together to audit broad exam-knowledge retention under quantization and sparsity.

Model Group	Variant	0-shot mean EN ZH		5-shot mean EN ZH		Across-shot mean EN ZH Overall
Qwen2.5-7B	FP16	0.5032	0.5249	0.5032	0.5249	0.5032	0.5249	0.5141
	BnB-4bit	0.5025	0.4826	0.5025	0.4826	0.5025	0.4826	0.4926
	AWQ	0.4959	0.5184	0.4976	0.5184	0.4967	0.5184	0.5076
	GPTQ-INT4	0.4895	0.5012	0.4895	0.5012	0.4895	0.5012	0.4954
	2:4-Sparse	0.3391	0.3139	0.3391	0.3139	0.3391	0.3139	0.3265
	Unstructured-Sparse	0.4125	0.4135	0.4125	0.4135	0.4125	0.4135	0.4130

Appendix

Table 9: RoleEval grouped slice means for the knowledge grounding and truthfulness dimension. Shot-specific and across-shot EN/ZH means show how role-grounded knowledge retention varies by language and compression setting.

Model Group	Variant	Benchmark Score	Parseable Answers
Qwen2.5-7B	FP16	72.86	169.0
	BnB-4bit	74.57	169.0
	AWQ	70.76	170.0
	GPTQ-INT4	70.07	170.0
	2:4-Sparse	57.22	167.0
	Unstructured-Sparse	65.11	170.5

Appendix

Table 10: EQ-Bench results for the social cognition and affective understanding dimension. Benchmark score measures affective judgment quality, while parseable-answer count tracks whether compressed models remain within the expected emotion-label output format.

Model Group	Variant	Belief	Desire	Emotion	Intention	Knowledge	Non-literal Comm.
Qwen2.5-7B	FP16	0.7642	0.5286	0.6666	0.6631	0.4914	0.6996
	BnB-4bit	0.7391	0.4750	0.6614	0.6736	0.4707	0.7047
	AWQ	0.7500	0.5535	0.6625	0.6397	0.4672	0.6862
	GPTQ-INT4	0.7550	0.5465	0.6844	0.6841	0.4689	0.6939
	2:4-Sparse	0.4983	0.4643	0.4896	0.4948	0.3759	0.6240
	Unstructured-Sparse	0.6100	0.5143	0.6156	0.6175	0.4362	0.6675

Appendix

Table 11: ToMBench ability-level scores for the social cognition and affective understanding dimension. EN/ZH scores are averaged within each official ability group to show broad social-cognitive profiles under compression.

Task (EN/ZH mean)	BnB	AWQ	GPTQ	2:4	Unstructured
Faux-pas Recognition	0.974	0.980	0.986	0.849	0.905
Unexpected Outcome	0.973	0.987	1.008	0.721	0.915
Persuasion Story	0.969	0.977	1.006	0.783	0.925
False Belief Task	0.982	0.986	1.004	0.682	0.825
Ambiguous Story	0.989	0.983	0.993	0.675	0.862
Strange Story	0.991	0.987	0.999	0.812	0.920

Appendix

Table 12: ToMBench task-level retention ratios for the social cognition and affective understanding dimension. Values are FP16-relative retention ratios after averaging EN/ZH within each task and base model, then averaging by compression method.

Subability	BnB	AWQ	GPTQ	2:4	Unstructured
Belief
Content false beliefs	0.976	1.018	1.020	0.762	0.895
Location false beliefs	0.986	0.957	0.988	0.612	0.768
Identity false beliefs	1.007	0.980	0.997	0.771	0.920
Second-order beliefs	0.963	0.975	0.981	0.608	0.724
Beliefs based action/emotions	0.990	0.986	0.988	0.700	0.864

Appendix

Table 13: ToMBench subability-level retention ratios for the social cognition and affective understanding dimension. Values are FP16-relative retention ratios

R

, so the FP16 parent is omitted and corresponds to

R=1

Model Group	Variant	Character Consistency KE KA KH PB PU Avg.						Conversational Ability Flu. Coh. Cons. Avg.				Role-playing Attractiveness HL CS ED Emp. Avg.
Qwen2.5-7B	FP16	2.419	2.977	3.009	3.183	3.049	2.927	3.533	3.936	3.612	3.694	3.325	3.494	2.547	3.257	3.155
	BnB-4bit	2.372	2.902	2.976	3.131	3.008	2.878	3.571	3.899	3.517	3.662	3.258	3.433	2.547	3.172	3.103
	AWQ	2.388	2.901	2.956	3.128	3.040	2.883	3.531	3.901	3.544	3.659	3.252	3.454	2.540	3.213	3.115
	GPTQ-INT4	2.412	2.944	2.970	3.185	3.042	2.910	3.539	3.913	3.539	3.664	3.217	3.519	2.600	3.199	3.134
	2:4-Sparse	1.995	2.749	2.379	2.269	2.421	2.363	2.737	3.040	2.533	2.770	2.459	2.630	1.728	2.702	2.380
	Unstructured-Sparse	2.217	2.809	2.679	2.565	2.711	2.596	3.180	3.545	3.094	3.273	2.905	3.084	2.030	2.963	2.746

Appendix

Table 14: CharacterEval full official grouped results for the persona fidelity and stylistic consistency dimension. The appendix restores the released grouped breakdown, including KE, KH, and Emp., together with workbook-provided group averages.

Model Group

Variant

DTDD Mach. Psych.

BFI Extra. Agree. Consc. Neuro. Open.

ECR-R Anx. Avoid.

LMS Rich Motiv. Import.

Empathy

Qwen2.5-7B

FP16

3.3

\pm

1.6

3.5

\pm

0.4

2.8

\pm

0.1

3.8

\pm

0.7

3.5

\pm

0.1

3.2

\pm

0.4

3.8

\pm

0.8

3.4

\pm

0.4

4.5

\pm

0.1

2.5

\pm

1.2

3.3

\pm

0.7

4.3

\pm

0.3

3.8

\pm

0.4

BnB-4bit

3.8

\pm

1.0

4.9

\pm

0.6

3.1

\pm

0.2

3.7

\pm

0.7

2.9

\pm

0.8

3.6

\pm

0.8

3.7

\pm

0.6

4.1

\pm

0.2

4.9

\pm

0.1

2.0

\pm

0.3

3.7

\pm

0.3

4.7

\pm

0.0

3.1

\pm

0.7

AWQ

4.8

\pm

0.3

5.1

\pm

0.4

2.4

\pm

0.5

3.6

\pm

0.9

2.8

\pm

1.0

3.6

\pm

0.8

3.2

\pm

0.1

3.6

\pm

0.3

4.5

\pm

0.2

2.7

\pm

1.4

3.3

\pm

0.7

3.7

\pm

0.7

3.7

\pm

0.0

GPTQ-INT4

5.0

\pm

0.5

5.9

\pm

0.4

3.1

\pm

0.2

4.1

\pm

0.4

3.9

\pm

0.1

3.2

\pm

0.4

3.0

\pm

0.2

3.4

\pm

0.9

4.4

\pm

0.1

2.5

\pm

0.2

4.2

\pm

0.5

4.2

\pm

0.5

3.2

\pm

0.2

2:4-Sparse

3.5

\pm

1.0

3.5

\pm

1.0

3.6

\pm

1.7

3.0

\pm

0.9

2.5

\pm

0.1

4.4

\pm

0.2

3.4

\pm

0.5

3.8

\pm

0.6

3.8

\pm

0.2

3.2

\pm

0.9

3.3

\pm

0.0

3.3

\pm

1.0

3.6

\pm

0.2

Unstructured-Sparse

3.2

\pm

1.8

3.8

\pm

0.4

2.7

\pm

0.4

2.5

\pm

0.7

4.0

\pm

0.3

2.8

\pm

0.1

3.1

\pm

0.4

3.8

\pm

0.5

3.8

\pm

0.3

3.3

\pm

1.0

3.7

\pm

0.0

3.3

\pm

1.0

3.7

\pm

0.1

Appendix

Table 15: PsychoBench representative-scale raw outputs for the persona fidelity and stylistic consistency dimension. The table groups DTDD, BFI, ECR-R, LMS, and Empathy in a unified view. Because the released file does not provide DTDD Narcissism or LMS Overall, only the available subscale and factor outputs are reported.

Model Group	Variant	Top-1 type	Count	Top-2 type	Count	Top-3 type	Count
Qwen2.5-7B	FP16	ENTJ-T	10	ESTP-T	10	–	–
	BnB-4bit	ESFJ-A	10	ESFP-T	9	ESTP-T	1
	AWQ	ESTP-T	10	ISTJ-T	10	–	–
	GPTQ-INT4	ESTP-T	10	ISTJ-T	10	–	–
	2:4-Sparse	ESTP-A	10	ISFJ-T	10	–	–
	Unstructured-Sparse	INFP-T	10	ISFJ-T	10	–	–

Appendix

Table 16: 16P aggregated dominant-type distribution for the persona fidelity and stylistic consistency dimension. Values are taken directly from the released aggregated file; each row reports the top personality types and counts for a model variant.

Model Group	Variant	Extraverted	Intuitive	Thinking	Judging	Assertive	Axis MAE vs FP16
Qwen2.5-7B	FP16	69.0	51.5	63.5	46.5	41.5	0.000
	BnB-4bit	70.3	38.6	40.3	55.3	48.5	0.106
	AWQ	58.0	45.0	58.5	48.0	38.5	0.054
	GPTQ-INT4	59.5	36.3	53.4	45.6	43.5	0.075
	2:4-Sparse	51.5	42.0	52.0	51.5	50.0	0.104
	Unstructured-Sparse	36.5	46.5	30.5	51.0	39.5	0.154

Appendix

Table 17: 16P weighted axis means for the persona fidelity and stylistic consistency dimension. The table reports count-weighted axis percentages and an appendix-side Axis MAE against each FP16 parent; this derived readout summarizes axis-level drift from the released aggregated file.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 280 papers

Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction
Jun 1, 2026Yujia Tong, Yuxi Wang, Yunyang Wan +3LLM Quantization LLM Evaluation
Wuhan University of Technology · Nanyang Technological University
The Asymmetric Harms of LLM Compression
Aug 20, 2026Yuan Wu, Mairui Li, Lesia Semenova +1LLM Compression Social Bias in Language Models
University of North Carolina at Chapel Hill
Capability-Guided Compression: Toward Interpretability-Aware Budget Allocation for Large Language Models
Mar 17, 2026Rishaank GuptaPhase Transitions in Neural Networks Model Compression
Independent Researcher

Jun 1, 2026J/K move · Enter open · S save

Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction

Yujia Tong, Yuxi Wang, Yunyang Wan, Tian Zhang+2

Wuhan University of Technology · Nanyang Technological University

LLM Quantization LLM Evaluation LLM Pruning LLM Compression Confidence Estimation in Language Models LLM Reliability+1

Open paper AskPDF arXiv

Model compression techniques such as quantization and pruning are widely used to reduce the deployment cost of large language models (LLMs), with existing evaluations focusing almost exclusively on accuracy preservation. However, in safety-critical applications, a model's ability to reliably quantify its own uncertainty is equally important. We ask: does compression preserve this ability? To answer this question, we benchmark 12 LLMs under various compression configurations across five NLP tasks, using conformal prediction to provide a rigorous, distribution-free measure of uncertainty. Our experiments reveal that: (I) compression frequently decouples accuracy from uncertainty; (II) larger models absorb compression-induced uncertainty far more effectively than smaller ones; and (III) uncertainty inflation is often threshold-like rather than gradual. These results suggest that accuracy-only evaluation is insufficient for assessing the deployment readiness of compressed LLMs, and that uncertainty-aware benchmarking should be a standard component of model compression pipelines.

Abstract

LLM Compression

Explore similar work

Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction

The Asymmetric Harms of LLM Compression

Capability-Guided Compression: Toward Interpretability-Aware Budget Allocation for Large Language Models