The Asymmetric Harms of LLM Compression

Yuan Wu; Mairui Li; Lesia Semenova; Chudi Zhong

cs.CLAug 20, 2026

The Asymmetric Harms of LLM Compression

Authors: Yuan Wu, Mairui Li, Lesia Semenova, Chudi Zhong

Organizations: University of North Carolina at Chapel Hill

Abstract

Large language models (LLMs) compression reduces deployment costs, but standard aggregate metrics like perplexity and accuracy often mask underlying behavioral shifts. In this work, we systematically evaluate 3 LLMs across 11 compression methods to investigate the effects of compression on knowledge retention, model confidence, and social bias. We find that compression disproportionately reduces the relative retention of head knowledge compared to tail knowledge. Furthermore, compressed models often remain substantially confident in their incorrect answers on newly lost knowledge. Finally, we demonstrate that stable aggregate bias scores can conceal substantial, opposing shifts in stereotypical preferences across demographic subgroups. Together, these findings reveal asymmetric behavioral changes that aggregate performance measures fail to capture, highlighting the need for granular evaluation of compressed models before deployment.

Figures & tables

Figure 1: Accuracy across different popularity groups on PopQA under different compression settings. Results for all methods and models are provided in Figures 8 , 9 , and 10 in Appendix A.2.1 .

Figure 2: Relative retention shift across PopQA popularity groups under different compression settings, reported in percentage points. Negative and positive $\mathrm{RS}_{g}$ indicate lower and higher relative retention than the dataset overall, respectively. Results for all methods and models are provided in Figures 14 , 15 , and 16 in Appendix A.2.1 .

Figure 3: Knowledge-loss rate across popularity groups on PopQA under different compression settings. Higher values indicate that a larger proportion of the knowledge correctly answered by the base model is incorrect after compression. Results for all methods and models are provided in Figures 20 , 21 , and 22 in Appendix A.2.2 .

Figure 4: Confidence on lost knowledge across popularity groups on PopQA under different compression settings. Each dot represents the median confidence; the thick vertical bar shows the 25th–75th percentile range, and the thin vertical bar shows the 5th–95th percentile range. Colors distinguish tail, middle, and head knowledge. Results for all methods and models are provided in Figures 26 , 27 , and 28 in Appendix A.2.2 .

Figure 5: Bias changes on WinoBias under different compression settings, reported in percentage points. Blue, orange, and black bars denote changes for male, female, and all examples, respectively, and error bars denote 95% confidence intervals. Positive values indicate increased stereotypical preference, negative values indicate decreased preference, and the dashed line marks no change from the base model. Results for all methods and models are provided in Figures 38 , 39 , and 40 in Appendix A.2.3 .

Appendix figures & tables43 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	GPTQ	AWQ	OmniQuant	AQLM
Bit width	2, 3, 4	2, 3, 4	2, 3, 4	2, 3, 4
Calibration dataset	C4	C4	C4	C4
Calibration samples	128	128	128	128
Calibration seed	42	42	42	42
Calibration sequence length	512	512	512	512
Group size	128	128	128	–

Appendix

Table 1: Quantization configurations.

Parameter	Magnitude	WANDA	SparseGPT	ShortGPT	Layer Dropping
Compression granularity	weights	weights	weights	decoder blocks	decoder blocks
Compression levels	30/50/70%	30/50/70%	30/50/70%	5/10/15/20/25%	5/10/15/20/25%
Semi-structured patterns	–	4:8, 2:4	4:8, 2:4	–	–
Calibration dataset	–	C4	C4	C4	C4
Calibration samples	–	32	32	32	32
Calibration sequence length	–	512	512	512	256

Appendix

Table 2: Pruning configurations.

Compression	Method	Setting	Perplexity (PPL)
			Llama-3.1- 8B-Instruct	Qwen-3- 8B	Gemma-2- 9B-it
None	Full precision	FP16	7.1253	9.5888	10.2117
Quantization	GPTQ	2-bit	1612.6539	141.3107	137.1019
		3-bit	10.0354	11.2652	12.3045
		4-bit	8.4459	9.9511	10.4903
	AWQ	2-bit	94354.1597	16508.5254	9407.2547

Appendix

Table 3: Perplexity of the three full-precision models and their compressed variants on WikiText-2.

Figure 6: Overall accuracy on PopQA under different compression methods and settings.

Figure 7: Overall accuracy on Head-to-Tail under different compression methods and settings.

Figure 8: Accuracy across different popularity groups on PopQA for Llama-3.1-8B-Instruct under additional compression methods and settings.

Figure 9: Accuracy across different popularity groups on PopQA for Qwen3-8B under additional compression methods and settings.

Figure 10: Accuracy across different popularity groups on PopQA for Gemma-2-9B under additional compression methods and settings.

Figure 11: Accuracy across different popularity groups on Head-to-Tail for Llama-3.1-8B-Instruct under different compression methods and settings.

Figure 12: Accuracy across different popularity groups on Head-to-Tail for Qwen3-8B under different compression methods and settings.

Figure 13: Accuracy across different popularity groups on Head-to-Tail for Gemma-2-9B under different compression methods and settings.

Figure 14: Relative retention shifts across different popularity groups on PopQA for Llama-3.1-8B-Instruct under additional compression methods and settings. Positive values indicate that a group retains a larger fraction of its base-model accuracy than the model overall, while negative values indicate lower relative retention. Values are reported in percentage points.

Figure 15: Relative retention shifts across different popularity groups on PopQA for Qwen3-8B under additional compression methods and settings. Positive values indicate that a group retains a larger fraction of its base-model accuracy than the model overall, while negative values indicate lower relative retention. Values are reported in percentage points.

Figure 16: Relative retention shifts across different popularity groups on PopQA for Gemma-2-9B under additional compression methods and settings. Positive values indicate that a group retains a larger fraction of its base-model accuracy than the model overall, while negative values indicate lower relative retention. Values are reported in percentage points.

Figure 17: Relative retention shifts across different popularity groups on Head-to-Tail for Llama-3.1-8B-Instruct under different compression methods and settings. Positive values indicate that a group retains a larger fraction of its base-model accuracy than the model overall, while negative values indicate lower relative retention. Values are reported in percentage points.

Figure 18: Relative retention shifts across different popularity groups on Head-to-Tail for Qwen3-8B under different compression methods and settings. Positive values indicate that a group retains a larger fraction of its base-model accuracy than the model overall, while negative values indicate lower relative retention. Values are reported in percentage points.

Figure 19: Relative retention shifts across different popularity groups on Head-to-Tail for Gemma-2-9B under different compression methods and settings. Positive values indicate that a group retains a larger fraction of its base-model accuracy than the model overall, while negative values indicate lower relative retention. Values are reported in percentage points.

Figure 20: Knowledge-loss rates across different popularity groups on PopQA for Llama-3.1-8B-Instruct under additional compression methods and settings.

Figure 21: Knowledge-loss rates across different popularity groups on PopQA for Qwen3-8B under different compression methods and settings.

Figure 22: Knowledge-loss rates across different popularity groups on PopQA for Gemma-2-9B under different compression methods and settings.

Figure 23: Knowledge-loss rates across different popularity groups on Head-to-Tail for Llama-3.1-8B-Instruct under different compression methods and settings.

Figure 24: Knowledge-loss rates across different popularity groups on Head-to-Tail for Qwen3-8B under different compression methods and settings.

Figure 25: Knowledge-loss rates across different popularity groups on Head-to-Tail for Gemma-2-9B under different compression methods and settings.

Figure 26: Calibrated confidence assigned to lost knowledge across different popularity groups on PopQA for Llama-3.1-8B-Instruct under additional compression methods and settings.

Figure 27: Calibrated confidence assigned to lost knowledge across different popularity groups on PopQA for Qwen3-8B under additional compression methods and settings.

Figure 28: Calibrated confidence assigned to lost knowledge across different popularity groups on PopQA for Gemma-2-9B under additional compression methods and settings.

Figure 29: Calibrated confidence assigned to lost knowledge across different popularity groups on Head-to-Tail for Llama-3.1-8B-Instruct under all compression methods and settings.

Figure 30: Calibrated confidence assigned to lost knowledge across different popularity groups on Head-to-Tail for Qwen3-8B under all compression methods and settings.

Figure 31: Calibrated confidence assigned to lost knowledge across different popularity groups on Head-to-Tail for Gemma-2-9B under all compression methods and settings.

Figure 32: Calibrated reliability diagrams on PopQA for Llama-3.1-8B-Instruct across all compression methods and settings. The diagonal line represents perfect calibration.

Figure 33: Calibrated reliability diagrams on PopQA for Qwen3-8B across all compression methods and settings. The diagonal line represents perfect calibration.

Figure 34: Calibrated reliability diagrams on PopQA for Gemma-2-9B across all compression methods and settings. The diagonal line represents perfect calibration.

Figure 35: Calibrated reliability diagrams on Head-to-Tail for Llama-3.1-8B-Instruct across all compression methods and settings. The diagonal line represents perfect calibration.

Figure 36: Calibrated reliability diagrams on Head-to-Tail for Qwen3-8B across all compression methods and settings. The diagonal line represents perfect calibration.

Figure 37: Calibrated reliability diagrams on Head-to-Tail for Gemma-2-9B across all compression methods and settings. The diagonal line represents perfect calibration.

Figure 38: Overall and gender-subgroup changes in stereotypical preference on WinoBias for Llama-3.1-8B-Instruct under additional compression methods and settings. Blue and orange bars show the changes for male and female subgroups, respectively, while black bars show the overall change. Error bars denote 95% confidence intervals. Positive values indicate increased stereotypical preference after compression. The dashed vertical line denotes no change from the base model. All values are reported in percentage points.

Figure 39: Overall and gender-subgroup changes in stereotypical preference on WinoBias for Qwen3-8B under different compression methods and settings. Blue and orange bars show the changes for male and female subgroups, respectively, while black bars show the overall change. Error bars denote 95% confidence intervals. Positive values indicate increased stereotypical preference after compression. The dashed vertical line denotes no change from the base model. All values are reported in percentage points.

Figure 40: Overall and gender-subgroup changes in stereotypical preference on WinoBias for Gemma-2-9B-it under different compression methods and settings. Blue and orange bars show the changes for male and female subgroups, respectively, while black bars show the overall change. Error bars denote 95% confidence intervals. Positive values indicate increased stereotypical preference after compression. The dashed vertical line denotes no change from the base model. All values are reported in percentage points.

Figure 41: Overall and subgroup-level changes in stereotypical preference on BBQ for Llama-3.1-8B-Instruct under different compression methods and settings. Colored bars show the changes for different demographic subgroups, while black bars show the overall change. Error bars denote 95% confidence intervals. Positive values indicate increased stereotypical preference after compression. The dashed vertical line denotes no change from the base model. All values are reported in percentage points.

Figure 42: Overall and subgroup-level changes in stereotypical preference on BBQ for Qwen3-8B under different compression methods and settings. Colored bars show the changes for different demographic subgroups, while black bars show the overall change. Error bars denote 95% confidence intervals. Positive values indicate increased stereotypical preference after compression. The dashed vertical line denotes no change from the base model. All values are reported in percentage points.

Figure 43: Overall and subgroup-level changes in stereotypical preference on BBQ for Gemma-2-9B-it under different compression methods and settings. Colored bars show the changes for different demographic subgroups, while black bars show the overall change. Error bars denote 95% confidence intervals. Positive values indicate increased stereotypical preference after compression. The dashed vertical line denotes no change from the base model. All values are reported in percentage points.

Rank	Gender	Occupation	Compression Configuration	Base $\boldsymbol{B_{g}}$ (%)	Comp. $\boldsymbol{B_{g}}$ (%)	$\boldsymbol{\Delta B_{g}}$ (pp)	$\boldsymbol{\Delta B}$ (pp)
Llama-3.1-8B-Instruct
1	Female	Secretary	SparseGPT N:M (2:4)	71.9	18.8	$-53.1$	$-2.2$
3	Female	Auditor	SparseGPT N:M (4:8)	82.1	32.1	$-50.0$	$+2.3$
8	Male	Analyst	SparseGPT N:M (2:4)	50.0	2.5	$-47.5$	$-2.2$
9	Female	Clerk	SparseGPT N:M (2:4)	78.6	32.1	$-46.4$	$-2.2$
Qwen3-8B

Appendix

Table 4: The ten largest absolute occupation-level bias changes on WinoBias among the retained compression configurations. Base and compressed subgroup bias scores are reported as percentages, while subgroup changes

\Delta B_{g}

and overall changes

\Delta B

are reported in percentage points.

Rank	Category	Subgroup	Compression Configuration	Base $\boldsymbol{B_{g}}$ (%)	Comp. $\boldsymbol{B_{g}}$ (%)	$\boldsymbol{\Delta B_{g}}$ (pp)	$\boldsymbol{\Delta B}$ (pp)
Llama-3.1-8B-Instruct
8	Disability	People with cognitive disabilities or mental illness	SparseGPT N:M (4:8)	40.6	59.4	$+18.8$	$-1.7$
9	Disability	People with cognitive disabilities or mental illness	WANDA N:M (2:4)	40.6	59.4	$+18.8$	$-3.1$
Qwen3-8B
1	Disability	Down’s syndrome	ShortGPT (10%)	37.5	75.0	$+37.5$	$-1.2$
2	Disability	Down’s syndrome	ShortGPT (15%)	37.5	75.0	$+37.5$	$-1.4$

Appendix

Table 5: The ten largest absolute subgroup-level bias changes on BBQ among the retained compression configurations. Base and compressed subgroup bias scores are reported as percentages, while subgroup changes

\Delta B_{g}

and overall changes

\Delta B

are reported in percentage points.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 283 papers

Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels
May 2, 2026Plawan Kumar Rath, Rahul MaliakkalLLM Quantization Social Bias in Language Models
Meta
Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction
Jun 1, 2026Yujia Tong, Yuxi Wang, Yunyang Wan +3LLM Quantization LLM Evaluation
Wuhan University of Technology · Nanyang Technological University
ACE-Compression: An Ability-Centric Evaluation of Compressed Large Language Models
Oct 20, 2026Jiahui Ning, Liang Chen, Yuchen Liang +2LLM Compression LLM Evaluation
City University of Hong Kong · Tongji University · Rensselaer Polytechnic Institute +1

May 2, 2026J/K move · Enter open · S save

Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels

Plawan Kumar Rath, Rahul Maliakkal

Abstract

LLM Compression

Explore similar work

Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels

Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction

ACE-Compression: An Ability-Centric Evaluation of Compressed Large Language Models