Effects of Model Scale and Quantization on Social Bias and Abstention in Question Answering

Kuniko Paxton; Ifeoluwa Wuraola; Dhavalkumar Thakker

cs.CLOct 21, 2026

Effects of Model Scale and Quantization on Social Bias and Abstention in Question Answering

Authors: Kuniko Paxton, Ifeoluwa Wuraola, Dhavalkumar Thakker

Organizations: University of Hull Cottingham Road, Hull, HU6 7RX, United Kingdom

Abstract

The deployment of small and quantized language models in resource-constrained settings raises an important fairness question: when a compressed model produces fewer biased responses, does this reflect fairer behavior or greater abstention? We study this problem in Question Answering (QA) using the BBQ benchmark across multiple model sizes and quantization settings from Qwen3, Gemma-3, and SmolLM2. We also examine whether the observed trends remain stable across alternative prompt formulations and when the ``Unknown'' option is removed in a forced-choice setting. Our results show that larger models achieved higher QA accuracy, but bias showed no simple scaling trend and quantization effects varied across configurations. Notably, 20 of the 26 configurations with fewer biased responses also show increased over-abstention, indicating that lower observed bias can partly reflect greater abstention. These findings show that lower observed bias can partly reflect greater abstention, and that fairness evaluation should consider biased responses, abstention, prompt formulation, and answer-option design jointly.

Figures & tables

Figure 1: Overview of an Experimental Framework for Evaluating Stereotype Bias Using BBQ. The process evaluates the system across various BBQ contexts using five different prompt settings, model variants, and quantization settings. Two-choice questions use prompts 1–2, while three-choice questions use all five prompts (prompts 1–5).

Qwen3
Model	Dense	FP8	FP8 Dyn.	INT8	INT8 Dyn.	INT4
0.6B	49.07	48.37	50.72	49.11	49.08	44.82
1.7B	68.25	66.64	68.33	67.39	67.36	64.77
4B	85.63	85.35	85.22	85.63	85.37	85.08
8B	89.31	89.32	89.48	89.36	88.89	87.12
Gemma-3

Table 1: Accuracy (%) by model family, size, and quantization configuration, macro-averaged across BBQ categories and averaged over five prompts.

Figure 2: Change in biased-response rate relative to the corresponding Dense model across model families, sizes, and quantization configurations. Values are averaged across five prompts and reported in percentage points (pp).

Figure 3: Changes in biased-response rate and over-abstention relative to Dense for Qwen3. Each point represents a model size and quantization configuration. Values are averaged across five prompts and reported in percentage points (pp).

Figure 4: Changes in biased-response rate and over-abstention relative to Dense for Gemma-3.

Figure 5: Changes in biased-response rate and over-abstention relative to Dense for SmolLM2.

Figure 6: Category-level changes in biased-response rate relative to the corresponding Dense models across Qwen3, Gemma-3, and SmolLM2. Values are averaged across five prompts and expressed in percentage points (pp).

Measure	Same direction	Mixed directions
Biased-response rate	14/50	36/50
Over-abstention rate	13/50	37/50
Both measures	12/50	38/50

Table 2: Direction of quantization-induced changes across five prompts.

Figure 7: Responses after reducing the options from three to two for cases originally answered “Unknown.” The top and bottom rows show anti-biased rates under the Negative condition and biased rates under the Non-negative condition, respectively, for Prompt 1 (left) and Prompt 2 (right). Color intensity indicates response rate and bubble size the number of valid cases; rows denote models and columns sensitive-attribute categories.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Items	Specs
CPU	Intel Core i9-14900F
GPU	NVIDIA GeForce RTX 4090
GPU Memory	24 GB
System Memory	64 GB
OS	Ubuntu 22.04.4 LTS (WSL2)
Python	3.11.15

Appendix

Table 3: Execution Environment

Comparison	Negative	Non-negative
Prompt 1 vs. Prompt 2	0.956 / 2.10	0.889 / 1.80
FP vs. INT4 Prompt 1	0.975 / 1.16	0.983 / 0.95
FP vs. INT4 Prompt 2	0.992 / 0.98	0.990 / 1.11

Appendix

Table 4: Stability of category-level response patterns across prompt formulations and INT4 quantization. Values are Pearson correlation / median absolute difference.

Social Bias in Language Models

Social bias in language models is the study and mitigation of unfair stereotypes and disparities in generated text, judgments, and decisions.

See all 199 papers

Investigating Social Bias Changes in Quantized Language Models
Feb 5, 2026Stanley Z. Hua, Sanae Lotfi, Irene Y. ChenLLM Quantization Social Bias in Language Models
UC Berkeley & UCSF · Meta Superintelligence Labs
QuantiBias: Benchmarking Quantization-Induced Bias in LLMs
Jul 23, 2026Emilio FerraraOpen-Ended Generation LLM Quantization
Thomas Lord Department of Computer Science University of Southern California
Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels
May 2, 2026Plawan Kumar Rath, Rahul MaliakkalLLM Quantization Social Bias in Language Models
Meta

Feb 5, 2026J/K move · Enter open · S save

Investigating Social Bias Changes in Quantized Language Models

Stanley Z. Hua, Sanae Lotfi, Irene Y. Chen

UC Berkeley & UCSF · Meta Superintelligence Labs

LLM Quantization Social Bias in Language Models Language Model Bias Evaluation Linguistic Bias in Language Models Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

Post-training quantization reduces the memory needed to run large language models but alters their social biases in ways that aggregate metrics fail to capture. We present the first large-scale study of 50 quantized models evaluated on PostTrainingBiasBench, a unified benchmark of 13 closed- and open-ended bias datasets. We identify a phenomenon we term quantization-induced bias flipping, in which quantization causes models to change responses from biased to unbiased and vice versa, up to 21% of the time, despite no change in aggregate bias scores. These flips are strongly associated with model uncertainty, where the responses with high uncertainty are 3-11x more likely to change than the confident ones. Quantization strength amplifies this effect, with 4-bit quantized models exhibiting 4-6x more behavioral changes than 8-bit quantized models. Critically, these changes create asymmetric impacts across demographic groups, where bias can worsen by up to 18.6% for some groups while improving by 14.1% for others, yielding misleadingly neutral aggregate outcomes. Larger models show no consistent robustness advantage, and group-specific shifts vary unpredictably across model families. Our findings demonstrate that compression fundamentally alters bias patterns, requiring crucial post-quantization evaluation and interventions to ensure reliability in practice.

Abstract

Social Bias in Language Models

Explore similar work

Investigating Social Bias Changes in Quantized Language Models

QuantiBias: Benchmarking Quantization-Induced Bias in LLMs

Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels