Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs

Shayan Shahrabi-Farahani; Dara Rahmati

cs.CLAug 19, 2026

Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs

Authors: Shayan Shahrabi-Farahani, Dara Rahmati

Organizations: Shahid Beheshti University · Shahid Beheshti University, Tehran, Iran

Abstract

Proactive interference (PI) is a documented failure mode in large language models in which retrieval of a repeatedly overwritten value degrades as prior overwrites accumulate, mirroring a classical phenomenon in human working memory. Post-training quantization (PTQ) is now the default deployment path for open-weight models, yet its effect on this failure mode has not been tested. We evaluate three precision levels (FP16, INT8, INT4/NF4, via bitsandbytes) across three architecturally distinct instruction-tuned models (Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, Phi-3.5-mini-instruct), holding the retrieval task fixed. INT4 quantization significantly reduces accuracy under high interference in every model (e.g., from 81.0% to 68.3% for Qwen), confirmed by paired McNemar's tests ( $p \le 2.6 \times 10^{-6}$ ) and a mixed-effects regression spanning all interference levels; INT8, often assumed safe, also carries a smaller but real penalty in two of three models. The effect is specific to semantically similar (word-type) distractors and reverses sign under a numeric control condition, and is mechanistically linked to a rise in same-key intrusion errors under INT4 (from 21.5% to 24.6% of trials, $p = 4.8 \times 10^{-7}$ ). A follow-up ablation shows the effect originates in the quantized transformer backbone rather than the output projection layer. These results suggest that bitsandbytes 4-bit quantization can impose an additional cost on applications relying on long, updatable, semantically dense contexts, even when aggregate benchmark accuracy appears largely unaffected. We release our code and tokenizer-verified vocabulary construction method at https://github.com/ShayanShahrabi/compress-and-forget

Figures & tables

Figure 1: Overview of the experimental pipeline and evaluation design. (A) Each of three instruction-tuned models is loaded at three precisions (FP16, INT8, INT4/NF4 via bitsandbytes) and evaluated on the PI-LLM retrieval benchmark, in which a subject’s attribute is overwritten $k$ times and the model is queried for only the most recent value. (B) Two attribute conditions are tested in parallel – semantic (word-type) attributes as the main condition and numeric attributes as a control – across interference levels from $k=1$ to $k=96$ , with exact-match accuracy and paired statistical tests (Fisher’s exact, Mantel–Haenszel, mixed-effects logistic regression) computed at each level.

Model	FP16	INT8	INT4
Qwen2.5-7B-Instruct	97.6%	97.6%	96.5%
Mistral-7B-Instruct-v0.3	75.6%	74.4%	72.4%
Phi-3.5-mini-instruct	57.2%	53.0%	59.5%

Table 1: Overall accuracy (all attributes and interference levels pooled) by model and quantization.

Figure 2: Word-type (semantic interference) accuracy vs. interference level, by model and quantization precision. Shaded region / error bars: omitted for clarity in the line plot; see Table 2 for exact accuracy and significance at each model’s key level.

Model	Level	FP16	INT4	$n_{10}$ / $n_{01}$	OR	$p$ (McNemar)
Qwen2.5-7B-Instruct	64	81.0%	68.3%	40 / 2	20.00	$4.1\times 10^{-10}$
Mistral-7B-Instruct-v0.3	16	44.0%	37.4%	129 / 50	2.58	$5.5\times 10^{-9}$
Phi-3.5-mini-instruct	8	61.8%	54.0%	54 / 15	3.60	$2.6\times 10^{-6}$

Table 2: FP16 vs. INT4 accuracy at each model’s most-powered high-interference level (word-type attributes), with McNemar’s paired test.

n_{10}

= trials correct under FP16 but not INT4;

n_{01}

= the reverse; OR

=n_{10}/n_{01}

n

= paired trials (pooled across 5 seeds). All three models show the same direction of effect in every individual seed (15/15 seed-comparisons). The unpaired Fisher’s exact test on the same cells agrees in direction and significance but is markedly less powered (

p=0.0005

0.0012

0.015

respectively), as expected once the trial-level pairing is discarded.

Model	Level	FP16	INT8	$n_{10}$ / $n_{01}$	OR	$p$ (McNemar)
Qwen2.5-7B-Instruct	64	81.0%	80.0%	10 / 7	1.43	$0.629$
Mistral-7B-Instruct-v0.3	16	44.0%	42.2%	46 / 25	1.84	$0.018$
Phi-3.5-mini-instruct	8	61.8%	57.6%	35 / 14	2.50	$0.004$

Table 3: FP16 vs. INT8 accuracy at the same levels as Table 2 (word-type attributes), McNemar’s paired test. The unpaired Fisher’s exact test at these same cells gives

p=0.837

0.410

0.197

respectively (all n.s.) – the paired test recovers a real effect in two of the three models that the unpaired test misses.

Model	Levels tested	FP16	INT8	INT4	INT4 % IES lost
Qwen2.5-7B-Instruct	1–64 (7 levels)	0.975	0.972	0.957	1.82%
Mistral-7B-Instruct-v0.3	1–32 (6 levels)	0.698	0.680	0.662	5.09%
Phi-3.5-mini-instruct	1–16 (5 levels)	0.820	0.791	0.777	5.16%

Table 4: Interference Endurance Score (IES; area under the accuracy-vs-log 2 -interference-level curve, normalized to

[0,1]

) by model and quantization, word-type attributes. % lost is relative to FP16. Tested interference-level ranges differ by model due to the vocabulary-size cap in Table 7 .

Figure 3: Accuracy vs. log 2 -scaled interference level, FP16/INT8/INT4 overlaid per model – the curve underlying the IES computation in Table 4 . Illustrates the cliff-dilution effect discussed in the text: Qwen’s near-flat curve through level 32 versus Mistral’s and Phi’s more gradual declines.

Term	Coef. ( $\beta$ )	OR	95% CI	$z$ ( $p$ )
Word-type attributes (main hypothesis)
int8 (main effect)	$-0.256$	0.77	[0.73, 0.82]	$-7.93$ ( $2.2\times 10^{-15}$ )
int4 (main effect)	$-0.143$	0.87	[0.81, 0.92]	$-4.44$ ( $9.1\times 10^{-6}$ )
int8 $\times\log_{2}(\text{level})$	$+0.023$	1.02	[1.01, 1.04]	$2.73$ ( $0.0063$ )
int4 $\times\log_{2}(\text{level})$	$\mathbf{-0.046}$	0.96	[0.94, 0.97]	$\mathbf{-5.43}$ ( $\mathbf{5.7\times 10^{-8}}$ )
Numeric attributes (specificity control)

Table 5: Mixed-effects logistic regression:

\text{correct}\sim\text{quant\_mode}*\log_{2}(\text{level})+(1\mid\text{model})+(1\mid\text{seed})+(1\mid\text{attribute})

. FP16 is the reference level. The key row in each panel is the INT4

\times

log-interference-level interaction: negative for word-type, positive for numeric – opposite signs, both highly significant, even after accounting for attribute-level difficulty heterogeneity via the added random intercept.

Figure 4: Numeric-attribute (control) accuracy vs. interference level, by model and quantization precision. Baseline interference is present, but no consistent quantization effect emerges at any level, in contrast to Figure 2 .

Figure 5: Accuracy at each model’s key high-interference level, word-type attributes only, with 95% confidence intervals. Brackets denote significance via McNemar’s paired test (our primary test, Section 3.3 ): the lower bracket compares FP16 vs. INT8, the upper bracket FP16 vs. INT4. ***

p<0.001

, **

p<0.01

, *

p<0.05

, n.s. not significant. Note the INT8 bracket is now significant for Mistral and Phi – the paired test recovers an effect the unpaired Fisher’s test (used in an earlier version of this analysis) missed.

Figure 6: Same-key intrusion rate (fraction of all word-type trials whose wrong answer exactly matches an earlier overwritten value), by quantization precision, pooled across all three models, seeds, attributes, and interference levels. Error bars: 95% CI. Bracket: McNemar’s paired test, FP16 vs. INT4 (

p=4.8\times 10^{-7}

Model	Level	$n$	INT4 (backbone)	INT4 (+lm_head)	lm_head share of gap	$p$
Qwen2.5-7B-Instruct	64	300	68.3%	68.0%	2.5%	$1.0$
Mistral-7B-Instruct-v0.3	16	1,200	37.4%	37.3%	1.3%	$1.0$
Phi-3.5-mini-instruct	8	500	54.0%	54.0%	0.0%	$1.0$

Table 6: Effect of additionally quantizing lm_head on top of the backbone-only INT4 condition, word-type attributes, at each model’s key interference level (same levels as Table 2 ). “INT4 (backbone)” is the condition used throughout the paper; “INT4 (+lm_head)” is the corrected ablation condition with lm_head also quantized.

p

-values are from McNemar’s paired test comparing the two conditions directly; none reach significance.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	mood	favorite color	favorite animal	occupation
Qwen2.5-7B-Instruct	64	32	32	32
Mistral-7B-Instruct-v0.3	32	16	16	16
Phi-3.5-mini-instruct	16	16	8	16

Appendix

Table 7: Maximum interference level tested per word-type attribute, by model (limited by tokenizer-filtered single-token vocabulary size).

Setting	Value
GPU	NVIDIA RTX 3090 (24 GB VRAM)
Device map	{"":0}
Low CPU memory usage	True
INT8 configuration
BitsAndBytes	load_in_8bit=True
LLM.int8 outlier threshold	6.0 (default)

Appendix

Table 8: Model loading and quantization configuration used in all experiments.

Level $k$	1	2	4	8	16	32	64	96
Trials	15	15	20	25	60	50	60	60

Appendix

Table 9: Trials per interference level, per (model, quantization, seed, attribute) cell.

	FP16	INT8	INT4
Accuracy	37.1%	30.7%	35.3%

Appendix

Table 10: Phi-3.5-mini-instruct at

k=16

: the one condition in our data where INT8 accuracy falls below INT4,

n=300

per cell.

Model	INT8 % IES lost	INT4 % IES lost
Qwen2.5-7B-Instruct	$-0.07\%$	$0.05\%$
Mistral-7B-Instruct-v0.3	$0.76\%$	$2.60\%$
Phi-3.5-mini-instruct	$5.49\%$	$-3.98\%$

Appendix

Table 11: INT8/INT4 IES penalty (% lost relative to FP16) on numeric (control) attributes, for comparison with the word-type results in Table 4 .

Explore similar work

Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels
May 2, 2026Plawan Kumar Rath, Rahul MaliakkalLLM Quantization Social Bias in Language Models
Meta
Quantization Degradation in Large Language Models: A Signal-Noise Perspective
Aug 8, 2026Chenxi Zhou, Pengfei Cao, Jinyu Ye +5LLM Quantization Weight-Only Quantization
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China +1
Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents
Jul 29, 2026Jiwon Jang, Kisu Yang, Heuiseok Lim +1AI Agent Reliability Benchmark Design
VAIV Company · Seoul National University · Korea University

May 2, 2026J/K move · Enter open · S save

Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels

Plawan Kumar Rath, Rahul Maliakkal

Abstract

Explore similar work

Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels

Quantization Degradation in Large Language Models: A Signal-Noise Perspective

Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents