Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs

Shayan Shahrabi-Farahani; Dara Rahmati

cs.CLAug 19, 2026

Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs

Authors: Shayan Shahrabi-Farahani, Dara Rahmati

Organizations: Shahid Beheshti University · Shahid Beheshti University, Tehran, Iran

Abstract

Proactive interference (PI) is a documented failure mode in large language models in which retrieval of a repeatedly overwritten value degrades as prior overwrites accumulate, mirroring a classical phenomenon in human working memory. Post-training quantization (PTQ) is now the default deployment path for open-weight models, yet its effect on this failure mode has not been tested. We evaluate three precision levels (FP16, INT8, INT4/NF4, via bitsandbytes) across three architecturally distinct instruction-tuned models (Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, Phi-3.5-mini-instruct), holding the retrieval task fixed. INT4 quantization significantly reduces accuracy under high interference in every model (e.g., from 81.0% to 68.3% for Qwen), confirmed by paired McNemar's tests ( $p \le 2.6 \times 10^{-6}$ ) and a mixed-effects regression spanning all interference levels; INT8, often assumed safe, also carries a smaller but real penalty in two of three models. The effect is specific to semantically similar (word-type) distractors and reverses sign under a numeric control condition, and is mechanistically linked to a rise in same-key intrusion errors under INT4 (from 21.5% to 24.6% of trials, $p = 4.8 \times 10^{-7}$ ). A follow-up ablation shows the effect originates in the quantized transformer backbone rather than the output projection layer. These results suggest that bitsandbytes 4-bit quantization can impose an additional cost on applications relying on long, updatable, semantically dense contexts, even when aggregate benchmark accuracy appears largely unaffected. We release our code and tokenizer-verified vocabulary construction method at https://github.com/ShayanShahrabi/compress-and-forget

Figures & tables

Figure 1: Overview of the experimental pipeline and evaluation design. (A) Each of three instruction-tuned models is loaded at three precisions (FP16, INT8, INT4/NF4 via bitsandbytes) and evaluated on the PI-LLM retrieval benchmark, in which a subject’s attribute is overwritten $k$ times and the model is queried for only the most recent value. (B) Two attribute conditions are tested in parallel – semantic (word-type) attributes as the main condition and numeric attributes as a control – across interference levels from $k=1$ to $k=96$ , with exact-match accuracy and paired statistical tests (Fisher’s exact, Mantel–Haenszel, mixed-effects logistic regression) computed at each level.

Model	FP16	INT8	INT4
Qwen2.5-7B-Instruct	97.6%	97.6%	96.5%
Mistral-7B-Instruct-v0.3	75.6%	74.4%	72.4%
Phi-3.5-mini-instruct	57.2%	53.0%	59.5%

Table 1: Overall accuracy (all attributes and interference levels pooled) by model and quantization.

Figure 2: Word-type (semantic interference) accuracy vs. interference level, by model and quantization precision. Shaded region / error bars: omitted for clarity in the line plot; see Table 2 for exact accuracy and significance at each model’s key level.

Model	Level	FP16	INT4	$n_{10}$ / $n_{01}$	OR	$p$ (McNemar)
Qwen2.5-7B-Instruct	64	81.0%	68.3%	40 / 2	20.00	$4.1\times 10^{-10}$
Mistral-7B-Instruct-v0.3	16	44.0%	37.4%	129 / 50	2.58	$5.5\times 10^{-9}$
Phi-3.5-mini-instruct	8	61.8%	54.0%	54 / 15	3.60	$2.6\times 10^{-6}$

Table 2: FP16 vs. INT4 accuracy at each model’s most-powered high-interference level (word-type attributes), with McNemar’s paired test.

n_{10}

= trials correct under FP16 but not INT4;

n_{01}

= the reverse; OR

=n_{10}/n_{01}

n

= paired trials (pooled across 5 seeds). All three models show the same direction of effect in every individual seed (15/15 seed-comparisons). The unpaired Fisher’s exact test on the same cells agrees in direction and significance but is markedly less powered (

p=0.0005

0.0012

0.015

respectively), as expected once the trial-level pairing is discarded.

Model	Level	FP16	INT8	$n_{10}$ / $n_{01}$	OR	$p$ (McNemar)
Qwen2.5-7B-Instruct	64	81.0%	80.0%	10 / 7	1.43	$0.629$
Mistral-7B-Instruct-v0.3	16	44.0%	42.2%	46 / 25	1.84	$0.018$
Phi-3.5-mini-instruct	8	61.8%	57.6%	35 / 14	2.50	$0.004$

Table 3: FP16 vs. INT8 accuracy at the same levels as Table 2 (word-type attributes), McNemar’s paired test. The unpaired Fisher’s exact test at these same cells gives

p=0.837

0.410

0.197

respectively (all n.s.) – the paired test recovers a real effect in two of the three models that the unpaired test misses.

Model	Levels tested	FP16	INT8	INT4	INT4 % IES lost
Qwen2.5-7B-Instruct	1–64 (7 levels)	0.975	0.972	0.957	1.82%
Mistral-7B-Instruct-v0.3	1–32 (6 levels)	0.698	0.680	0.662	5.09%
Phi-3.5-mini-instruct	1–16 (5 levels)	0.820	0.791	0.777	5.16%

Table 4: Interference Endurance Score (IES; area under the accuracy-vs-log 2 -interference-level curve, normalized to

[0,1]

) by model and quantization, word-type attributes. % lost is relative to FP16. Tested interference-level ranges differ by model due to the vocabulary-size cap in Table 7 .

Figure 3: Accuracy vs. log 2 -scaled interference level, FP16/INT8/INT4 overlaid per model – the curve underlying the IES computation in Table 4 . Illustrates the cliff-dilution effect discussed in the text: Qwen’s near-flat curve through level 32 versus Mistral’s and Phi’s more gradual declines.

Term	Coef. ( $\beta$ )	OR	95% CI	$z$ ( $p$ )
Word-type attributes (main hypothesis)
int8 (main effect)	$-0.256$	0.77	[0.73, 0.82]	$-7.93$ ( $2.2\times 10^{-15}$ )
int4 (main effect)	$-0.143$	0.87	[0.81, 0.92]	$-4.44$ ( $9.1\times 10^{-6}$ )
int8 $\times\log_{2}(\text{level})$	$+0.023$	1.02	[1.01, 1.04]	$2.73$ ( $0.0063$ )
int4 $\times\log_{2}(\text{level})$	$\mathbf{-0.046}$	0.96	[0.94, 0.97]	$\mathbf{-5.43}$ ( $\mathbf{5.7\times 10^{-8}}$ )
Numeric attributes (specificity control)

Table 5: Mixed-effects logistic regression:

\text{correct}\sim\text{quant\_mode}*\log_{2}(\text{level})+(1\mid\text{model})+(1\mid\text{seed})+(1\mid\text{attribute})

. FP16 is the reference level. The key row in each panel is the INT4

\times

log-interference-level interaction: negative for word-type, positive for numeric – opposite signs, both highly significant, even after accounting for attribute-level difficulty heterogeneity via the added random intercept.

Figure 4: Numeric-attribute (control) accuracy vs. interference level, by model and quantization precision. Baseline interference is present, but no consistent quantization effect emerges at any level, in contrast to Figure 2 .

Figure 5: Accuracy at each model’s key high-interference level, word-type attributes only, with 95% confidence intervals. Brackets denote significance via McNemar’s paired test (our primary test, Section 3.3 ): the lower bracket compares FP16 vs. INT8, the upper bracket FP16 vs. INT4. ***

p<0.001

, **

p<0.01

, *

p<0.05

, n.s. not significant. Note the INT8 bracket is now significant for Mistral and Phi – the paired test recovers an effect the unpaired Fisher’s test (used in an earlier version of this analysis) missed.

Figure 6: Same-key intrusion rate (fraction of all word-type trials whose wrong answer exactly matches an earlier overwritten value), by quantization precision, pooled across all three models, seeds, attributes, and interference levels. Error bars: 95% CI. Bracket: McNemar’s paired test, FP16 vs. INT4 (

p=4.8\times 10^{-7}

Model	Level	$n$	INT4 (backbone)	INT4 (+lm_head)	lm_head share of gap	$p$
Qwen2.5-7B-Instruct	64	300	68.3%	68.0%	2.5%	$1.0$
Mistral-7B-Instruct-v0.3	16	1,200	37.4%	37.3%	1.3%	$1.0$
Phi-3.5-mini-instruct	8	500	54.0%	54.0%	0.0%	$1.0$

Table 6: Effect of additionally quantizing lm_head on top of the backbone-only INT4 condition, word-type attributes, at each model’s key interference level (same levels as Table 2 ). “INT4 (backbone)” is the condition used throughout the paper; “INT4 (+lm_head)” is the corrected ablation condition with lm_head also quantized.

p

-values are from McNemar’s paired test comparing the two conditions directly; none reach significance.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	mood	favorite color	favorite animal	occupation
Qwen2.5-7B-Instruct	64	32	32	32
Mistral-7B-Instruct-v0.3	32	16	16	16
Phi-3.5-mini-instruct	16	16	8	16

Appendix

Table 7: Maximum interference level tested per word-type attribute, by model (limited by tokenizer-filtered single-token vocabulary size).

Setting	Value
GPU	NVIDIA RTX 3090 (24 GB VRAM)
Device map	{"":0}
Low CPU memory usage	True
INT8 configuration
BitsAndBytes	load_in_8bit=True
LLM.int8 outlier threshold	6.0 (default)

Appendix

Table 8: Model loading and quantization configuration used in all experiments.

Level $k$	1	2	4	8	16	32	64	96
Trials	15	15	20	25	60	50	60	60

Appendix

Table 9: Trials per interference level, per (model, quantization, seed, attribute) cell.

	FP16	INT8	INT4
Accuracy	37.1%	30.7%	35.3%

Appendix

Table 10: Phi-3.5-mini-instruct at

k=16

: the one condition in our data where INT8 accuracy falls below INT4,

n=300

per cell.

Model	INT8 % IES lost	INT4 % IES lost
Qwen2.5-7B-Instruct	$-0.07\%$	$0.05\%$
Mistral-7B-Instruct-v0.3	$0.76\%$	$2.60\%$
Phi-3.5-mini-instruct	$5.49\%$	$-3.98\%$

Appendix

Table 11: INT8/INT4 IES penalty (% lost relative to FP16) on numeric (control) attributes, for comparison with the word-type results in Table 4 .

Explore similar work

May 2, 2026

Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels

Large Language Models are routinely compressed via post-training quantization to reduce inference costs and memory footprint for cloud and edge deployment, yet the impact of this compression on model quality remains poorly understood. Existing studies typically compare only two conditions (full-precision vs. a single quantized variant), rely on aggregate bias metrics, and evaluate a single model family, making it impossible to distinguish gradual degradation from threshold-dependent safety failures. We conduct a controlled empirical study of three instruction-tuned models (Qwen2.5-7B, Mistral-7B, Phi-3.5-mini) at five precision levels (BF16 through 3-bit) on 12,148 BBQ bias benchmark items across 5 random seeds, totaling 911,100 inference records. Our results reveal that 3-bit quantization causes 6-21% of previously unbiased items to develop new stereotypical behaviors, following a clear dose-response pattern confirmed via logistic regression, while models' willingness to select "unknown" answers declines by 17.4%. Crucially, these item-level changes are invisible to standard quality metrics: perplexity increases by less than 0.5% at 8-bit and under 3% at 4-bit across all three models, yet 2.5-5.6% of items already develop new biases at 4-bit. These findings demonstrate that aggregate evaluation metrics systematically miss fairness-critical degradation, underscoring the need for quality-aware compression protocols that explicitly test for bias emergence before deployment.

Plawan Kumar Rath, Rahul Maliakkal

Quantization Degradation in Large Language Models: A Signal-Noise Perspective

Post-training quantization reduces the deployment cost of large language models, yet how severely a quantized model degrades is not determined by bit-width alone. We systematically study weight-only post-training quantization across bit-widths, quantization methods, model scales and downstream tasks on multiple model families. We observe that such degradation varies substantially across these factors: 4-bit quantization usually preserves performance, 2-bit often causes broad degradation, and at 3-bit, degradation becomes apparent but varies markedly with task type, quantization method and model scale. To explain this variability, we use the signal-to-noise ratio (SNR) to measure how strongly quantization perturbs full-precision representations. We trace degradation back to two linked processes: how quantization errors arise within individual modules, and how they accumulate across layers. First, a source SNR decomposition shows that newly introduced errors depend on three factors: the magnitude of the weight error, the strength of the task-specific signal, and how strongly the quantization error aligns with task-specific activations. Different factors affect these components in distinct ways. Second, a cross-layer propagation analysis shows that these errors can be attenuated, preserved, or amplified as they pass across layers, and that larger models benefit from weaker error amplification. Together, these results establish that quantization degradation is governed by how errors are introduced at the source and how they accumulate across the network.

Chenxi Zhou, Pengfei Cao, Jinyu Ye +5

School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China +1

LLM Quantization Weight-Only Quantization Post-Training Quantization Neural Network Quantization

Ask PDF

Jul 29, 2026

Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents

Post-training quantization to 4-bit weights is widely reported to be nearly lossless. We test this claim for multi-turn, tool-calling agents, where it now matters most. On

τ^2

-bench, across two open-weight model families in dense and MoE variants and two domains (eight cells, 456 episodes each, at 16-, 8-, and 4-bit weights), quantization indeed looks free on the standard metric. No cell shows a score change that survives multiple-comparison correction, and in the cell that carries the largest process damage, equivalence testing bounds the change within

\pm

7.5 points. The process tells a different story. Quantization amplifies the failure the model already exhibits at full precision (tool-name hallucination in telecom, with the same directional trend in retail entity errors) by up to 2.5

\times

in volume (+17.6 points per task), while creating essentially no new failures. The failure set is the same at every precision (rank correlation

\geq

0.94, 0.18% novel events). The score stays flat because the benchmark's ten-error budget absorbs the extra failures. Shrinking the budget to two errors re-exposes a score gap of 17 points, and it does so only in the one cell where quantization added error volume, exactly as the masking account predicts. A targeted error-repair prompt, run for five telecom models at every precision, removes the damage exactly and only where it lives. Both diagnostics, the per-channel error rate and success under a shrinking budget, come from logs benchmarks already collect; we suggest reporting them alongside task reward.

Jiwon Jang, Kisu Yang, Heuiseok Lim +1

VAIV Company · Seoul National University · Korea University

AI Agent Reliability Benchmark Design 4-Bit Quantization AI Agent Benchmarks LLM Tool Use Post-Training Quantization+1

Ask PDF