Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs

Saurav Bhandari; Benjamin Wade

cs.LGSep 6, 2026

Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs

Authors: Saurav Bhandari, Benjamin Wade

Organizations: Department of Electronics and Computer Engineering Institute of Engineering, Thapathali Campus Tribhuvan University, Kathmandu, Nepal · Independent Researcher

Abstract

Inference-time activation steering enables behavioral control of large language models without parameter modification, while post-training quantization reduces memory and compute costs for deployment. Despite their growing convergence in practice, the interaction between these two techniques remains uncharacterized. We systematically study activation steering under weight-only quantization (INT8 and NF4) across four open-weight 7-9B models and two behavioral targets: judged sentiment and judge-free reasoning length. Using an iso-effect framework that compares capability costs at matched behavioral effect, we find that sentiment steering survives quantization intact. After correcting a GSM8K parser artifact with a uniform v2.3.1 rescore, the pooled INT8 contrast is -0.010 (90% CI [-0.026, +0.007]), descriptively Equivalent under the preregistered three-label rule, while NF4 remains Inconclusive at -0.017 ([-0.067, +0.033]). In contrast, reasoning length exhibits a surprising asymmetric dose-response: lengthening is graded but terminates in cap-runaway and collapse, while shortening is a step function with only 12-30% shortening (model-dependent) before discontinuous failure. We expose a methodological pitfall: the naive iso-effect ladder anchors on the collapse floor for floor-bounded targets, and we introduce a censored construction that restores interpretable crossings. We also quantify a substantial baseline capability shift for Mistral-NF4 (0.545 to 0.365 GSM8K at alpha=0), demonstrating that compression can dominate the steering intervention. Despite this, steering vectors remain highly collinear with their FP16 siblings (cosine similarity 0.989-0.998 for INT8, 0.945-0.990 for NF4), confirming that the behavioral direction survives quantization even when the cost structure does not. All code and data are released.

Figures & tables

Figure 1: Experimental pipeline: per-model FP16, INT8 and NF4 evaluation with per-precision steering vectors (5 resamples). Behavioral dose-response for sentiment (RoBERTa) and length (survivor-median). Capability at matched effect via GSM8K (N=200, CoT) with MMLU as secondary. The current v1 analysis reports cross-model sentiment results descriptively at $k=3$ models and length results descriptively at $k=4$ models; the confirmatory $K=12$ analysis is deferred.

Model	Params	Layer $L$	Targets	Residual norm
Qwen2.5-7B-Instruct	7.6B	14 (of 28)	sentiment, length	267.1
Llama-3.1-8B-Instruct	8.0B	16 (of 32)	sentiment, length	9.4
Mistral-7B-Instruct-v0.3	7.2B	16 (of 32)	sentiment, length	10.4
Gemma-2-9B-it	9.2B	21 (of 42)	length only	403.3

Table 1: Design matrix. Residual norm is the mean

\ell_{2}

norm of the residual stream at the steering layer, averaged over the collected length-target runs per model.

Model	INT8 contrast [90% CI]	NF4 contrast [90% CI]
Qwen2.5-7B	$-0.011$ $[-0.039,+0.016]$	$+0.014$ $[-0.016,+0.043]$
Llama-3.1-8B	$+0.012$ $[-0.039,+0.063]$	$+0.012$ $[-0.055,+0.079]$
Mistral-7B-v0.3	$-0.024$ $[-0.070,+0.022]$	$-0.080$ $[-0.135,-0.024]$
Pooled (3 models)	$-0.010$ $[-0.026,+0.007]$	$-0.017$ $[-0.067,+0.033]$

Table 2: Sentiment IECC contrasts,

\mathrm{IECC}(\text{scheme})-\mathrm{IECC}(\text{FP16})

, Option C run-level combine (REML

\tau^{2}

+ modified Knapp–Hartung; 90% CI;

k=5

runs). Per-model labels are all Inconclusive; the pooled INT8 interval lies entirely within

\pm\delta

(descriptive Equivalent).

Figure 2: Sentiment iso-effect capability cost contrasts versus FP16 (90% CIs; diamonds are cross-model pooled estimates; shaded band is the preregistered equivalence margin

\pm\delta=0.03

). Negative values mean the quantized scheme is cheaper than FP16 at matched effect. Per-model contrasts are Inconclusive; the pooled INT8 interval lies entirely within the margin, meeting the Equivalent label at the descriptive level.

Figure 3: Length dose–response (top: survivor-median generated tokens; bottom: termination-failure rate) for four models

\times

three schemes, averaged over five resamples. Solid segments lie in the clean region (failure

\leq 10\%

); hollow markers are survivor-biased points beyond the censoring threshold (dashed line, bottom panels). The asymmetry is model-general: graded lengthening into cap-runaway on the positive arm; a shallow shortening plateau then discontinuous collapse to empty generations on the negative arm. Quantization (color) preserves the structure throughout.

Figure 4: Anatomy of the failure and the repair (Qwen2.5-7B FP16, resample 5, positive arm). As coded, the ladder’s extreme is the collapse floor, so

E^{\ast}(0.5)

falls below baseline and its crossing is an interpolation across the cliff between the last surviving and first all-fail alpha. The censored construction anchors on the maximum surviving effect, placing the level and crossing in the clean region.

Figure 5: Unsteered (

\alpha=0

) GSM8K accuracy by model and scheme (means over five resamples, length campaign). Three models hold within four points across schemes; Mistral loses eighteen points under NF4 before any steering is applied. The shift replicates in the sentiment campaign.

Activation Steering

Activation steering changes model behavior by intervening directly on internal representations during inference, often without updating weights.

See all 376 papers

May 2, 2026

Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels

Large Language Models are routinely compressed via post-training quantization to reduce inference costs and memory footprint for cloud and edge deployment, yet the impact of this compression on model quality remains poorly understood. Existing studies typically compare only two conditions (full-precision vs. a single quantized variant), rely on aggregate bias metrics, and evaluate a single model family, making it impossible to distinguish gradual degradation from threshold-dependent safety failures. We conduct a controlled empirical study of three instruction-tuned models (Qwen2.5-7B, Mistral-7B, Phi-3.5-mini) at five precision levels (BF16 through 3-bit) on 12,148 BBQ bias benchmark items across 5 random seeds, totaling 911,100 inference records. Our results reveal that 3-bit quantization causes 6-21% of previously unbiased items to develop new stereotypical behaviors, following a clear dose-response pattern confirmed via logistic regression, while models' willingness to select "unknown" answers declines by 17.4%. Crucially, these item-level changes are invisible to standard quality metrics: perplexity increases by less than 0.5% at 8-bit and under 3% at 4-bit across all three models, yet 2.5-5.6% of items already develop new biases at 4-bit. These findings demonstrate that aggregate evaluation metrics systematically miss fairness-critical degradation, underscoring the need for quality-aware compression protocols that explicitly test for bias emergence before deployment.

Plawan Kumar Rath, Rahul Maliakkal

The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs

Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity. We show that these metrics fail to capture behavioral changes induced by quantization. We introduce correctness agreement, a decision-level metric that measures overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy. Across multiple models and quantization schemes from 8-bit to 2-bit, we find that behavioral divergence emerges under moderate quantization even when task performance appears preserved. To explain this effect, we analyze quantization as a structural operator on attention weights and quantify layer-wise distortions using statistical and distributional measures. Our results reveal non-linear breakpoints at low bit-widths and show that query and key projections are consistently more sensitive than value and output projections. These findings expose an illusion of equivalence between base and quantized models and motivate behavioral evaluation beyond conventional performance metrics.

Baha Rababah, Cuneyt Gurcan Akcora, Carson K. Leung

Department of Computer Science, University of Manitoba, Canada · AI Initiative, University of Central Florida, USA

LLM Evaluation Quantization Sensitivity Analysis Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Ask PDF

Aug 19, 2026

Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs

Proactive interference (PI) is a documented failure mode in large language models in which retrieval of a repeatedly overwritten value degrades as prior overwrites accumulate, mirroring a classical phenomenon in human working memory. Post-training quantization (PTQ) is now the default deployment path for open-weight models, yet its effect on this failure mode has not been tested. We evaluate three precision levels (FP16, INT8, INT4/NF4, via bitsandbytes) across three architecturally distinct instruction-tuned models (Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, Phi-3.5-mini-instruct), holding the retrieval task fixed. INT4 quantization significantly reduces accuracy under high interference in every model (e.g., from 81.0% to 68.3% for Qwen), confirmed by paired McNemar's tests (

p \le 2.6 \times 10^{-6}

) and a mixed-effects regression spanning all interference levels; INT8, often assumed safe, also carries a smaller but real penalty in two of three models. The effect is specific to semantically similar (word-type) distractors and reverses sign under a numeric control condition, and is mechanistically linked to a rise in same-key intrusion errors under INT4 (from 21.5% to 24.6% of trials,

p = 4.8 \times 10^{-7}

). A follow-up ablation shows the effect originates in the quantized transformer backbone rather than the output projection layer. These results suggest that bitsandbytes 4-bit quantization can impose an additional cost on applications relying on long, updatable, semantically dense contexts, even when aggregate benchmark accuracy appears largely unaffected. We release our code and tokenizer-verified vocabulary construction method at https://github.com/ShayanShahrabi/compress-and-forget

Shayan Shahrabi-Farahani, Dara Rahmati

Shahid Beheshti University · Shahid Beheshti University, Tehran, Iran

LLM Quantization Post-Training Quantization Low-Bit Quantization Knowledge Conflicts in Language Models