cs.LGSep 2, 2026

The Effect of Quantization on Clinical Benchmarks: Accuracy and Safety Across Model Families

Authors: Leonard Twagirayezu, Prasenjit Mitra

Organizations: Kigali, Rwanda

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Quantization enables deployment of large language models on resource-constrained clinical edge devices, but its effect on clinical accuracy and safety remains understudied. We evaluate five 7-8B parameter models at FP16, GPTQ-INT8, and GPTQ-INT4 precision across five benchmarks: MedQA, MedMCQA, Med-HALT, a risk-stratified sample of HealthBench, and MedSafetyBench. The study jointly varies quantization bit width, model family, and clinical task type, with explicit risk stratification and safety measures. INT8 GPTQ is universally safe (max. degradation -1.9%-1.9%), while INT4 degradation is substantial and model-dependent: BioMistral-7B, clinically fine-tuned, loses 19.7% on MedMCQA, more than any general-purpose model, showing clinical fine-tuning does not confer compression robustness. MedMCQA degrades more than MedQA under INT4; Med-HALT is largely unaffected. On HealthBench's emergency-risk subgroup, Qwen2.5-7B degrades by 26.8% under INT4, suggesting high-risk scenarios are disproportionately vulnerable to compression. On MedSafetyBench, the model family dominates over precision (refusal rates range 10.2%-74.9% at FP16), though Qwen2.5-7B (-17.8%) and Meditron-7B (-28.3%) show substantial INT4 safety degradation; notably, Qwen2.5-7B is simultaneously the most accuracy-robust model, demonstrating that accuracy and safety robustness are independent properties. We additionally test two recovery methods, clinical calibration substitution and QLoRA fine-tuning, both producing the same trade-off: MedMCQA recovers while MedQA further degrades, indicating recovery strategies require task-specific validation rather than being assumed universally beneficial. These findings indicate INT8 is broadly safe for clinical deployment, while INT4 safety must be assessed per-model and per-task, and that safety alignment is determined primarily by instruction tuning rather than clinical domain adaptation.

Figures & tables

Figure 1: Quantization degradation heatmap: Δ%\Delta\% from FP16 baseline across five models, two precision levels, and four benchmarks.
ModelBenchmarkFP16CI 95INT8Δ\Delta %INT4Δ\Delta %
Llama-3.1-8B  ∘\,\circMedQA0.533[0.51, 0.56]0.533+0.0+0.00.531−0.4-0.4
MedMCQA0.178[0.15, 0.20]0.176−1.1-1.10.158−11.2-11.2
Med-HALT0.529[0.50, 0.56]0.530+0.2+0.20.520−1.7-1.7
HealthBench0.494[0.44, 0.55]0.477−3.5-3.50.380−23.0-23.0
risk-wt-5 †0.471[0.25, 0.70]0.456−3.1-3.10.458−2.6-2.6
Qwen2.5-7B  ∘\,\circMedQA0.596[0.57, 0.62]0.596+0.0+0.00.589−1.2-1.2
Table 1: Accuracy and degradation ( Δ%\Delta\% from FP16) across five models, three precisions, and four accuracy benchmarks (MedQA, MedMCQA, Med-HALT, HealthBench). MedSafetyBench safety-refusal results are reported separately in Table 2 . CI 95 computed via binomial standard error for MCQ benchmarks and mean ±\pm SE for HealthBench. BioMistral-7B and Meditron-7B excluded from HealthBench. †\dagger risk-wt-5 = emergency-referral subgroup ( n≈19n{\approx}19 ). ∘\circ general-purpose; ∙\bullet clinical.
Figure 2: MedSafetyBench refusal rates across precision levels.
ModelFamilyFP16INT8Δ\Delta %INT4Δ\Delta %
Llama-3.1-8BGeneral0.7490.747−0.3-0.30.791+5.6∗+5.6^{*}
Qwen2.5-7BGeneral0.3730.376+0.6+0.60.307−17.8-17.8
BioMistral-7BClinical0.3090.300−2.9-2.90.316+2.2+2.2
Meditron-7B †Clinical0.1020.100−2.2-2.20.073−28.3-28.3
Med42-8BClinical0.6600.658−0.3-0.30.624−5.4-5.4
Table 2: MedSafetyBench refusal rates (safety scores) across five models and three precision levels. Δ%\Delta\% computed relative to FP16 baseline. ∗ Llama-3.1-8B INT4 gain flagged as possible classifier artifact; see Section Limitations and Future Work . † Meditron-7B’s refusal rate is likely modestly higher than reported due to a known classifier limitation; see Section Limitations and Future Work .
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

ModelBMFP16WikiMedW Δ\DeltaM Δ\Delta
Llama-3.1-8BMedQA0.5330.5310.269−0.4-0.4−49.6 -49.6\,
MedMCQA0.1780.1580.149−11.2-11.2−16.3 -16.3\,
Qwen2.5-7BMedQA0.5960.5890.594−1.2-1.2−0.4 -0.4\,
MedMCQA0.2970.2920.306−1.7-1.7+3.0 +3.0\,
BioMistral-7BMedQA0.4100.4130.391+0.8+0.8−4.6 -4.6\,
MedMCQA0.2130.1710.235−19.7-19.7+10.3 +10.3\,
Appendix
Table 3: Effect of clinical calibration data on INT4 GPTQ accuracy. BM = benchmark. Wiki = GPTQ-INT4-Wiki (WikiText-2 calibration, standard). Med = GPTQ-INT4-Med (128 MedQA training samples as calibration). W Δ\Delta and M Δ\Delta are Δ%\Delta\% from FP16 for Wiki-INT4 and Med-INT4 respectively.
ModelBMFP16I4QLI4 Δ\DeltaQL Δ\Delta
Llama-3.1-8BMedQA0.5330.5310.508−0.4-0.4−4.9-4.9
MedMCQA0.1780.1580.360−11.2-11.2+102.2+102.2
Qwen2.5-7BMedQA0.5960.5890.539−1.2-1.2−9.6-9.6
MedMCQA0.2970.2920.353−1.7-1.7+18.9+18.9
BioMistral-7BMedQA0.4100.4130.312+0.8+0.8−23.9-23.9
MedMCQA0.2130.1710.305−19.7-19.7+43.2+43.2
Appendix
Table 4: QLoRA fine-tuning recovery results. LoRA rank 16, 1,000 MedQA training examples, 3 epochs. BM = benchmark. I4 = INT4 (GPTQ, pre-recovery). QL = QLoRA fine-tuned. I4 Δ\Delta and QL Δ\Delta are Δ%\Delta\% from FP16 for INT4 and QLoRA respectively. Recovery = QLoRA −- INT4.
More in this topic

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 381 papers

Explore similar work

CardsList
May 2, 2026

Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels

Large Language Models are routinely compressed via post-training quantization to reduce inference costs and memory footprint for cloud and edge deployment, yet the impact of this compression on model quality remains poorly understood. Existing studies typically compare only two conditions (full-precision vs. a single quantized variant), rely on aggregate bias metrics, and evaluate a single model family, making it impossible to distinguish gradual degradation from threshold-dependent safety failures. We conduct a controlled empirical study of three instruction-tuned models (Qwen2.5-7B, Mistral-7B, Phi-3.5-mini) at five precision levels (BF16 through 3-bit) on 12,148 BBQ bias benchmark items across 5 random seeds, totaling 911,100 inference records. Our results reveal that 3-bit quantization causes 6-21% of previously unbiased items to develop new stereotypical behaviors, following a clear dose-response pattern confirmed via logistic regression, while models' willingness to select "unknown" answers declines by 17.4%. Crucially, these item-level changes are invisible to standard quality metrics: perplexity increases by less than 0.5% at 8-bit and under 3% at 4-bit across all three models, yet 2.5-5.6% of items already develop new biases at 4-bit. These findings demonstrate that aggregate evaluation metrics systematically miss fairness-critical degradation, underscoring the need for quality-aware compression protocols that explicitly test for bias emergence before deployment.

Plawan Kumar Rath, Rahul Maliakkal

Meta

LLM QuantizationSocial Bias in Language ModelsLanguage Model Bias EvaluationPost-Training Quantization
AskPDF
Oct 13, 2026

Detecting Unreliable Classifications from Quantized Models

Post-training INT8 quantization is typically validated by checking accuracy retention, which requires ground-truth labels often unavailable in low-resource clinical settings. We show a label-free alternative: comparing a model's own output before and after quantization. We compute cosine similarity between the FP32 and INT8 models' output probability vectors for each prediction, which requires only a single forward pass. This output-agreement similarity separates lower-accuracy predictions across four evaluated architectures (ResNet-50, VGG-16-BN, Wide ResNet-50, and ViT-B/16) under fixed-percentile splits, outperforming a confidence-only baseline for all four. These results show that quantized models can be compared to their baselines to assess their drift and determine whether they are well suited for clinical deployment.

Ananya Kotla, Jorge T. Gomez

Post-Training QuantizationClassification
AskPDF
Jun 28, 2026

The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysis

Modern LLM deployments often combine quantization with higher sampling temperatures to reduce cost, latency, or repetition, yet safety evaluations usually treat these as fixed implementation details. We test whether models that are safe at FP16 with greedy decoding remain safe after quantization and stochastic sampling, or whether the two factors amplify each other. We evaluate 8 instruction-tuned models from five families across 3 precisions and 6 temperatures, covering 144 configurations on 7 harmfulness benchmarks and generating about 2.0 million responses, which are scored by a six-judge safety ensemble. Contrary to concerns that low-bit deployment erodes alignment, we find that standard quantization is approximately safety-neutral: for 7 of 8 models, AWQ INT4 keeps attack success within about 1.6 percentage points of FP16 or lowers it, with clear degradation only for SmolLM3-3B (34.5% to 44.1%). However, the larger risk comes from sampling: higher temperatures sharply increase decision instability, with DFR reaching 41.9% at T = 1.0, even when average ASR changes only modestly. The two factors do not compound: our Compound Degradation Index remains sub-additive (-0.071 to +0.018), indicating that quantization partially offsets rather than amplifies temperature-induced degradation. Finally, a per-benchmark breakdown shows that single-benchmark evaluation badly understates risk: several models scoring 0% on AdvBench exceed 80% on ManyHarm. Standard INT4/INT8 quantization can therefore be reasonable for well-aligned models, but safety claims should report multi-sample stability across multiple benchmarks rather than rely on a single benchmark at greedy decoding.

Hari Prasad, Ritam Pal

Conscious Engines

LLM Safety BenchmarksLLM QuantizationLLM Safety AlignmentLLM Safety EvaluationLanguage Model Robustness
AskPDF