cs.LGNov 8, 2026

When Calibration Data Matter in Sub-4-Bit Language Model Quantization

Authors: Yehong Jiang, Yen-Kuang Chen, Xinmin Tian

Organizations: Massachusetts Institute of Technology · Intel Corporation

Abstract

Low-bit post-training quantization reduces language-model memory but can sharply degrade downstream task accuracy. Calibration usually relies on generic text, but when task-informed data recover accuracy and which quantizers benefit remain poorly understood. We introduce FragCal-DC, which supplements generic text with training documents from task families with low retention of full-precision accuracy in a two-bit scan. We use a two- to four-bit sweep to locate functional regimes and four-seed calibration comparisons to evaluate six data-dependent methods across six dense models (1.7B-8B) and ten tasks. At their lowest tested functional bit widths, iterative SignRound V2 and OmniQuant improve at two bits, whereas single-pass GPTQ and AWQ show small mean responses at three bits. At two bits, SignRound V2 gains 1.83 percentage points in mean ten-task accuracy over generic-text calibration across six models. At 70 billion parameters, GPTQ and SignRound V2 both improve at two bits, with functional generic-calibration baselines. For two-bit SignRound V2, task-level controls show that recovery concentrates on families included in calibration, while incorrect answer designations retain about 90% of the mean accuracy gain. Changes in reconstruction error do not consistently track accuracy gains. These findings support task-informed calibration as a design principle for preserving downstream accuracy in low-bit quantization.

Figures & tables

Appendix figures & tables54 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Target-Aware Calibration Data Selection for Preserving Uncertainty in Quantized Language Models

    Aug 21, 2026Zhen Yang, Sizai Hou, Kaiwen Zheng +4LLM QuantizationUncertainty Calibration

  2. Beyond 4 bits: Data-Centric Post-training Quantization and Recovery

    Oct 20, 2026Rhea Sanjay Sukthanker, Riccardo Grazzi, James Hensman +1LLM QuantizationPost-Training Quantization