cs.LGNov 17, 2026

TempQuant: Learning Adaptive Temperatures for Quantized Language Models

Authors: Mukund Athmaram, Liang Ma, Manoj Rao

Organizations: AMD · Cornell University

Abstract

Large language models are increasingly deployed at low precision, yet quantization can degrade their performance. We study how quantization interacts with decoding settings such as the sampling temperature and top-p by examining how it distorts the model's next-token probabilities. Measuring entropy during generation across deployed temperatures, we find that quantization warms the model on average, and that temperature matters much more for the performance of a quantized model than for its full-precision counterpart. Because quantization and temperature both shape the next-token distribution, we investigate how much the sampling settings can undo what quantization changes, through our method TempQuant, which learns the quantized model's temperature and top-p to bring its next-token probabilities closer to the full-precision model's. We learn these settings at two granularities. The first is a single global setting per requested temperature, which finds nearly the same temperatures that tuning on labeled answers would, without any labels. Model cards typically recommend sampling a quantized model with the same settings as its full-precision counterpart, while we find that different settings can substantially improve its accuracy. On 3-bit Qwen3-4B, for example, global TempQuant raises MATH accuracy from 34% under the configuration recommended for Qwen's quantized releases to 51%. The second sets the temperature and top-p for every token, motivated by our finding that tokens whose rankings flip under quantization tend to need warming, while tokens whose rankings are preserved tend to need cooling. When quantization degrades accuracy substantially, TempQuant recovers much of it, and when degradation is lighter, it still brings the quantized model's behavior closer to the full-precision model's.

Figures & tables

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. OnlineQAT: On-Policy Distillation for Ultra-Low-Bit Large Language Models

    Oct 7, 2026Wenjun Wang, Heng Li, Yanggan Gu +1LLM QuantizationQuantization

  2. Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories

    Oct 21, 2026Boyuan Zheng, Jing Xiong, Chenwei Wu +8LLM QuantizationQuantization-Aware Training