TempQuant: Learning Adaptive Temperatures for Quantized Language Models
Organizations: AMD · Cornell University
Abstract
Large language models are increasingly deployed at low precision, yet quantization can degrade their performance. We study how quantization interacts with decoding settings such as the sampling temperature and top-p by examining how it distorts the model's next-token probabilities. Measuring entropy during generation across deployed temperatures, we find that quantization warms the model on average, and that temperature matters much more for the performance of a quantized model than for its full-precision counterpart. Because quantization and temperature both shape the next-token distribution, we investigate how much the sampling settings can undo what quantization changes, through our method TempQuant, which learns the quantized model's temperature and top-p to bring its next-token probabilities closer to the full-precision model's. We learn these settings at two granularities. The first is a single global setting per requested temperature, which finds nearly the same temperatures that tuning on labeled answers would, without any labels. Model cards typically recommend sampling a quantized model with the same settings as its full-precision counterpart, while we find that different settings can substantially improve its accuracy. On 3-bit Qwen3-4B, for example, global TempQuant raises MATH accuracy from 34% under the configuration recommended for Qwen's quantized releases to 51%. The second sets the temperature and top-p for every token, motivated by our finding that tokens whose rankings flip under quantization tend to need warming, while tokens whose rankings are preserved tend to need cooling. When quantization degrades accuracy substantially, TempQuant recovers much of it, and when degradation is lighter, it still brings the quantized model's behavior closer to the full-precision model's.
Figures & tables
| MATH | GPQA | |||||
|---|---|---|---|---|---|---|
| Sampler | Accuracy | Strict | At limit | Accuracy | Strict | At limit |
| Default | 23.7 | 9.5 | 89.0 | 11.1 | 1.5 | 93.4 |
| Qwen recipe, presence penalty | 33.8 | 26.0 | 67.0 | 17.7 | 11.1 | 65.7 |
| Global TempQuant | 51.0 | 36.7 | 47.5 | 23.7 | 21.7 | 12.1 |
| Global TempQuant + presence penalty | 53.5 | 46.7 | 29.0 | 23.7 | 23.2 | 7.1 |
| Accuracy (%) | Mean | At length | |||||
| Policy | GSM8K | MATH | AIME | GPQA | accuracy | limit (%) | Closeness |
| Temperature ( ) = 0.3 | |||||||
| BF16 | 94.33 | 95.05 | 66.56 | 40.34 | 74.07 | 5.5 | 100 |
| GPTQ3 | 18.30 | 15.20 | 0.73 | 6.69 | 10.23 | 97.6 | 43.7 |
| Global TempQuant | 81.71 | 53.00 | 1.98 | 25.00 | 40.42 | 43.5 | 71.0 |
| Per-step TempQuant, forward KL | 81.33 | 52.10 | 1.43 | 26.07 | 40.23 | 44.4 | 71.2 |
| Policy | Unique summaries (%) | Meaning diversity | ROUGE-2 | Closeness |
|---|---|---|---|---|
| Temperature ( ) = 0.3 | ||||
| BF16 (value) | 50 | 0.037 | 6.81 | 100 |
| GPTQ3 | +22 | +0.045 | 0.78 | 15.2 |
| Per-step TempQuant, forward KL | +49 | +0.141 | 1.22 | 21.3 |
| Per-step TempQuant, JSD | +43 | +0.090 | 0.99 | 17.8 |
| Per-step TempQuant, reverse KL | +13 | +0.032 | 0.71 | 14.8 |
Appendix figures & tables42 assets
Supplementary material from the paper’s appendix.
Appendix
| Training divergence | |||
|---|---|---|---|
| Forward KL | |||
| JSD | |||
| Reverse KL |
| Controller architecture | |
|---|---|
| Model | Qwen3-4B GPTQ3 (main results) or native MXFP4 W4A4; BF16 KV |
| Reference | BF16 temperatures , top- |
| Input | Ten distribution summaries followed by 32 ranked logit features |
| Temperature head | , Tanh hidden activations |
| Top- head | , Tanh |
| Trainable parameters | 165,048 |
| Coordinate | Feature | Definition |
|---|---|---|
| 1 | Gap between the first- and second-ranked logits | |
| 2 | Gap between the first- and eighth-ranked logits | |
| 3 | Gap between the first- and 32nd-ranked logits | |
| 4 | Logit sample standard deviation | |
| 5 | Mean logit | |
| 6 | Maximum logit after centering |
| Training configuration | |
|---|---|
| Objectives | Smoothed forward KL, smoothed reverse KL, symmetric JSD |
| Loss scale | for either KL objective; for JSD |
| Uniform mixture | for KL; none for JSD |
| Smooth top- width | |
| Action-variance weight | |
| Optimizer | Adam: , , , weight decay ; without AMSGrad |
| Forward KL | Mean chosen | |||||
|---|---|---|---|---|---|---|
| Training variant | ||||||
| As in the paper | 0.594 | 0.483 | 0.482 | 1.090 | 1.404 | 1.371 |
| KL smoothing | 0.594 | 0.482 | 0.487 | 1.096 | 1.407 | 1.344 |
| KL smoothing | 0.594 | 0.482 | 0.488 | 1.097 | 1.407 | 1.345 |
| Smooth top- sharpness | 0.595 | 0.495 | 0.505 | 1.080 | 1.373 | 1.316 |
| Smooth top- sharpness | 0.595 | 0.467 | 0.468 | 1.105 | 1.428 | 1.344 |
| Task | Items | Trials | Output limit (tokens) |
|---|---|---|---|
| GSM8K test | 1,319 | 4 | 16,384 |
| MATH-500 test | 500 | 4 | 32,768 |
| AIME-120 | 120 | 8 | 32,768 |
| GPQA-Diamond | 198 | 8 | 32,768 |
| XSum validation subset | 128 | 8 | 512 |
| Task | Global GPTQ3 | Per-step GPTQ3 | Per-step global | |
|---|---|---|---|---|
| GSM8K | 0.3 | [+61.7, +65.2] | [+61.4, +64.8] | [-1.3, +0.5] |
| 0.6 | [+28.6, +32.3] | [+27.5, +31.2] | [-2.1, -0.1] | |
| 1.0 | [-2.9, -0.4] | [-0.4, +1.6] | [+1.1, +3.3] | |
| MATH | 0.3 | [+34.5, +41.6] | [+33.7, +40.6] | [-2.6, +0.8] |
| 0.6 | [+19.6, +25.5] | [+18.8, +24.4] | [-2.7, +0.7] | |
| 1.0 | [-3.5, +0.6] | [-0.2, +3.2] | [+1.3, +4.5] |
| GSM8K | MATH | GPQA | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Policy | Strict | Fin. acc. | Fin. | Strict | Fin. acc. | Fin. | Strict | Fin. acc. | Fin. | |
| 0.3 | BF16 | 94.28 | 94.7 | 99.6 | 94.90 | 96.5 | 98.7 | 38.83 | 41.9 | 92.7 |
| 0.3 | GPTQ3 | 5.46 | 92.3 | 5.9 | 2.67 | 96.4 | 2.8 | 0.06 | 7.7 | 0.8 |
| 0.3 | Global TempQuant | 66.22 | 88.1 | 75.2 | 39.12 | 75.7 | 51.8 | 22.35 | 26.2 | 85.2 |
| 0.3 | Per-step TempQuant (FKL) | 68.43 | 87.7 | 78.1 | 40.14 | 77.7 | 51.7 | 23.30 | 28.9 | 80.5 |
| 0.6 | BF16 | 94.83 | 94.9 | 99.9 | 95.65 | 96.4 | 99.5 | 41.29 | 41.7 | 99.1 |
| Model | Policy | GSM8K | MATH | GPQA | At length limit (%) GPTQ3 global | Forward KL removed (%) |
|---|---|---|---|---|---|---|
| Qwen3-4B | BF16 | 94.83 | 95.75 | 41.48 | ||
| GPTQ3 | 50.36 | 29.90 | 10.98 | 76.7 29.9 | 73 | |
| Global TempQuant | 80.80 | 52.35 | 25.19 | |||
| Per-step TempQuant | 79.68 | 51.35 | 24.08 | |||
| Qwen3-8B | BF16 | 96.50 | 95.75 | 42.68 | ||
| GPTQ3 | 88.67 | 60.83 | 22.22 | 29.2 11.7 | 70 |
| Model | Policy | Forward KL | Reverse KL | JSD | TV |
|---|---|---|---|---|---|
| Qwen3-8B | GPTQ3 | .915 | 1.310 | .0530 | .119 |
| Global TempQuant | .275 | 1.860 | .0590 | .141 | |
| Per-step TempQuant, forward KL | .424 | 1.922 | .0592 | .144 | |
| Per-step TempQuant, JSD | .874 | 1.353 | .0531 | .121 | |
| Per-step TempQuant, reverse KL | 1.538 | 1.122 | .0583 | .120 | |
| Oracle per-step TempQuant ( only) | .162 | 1.073 | .0363 | .086 |
| Forward KL at smoothing weight | BF16 token | ||||
|---|---|---|---|---|---|
| Policy ( ) | cut (%) | ||||
| Uncorrected | 1.793 | 1.643 | 1.493 | 1.342 | 14.5 |
| Global TempQuant (top- only) | 0.548 | 0.548 | 0.548 | 0.547 | 0.0 |
| Global TempQuant | 0.397 | 0.397 | 0.397 | 0.396 | 0.0 |
| Per-step TempQuant ( only) | 0.546 | 0.528 | 0.510 | 0.491 | 2.8 |
| Per-step TempQuant | 0.513 | 0.498 | 0.483 | 0.468 | 2.3 |
| Removed (%) | Top- | Full | Top- | Full | Top- | Full |
|---|---|---|---|---|---|---|
| Global TempQuant | 81 | 82 | 73 | 75 | 60 | 65 |
| Per-step TempQuant | 75 | 76 | 68 | 69 | 45 | 49 |
| Oracle per-step TempQuant | 89 | 89 | 84 | 84 | 75 | 76 |
| Fit on | Chosen | All tokens | Preserved | Flipped | |
|---|---|---|---|---|---|
| 0.3 | All tokens | 0.451 | 0.156 | 1.875 | |
| Preserved only | 0.733 | 0.122 | 3.676 | ||
| 0.6 | All tokens | 0.397 | 0.151 | 1.579 | |
| Preserved only | 0.527 | 0.129 | 2.445 | ||
| 1.0 | All tokens | 0.346 | 0.148 | 1.298 | |
| Preserved only | 0.402 | 0.136 | 1.684 |
| Tuned minus global TempQuant (accuracy points) | |||
| Labeled questions | Mean | 10th–90th percentile | Worse by 3+ points (%) |
| 10 | +0.4 | [ 3.7, +3.3] | 16 |
| 20 | +0.8 | [ 3.4, +3.2] | 12 |
| 50 | +1.0 | [ 1.4, +3.2] | 6 |
| 100 | +0.7 | [ 2.3, +3.8] | 7 |
| Qwen3-4B GPTQ3 setting | Tuning MATH | New MATH | GSM8K | GPQA |
| Global TempQuant, , | 49.00 | 53.92 | 80.80 | 25.63 |
| Global TempQuant, , | 48.75 | 54.67 | 81.71 | 24.24 |
| Tuned on answers | 47.75 | 55.83 | 81.12 | 25.63 |
| Tuned on answers | 49.38 | 54.33 | 80.40 | 24.75 |
| R1-Distill-Llama-8B GPTQ3 setting | Tuning MATH | New MATH | GSM8K | GPQA |
| Default | 64.48 | 69.50 | 76.57 | 22.73 |
| Text so far | “…if a number is divisible by both 6 and 10, that means it has to be a” | “…if the number is 1, then it has 1 divisor, but 1 isn” | ||
|---|---|---|---|---|
| BF16’s next token | common | ’t | ||
| Policy | ||||
| BF16 | 0.60 | 1.00 | 0.60 | 1.00 |
| GPTQ3 | 0.60 | 0.00 | 0.60 | 0.79 |
| Global TempQuant | 1.13 | 0.14 | 1.13 | 0.18 |
| Per-step TempQuant | 2.19 | 0.29 | 0.64 | 0.73 |
| Policy | Forward KL | Reverse KL | JSD | Total variation |
| Temperature ( ) = 0.3 | ||||
| Global TempQuant (forward KL) | -81 | +36 | +4 | +27 |
| Global TempQuant (JSD) | -48 | +7 | -6 | +5 |
| Global TempQuant (reverse KL) | +26 | -1 | +5 | 0 |
| Global TempQuant (top- only, forward KL) | -57 | 0 | -2 | 0 |
| Per-step TempQuant (forward KL) | -75 | +33 | +2 | +25 |
| Training loss | Forward KL | Reverse KL | JSD | |
|---|---|---|---|---|
| Forward KL | 0.3 | |||
| 0.6 | ||||
| 1.0 | ||||
| Reverse KL | 0.3 | |||
| 0.6 | ||||
| 1.0 |
| BF16 probability the policy cuts (%) | Policy probability on tokens BF16 cuts (%) | Mean temperature | Mean top- | |
| GPTQ3 | 11.1 | 12.3 | 0.30 | 0.950 |
| Forward KL | 1.0 | 18.1 | 1.09 | 0.975 |
| JSD | 5.6 | 13.7 | 0.62 | 0.962 |
| Reverse KL | 14.0 | 12.1 | 0.48 | 0.942 |
| Per-step TempQuant | Forward KL | Reverse KL | JSD | Total variation |
|---|---|---|---|---|
| Temperature only | 69.4 | +33.4 | +1.7 | +24.3 |
| Temperature and top- | 75.4 | +33.3 | +1.8 | +24.6 |
| Temperature only | 65.9 | +64.3 | +18.5 | +33.2 |
| Temperature and top- | 67.7 | +55.6 | +14.5 | +28.5 |
| Policy | GSM8K | MATH | AIME | GPQA |
|---|---|---|---|---|
| Global TempQuant, | ||||
| Top- only | 62.06 | 35.40 | 0.47 | 15.21 |
| Temperature and top- | 80.80 | 52.35 | 1.72 | 25.19 |
| Per-step TempQuant, | ||||
| Temperature only | 78.93 | 51.91 | 1.25 | – |
| Temperature and top- | 79.68 | 51.35 | 1.43 | – |
| Policy ( ) | Word overlap with BF16 | Self-BLEU | Repeated 4-grams (%) | At length limit (%) | Closeness |
|---|---|---|---|---|---|
| GPTQ3, reasoning (mean of four tasks) | |||||
| BF16 | 66.1 | 72.0 | 35.3 | 2.6 | 100 |
| GPTQ3 | 7.3 | 55.6 | 89.6 | 83.7 | 47.3 |
| Global TempQuant | 18.0 | 64.8 | 59.2 | 42.7 | 70.2 |
| Per-step TempQuant, forward KL | 16.6 | 62.6 | 58.6 | 48.2 | 68.1 |
| Per-step TempQuant, JSD | 9.0 | 56.9 | 85.5 | 76.8 | 50.9 |
| Task | BF16 vs BF16 | GPTQ3 | Global TempQuant | Per-step TempQuant | |
|---|---|---|---|---|---|
| MATH | 0.3 | .939 | .579 | .879 | .883 |
| 0.6 | .967 | .797 | .881 | .830 | |
| 1.0 | .983 | .898 | .851 | .902 | |
| GSM8K | 0.3 | .951 | .458 | .662 | .652 |
| 0.6 | .958 | .485 | .582 | .657 | |
| 1.0 | .961 | .579 | .637 | .538 |
| Text so far: “…So maybe I should start by finding the LCM of 6 and 10. Then, once I have that, I can figure out the smallest multiple of that LCM that’s greater than 100.” | |||
| Policy | Start of one continuation | Opens with doubt | |
| BF16 | 0.60 | Alright, let me recall how to calculate the LCM. I think the formula is LCM … | 0/16 |
| GPTQ3 | 0.60 | Wait , but let me check. The LCM of 6 and 10… Hmm , 6 and 10. Let me recall how to calculate the LCM… | 16/16 |
| Global TempQuant | 1.13 | Wait , but let me check. Let me remember that LCM of 6 and 10. But 6 and 10 have a common factor… | 14/16 |
| Per-step TempQuant | 1.91 | Alright, so let’s start with finding the LCM of 6 and 10. To find the LCM of two numbers, I think I need to find their prime factors… | 10/16 |
| Oracle per-step TempQuant | 2.58 | Hmm , right. Let me recall how to find the LCM. To find the LCM of two numbers, I can use the formula… | 9/16 |
| Policy | Starts with BF16’s word (%) | Opens with doubt (%) | “Wait” per continuation | Markers per continuation |
|---|---|---|---|---|
| BF16 | 83 | 18 | 1.4 | 8.0 |
| Raw GPTQ3 | 5 | 75 | 5.4 | 18.4 |
| Global TempQuant | 12 | 63 | 5.1 | 18.5 |
| Per-step TempQuant | 13 | 60 | 5.0 | 17.5 |
| Oracle per-step TempQuant | 15 | 52 | 4.5 | 15.8 |
| Policy | |||
|---|---|---|---|
| Uncorrected | 1.226 | 0.722 | 0.582 |
| Top- off | 0.488 | 0.264 | 0.172 |
| Global TempQuant ( only) | 0.330 | 0.282 | 0.255 |
| Global TempQuant | 0.236 | 0.189 | 0.153 |
| Per-step TempQuant | 0.330 | 0.246 | 0.211 |
| Accuracy (%) | Mean | At length | ||||
| Policy | GSM8K | MATH | AIME | GPQA | accuracy | limit (%) |
| Temperature ( ) = 0.3 | ||||||
| BF16 | 94.47 | 95.40 | 65.6 | 39.8 | 73.82 | 5.6 |
| MXFP4 | 93.67 | 91.95 | 43.2 | 41.5 | 67.58 | 3.5 |
| Global TempQuant | 93.76 | 93.20 | 47.6 | 40.8 | 68.85 | 1.8 |
| Per-step TempQuant, forward KL | 93.75 | 92.48 | 47.9 | 40.1 | 68.56 | 1.8 |
| Policy | Self-BLEU | ROUGE-2 | Closeness |
|---|---|---|---|
| Temperature ( ) = 0.3 | |||
| BF16 (value) | 95.4 | 6.76 | 100 |
| MXFP4 | 8.6 | 1.14 | 14.0 |
| Per-step TempQuant, forward KL | 22.4 | 1.20 | 17.7 |
| Per-step TempQuant, JSD | 13.6 | 1.16 | 15.2 |
| Per-step TempQuant, reverse KL | 6.9 | 1.18 | 13.5 |
| Quantized policy | Accuracy (%) | BF16 correct, quantized wrong | BF16 wrong, quantized correct | Total disagreements |
|---|---|---|---|---|
| Temperature | ||||
| Native MXFP4 | 92.20 | 92 | 23 | 115 |
| Forward KL | 92.73 | 88 | 29.5 | 117.5 |
| Reverse KL | 92.40 | 88.5 | 23.5 | 112 |
| JSD | 92.96 | 79 | 25 | 104 |
| Global TempQuant | 93.20 | 74 | 30 | 104 |
| Policy | Distance to BF16 | Reduction [95% CI] | Holm | |
|---|---|---|---|---|
| Native MXFP4 | 0.1301 | |||
| Per-step TempQuant, JSD | 0.1242 | 0.0059 [0.0013, 0.0106] | 0.011 | 0.021 |
| Per-step TempQuant, reverse KL | 0.1303 | 0.0002 [ 0.0054, 0.0051] | 0.94 | 0.94 |
| Per-step TempQuant, forward KL | 0.1263 | 0.0038 [ 0.0015, 0.0091] | 0.16 |
| Setting for AWQ | Temperature | Forward KL |
|---|---|---|
| Requested, | 0.60 | 0.237 |
| Top- off, | 0.60 | 0.063 |
| AWQ’s global TempQuant, | 0.71 | 0.061 |
| GPTQ3’s global TempQuant, | 1.13 | 0.088 |
| GPTQ3’s per-step predictor | 1.7 (mean) | 0.148 |
| Policy | GSM8K | MATH | GPQA | At length limit (%) |
|---|---|---|---|---|
| AWQ, | 94.7 | 95.3 | 41.2 | 0.3 |
| Global TempQuant, | 94.5 | 94.7 | 45.2 | 0.1 |
| Policy | |||
| Uncorrected | 2.409 | 1.493 | 0.867 |
| Global TempQuant (top- only) | 1.037 | 0.548 | 0.364 |
| Global TempQuant ( only) | 0.564 | 0.513 | 0.474 |
| Global TempQuant | 0.451 | 0.397 | 0.346 |
| Oracle per-step TempQuant ( only) | 0.288 | 0.264 | 0.255 |
| Oracle per-step TempQuant | 0.272 | 0.240 | 0.220 |
| GSM8K (96 questions) | MATH (64 questions) | |||||
| Policy | Accuracy | Length (K) | At limit | Accuracy | Length (K) | At limit |
| BF16, same questions | 96.6 | 2.3 | 0.3 | 93.4 | 4.9 | 1.2 |
| GPTQ3 | 47.9 | 12.2 | 63.5 | 25.0 | 29.8 | 89.1 |
| Global TempQuant | 84.4 | 7.6 | 19.8 | 53.1 | 19.0 | 42.2 |
| Per-step TempQuant | 79.7 | 8.3 | 25.5 | 56.3 | 20.4 | 46.9 |
| Oracle per-step TempQuant | 92.7 | 4.8 | 3.1 | 78.1 | 13.3 | 21.9 |
| Top token kept (83%) | Top token changed (17%) | Share on changed steps (%) | ||||
| GPTQ3 | Oracle | GPTQ3 | Oracle | GPTQ3 | Oracle | |
| 0.3 | 0.28 | 0.03 | 12.66 | 1.45 | 90 | 92 |
| 0.6 | 0.35 | 0.04 | 7.01 | 1.22 | 81 | 87 |
| 1.0 | 0.31 | 0.05 | 3.55 | 1.02 | 70 | 80 |
| Steps grouped by GPTQ3 rank of BF16 top token | Steps (%) | Raw forward KL (%) | Oracle residual (%) | Removed by oracle (%) |
| Temperature ( ) = 0.3; oracle removes 89% overall | ||||
| Top token kept, order kept | 80.1 | 5.1 | 1.3 | 97 |
| Top token kept, order changed | 3.3 | 3.7 | 6.0 | 82 |
| BF16 top ranked 2nd | 9.9 | 44.4 | 41.2 | 90 |
| BF16 top ranked 3rd–5th | 4.9 | 33.6 | 33.3 | 89 |
| BF16 top ranked below 5th | 1.8 | 13.3 | 18.1 | 85 |
| Batch size | Requested settings | Global TempQuant | Per-step TempQuant | Per-step increase |
|---|---|---|---|---|
| 1 | 2.93 | 2.89 | 3.47 | 18% |
| 8 | 3.48 | 3.41 | 4.64 | 33% |
| 64 | 6.09 | 5.89 | 8.19 | 34% |
LLM Quantization
Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.
See all 443 papers