CoreQ: Mismatch-Corrected Successive Rounding for LLM Quantization
Organizations: The University of Texas at Austin · DEVCOM Army Research Laboratory
Abstract
Post-training quantization (PTQ) enables efficient deployment of large language models by mapping pretrained weights to low-bit formats without retraining, typically using a small calibration set to minimize a layer-wise calibration objective. However, this sequential procedure induces a mismatch: errors from earlier quantized layers alter the inputs received by later layers, causing the activations to deviate from those of the full-precision model. Recent approaches introduce mismatch-aware calibration objectives to compensate for this effect, but leave open how much of the observed mismatch should shift each layer's calibration target. We show that fully applying the empirical mismatch correction can overfit finite calibration data, while using a fixed correction coefficient ignores variation in mismatch estimates across layers. To address these limitations, we propose CoreQ, a learning-free PTQ framework that uses a geometric decomposition of the mismatch to compute a closed-form, layer-adaptive correction coefficient. This coefficient controls how much of the observed mismatch enters each layer's calibration target, reducing overfitting to limited calibration data without coefficient tuning. CoreQ then applies efficient greedy successive rounding to the resulting triangular least-squares objective. Its bounded beam-search extension, K-CoreQ, trades modest additional compute for improved performance. Across multiple LLM families, scales, bit-widths, and quantization settings, CoreQ and K-CoreQ achieve lower perplexity and higher downstream accuracy than strong PTQ baselines. Code is available at GitHub.
Figures & tables
| Per-channel | Per-group (g128) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Method | Wiki2 ( ) | C4 ( ) | Avg.Acc ( ) | Time (s) | Wiki2 ( ) | C4 ( ) | Avg.Acc ( ) | Time (s) |
| L3-8B | FP16 | 6.14 | 8.93 | 74.31 | 6.14 | 8.93 | 74.31 | ||
| GPTQ | 24.80 3.20 | 23.59 1.77 | 51.68 1.21 | 550.1 | 9.87 0.28 | 12.74 0.40 | 66.82 0.65 | 547.5 | |
| GPTAQ | 33.47 12.40 | 22.00 3.10 | 51.61 1.80 | 701.4 | 9.07 0.33 | 12.15 0.13 | 65.77 1.28 | 715.0 | |
| LDLQ | 24.65 4.79 | 23.16 1.92 | 51.02 1.69 | 457.4 | 14.99 2.85 | 14.12 1.47 | 65.20 2.31 | 472.5 | |
| GuidedQ | 49.79 10.56 | 40.03 6.55 | 48.58 0.96 | 2110.1 | — | — | — | — | |
| W4A4KV4 | W4A4KV16 | |||||||||||
| Methods | L3-8B | L2-7B | L2-13B | L3-8B | L2-7B | L2-13B | ||||||
| Wiki2 | T (s) | Wiki2 | T (s) | Wiki2 | T (s) | Wiki2 | T (s) | Wiki2 | T (s) | Wiki2 | T (s) | |
| OmniQuant | – | – | 15.12 0.21 | 3848 | 11.82 0.06 | 6654 | – | – | 12.21 0.09 | 3640 | 9.95 0.12 | 6337 |
| QuaRot+GPTQ | 8.83 0.19 | 811 | 6.21 0.02 | 1445 | 5.48 0.02 | 2131 | 7.86 0.06 | 793 | 6.08 0.02 | 1317 | 5.35 0.01 | 1977 |
| QuaRot+GPTAQ | 8.08 0.02 | 954 | 5.97 0.00 | 710 | 5.30 0.01 | 1725 | 7.37 0.02 | 918 | 5.87 0.01 | 720 | 5.17 0.01 | 2157 |
| QuaRot+CoreQ | 7.96 0.02 | 892 | 5.93 0.01 | 1420 | 5.26 0.00 | 2050 | 7.20 0.01 | 885 | 5.82 0.00 | 1366 | 5.14 0.00 | 2068 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Tensor | CoreQ ( ) | -CoreQ (beam) |
|---|---|---|
| Corrected rounding center | ||
| Cholesky factor | ||
| Quantized weight | ||
| Beam scores | — | |
| Suffix decisions | — | up to |
| Per-step expansion cost | — |
| WikiText-2 | C4 | |||||
|---|---|---|---|---|---|---|
| Correction | ||||||
| Stage | Time (s) | Share (%) |
|---|---|---|
| collection and | 193.2 | 28.8 |
| collection | 23.1 | 3.4 |
| 122.1 | 18.2 | |
| computation | 7.7 | 1.1 |
| Greedy successive rounding | 324.5 | 48.4 |
| Total profiled GPU time | 670.6 | 100.0 |
| Wiki2 PPL ( ) | C4 PPL ( ) | Avg. Acc. (%, ) | ||||
|---|---|---|---|---|---|---|
| # Samples | GPTAQ | CoreQ | GPTAQ | CoreQ | GPTAQ | CoreQ |
| 16 | 11.09 0.34 | 11.03 1.00 | 12.13 0.19 | 11.34 0.14 | 58.17 0.48 | 59.28 0.61 |
| 32 | 10.37 1.35 | 9.17 0.31 | 10.80 0.10 | 10.06 0.07 | 60.10 0.23 | 60.96 0.70 |
| 64 | 9.28 0.57 | 9.06 0.65 | 10.26 0.02 | 9.82 0.22 | 60.86 0.64 | 61.01 0.80 |
| 128 | 8.84 0.23 | 8.41 0.21 | 10.04 0.06 | 9.61 0.12 | 60.91 0.52 | 61.46 0.50 |
| Metric | W4A16KV16 | W4A4KV16 | W4A4KV4 |
|---|---|---|---|
| WikiText-2 / C4 PPL | / | / | / |
| LLaMA-2-7B | LLaMA-2-13B | LLaMA-3-8B | |||||||
| Method | Wiki2 | C4 | Time | Wiki2 | C4 | Time | Wiki2 | C4 | Time |
| Per-channel (W3) | |||||||||
| AWQ | 16.93 0.33 | 18.55 0.45 | 801 | 6.48 0.02 | 7.92 0.02 | 1326 | 12.18 0.16 | 16.48 0.20 | 777 |
| OmniQuant | 6.63 0.01 | 8.64 0.02 | 2309 | 5.60 0.02 | 7.46 0.03 | 3975 | 15.11 0.47 | 21.10 0.74 | 2338 |
| QuaRot + CoreQ | 5.84 0.01 | 7.70 0.01 | 570 | 5.17 0.00 | 6.99 0.00 | 956 | 7.14 0.01 | 11.09 0.04 | 679 |
| Group-wise, (W3) | |||||||||
| Method | DeiT-S | DeiT-B | ||
|---|---|---|---|---|
| Acc (%) | Q.Time (s) | Acc (%) | Q.Time (s) | |
| FP16 | 79.9 | – | 81.8 | – |
| GPTQ | 71.94 0.06 | 13.86 2.09 | 77.60 0.04 | 18.03 0.30 |
| GPTAQ | 74.09 0.16 | 15.40 0.70 | 78.18 0.12 | 24.85 0.75 |
| CoreQ | 75.41 0.07 | 14.75 0.06 | 78.55 0.09 | 21.20 0.70 |
| 4-CoreQ | 75.52 0.08 | 17.41 0.31 | 78.56 0.12 | 24.74 1.87 |
| Qwen3-8B | Phi-3-3.8B | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Granularity | Method | Wiki2 ( ) | C4 ( ) | Avg. Acc ( ) | Q.Time (s) | Wiki2 ( ) | C4 ( ) | Avg. Acc ( ) | Q.Time (s) |
| FP16 | 9.73 | 14.65 | 74.15 | — | 6.01 | 9.11 | 75.41 | — | |
| Per-channel | GPTQ | 18.75 0.45 | 19.69 0.12 | 54.17 0.51 | 499.2 17.7 | 15.46 0.50 | 15.80 0.05 | 54.48 1.48 | 266.5 7.6 |
| GPTAQ | 17.63 0.59 | 19.08 0.28 | 57.07 0.90 | 647.2 19.1 | 12.99 0.28 | 14.21 0.15 | 57.19 0.85 | 335.7 7.7 | |
| LDLQ | 16.18 0.18 | 18.49 0.08 | 56.19 0.81 | 375.4 13.4 | 15.75 0.74 | 16.03 0.48 | 54.49 0.65 | 216.1 2.8 | |
| CoreQ | 15.44 0.23 | 17.51 0.08 | 63.58 0.93 | 553.9 12.0 | 11.86 0.34 | 13.68 0.32 | 60.51 1.30 | 302.1 5.8 | |
| Per-channel | Per-group ( ) | ||||||
|---|---|---|---|---|---|---|---|
| Bits | Method | Wiki2 ( ) | C4 ( ) | Q.Time (s) | Wiki2 ( ) | C4 ( ) | Q.Time (s) |
| FP16 | — | 3.32 | 5.52 | — | 3.32 | 5.52 | — |
| W2 | GPTQ | 181.31 19.81 | 83.17 5.14 | 3301.2 73.0 | 12.38 1.42 | 11.08 0.10 | 3271.6 9.5 |
| GPTAQ | 251.82 80.72 | 83.59 11.38 | 4663.4 3.0 | 12.28 3.07 | 11.85 2.16 | 4846.5 43.9 | |
| LDLQ | 193.51 36.21 | 91.32 10.08 | 2753.5 26.7 | 8.68 0.29 | 9.90 0.22 | 2939.7 76.5 | |
| CoreQ | 94.97 12.49 | 38.46 2.37 | 4731.9 69.3 | 7.63 0.49 | 8.95 0.32 | 4772.1 89.6 | |
| Per-channel | Per-group ( ) | ||||||
|---|---|---|---|---|---|---|---|
| Bits | Method | Wiki2 ( ) | MMLU ( ) | GSM8K-CoT ( ) | Wiki2 ( ) | MMLU ( ) | GSM8K-CoT ( ) |
| FP16 | — | 6.24 | 67.95 | 75.66 | 6.24 | 67.95 | 75.66 |
| W3 | GPTQ | 154.03 27.68 | 30.00 3.07 | 0.28 0.48 | 11.06 0.48 | 56.33 0.51 | 36.52 1.38 |
| GPTAQ | 33.79 11.11 | 32.77 2.31 | 2.86 1.73 | 9.95 0.22 | 56.20 1.11 | 42.53 3.21 | |
| LDLQ | 19.89 1.53 | 39.62 1.40 | 4.17 0.92 | 9.72 0.20 | 59.53 0.62 | 45.69 2.25 | |
| CoreQ | 18.75 2.98 | 39.76 2.83 | 8.62 1.89 | 9.32 0.14 | 59.95 0.36 | 48.14 1.67 | |
LLM Quantization
Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.
See all 446 papers