AutoQS: Automated Quantization Strategy Design with Large Language Models
Organizations: Xi’an Jiaotong University · The Hong Kong University of Science and Technology · Zhongguancun Academy
Abstract
Quantization reduces LLM deployment costs, but effective strategies must balance model quality and storage. LLM-driven search can automate strategy design through repeated proposal and evaluation. This process must identify useful strategies within a limited evaluation budget. The key challenge is that actual quantization is expensive, while inexpensive proxies may not reflect quantized-model performance. To address this challenge, we present AutoQS, an Automated Quantization Strategy Design with LLMs. AutoQS keeps weight precision fixed and searches group-size allocations across layers and operators under deployment constraints. Evaluation history guides coordinated LLM proposals, while deterministic validation ensures compatible allocations. An inexpensive quality proxy evaluates candidates, and an candidate repository retains alternative allocations to inform subsequent proposals. Periodic actual quantization complements these proxy evaluations with measured quality and checkpoint storage, providing feedback for further search within the evaluation budget. Experiments on Qwen2.5-7B-Instruct show that AutoQS improves seven-task average accuracy over uniform-group GPTQ and AWQ by 0.70 and 1.12 percentage points, respectively, at four-bit weight precision.
Figures & tables
| Method | W-bit | Group | MMLU | HellaSwag | ARC-E | ARC-C | PIQA | WinoGrande | BoolQ | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| FP16 | 16 | – | 71.94 | 78.94 | 80.43 | 51.02 | 79.71 | 73.01 | 84.68 | 74.25 |
| RTN ( Liu et al., 2025 ) | 4 | 128 | 71.26 | 75.84 | 77.05 | 48.14 | 77.26 | 70.51 | 80.79 | 71.55 |
| GPTQ ( Frantar et al., 2023 ) | 4 | 128 | 69.78 | 76.68 | 78.21 | 49.58 | 76.15 | 71.08 | 83.16 | 72.09 |
| AWQ ( Lin et al., 2024 ) | 4 | 128 | 71.31 | 78.26 | 79.17 | 49.15 | 79.60 | 71.67 | 82.91 | 73.15 |
| OmniQuant ( Shao et al., 2024 ) | 4 | 128 | 71.01 | 78.25 | 79.46 | 50.54 | 78.72 | 72.29 | 83.61 | 73.41 |
| CALDERA ( Saha et al., 2024 ) | 4 | 128 | 64.13 | 75.61 | 73.50 | 45.54 | 72.83 | 70.03 | 79.83 | 68.78 |
| Setting | Proposer | Feedback | Best iter. | Best Macro3 | Iter.-100 Macro3 | Mean policy change (%) |
|---|---|---|---|---|---|---|
| Full | Qwen-Plus | Yes | 100 | 73.8521 | 73.8521 | 26.53 |
| No-feedback | Qwen-Plus | No | 45 | 72.5777 | 72.2177 | 29.23 |
| Random proposer | Legal random mutation | Yes | 15 | 73.4465 | 72.0465 | 11.91 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | Value |
|---|---|
| Search iterations | 100 |
| Physical-evaluation | every 5 iterations |
| Maximum physical checkpoints | 20 |
| Physical-evaluation batch size | 1 |
| Default LLM proposer | Qwen-Plus |
| Post-search Final-7 batch size | 1 |
| Method | W-bit | Group | MMLU | HellaSwag | ARC-E | ARC-C | PIQA | WinoGrande | BoolQ | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| FP16 | 16 | – | 71.76 | 80.48 | 81.06 | 55.03 | 80.36 | 70.72 | 86.45 | 75.12 |
| GPTQ | 4 | 128 | 70.77 | 79.73 | 80.13 | 54.35 | 80.25 | 70.56 | 85.41 | 74.46 |
| AWQ | 4 | 128 | 70.80 | 79.61 | 78.75 | 54.69 | 78.56 | 69.61 | 86.27 | 74.04 |
| AutoQS | 4 | Mixed | 70.71 | 79.68 | 80.64 | 57.94 | 80.30 | 70.48 | 86.36 | 75.16 |
| Method | W-bit | Group | MMLU | HellaSwag | ARC-E | ARC-C | PIQA | WinoGrande | BoolQ | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| FP16 | 16 | – | 41.84 | 76.00 | 76.35 | 46.25 | 79.11 | 69.06 | 77.71 | 66.62 |
| RTN | 4 | 128 | 39.45 | 74.36 | 74.22 | 42.22 | 77.60 | 66.82 | 74.55 | 64.17 |
| GPTQ | 4 | 128 | 38.73 | 74.76 | 75.00 | 43.09 | 78.45 | 68.27 | 75.14 | 64.78 |
| AWQ | 4 | 128 | 40.75 | 75.57 | 75.59 | 45.31 | 78.78 | 68.35 | 78.23 | 66.08 |
| OmniQuant | 4 | 128 | 41.35 | 74.88 | 75.21 | 44.54 | 78.94 | 68.19 | 77.16 | 65.75 |
| CALDERA | 4 | 128 | 39.41 | 75.79 | 75.70 | 43.84 | 81.02 | 68.45 | 76.96 | 65.88 |
| Model | Setting | MMLU | HellaSwag | ARC-E | ARC-C | PIQA | WinoGrande | BoolQ | Macro7 |
|---|---|---|---|---|---|---|---|---|---|
| LLaMA-7B | Dense | 32.17 | 76.20 | 72.85 | 44.80 | 79.16 | 69.93 | 75.17 | 64.33 |
| LLaMA-7B | Source searched W4 | 27.03 | 73.82 | 71.42 | 42.92 | 79.27 | 70.09 | 74.28 | 62.69 |
| Llama2-7B | Dense (local) | 41.83 | 76.00 | 74.49 | 46.16 | 79.05 | 68.90 | 77.71 | 66.31 |
| Llama2-7B | Uniform W4-G128 | 37.96 | 75.71 | 71.17 | 43.60 | 78.51 | 67.48 | 77.00 | 64.49 |
| Llama2-7B | Zero-shot transfer | 39.83 | 75.49 | 70.33 | 42.92 | 78.62 | 68.75 | 75.99 | 64.56 |
| Vicuna-7B-v1.5 | Dense (local) | 48.75 | 73.77 | 71.25 | 45.90 | 77.97 | 69.53 | 80.92 | 66.87 |
| W-bit | MMLU | HellaSwag | ARC-E | ARC-C | PIQA | WinoGrande | BoolQ | Macro7 |
|---|---|---|---|---|---|---|---|---|
| 2 | 25.02 | 26.18 | 25.13 | 25.26 | 51.69 | 50.20 | 50.09 | 36.22 |
| 3 | 58.66 | 70.06 | 65.36 | 45.73 | 73.29 | 58.96 | 74.50 | 63.79 |
| 4 | 70.43 | 78.69 | 80.35 | 56.57 | 79.54 | 68.43 | 85.44 | 74.21 |
| LLM API | W-bit | Search seed | MMLU | HellaSwag | ARC-E | ARC-C | PIQA | WinoGrande | BoolQ | Macro7 |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen-Plus | 4 | 2026 | 70.18 | 78.24 | 79.29 | 54.18 | 79.82 | 66.38 | 84.83 | 73.27 |
| Qwen-Plus | 4 | 1234 | 70.43 | 78.69 | 80.35 | 56.57 | 79.54 | 68.43 | 85.44 | 74.21 |
| DeepSeek | 4 | 1234 | 70.33 | 78.44 | 79.46 | 54.52 | 79.76 | 68.35 | 84.95 | 73.69 |
| Qwen-Plus | 2 | 2026 | 25.49 | 26.12 | 24.87 | 27.99 | 51.96 | 49.57 | 48.78 | 36.40 |
| DeepSeek | 2 | 1234 | 25.86 | 26.34 | 25.00 | 27.30 | 53.10 | 50.51 | 49.97 | 36.87 |
| Iteration | MMLU | HellaSwag | ARC-E | ARC-C | PIQA | WinoGrande | BoolQ | Macro7 |
|---|---|---|---|---|---|---|---|---|
| 10 | 70.48 | 78.37 | 80.60 | 55.80 | 79.27 | 68.03 | 84.43 | 73.86 |
| 25 | 70.10 | 77.93 | 80.56 | 54.95 | 78.29 | 67.56 | 85.47 | 73.55 |
| 60 | 69.98 | 78.52 | 80.89 | 56.23 | 80.03 | 67.01 | 84.43 | 73.87 |
| 70 | 69.91 | 78.43 | 80.51 | 56.06 | 79.65 | 67.25 | 84.65 | 73.78 |
| 100 | 70.43 | 78.69 | 80.35 | 56.57 | 79.54 | 68.43 | 85.44 | 74.21 |
| Slot | Iter. | Macro3 | Macro7 | Search rank | Macro7 rank |
|---|---|---|---|---|---|
| 4 | 20 | 64.9683 | 62.9818 | 8 | 1 |
| 6 | 30 | 65.3866 | 62.8990 | 2 | 2 |
| 9 | 45 | 65.0425 | 62.6471 | 6 | 7 |
| 10 | 50 | 64.9990 | 62.5527 | 7 | 8 |
| 17 | 85 | 65.3613 | 62.7483 | 3 | 4 |
| 18 | 90 | 65.2977 | 62.8870 | 4 | 3 |
| Model | Setting | Modules | Non-G128 (%) | |||||
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-7B | Full, seed 1234 | 196 | 12 | 14 | 15 | 15 | 140 | 28.57 |
| Qwen2.5-7B | No-feedback, seed 1234 | 196 | 1 | 14 | 13 | 15 | 153 | 21.94 |
| Qwen2.5-7B | Random proposer, seed 1234 | 196 | 8 | 5 | 4 | 3 | 176 | 10.20 |
| LLaMA-7B | Source searched W4 | 224 | 11 | 15 | 15 | 15 | 168 | 25.00 |
| Llama2-7B | Target-native best physical | 224 | 15 | 11 | 24 | 18 | 156 | 30.36 |
| Policy pair | Different modules | Fraction (%) |
|---|---|---|
| Full vs. No-feedback | 86 / 196 | 43.88 |
| Full vs. Random proposer | 63 / 196 | 32.14 |
| No-feedback vs. Random proposer | 63 / 196 | 32.14 |
| Method | Primary optimization object | Typical adaptation level | Uses calibration signal | Grouping in our tables | Relationship to AutoQS |
|---|---|---|---|---|---|
| RTN | Direct rounding to the target grid | Weight / quantization group | No learned optimization | Fixed G128 | Reference fixed policy |
| GPTQ | Sequential one-shot weight quantization using approximate second-order information | Weight / block | Yes | Fixed G128 | Physical quantizer used after policy search |
| AWQ | Activation-aware protection through channel scaling | Channel | Yes, activation statistics | Fixed G128 | Orthogonal to module-level allocation |
| OmniQuant | Learnable clipping and equivalent transformations | Block / channel | Yes, differentiable calibration | Fixed G128 | Optimizes quantization parameters rather than allocation topology |
| CALDERA | Low-precision plus low-rank matrix decomposition | Layer / matrix | Yes | Fixed G128 in reported comparison | Changes the weight representation itself |
| SVDQuant | SVD/low-rank-oriented baseline as reported by FBQuant | Method-specific | Method-specific | Fixed G128 in reported comparison | No module-level group policy inferred here |
| Stage | Manual heterogeneous-policy tuning | AutoQS |
|---|---|---|
| Choose modules to modify | Researcher manually selects target modules | Candidate generation and mutation are performed automatically |
| Choose group size | Researcher chooses a value for each target | Values are selected from the legal set under the search protocol |
| Validate legality | Researcher or ad-hoc scripts verify compatibility | Semantic and structural checks reject invalid candidates automatically |
| Low-cost screening | Researcher designs and runs exploratory tests | Stage-1 evaluation is applied to every valid unseen candidate |
| High-fidelity test | Researcher manually launches quantization and evaluation | Physical GPTQ + Macro3 is scheduled automatically |
| Use previous result | Researcher inspects results and chooses the next modification | Physical feedback can influence later search decisions |
LLM Quantization
Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.
See all 434 papers