Quantization reduces LLM deployment costs, but effective strategies must balance model quality and storage. LLM-driven search can automate strategy design through repeated proposal and evaluation. This process must identify useful strategies within a limited evaluation budget. The key challenge is that actual quantization is expensive, while inexpensive proxies may not reflect quantized-model performance. To address this challenge, we present AutoQS, an Automated Quantization Strategy Design with LLMs. AutoQS keeps weight precision fixed and searches group-size allocations across layers and operators under deployment constraints. Evaluation history guides coordinated LLM proposals, while deterministic validation ensures compatible allocations. An inexpensive quality proxy evaluates candidates, and an candidate repository retains alternative allocations to inform subsequent proposals. Periodic actual quantization complements these proxy evaluations with measured quality and checkpoint storage, providing feedback for further search within the evaluation budget. Experiments on Qwen2.5-7B-Instruct show that AutoQS improves seven-task average accuracy over uniform-group GPTQ and AWQ by 0.70 and 1.12 percentage points, respectively, at four-bit weight precision.
Figures & tables
Figure 1: Overview of the AutoQS framework. AutoQS first defines a structured quantization policy space under a fixed weight bit-width, where layer- and operator-wise group sizes are searched. An LLM-guided search procedure generates constrained policy modifications, which are validated and evaluated using a low-cost proxy while a diversity-preserving candidate records maintains previously explored allocation patterns. Every K=5 search iterations, the procedure selects the latest eligible candidate in the current five-iteration window for fresh GPTQ quantization. This target selection is independent of the proxy-score ordering. The resulting checkpoint is evaluated for task accuracy and storage, providing measured outcomes for subsequent policy selection.
Method
W-bit
Group
MMLU
HellaSwag
ARC-E
ARC-C
PIQA
WinoGrande
BoolQ
Avg. ↑
FP16
16
–
71.94
78.94
80.43
51.02
79.71
73.01
84.68
74.25
RTN ( Liu et al., 2025 )
4
128
71.26
75.84
77.05
48.14
77.26
70.51
80.79
71.55
GPTQ ( Frantar et al., 2023 )
4
128
69.78
76.68
78.21
49.58
76.15
71.08
83.16
72.09
AWQ ( Lin et al., 2024 )
4
128
71.31
78.26
79.17
49.15
79.60
71.67
82.91
73.15
OmniQuant ( Shao et al., 2024 )
4
128
71.01
78.25
79.46
50.54
78.72
72.29
83.61
73.41
CALDERA ( Saha et al., 2024 )
4
128
64.13
75.61
73.50
45.54
72.83
70.03
79.83
68.78
Table 1: Main 4-bit results on Qwen2.5-7B. External baseline results are taken from Table 8 of FBQuant ( Liu et al., 2025 ) . All external quantized methods use group size 128, whereas AutoQS uses a searched heterogeneous group-size configuration. Bold denotes the best result among 4-bit methods.
Setting
Proposer
Feedback
Best iter.
Best Macro3 ↑
Iter.-100 Macro3
Mean policy change (%)
Full
Qwen-Plus
Yes
100
73.8521
73.8521
26.53
No-feedback
Qwen-Plus
No
45
72.5777
72.2177
29.23
Random proposer
Legal random mutation
Yes
15
73.4465
72.0465
11.91
Table 2: Matched-seed search-mechanism ablation on Qwen2.5-7B W4. All runs use search seed 1234, T=100 iterations, and one physical evaluation every five iterations. “Best Macro3” denotes the best observed physical candidate under the predefined search-time criterion.
Figure 2: Module-level group-size composition of the analyzed frozen W4 policies. (a) Full distributions across all five group-size settings. (b) Detail of the four lower-share settings ( G=−1,16,32,64 ). The three Qwen2.5-7B variants use the same search seed of 1234. “LLaMA source” denotes the frozen source policy used for zero-shot transfer, while “Llama2 native” denotes the best observed physical policy under Macro3 among the 20 physical evaluations of the target-native Llama2-7B search.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Value
Search iterations
100
Physical-evaluation
every 5 iterations
Maximum physical checkpoints
20
Physical-evaluation batch size
1
Default LLM proposer
Qwen-Plus
Post-search Final-7 batch size
1
Appendix
Table 3: Compute and evaluation budget of the default 100-iteration AutoQS search. Only quantities directly supported by the frozen run artifacts are reported.
Method
W-bit
Group
MMLU
HellaSwag
ARC-E
ARC-C
PIQA
WinoGrande
BoolQ
Avg. ↑
FP16
16
–
71.76
80.48
81.06
55.03
80.36
70.72
86.45
75.12
GPTQ
4
128
70.77
79.73
80.13
54.35
80.25
70.56
85.41
74.46
AWQ
4
128
70.80
79.61
78.75
54.69
78.56
69.61
86.27
74.04
AutoQS
4
Mixed
70.71
79.68
80.64
57.94
80.30
70.48
86.36
75.16
Appendix
Table 4: 4-bit results on Qwen2.5-7B-Instruct. Dense FP16, GPTQ-Int4, AWQ, and AutoQS are evaluated using the same seven-task post-search protocol. Bold denotes the best result among 4-bit methods.
Method
W-bit
Group
MMLU
HellaSwag
ARC-E
ARC-C
PIQA
WinoGrande
BoolQ
Avg. ↑
FP16
16
–
41.84
76.00
76.35
46.25
79.11
69.06
77.71
66.62
RTN
4
128
39.45
74.36
74.22
42.22
77.60
66.82
74.55
64.17
GPTQ
4
128
38.73
74.76
75.00
43.09
78.45
68.27
75.14
64.78
AWQ
4
128
40.75
75.57
75.59
45.31
78.78
68.35
78.23
66.08
OmniQuant
4
128
41.35
74.88
75.21
44.54
78.94
68.19
77.16
65.75
CALDERA
4
128
39.41
75.79
75.70
43.84
81.02
68.45
76.96
65.88
Appendix
Table 5: 4-bit results on Llama2-7B. External baseline results are taken from Table 3 of FBQuant ( Liu et al., 2025 ) . The external quantized configurations use group size 128. AutoQS-Transfer applies the frozen allocation searched on the original LLaMA-7B source model without target-side allocation search or policy adaptation.
Model
Setting
MMLU
HellaSwag
ARC-E
ARC-C
PIQA
WinoGrande
BoolQ
Macro7 ↑
LLaMA-7B
Dense
32.17
76.20
72.85
44.80
79.16
69.93
75.17
64.33
LLaMA-7B
Source searched W4
27.03
73.82
71.42
42.92
79.27
70.09
74.28
62.69
Llama2-7B
Dense (local)
41.83
76.00
74.49
46.16
79.05
68.90
77.71
66.31
Llama2-7B
Uniform W4-G128
37.96
75.71
71.17
43.60
78.51
67.48
77.00
64.49
Llama2-7B
Zero-shot transfer
39.83
75.49
70.33
42.92
78.62
68.75
75.99
64.56
Vicuna-7B-v1.5
Dense (local)
48.75
73.77
71.25
45.90
77.97
69.53
80.92
66.87
Appendix
Table 6: Zero-shot quantization-policy transfer from original LLaMA-7B to related 7B models. The source policy is selected on LLaMA-7B using source Macro3 only and is then frozen. Target models perform no allocation search or policy adaptation; GPTQ calibration and packing are freshly performed on target weights.
W-bit
MMLU
HellaSwag
ARC-E
ARC-C
PIQA
WinoGrande
BoolQ
Macro7 ↑
2
25.02
26.18
25.13
25.26
51.69
50.20
50.09
36.22
3
58.66
70.06
65.36
45.73
73.29
58.96
74.50
63.79
4
70.43
78.69
80.35
56.57
79.54
68.43
85.44
74.21
Appendix
Table 7: Bit-width analysis of AutoQS on Qwen2.5-7B. Each row reports an independently evaluated fixed-bit search. Within a run, only the heterogeneous group-size allocation is searched Liu et al. (2026d) ; Liu et al. (2026c) .
LLM API
W-bit
Search seed
MMLU
HellaSwag
ARC-E
ARC-C
PIQA
WinoGrande
BoolQ
Macro7 ↑
Qwen-Plus
4
2026
70.18
78.24
79.29
54.18
79.82
66.38
84.83
73.27
Qwen-Plus
4
1234
70.43
78.69
80.35
56.57
79.54
68.43
85.44
74.21
DeepSeek
4
1234
70.33
78.44
79.46
54.52
79.76
68.35
84.95
73.69
Qwen-Plus
2
2026
25.49
26.12
24.87
27.99
51.96
49.57
48.78
36.40
DeepSeek
2
1234
25.86
26.34
25.00
27.30
53.10
50.51
49.97
36.87
Appendix
Table 8: Effect of LLM proposer, search seed, and weight precision on Qwen2.5-7B. The reported seed is the search/evolution seed. Post-search Final-7 evaluation uses seed 1234. All entries correspond to completed Final-7 evaluations.
Figure 3: Matched-seed evolutionary search dynamics on Qwen2.5-7B W4. All three runs use search seed 1234, 100 search iterations, and one physical evaluation every five iterations. (a) Physical Macro3 at each physical boundary; stars mark the best observed physical candidate under the predefined Macro3 criterion. (b) Best-so-far Macro3. (c) Fraction of module-level group-size assignments changed relative to the preceding physical policy; the first point is set to zero by convention. (d) Fraction of modules assigned group size 128.
Iteration
MMLU
HellaSwag
ARC-E
ARC-C
PIQA
WinoGrande
BoolQ
Macro7 ↑
10
70.48
78.37
80.60
55.80
79.27
68.03
84.43
73.86
25
70.10
77.93
80.56
54.95
78.29
67.56
85.47
73.55
60
69.98
78.52
80.89
56.23
80.03
67.01
84.43
73.87
70
69.91
78.43
80.51
56.06
79.65
67.25
84.65
73.78
100
70.43
78.69
80.35
56.57
79.54
68.43
85.44
74.21
Appendix
Table 9: Post-search Final-7 performance of selected frozen physical checkpoints from the same 100-iteration Qwen2.5-7B W4 trajectory. These checkpoints are intermediate states of one search and should not be interpreted as independent truncated-budget experiments.
Slot
Iter.
Macro3
Macro7
Search rank
Macro7 rank
4
20
64.9683
62.9818
8
1
6
30
65.3866
62.8990
2
2
9
45
65.0425
62.6471
6
7
10
50
64.9990
62.5527
7
8
17
85
65.3613
62.7483
3
4
18
90
65.2977
62.8870
4
3
Appendix
Table 10: Search-time Macro3 and post-search Macro7 for eight frozen physical candidates from the LLaMA-7B W4 trajectory. The search rank is determined exclusively by Macro3, whereas the Macro7 rank is computed post hoc for analysis.
Model
Setting
Modules
G=−1
G=16
G=32
G=64
G=128
Non-G128 (%)
Qwen2.5-7B
Full, seed 1234
196
12
14
15
15
140
28.57
Qwen2.5-7B
No-feedback, seed 1234
196
1
14
13
15
153
21.94
Qwen2.5-7B
Random proposer, seed 1234
196
8
5
4
3
176
10.20
LLaMA-7B
Source searched W4
224
11
15
15
15
168
25.00
Llama2-7B
Target-native best physical
224
15
11
24
18
156
30.36
Appendix
Table 11: Module-level group-size structure of frozen W4 policies. The three Qwen2.5-7B search-mechanism rows use the same search seed of 1234. The LLaMA-7B row is the frozen source policy used in the transfer experiment. The Llama2-7B row is the best observed physical policy under Macro3 among the 20 physical evaluations of the target-native search.
Policy pair
Different modules
Fraction (%)
Full vs. No-feedback
86 / 196
43.88
Full vs. Random proposer
63 / 196
32.14
No-feedback vs. Random proposer
63 / 196
32.14
Appendix
Table 12: Pairwise module-level disagreement between matched-seed Qwen2.5-7B W4 policies.
Method
Primary optimization object
Typical adaptation level
Uses calibration signal
Grouping in our tables
Relationship to AutoQS
RTN
Direct rounding to the target grid
Weight / quantization group
No learned optimization
Fixed G128
Reference fixed policy
GPTQ
Sequential one-shot weight quantization using approximate second-order information
Weight / block
Yes
Fixed G128
Physical quantizer used after policy search
AWQ
Activation-aware protection through channel scaling
Channel
Yes, activation statistics
Fixed G128
Orthogonal to module-level allocation
OmniQuant
Learnable clipping and equivalent transformations
Block / channel
Yes, differentiable calibration
Fixed G128
Optimizes quantization parameters rather than allocation topology
CALDERA
Low-precision plus low-rank matrix decomposition
Layer / matrix
Yes
Fixed G128 in reported comparison
Changes the weight representation itself
SVDQuant
SVD/low-rank-oriented baseline as reported by FBQuant
Method-specific
Method-specific
Fixed G128 in reported comparison
No module-level group policy inferred here
Appendix
Table 13: Comparison of the optimization dimensions represented by the quantization baselines and AutoQS. The “Grouping in our tables” column refers specifically to the configurations evaluated in this paper.
Stage
Manual heterogeneous-policy tuning
AutoQS
Choose modules to modify
Researcher manually selects target modules
Candidate generation and mutation are performed automatically
Choose group size
Researcher chooses a value for each target
Values are selected from the legal set under the search protocol
Validate legality
Researcher or ad-hoc scripts verify compatibility
Semantic and structural checks reject invalid candidates automatically
Low-cost screening
Researcher designs and runs exploratory tests
Stage-1 evaluation is applied to every valid unseen candidate
High-fidelity test
Researcher manually launches quantization and evaluation
Physical GPTQ + Macro3 is scheduled automatically
Use previous result
Researcher inspects results and chooses the next modification
Physical feedback can influence later search decisions
Appendix
Table 14: Conceptual comparison of manual heterogeneous-policy tuning and the automated AutoQS workflow. No numerical human-hours speedup is claimed because manual research time was not measured under a controlled protocol.
Quantization-aware training (QAT) leverages lower-precision arithmetic to reduce the cost of LLM deployment, but aggressive quantization degrades final model performance. A common remedy is mixed-precision training, in which high precision is assigned to some of the layers to maintain performance while keeping the cost constrained. This approach then requires precision assignments for model layers during training. We provide a new approach, called Q-PACE, consisting of a second-order sensitivity model that predicts the loss increase as a sum of quantization noise MSE weighted by per-layer curvature coefficients. During training, we periodically re-compute these coefficients using perturbations across layers, and re-assign precision. Pretraining and supervised fine-tuning experiments on LLMs of up to 4B parameters show that Q-PACE consistently improves over existing mixed-precision training recipes, and achieves comparable loss at substantially lower total memory budgets. We further find that quantization sensitivity is highly predictable by depth and layer type, and its stability during training allows for infrequent, cheap recalibration.
Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2
Prefill and decode reward different approaches to quantization: low-precision arithmetic accelerates prompt processing, while compact weights reduce memory traffic during generation. We propose "disaggregated quantization" (DQ), which specializes computation formats, weights and storage placement to both of these phases. On Qwen 3 and Gemma 3, removing activation quantization specifically on decode improves accuracy on decode-heavy tasks without increasing inference cost. Training separate compute-native prefill weights accelerates prompt processing relative to weight-only inference while matching or exceeding its accuracy at 2-3-bit decode on both decode-heavy and prefill-heavy tasks. With released Qwen3.8-27B GGUF decoders, training an NVFP4 prefiller improves 1-bit accuracy by 32.5 points on MMLU-Pro and 35.3 on MMMU-Pro without modifying the decode checkpoint. To accommodate the additional checkpoint on a single device, offloaded disaggregated prefill (ODP) streams its weights from SSD, amortizing loading over prompt length. On the same 27B model, ODP delivers a 1.78x time-to-first-token speedup over the weight-only baseline at 8K prompt length in llama.cpp. We evaluate accuracy under disaggregated serving in vLLM and further validate shared-weight format disaggregation through post-training quantization on models up to 2.8T parameters.
Andrei Panferov, Maximilian Kleinegger, Sweta Priyadarshi +2
In this paper, we address post-training quantization (PTQ) for large language models (LLMs) from an overlooked perspective: given a pre-trained high-precision LLM, the predominant sequential quantization framework treats different layers equally, but this may be not optimal in challenging bit-width settings. We empirically study the quantization impact of different layers on model accuracy, and observe that: (1) shallow/deep layers are usually more sensitive to quantization than intermediate layers; (2) among shallow/deep layers, the most sensitive one is the first/last layer, which exhibits significantly larger quantization error than others. These empirical observations imply that the quantization design for different layers of LLMs is required on multiple levels instead of a single level shared to all layers. Motivated by this, we propose a new PTQ framework termed Sliding-layer Quantization (SliderQuant) that relies on a simple adaptive sliding quantization concept facilitated by few learnable parameters. The base component of SliderQuant is called inter-layer sliding quantization, which incorporates three types of novel sliding window designs tailored for addressing the varying quantization sensitivity of shallow, intermediate and deep layers. The other component is called intra-layer sliding quantization that leverages an incremental strategy to quantize each window. As a result, SliderQuant has a strong ability to reduce quantization errors across layers. Extensive experiments on basic language generation, zero-shot commonsense reasoning and challenging math and code tasks with various LLMs, including Llama/Llama2/Llama3/Qwen2.5 model families, DeepSeek-R1 distilled models and large MoE models, show that our method outperforms existing PTQ methods (including the latest PTQ methods using rotation transformations) for both weight-only quantization and weight-activation quantization.