Post-training quantization is the standard way to serve large language models at low cost. However, the best quantization recipe differs across models and precisions. As a result, experts must redesign it for every new model, which is time-consuming and often sub-optimal. Fortunately, an LLM agent can take over this design by refining the recipe over rounds of building and evaluating candidates, so that more rounds lead to better recipes. However, evaluating one quantized reasoning model takes tens of GPU-hours, and combining methods from different codebases means porting each to the new architecture by hand. We present QDA, a system in which an agent designs the recipe for each model and precision. To make evaluation fast, QDA scores each candidate in minutes by its output KL divergence from the original model on reasoning traces that the agent never sees, and checks promising ones with long generations to catch failures such as repetition loops. To make building fast, QDA provides a library of public PTQ methods that are compatible with one another and with new model architectures. Across four reasoning models, QDA improves accuracy over the strongest public recipe by 0.9, 1.4 and 2.2 percentage points on average at W4A16, W4A8 and W3A16, using 64 GPU-hours per search. On the 320B-parameter MoE model GLM-5.3-Flash at NVFP4 W4A4, it recovers 98.3% of the FP8 model's accuracy, against 96.9% for the best vendor checkpoint. Every QDA checkpoint also runs on existing serving kernels with no inference overhead. We will release the library, the evaluator and the harness.
Figures & tables
Figure 1: QDA beats the strongest public recipe at low cost. Left: average accuracy of four reasoning models at each precision, for the strongest public recipe and for QDA. Middle: QDA checkpoints run on existing kernels with no inference overhead. Right: the GPU-hours and agent tokens of one search.
Figure 2: The best recipe differs across models and precisions, and a full evaluation costs hundreds of times more than a KL score. (a) Accuracy change from BF16 of three public recipes and QDA at W4A16 on four models. A star marks the most accurate public recipe. On Q3.5-9B at W3A16, AWQ and GPTQ swap order. (b) Accuracy change at W4A8 of FP8-aware GPTQ, which fits GPTQ’s codes to the kernel’s FP8 scales instead of BF16 scales. This helps Q3.5-9B but hurts the other three. (c) Left: share of 44 checkpoint pairs that each measure ranks in the same order as accuracy. Right: GPU-minutes to compute each measure on one R1-8B checkpoint. Q3.5-9B, Q3.6-35B, Q3.8-27B and R1-8B denote Qwen3.5-9B, Qwen3.6-35B-A3B, Qwen3.8-27B and DeepSeek-R1-Distill-Llama-8B.
Precision
Weights
Activations
Kernel
W4A16
INT4 g128 asym
BF16
Marlin
W3A16
INT3 g128 sym; head + embeddings INT4 g64
BF16
Humming
W4A8
INT4 g128 sym
FP8 (E4M3) per-token
CUTLASS W4A8
NVFP4 W4A4
FP4 (E2M1) g16, FP8 scales
FP4 (E2M1) g16
NVFP4 GEMM
Table 1: Precisions evaluated. The first three run on GH200 with vLLM 0.27.1 for the four main models, and NVFP4 runs on GB300 with vLLM 0.31 for GLM-5.3-Flash.
Figure 3: The QDA loop. Every candidate gets a KL score within minutes, a few promising ones also get a long-generation check, and the agent combines both to decide what to change next. (a) The four steps of one round. The numbers match steps (1)–(4) in § 4 . (b) The KL of every checkpoint in one search, relative to the public GPTQ checkpoint. The line shows the lowest KL so far. The blue label marks a long-generation check, and the black labels mark two changes that the agent made (§ 4.3 ).
Figure 4: System architecture of QDA. QDA has three components: the composable library (§ 5.1 ), the isolated evaluator (§ 5.2 ) and the harness (§ 5.3 ), which runs the agent in a sandbox and schedules its GPU jobs. From the sandbox, the agent can reach the rest of the system only through four commands: (1) qda builds a recipe in the library and returns its checkpoint, (2) kl sends a checkpoint to the evaluator and returns its KL, (3) qsched submits other jobs, such as long-generation checks, to the scheduler, and (4) handin submits the final checkpoints, which are evaluated on the test benchmarks after the search.
Prec.
Method
Qwen3.5-9B
Qwen3.6-35B-A3B
Qwen3.8-27B
DeepSeek-R1-Distill-Llama-8B
Math
GPQA
LCB
IFB
Avg
Math
GPQA
LCB
IFB
Avg
Math
GPQA
LCB
IFB
Avg
Math
GPQA
LCB
IFB
Avg
BF16
–
77.0
82.3
70.9
66.7
74.2
86.1
85.4
81.6
66.0
79.8
80.1
82.7
87.2
67.9
79.5
30.5
49.0
38.9
63.8
45.6
W4A16
GPTQ
62.5
78.6
53.2
66.7
65.3
82.8
83.9
79.8
64.5
77.7
79.2
81.3
81.6
66.8
77.2
28.3
47.5
36.8
60.9
43.3
AWQ
72.8
79.6
63.5
65.5
70.4
80.9
83.5
79.5
63.1
76.8
78.3
82.1
85.2
67.6
78.3
28.1
48.1
36.6
59.8
43.1
ParoQuant
73.9
80.2
68.7
64.2
71.8
82.7
84.0
79.2
65.1
77.7
75.6
80.9
83.5
67.6
76.9
28.3
46.7
34.9
62.4
43.0
QDA
74.5
80.0
68.3
65.7
72.1
84.5
85.0
81.0
65.3
79.0
82.7
81.9
86.3
68.2
79.8
27.6
48.1
37.8
62.7
44.1
Table 2: Main results. QDA improves the average accuracy over the strongest public recipe by 0.9, 1.4 and 2.2 points at W4A16, W4A8 and W3A16, and by up to 6.1 points (DeepSeek-R1-Distill-Llama-8B at W3A16). Accuracy (%) of four reasoning models on math, GPQA-Diamond (GPQA), LiveCodeBench (LCB) and IFBench (IFB; IFEval for R1-Llama), and their average (Avg). Each QDA row is the final checkpoint of one 64-GPU-hour search, and the Qwen3.6-35B-A3B rows are means of three searches. In each Avg column, the best quantized result is in bold and the second best is underlined. Within each precision, all rows except ParoQuant, which adds an online rotation, run on the same kernel ( Table 1 ). QuaRot+GPTQ uses only QuaRot’s offline rotations (R1 on the Qwen models, R1 and R2 on R1-Llama). At W3A16, all Qwen searches but one on Qwen3.6-35B-A3B start from the GSQ checkpoint in this table and refine it.
Benchmark
FP8
NVIDIA
Red Hat AI
QDA
Math (93, × 16)
84.8
80.9
78.8
83.3
GPQA-D (198, × 12)
86.1
83.3
84.0
83.9
LCB v6 (454, × 4)
71.3
69.3
69.2
70.4
SuperGPQA (9k, × 1)
68.7
67.8
67.4
67.9
Avg
77.7
75.3
74.8
76.4
% of FP8
100
96.9
96.3
98.3
Table 3: GLM-5.3-Flash at NVFP4. QDA recovers 98.3% of the FP8 accuracy on average, against 96.9% and 96.3% for the NVFP4 releases of NVIDIA and Red Hat AI. Parentheses give the number of problems and samples per problem. Bold and underline mark the best and second-best NVFP4 results.
Prec.
Best public
Agent + ModelOpt
QDA
W4A16
77.7
78.5
79.0
W4A8
77.4
76.2
77.5
W3A16
75.9
74.5
75.7
Table 4: QDA beats the same agent with ModelOpt’s workflow and tools by 0.5–1.3 points on Qwen3.6-35B-A3B, with the same 64 GPU-hours. Four-category mean accuracy (%).
Figure 5: What the searches found. No published practice helps every model, and the step that lowers the KL most differs from search to search. (a) Change in KL at W4A16 when one published practice is added to a recipe, measured between two checkpoints of the same search whose recipes differ only in that practice. Lower is better. Each practice lowers the KL on one model but not on another. (b) The step with the largest KL drop in each of the 12 final recipes. Bold notes mark problems that the agent diagnosed itself.
(a) Ablation on Qwen3.6-35B-A3B
Prec.
BF16
Greedy
ASHA
w/o infra.
QDA
W4A16
79.8
78.9
78.9
78.8
79.0
W4A8
79.8
76.5
76.3
77.0
77.5
W3A16
79.8
75.0
75.5
75.5
75.7
Table 5: Ablation and a fixed recipe. (a) Replacing the agent with a predefined search algorithm (greedy search, ASHA) or replacing the library and scheduler with public tools and a plain GPU queue (w/o infra.), on Qwen3.6-35B-A3B with the same 64 GPU-hours, KL score and submission rule. QDA and w/o infra. are means of three searches. (b) A fixed recipe that came out of QDA’s searches: the pattern that five final recipes follow, applied with fixed settings and no search. It calibrates GPTQ on 640 answers that the original model wrote and then runs two rounds of scale distillation. Four-category mean accuracy (%). In each row, the best quantized result is in bold and the second best underlined.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Attention
MLP
Qwen3.5-9B
hybrid, Gated DeltaNet
dense
Qwen3.6-35B-A3B
hybrid, Gated DeltaNet
MoE, 8 of 256
Qwen3.8-27B
hybrid, Gated DeltaNet
dense
DeepSeek-R1-Distill-Llama-8B
full
dense
Appendix
Table 6: Models in the main suite. Hybrid models interleave full-attention layers with linear-attention (Gated DeltaNet) layers.
Batch
ParoQuant
GPTQ
QDA
Speedup
1
60
105
105
1.75 ×
16
805
1,262
1,263
1.57 ×
64
1,789
2,330
2,333
1.30 ×
Appendix
Table 7: Decode throughput (output tokens/s) of the W4A16 checkpoints of Qwen3.8-27B, measured together on one GH200 with vLLM 0.27.1. QDA runs on the same kernel as GPTQ and decodes within 0.2% of it, whereas ParoQuant adds an online rotation. Speedup is QDA over ParoQuant.
Method
Source
Full-attn.
Hybrid
Hybrid MoE
Transformations
Hadamard rotation (residual)
QuaRot
✓
✓
✓
Hadamard rotation (value heads)
QuaRot
✓
–
–
Channel permutation
deepcompressor
✓
✓
✓
Activation-aware scaling
AWQ
✓
✓
✓
Rounding
Appendix
Table 8: The 21 methods of the library , grouped by the decisions of § 2.2 (three clipping searches share a row), and the architectures that their code handles: full-attention (DeepSeek-R1-Distill-Llama-8B), Gated DeltaNet hybrid (Qwen3.5-9B, Qwen3.8-27B) and hybrid MoE (Qwen3.6-35B-A3B).
Figure 6: Final recipes of the 12 searches. Each row is one final checkpoint, read left to right. “Self-generated” calibration mixes reasoning traces that the BF16 model wrote with public text, scale KD distills the BF16 model into the quantized one by tuning the scales (and, where marked, the norms), and ×n marks n rounds of it.
Figure 7: Where the 64 GPU-hours of each ablation search go (Qwen3.6-35B-A3B). “w/o agent” is greedy search of Table 5 and “w/o infra.” is the agent without the infrastructure. “Generation check” is the agent’s checks of its own checkpoints, “other” is mostly calibration and training data that the agent generates, and “failed” is every job that errored, ran out of memory or was cancelled.
Slot
Steps (grids)
Start
BF16 model; public AWQ and GPTQ checkpoints
Transform
AWQ scale search writing a new BF16 base; channel permutation; Hadamard rotation of the residual stream
Solver
RTN, two implementations (min-max, MSE, importance-weighted MSE); GPTQ (min-max, MSE, Hessian-weighted MSE, importance-weighted MSE); GPTQ with router-weighted expert Hessians (same four); AWQ then RTN (min-max, MSE, importance-weighted MSE); MSE and importance-weighted clip search with RTN; AutoRound (min-max)
Refinement
End-to-end scale tuning by distillation from the BF16 model (codes and zero points frozen)
Appendix
Table 9: Search space of greedy search and ASHA on Qwen3.6-35B-A3B at W4A16, taken from the library’s method table. Steps whose method does not support this precision or architecture are left out.
Section
What it says
Goal
The best W4A16 checkpoint by the deadline. It is scored later on math, code, science and instruction following, which count equally, and nothing measurable in the workspace is that evaluation.
Clock and resources
Two nodes of four GH200s until the deadline, when every task is cancelled. Plan for idle GPUs in the last hour. Run long-generation checks (up to 81,920 tokens) on the top checkpoints in all four domains before submitting. Training data may include the quantized model’s own answers scored by the BF16 model. When the latest checkpoints of the current line of recipes tie in KL, try another line. Reread the notes and failed logs every couple of hours.
Workspace and tools
The quantization and serving environments, offline datasets, a format-compatibility package with no quantization method, and the qda library: atomic transforms, solvers and refinements to stack, plus measurements ( proxy , profiler ), diagnostics ( probe ) and mix . The method cards, loaded as a Claude Code skill. New methods and subagents are allowed.
Prior
The BF16 model and the public checkpoints with their methods. ParoQuant is listed for reference only, since it needs extra kernels. A public checkpoint cannot be submitted unchanged.
Format
INT4, g=128 , asymmetric, on the Marlin kernels; the same modules and bits as the public checkpoints; nothing added at inference. Forbidden: other bit widths or per-module bits, modules from foreign checkpoints, mixed symmetric and asymmetric modules.
Data
No build data may contain an evaluation item or a text of the reference traces.
Appendix
Table 10: The task statement for Qwen3.6-35B-A3B at W4A16, abridged. Every variant of Table 5 receives the same text, except that the variant without the infrastructure receives it without the lines about the library.