QDA: Closed-Loop LLM Quantization Design with Agents
Abstract
Post-training quantization is the standard way to serve large language models at low cost. However, the best quantization recipe differs across models and precisions. As a result, experts must redesign it for every new model, which is time-consuming and often sub-optimal. Fortunately, an LLM agent can take over this design by refining the recipe over rounds of building and evaluating candidates, so that more rounds lead to better recipes. However, evaluating one quantized reasoning model takes tens of GPU-hours, and combining methods from different codebases means porting each to the new architecture by hand. We present QDA, a system in which an agent designs the recipe for each model and precision. To make evaluation fast, QDA scores each candidate in minutes by its output KL divergence from the original model on reasoning traces that the agent never sees, and checks promising ones with long generations to catch failures such as repetition loops. To make building fast, QDA provides a library of public PTQ methods that are compatible with one another and with new model architectures. Across four reasoning models, QDA improves accuracy over the strongest public recipe by 0.9, 1.4 and 2.2 percentage points on average at W4A16, W4A8 and W3A16, using 64 GPU-hours per search. On the 320B-parameter MoE model GLM-5.3-Flash at NVFP4 W4A4, it recovers 98.3% of the FP8 model's accuracy, against 96.9% for the best vendor checkpoint. Every QDA checkpoint also runs on existing serving kernels with no inference overhead. We will release the library, the evaluator and the harness.
Figures & tables
| Precision | Weights | Activations | Kernel |
| W4A16 | INT4 g128 asym | BF16 | Marlin |
| W3A16 | INT3 g128 sym; head + embeddings INT4 g64 | BF16 | Humming |
| W4A8 | INT4 g128 sym | FP8 (E4M3) per-token | CUTLASS W4A8 |
| NVFP4 W4A4 | FP4 (E2M1) g16, FP8 scales | FP4 (E2M1) g16 | NVFP4 GEMM |
| Prec. | Method | Qwen3.5-9B | Qwen3.6-35B-A3B | Qwen3.8-27B | DeepSeek-R1-Distill-Llama-8B | ||||||||||||||||
| Math | GPQA | LCB | IFB | Avg | Math | GPQA | LCB | IFB | Avg | Math | GPQA | LCB | IFB | Avg | Math | GPQA | LCB | IFB | Avg | ||
| BF16 | – | 77.0 | 82.3 | 70.9 | 66.7 | 74.2 | 86.1 | 85.4 | 81.6 | 66.0 | 79.8 | 80.1 | 82.7 | 87.2 | 67.9 | 79.5 | 30.5 | 49.0 | 38.9 | 63.8 | 45.6 |
| W4A16 | GPTQ | 62.5 | 78.6 | 53.2 | 66.7 | 65.3 | 82.8 | 83.9 | 79.8 | 64.5 | 77.7 | 79.2 | 81.3 | 81.6 | 66.8 | 77.2 | 28.3 | 47.5 | 36.8 | 60.9 | 43.3 |
| AWQ | 72.8 | 79.6 | 63.5 | 65.5 | 70.4 | 80.9 | 83.5 | 79.5 | 63.1 | 76.8 | 78.3 | 82.1 | 85.2 | 67.6 | 78.3 | 28.1 | 48.1 | 36.6 | 59.8 | 43.1 | |
| ParoQuant | 73.9 | 80.2 | 68.7 | 64.2 | 71.8 | 82.7 | 84.0 | 79.2 | 65.1 | 77.7 | 75.6 | 80.9 | 83.5 | 67.6 | 76.9 | 28.3 | 46.7 | 34.9 | 62.4 | 43.0 | |
| QDA | 74.5 | 80.0 | 68.3 | 65.7 | 72.1 | 84.5 | 85.0 | 81.0 | 65.3 | 79.0 | 82.7 | 81.9 | 86.3 | 68.2 | 79.8 | 27.6 | 48.1 | 37.8 | 62.7 | 44.1 | |
| Benchmark | FP8 | NVIDIA | Red Hat AI | QDA |
| Math (93, 16) | 84.8 | 80.9 | 78.8 | 83.3 |
| GPQA-D (198, 12) | 86.1 | 83.3 | 84.0 | 83.9 |
| LCB v6 (454, 4) | 71.3 | 69.3 | 69.2 | 70.4 |
| SuperGPQA (9k, 1) | 68.7 | 67.8 | 67.4 | 67.9 |
| Avg | 77.7 | 75.3 | 74.8 | 76.4 |
| % of FP8 | 100 | 96.9 | 96.3 | 98.3 |
| Prec. | Best public | Agent + ModelOpt | QDA |
| W4A16 | 77.7 | 78.5 | 79.0 |
| W4A8 | 77.4 | 76.2 | 77.5 |
| W3A16 | 75.9 | 74.5 | 75.7 |
| (a) Ablation on Qwen3.6-35B-A3B | |||||
| Prec. | BF16 | Greedy | ASHA | w/o infra. | QDA |
| W4A16 | 79.8 | 78.9 | 78.9 | 78.8 | 79.0 |
| W4A8 | 79.8 | 76.5 | 76.3 | 77.0 | 77.5 |
| W3A16 | 79.8 | 75.0 | 75.5 | 75.5 | 75.7 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Attention | MLP |
| Qwen3.5-9B | hybrid, Gated DeltaNet | dense |
| Qwen3.6-35B-A3B | hybrid, Gated DeltaNet | MoE, 8 of 256 |
| Qwen3.8-27B | hybrid, Gated DeltaNet | dense |
| DeepSeek-R1-Distill-Llama-8B | full | dense |
| Batch | ParoQuant | GPTQ | QDA | Speedup |
| 1 | 60 | 105 | 105 | 1.75 |
| 16 | 805 | 1,262 | 1,263 | 1.57 |
| 64 | 1,789 | 2,330 | 2,333 | 1.30 |
| Method | Source | Full-attn. | Hybrid | Hybrid MoE |
| Transformations | ||||
| Hadamard rotation (residual) | QuaRot | ✓ | ✓ | ✓ |
| Hadamard rotation (value heads) | QuaRot | ✓ | – | – |
| Channel permutation | deepcompressor | ✓ | ✓ | ✓ |
| Activation-aware scaling | AWQ | ✓ | ✓ | ✓ |
| Rounding | ||||
| Slot | Steps (grids) |
| Start | BF16 model; public AWQ and GPTQ checkpoints |
| Transform | AWQ scale search writing a new BF16 base; channel permutation; Hadamard rotation of the residual stream |
| Solver | RTN, two implementations (min-max, MSE, importance-weighted MSE); GPTQ (min-max, MSE, Hessian-weighted MSE, importance-weighted MSE); GPTQ with router-weighted expert Hessians (same four); AWQ then RTN (min-max, MSE, importance-weighted MSE); MSE and importance-weighted clip search with RTN; AutoRound (min-max) |
| Refinement | End-to-end scale tuning by distillation from the BF16 model (codes and zero points frozen) |
| Section | What it says |
| Goal | The best W4A16 checkpoint by the deadline. It is scored later on math, code, science and instruction following, which count equally, and nothing measurable in the workspace is that evaluation. |
| Clock and resources | Two nodes of four GH200s until the deadline, when every task is cancelled. Plan for idle GPUs in the last hour. Run long-generation checks (up to 81,920 tokens) on the top checkpoints in all four domains before submitting. Training data may include the quantized model’s own answers scored by the BF16 model. When the latest checkpoints of the current line of recipes tie in KL, try another line. Reread the notes and failed logs every couple of hours. |
| Workspace and tools | The quantization and serving environments, offline datasets, a format-compatibility package with no quantization method, and the qda library: atomic transforms, solvers and refinements to stack, plus measurements ( proxy , profiler ), diagnostics ( probe ) and mix . The method cards, loaded as a Claude Code skill. New methods and subagents are allowed. |
| Prior | The BF16 model and the public checkpoints with their methods. ParoQuant is listed for reference only, since it needs extra kernels. A public checkpoint cannot be submitted unchanged. |
| Format | INT4, , asymmetric, on the Marlin kernels; the same modules and bits as the public checkpoints; nothing added at inference. Forbidden: other bit widths or per-module bits, modules from foreign checkpoints, mixed symmetric and asymmetric modules. |
| Data | No build data may contain an evaluation item or a text of the reference traces. |
LLM Quantization
Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.
See all 448 papers