DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs

Mahsa Salmani; Sayeh Sharify; Hesham Mostafa

cs.LGOct 9, 2026

DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs

Authors: Mahsa Salmani, Sayeh Sharify, Hesham Mostafa

Organizations: d-Matrix

Abstract

Low-bit inference on accelerators with native 4-bit support requires quantizing both weights and activations to realize the full throughput gains these formats provide. While post-training weight quantization has been extensively studied, activation quantization remains considerably more challenging and typically requires calibration data. In this work, we introduce DUALQUANT, a dual-scale, calibration-free post-training quantization framework that directly minimizes the weight-reconstruction Mean Squared Error (MSE) and, as an immediate by-product, yields per-channel activation rescalings prior to activation quantization. Our key observation is that the scaling factors obtained by minimizing weight-reconstruction error provide effective activation rescalings, enabling competitive activation quantization without calibration data. We formulate DUALQUANT as a non-differentiable weight-reconstruction problem and develop an efficient solver based on variable splitting and block coordinate descent, where each subproblem admits a closed-form update. DUALQUANT provides a unified algorithm supporting a wide range of formats, including the full MX family and NVFP4. To our knowledge, DUALQUANT is the first dual-scaling method to formally support MX and NV block formats. Across Llama and Qwen models evaluated on WikiText-2, C4, and zero-shot downstream tasks, DUALQUANT matches or exceeds the accuracy of calibration-based SmoothQuant across most models and formats, and substantially outperforms calibration-free baselines, with large gains for MX and NV block formats. We validate these gains with a real packed W4A4 inference path for Llama-3.1-8B on an RTX 4090. The resulting implementation closely matches the accuracy of fake quantization while delivering up to 2.61x memory reduction and up to 2.46x throughput speedup over BF16.

Figures & tables

Figure 1: Per-element quantization error $\scriptstyle|\mathbf{W}-\hat{\mathbf{W}}|$ under different scale designs. Each panel reports the MSE ratio ( $\uparrow$ ) relative to single-scale and the matrix imbalance (SINQ’s objective). (a) Original weight tile. (b) Single-scale: one outlier inflates the entire row. (c) SINQ minimizes matrix imbalance ( $1.65$ ) but achieves only $3.32\times$ MSE reduction. (d) DualQuant minimizes MSE, achieving $12.54\times$ reduction, illustrating that low imbalance does not imply low reconstruction error.

Figure 2: (a) Relative change

\scriptstyle\|\hat{\mathbf{Q}}^{(t)}-\hat{\mathbf{Q}}^{(t-1)}\|_{F}/\|\hat{\mathbf{Q}}^{(t-1)}\|_{F}

vs. BCD iteration for five transformer blocks. The change decays to near zero by

t{\approx}16

for all layers and blocks, validating the stabilization condition of Theorem 1 . (b) Normalized MSE

\scriptstyle\|\mathbf{W}-\hat{\mathbf{W}}\|_{F}^{2}/\|\mathbf{W}\|_{F}^{2}

vs. BCD iteration for the same blocks. All curves decrease monotonically and plateau within

{\approx}10

iterations, confirming Theorem 1 (i).

Bits	Format	Method	CF	WikiText-2 PPL $\downarrow$			C4 PPL $\downarrow$
Bits	Format	Method	CF	Llama-2-7B	Llama-3.1-8B	Llama-3.2-1B	Llama-2-7B	Llama-3.1-8B	Llama-3.2-1B
BF16 (reference)			–	5.11	5.62	8.67	7.04	8.82	13.53
4-bit	A: INT4 W: INT4 $(G{=}64)$	RTN	✓	5.73	7.11	12.06	7.97	11.99	19.69
		AWQ+	✗	5.57	6.83	12.46	7.73	11.51	20.24
		SmoothQuant	✗	5.68	6.98	13.37	7.91	11.75	21.99
		SINQ+	✓	5.61	6.83	12.36	7.80	11.45	20.68

Table 1: WikiText-2 and C4 perplexity (

\downarrow

) for joint W+A quantization at 4-bit precision on the Llama family. DualQuant applies

\bm{\beta}

-rescaling to activations without calibration data. SmoothQuant and AWQ+ use activation statistics from 256 calibration samples. SINQ+ is our extension of SINQ to W+A. QuaRot rotates activations/weights online with a Hadamard transform (no calibration). Bold : best calibration-free ( CF ) result. (

\dagger

): degenerate result

(PPL>1000)

; method incompatible with format. ✓ : calibration-free. ✗ : requires calibration data.

Llama-3.1-8B
Format	Method	CF	MMLU	HellaSwag	PIQA	ARC-Easy	WinoGrande
INT4 (G=64)	RTN	✓	54.89	56.44	77.86	77.40	68.90
	AWQ+	✗	55.78	56.65	78.51	77.19	69.06
	SmoothQuant	✗	55.24	55.93	77.31	77.44	68.75
	SINQ+	✓	55.50	56.68	77.09	77.65	67.72
	QuaRot	✓	54.21	56.16	77.04	77.40	70.17

Table 2: Llama-3.1-8B downstream accuracy (0-shot, %, higher is better

\uparrow

) for RTN, AWQ+, SmoothQuant, SINQ+, DualQuant , and QuaRot, across all four evaluated formats; QuaRot is omitted entirely for NVFP4, since QuaRot’s Hadamard rotation redistributes activation magnitude in a way that collapses NVFP4’s narrow-range E4M3 block scale. Bold : best calibration-free (CF) result per column within each model+format block. ✓ : calibration-free. ✗ : requires calibration data.

Workload	BF16	W4A4	Reduction	BF16	W4A4	Speedup
Workload	Memory (GB)	Memory (GB)	Memory	Throughput (tok/s)	Throughput (tok/s)
Prefill $(b{=}1)$	16.33	6.25	$2.61\times$	3904.6	6421.4	$1.64\times$
Prefill $(b{=}16)$	16.66	6.58	$2.53\times$	8940.3	22003.4	$2.46\times$
Decode $(b{=}1)$	16.34	6.26	$2.61\times$	56.6	83.7	$1.48\times$
Decode $(b{=}16)$	16.75	6.67	$2.51\times$	694.5	964.1	$1.39\times$

Table 3: End-to-end memory usage and throughput of the real packed W4A4 deployment versus BF16 for Llama-3.1-8B on an RTX 4090 with INT4,

G{=}64

. Prefill uses an 82-token prompt without a KV cache. Decode generates

100

tokens with a 182-token KV cache. Memory includes weights, scales, unquantized embedding and lm_head, activations, and the KV cache for decode.

Table 6

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: (a) Relative change

\scriptstyle\|\hat{\mathbf{Q}}^{(t)}-\hat{\mathbf{Q}}^{(t-1)}\|_{F}/\|\hat{\mathbf{Q}}^{(t-1)}\|_{F}

vs. BCD iteration for five transformer blocks. The

\hat{\mathbf{Q}}^{(t)}

change decays to near zero by

t{\approx}16

for all layers and blocks, validating the stabilization condition of Theorem 1 . (b) Normalized MSE

\scriptstyle\|\mathbf{W}-\hat{\mathbf{W}}\|_{F}^{2}/\|\mathbf{W}\|_{F}^{2}

vs. BCD iteration for the same blocks. All curves decrease monotonically and plateau within

{\approx}10

iterations, confirming Theorem 1 (i). Together with the Qwen2.5-7B results in Figure 2 , this illustrates that the convergence behavior of DualQuant is consistent across the two model families evaluated here. Notably, down_proj consistently shows the highest residual MSE, consistent with it being among the harder projections to quantize ( Frantar et al., 2023 ) .

Projection	$\beta$ applied	$\beta$ not applied	$\Delta$ (dB)
k_proj	17.61	16.80	$+0.81$
v_proj	17.45	16.80	$+0.65$
up_proj	18.48	18.20	$+0.27$
q_proj	17.05	16.80	$+0.25$
gate_proj	18.36	18.20	$+0.15$
o_proj	17.40	17.34	$+0.07$

Appendix

Table 6: Per-projection activation SQNR (dB,

\uparrow

) with and without DualQuant ’s

\beta

, on Llama-3.1-8B block 0 (INT4,

G{=}64

). Quantized weights

\hat{Q}

are held fixed; only the presence of the

\beta

pre-scaling varies.

\Delta>0

indicates

\beta

reduces activation quantization error. Each

\Delta

is computed at full precision and may differ slightly from the difference of the rounded columns. Other blocks exhibit the same qualitative behavior.

Bits	Format	Method	CF	Qwen2.5-7B	Qwen3-0.6B	Qwen3-14B	Qwen2.5-7B	Qwen3-0.6B	Qwen3-14B
				WikiText-2 PPL $\downarrow$			C4 PPL $\downarrow$
BF16 (reference)			–	6.15	18.14	7.54	9.12	27.30	11.06
4-bit	A: INT4 W: INT4 $(G{=}64)$	RTN	✓	7.47	49.44	8.39	13.50	74.78	14.83
		AWQ+	✗	7.36	38.80	8.19	13.21	59.20	14.43
		SmoothQuant	✗	7.10	30.67	8.12	12.74	50.73	14.21
		SINQ+	✓	7.28	30.21	8.02	12.94	49.05	14.15

Appendix

Table 7: WikiText-2 and C4 perplexity (

\downarrow

) for joint weight-and-activation quantization at 4-bit precision on the Qwen family. DualQuant applies

\bm{\beta}

\dagger

): degenerate result

(PPL>1000)

; method incompatible with format. ✓ : calibration-free. ✗ : requires calibration data.

Qwen3-14B
Format	Method	CF	MMLU	HellaSwag	PIQA	ARC-Easy	WinoGrande
INT4 (G=64)	RTN	✓	70.33	75.23	77.64	77.36	69.06
	AWQ+	✗	71.92	75.03	77.86	79.08	69.53
	SmoothQuant	✗	73.04	76.74	77.91	80.77	71.51
	SINQ+	✓	72.30	76.23	77.80	79.59	70.17
	QuaRot	✓	71.30	73.52	76.77	77.99	67.88

Appendix

Table 8: Qwen3-14B downstream accuracy (0-shot, %, higher is better

\uparrow

) for RTN, AWQ+, SmoothQuant, SINQ+, DualQuant , and QuaRot, across all 4 weight formats; QuaRot is omitted entirely for NVFP4, since QuaRot’s Hadamard rotation redistributes activation magnitude in a way that collapses NVFP4’s narrow-range E4M3 block scale. Bold : best calibration-free (CF) result per column within each model+format block. ✓ : calibration-free. ✗ : requires calibration data.

Bits	Format	Method	CF	Qwen2.5-7B	Qwen3-0.6B	Qwen3-14B	Llama-3.1-8B
BF16 (reference)			–	6.15	18.14	7.54	5.62
8-bit	A: INT8 W: INT8 $(G{=}128)$	RTN	✓	6.27	18.42	7.55	5.72
		SmoothQuant	✗	6.25	18.28	7.50	5.70
		SINQ+	✓	6.67	18.33	7.49	5.70
		DualQuant	✓	6.64	18.24	7.77	5.69
	A: MXINT8 W: MXINT8 $(G{=}32)$	RTN	✓	6.15	18.06	7.54	5.63

Appendix

Table 9: WikiText-2 perplexity (

\downarrow

) for 8-bit W+A quantization. DualQuant applies

\bm{\beta}

-rescaling without calibration data. SmoothQuant uses 256 calibration samples. SINQ+ is our extension of SINQ to W+A. Bold : best calibration-free ( CF ) result. ✓ : calibration-free. ✗ : requires calibration data.

Bits	Format	Method	CF	Qwen2.5-7B	Qwen3-0.6B	Qwen3-14B	Llama-3.1-8B
BF16 (reference)			–	6.15	18.14	7.54	5.62
4-bit	W: INT4 $(G{=}64)$	RTN	✓	6.67	26.33	7.93	6.32
		AWQ	✗	6.50	21.83	7.72	6.12
		GPTQ	✗	6.44	20.39	7.73	5.56
		SINQ	✓	6.57	22.21	7.76	6.16
		DualQuant	✓	6.55	21.35	7.88	6.12

Appendix

Table 10: WikiText-2 perplexity (

\downarrow

) for 4-bit weight-only quantization. Activations in BF16. DualQuant applies

\bm{\beta}

-rescaling without calibration data. AWQ and GPTQ use 256 calibration samples. (

\dagger

): degenerate result (PPL

>1000

); method incompatible with format. Bold : best calibration-free ( CF ) result. ✓ : calibration-free. ✗ : requires calibration data.

Component	Prefill ( $b{=}1$ )		Prefill ( $b{=}16$ )		Decode ( $b{=}1$ )		Decode ( $b{=}16$ )
Component	BF16	W4A4	BF16	W4A4	BF16	W4A4	BF16	W4A4
Quantized linears (224)	17.33	8.82	121.49	32.98	14.75	8.24	16.32	9.18
Activation quantization (incl. $\beta$ )	–	0.82	–	3.09	–	0.70	–	0.72
lm_head (BF16)	1.19	1.19	9.26	9.26	1.07	1.07	1.10	1.10
Attention	0.19	0.20	0.93	0.94	0.42	0.42	1.73	1.74
Norms / RoPE / KV cache / residual	2.43	2.33	14.03	13.37	1.60	1.64	4.18	4.14

Appendix

Table 11: GPU time (ms) per component for BF16 and DualQuant W4A4, INT4 (

G{=}64

), on Llama-3.1-8B using an RTX 4090.

Component	Prefill ( $b{=}1$ )		Prefill ( $b{=}16$ )		Decode ( $b{=}1$ )		Decode ( $b{=}16$ )
Component	BF16	W4A4	BF16	W4A4	BF16	W4A4	BF16	W4A4
Quantized linear weights	13.96	3.49	13.96	3.49	13.96	3.49	13.96	3.49
Per-group weight scales (BF16)	–	0.22	–	0.22	–	0.22	–	0.22
Embedding + lm_head (BF16)	2.10	2.10	2.10	2.10	2.10	2.10	2.10	2.10
Runtime/allocator overhead	0.14	0.11	0.14	0.11	0.14	0.11	0.14	0.11
Weights subtotal	16.20	5.92	16.20	5.92	16.20	5.92	16.20	5.92

Appendix

Table 12: End-to-end memory breakdown (GB) for BF16 and DualQuant W4A4 on Llama-3.1-8B using an RTX 4090.

Category	Assets and Licenses
Models	Llama-2-7B (Llama 2 Community); Llama-3.1-8B (Llama 3.1 Community); Llama-3.2-1B (Llama 3.2 Community); Qwen2-1.5B, Qwen2.5-7B, Qwen3-0.6B, Qwen3-14B (Apache 2.0)
Datasets	WikiText-2 (CC BY-SA 4.0); C4 (ODC-BY 1.0); MMLU (MIT); HellaSwag (MIT); PIQA (AFL-3.0); ARC-Easy (CC BY-SA 4.0); WinoGrande (CC-BY)

Appendix

Table 13: Licenses of existing assets used in this work.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 381 papers

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization
May 9, 2026Beshr IslamBouli, David JinLLM Quantization 4-Bit Quantization
Massachusetts Institute of Technology, Cambridge, MA, USA.
SoloQ: Calibration-Free Quantization for Diffusion Language Models
Oct 5, 2026Donghyun Lee, Arkapravo Ghosh, Varun Manjunath +5Diffusion Model Quantization KV-Cache Quantization
University of Southern California · Yale University · Korea University
Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference
May 13, 2026Lingchao Zheng, Yuwei Fan, Jun Li +5LLM Quantization LLM Inference Acceleration
Huawei

May 9, 2026J/K move · Enter open · S save

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization

Beshr IslamBouli, David Jin

Massachusetts Institute of Technology, Cambridge, MA, USA.

LLM Quantization 4-Bit Quantization Weight-Only Quantization Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference. Existing PTQ methods, such as AWQ and GPTQ, improve how weights are mapped onto a fixed 4-bit grid through scaling, clipping, or error compensation. To further improve accuracy, methods such as OmniQuant and QuIP# uses gradient-assisted algorithms at the cost of hours of quantization time. In this work, we propose AAAC (Activation-Aware Adaptive Codebooks), a lightweight method for 4-bit LLM weight quantization. AAAC replaces the fixed scalar codebook used in standard quantization with two small learned scalar codebooks (64 bytes) per layer. Each group of weights selects the codebook that minimizes activation-weighted reconstruction error, encoding the choice in the unused sign bit of the group's positive scale and adding zero storage overhead. AAAC completes in 3--30 minutes on a single GPU, and adds no memory beyond the model itself. We evaluate against AWQ, GPTQ, IF4, GPTVQ, OmniQuant, SqueezeLLM, and QuIP# across model families. AAAC outperforms baselines at orders-of-magnitude less quantization time.

Abstract

LLM Quantization

Explore similar work

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization

SoloQ: Calibration-Free Quantization for Diffusion Language Models

Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference