DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs

Mahsa Salmani; Sayeh Sharify; Hesham Mostafa

cs.LGOct 9, 2026

DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs

Authors: Mahsa Salmani, Sayeh Sharify, Hesham Mostafa

Organizations: d-Matrix

Abstract

Low-bit inference on accelerators with native 4-bit support requires quantizing both weights and activations to realize the full throughput gains these formats provide. While post-training weight quantization has been extensively studied, activation quantization remains considerably more challenging and typically requires calibration data. In this work, we introduce DUALQUANT, a dual-scale, calibration-free post-training quantization framework that directly minimizes the weight-reconstruction Mean Squared Error (MSE) and, as an immediate by-product, yields per-channel activation rescalings prior to activation quantization. Our key observation is that the scaling factors obtained by minimizing weight-reconstruction error provide effective activation rescalings, enabling competitive activation quantization without calibration data. We formulate DUALQUANT as a non-differentiable weight-reconstruction problem and develop an efficient solver based on variable splitting and block coordinate descent, where each subproblem admits a closed-form update. DUALQUANT provides a unified algorithm supporting a wide range of formats, including the full MX family and NVFP4. To our knowledge, DUALQUANT is the first dual-scaling method to formally support MX and NV block formats. Across Llama and Qwen models evaluated on WikiText-2, C4, and zero-shot downstream tasks, DUALQUANT matches or exceeds the accuracy of calibration-based SmoothQuant across most models and formats, and substantially outperforms calibration-free baselines, with large gains for MX and NV block formats. We validate these gains with a real packed W4A4 inference path for Llama-3.1-8B on an RTX 4090. The resulting implementation closely matches the accuracy of fake quantization while delivering up to 2.61x memory reduction and up to 2.46x throughput speedup over BF16.

Figures & tables

Figure 1: Per-element quantization error $\scriptstyle|\mathbf{W}-\hat{\mathbf{W}}|$ under different scale designs. Each panel reports the MSE ratio ( $\uparrow$ ) relative to single-scale and the matrix imbalance (SINQ’s objective). (a) Original weight tile. (b) Single-scale: one outlier inflates the entire row. (c) SINQ minimizes matrix imbalance ( $1.65$ ) but achieves only $3.32\times$ MSE reduction. (d) DualQuant minimizes MSE, achieving $12.54\times$ reduction, illustrating that low imbalance does not imply low reconstruction error.

Figure 2: (a) Relative change

\scriptstyle\|\hat{\mathbf{Q}}^{(t)}-\hat{\mathbf{Q}}^{(t-1)}\|_{F}/\|\hat{\mathbf{Q}}^{(t-1)}\|_{F}

vs. BCD iteration for five transformer blocks. The change decays to near zero by

t{\approx}16

for all layers and blocks, validating the stabilization condition of Theorem 1 . (b) Normalized MSE

\scriptstyle\|\mathbf{W}-\hat{\mathbf{W}}\|_{F}^{2}/\|\mathbf{W}\|_{F}^{2}

vs. BCD iteration for the same blocks. All curves decrease monotonically and plateau within

{\approx}10

iterations, confirming Theorem 1 (i).

Bits	Format	Method	CF	WikiText-2 PPL $\downarrow$			C4 PPL $\downarrow$
Bits	Format	Method	CF	Llama-2-7B	Llama-3.1-8B	Llama-3.2-1B	Llama-2-7B	Llama-3.1-8B	Llama-3.2-1B
BF16 (reference)			–	5.11	5.62	8.67	7.04	8.82	13.53
4-bit	A: INT4 W: INT4 $(G{=}64)$	RTN	✓	5.73	7.11	12.06	7.97	11.99	19.69
		AWQ+	✗	5.57	6.83	12.46	7.73	11.51	20.24
		SmoothQuant	✗	5.68	6.98	13.37	7.91	11.75	21.99
		SINQ+	✓	5.61	6.83	12.36	7.80	11.45	20.68

Table 1: WikiText-2 and C4 perplexity (

\downarrow

) for joint W+A quantization at 4-bit precision on the Llama family. DualQuant applies

\bm{\beta}

-rescaling to activations without calibration data. SmoothQuant and AWQ+ use activation statistics from 256 calibration samples. SINQ+ is our extension of SINQ to W+A. QuaRot rotates activations/weights online with a Hadamard transform (no calibration). Bold : best calibration-free ( CF ) result. (

\dagger

): degenerate result

(PPL>1000)

; method incompatible with format. ✓ : calibration-free. ✗ : requires calibration data.

Llama-3.1-8B
Format	Method	CF	MMLU	HellaSwag	PIQA	ARC-Easy	WinoGrande
INT4 (G=64)	RTN	✓	54.89	56.44	77.86	77.40	68.90
	AWQ+	✗	55.78	56.65	78.51	77.19	69.06
	SmoothQuant	✗	55.24	55.93	77.31	77.44	68.75
	SINQ+	✓	55.50	56.68	77.09	77.65	67.72
	QuaRot	✓	54.21	56.16	77.04	77.40	70.17

Table 2: Llama-3.1-8B downstream accuracy (0-shot, %, higher is better

\uparrow

) for RTN, AWQ+, SmoothQuant, SINQ+, DualQuant , and QuaRot, across all four evaluated formats; QuaRot is omitted entirely for NVFP4, since QuaRot’s Hadamard rotation redistributes activation magnitude in a way that collapses NVFP4’s narrow-range E4M3 block scale. Bold : best calibration-free (CF) result per column within each model+format block. ✓ : calibration-free. ✗ : requires calibration data.

Workload	BF16	W4A4	Reduction	BF16	W4A4	Speedup
Workload	Memory (GB)	Memory (GB)	Memory	Throughput (tok/s)	Throughput (tok/s)
Prefill $(b{=}1)$	16.33	6.25	$2.61\times$	3904.6	6421.4	$1.64\times$
Prefill $(b{=}16)$	16.66	6.58	$2.53\times$	8940.3	22003.4	$2.46\times$
Decode $(b{=}1)$	16.34	6.26	$2.61\times$	56.6	83.7	$1.48\times$
Decode $(b{=}16)$	16.75	6.67	$2.51\times$	694.5	964.1	$1.39\times$

Table 3: End-to-end memory usage and throughput of the real packed W4A4 deployment versus BF16 for Llama-3.1-8B on an RTX 4090 with INT4,

G{=}64

. Prefill uses an 82-token prompt without a KV cache. Decode generates

100

tokens with a 182-token KV cache. Memory includes weights, scales, unquantized embedding and lm_head, activations, and the KV cache for decode.

Table 6

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: (a) Relative change

\scriptstyle\|\hat{\mathbf{Q}}^{(t)}-\hat{\mathbf{Q}}^{(t-1)}\|_{F}/\|\hat{\mathbf{Q}}^{(t-1)}\|_{F}

vs. BCD iteration for five transformer blocks. The

\hat{\mathbf{Q}}^{(t)}

change decays to near zero by

t{\approx}16

for all layers and blocks, validating the stabilization condition of Theorem 1 . (b) Normalized MSE

\scriptstyle\|\mathbf{W}-\hat{\mathbf{W}}\|_{F}^{2}/\|\mathbf{W}\|_{F}^{2}

vs. BCD iteration for the same blocks. All curves decrease monotonically and plateau within

{\approx}10

iterations, confirming Theorem 1 (i). Together with the Qwen2.5-7B results in Figure 2 , this illustrates that the convergence behavior of DualQuant is consistent across the two model families evaluated here. Notably, down_proj consistently shows the highest residual MSE, consistent with it being among the harder projections to quantize ( Frantar et al., 2023 ) .

Projection	$\beta$ applied	$\beta$ not applied	$\Delta$ (dB)
k_proj	17.61	16.80	$+0.81$
v_proj	17.45	16.80	$+0.65$
up_proj	18.48	18.20	$+0.27$
q_proj	17.05	16.80	$+0.25$
gate_proj	18.36	18.20	$+0.15$
o_proj	17.40	17.34	$+0.07$

Appendix

Table 6: Per-projection activation SQNR (dB,

\uparrow

) with and without DualQuant ’s

\beta

, on Llama-3.1-8B block 0 (INT4,

G{=}64

). Quantized weights

\hat{Q}

are held fixed; only the presence of the

\beta

pre-scaling varies.

\Delta>0

indicates

\beta

reduces activation quantization error. Each

\Delta

is computed at full precision and may differ slightly from the difference of the rounded columns. Other blocks exhibit the same qualitative behavior.

Bits	Format	Method	CF	Qwen2.5-7B	Qwen3-0.6B	Qwen3-14B	Qwen2.5-7B	Qwen3-0.6B	Qwen3-14B
				WikiText-2 PPL $\downarrow$			C4 PPL $\downarrow$
BF16 (reference)			–	6.15	18.14	7.54	9.12	27.30	11.06
4-bit	A: INT4 W: INT4 $(G{=}64)$	RTN	✓	7.47	49.44	8.39	13.50	74.78	14.83
		AWQ+	✗	7.36	38.80	8.19	13.21	59.20	14.43
		SmoothQuant	✗	7.10	30.67	8.12	12.74	50.73	14.21
		SINQ+	✓	7.28	30.21	8.02	12.94	49.05	14.15

Appendix

Table 7: WikiText-2 and C4 perplexity (

\downarrow

) for joint weight-and-activation quantization at 4-bit precision on the Qwen family. DualQuant applies

\bm{\beta}

\dagger

): degenerate result

(PPL>1000)

; method incompatible with format. ✓ : calibration-free. ✗ : requires calibration data.

Qwen3-14B
Format	Method	CF	MMLU	HellaSwag	PIQA	ARC-Easy	WinoGrande
INT4 (G=64)	RTN	✓	70.33	75.23	77.64	77.36	69.06
	AWQ+	✗	71.92	75.03	77.86	79.08	69.53
	SmoothQuant	✗	73.04	76.74	77.91	80.77	71.51
	SINQ+	✓	72.30	76.23	77.80	79.59	70.17
	QuaRot	✓	71.30	73.52	76.77	77.99	67.88

Appendix

Table 8: Qwen3-14B downstream accuracy (0-shot, %, higher is better

\uparrow

) for RTN, AWQ+, SmoothQuant, SINQ+, DualQuant , and QuaRot, across all 4 weight formats; QuaRot is omitted entirely for NVFP4, since QuaRot’s Hadamard rotation redistributes activation magnitude in a way that collapses NVFP4’s narrow-range E4M3 block scale. Bold : best calibration-free (CF) result per column within each model+format block. ✓ : calibration-free. ✗ : requires calibration data.

Bits	Format	Method	CF	Qwen2.5-7B	Qwen3-0.6B	Qwen3-14B	Llama-3.1-8B
BF16 (reference)			–	6.15	18.14	7.54	5.62
8-bit	A: INT8 W: INT8 $(G{=}128)$	RTN	✓	6.27	18.42	7.55	5.72
		SmoothQuant	✗	6.25	18.28	7.50	5.70
		SINQ+	✓	6.67	18.33	7.49	5.70
		DualQuant	✓	6.64	18.24	7.77	5.69
	A: MXINT8 W: MXINT8 $(G{=}32)$	RTN	✓	6.15	18.06	7.54	5.63

Appendix

Table 9: WikiText-2 perplexity (

\downarrow

) for 8-bit W+A quantization. DualQuant applies

\bm{\beta}

-rescaling without calibration data. SmoothQuant uses 256 calibration samples. SINQ+ is our extension of SINQ to W+A. Bold : best calibration-free ( CF ) result. ✓ : calibration-free. ✗ : requires calibration data.

Bits	Format	Method	CF	Qwen2.5-7B	Qwen3-0.6B	Qwen3-14B	Llama-3.1-8B
BF16 (reference)			–	6.15	18.14	7.54	5.62
4-bit	W: INT4 $(G{=}64)$	RTN	✓	6.67	26.33	7.93	6.32
		AWQ	✗	6.50	21.83	7.72	6.12
		GPTQ	✗	6.44	20.39	7.73	5.56
		SINQ	✓	6.57	22.21	7.76	6.16
		DualQuant	✓	6.55	21.35	7.88	6.12

Appendix

Table 10: WikiText-2 perplexity (

\downarrow

) for 4-bit weight-only quantization. Activations in BF16. DualQuant applies

\bm{\beta}

-rescaling without calibration data. AWQ and GPTQ use 256 calibration samples. (

\dagger

): degenerate result (PPL

>1000

); method incompatible with format. Bold : best calibration-free ( CF ) result. ✓ : calibration-free. ✗ : requires calibration data.

Component	Prefill ( $b{=}1$ )		Prefill ( $b{=}16$ )		Decode ( $b{=}1$ )		Decode ( $b{=}16$ )
Component	BF16	W4A4	BF16	W4A4	BF16	W4A4	BF16	W4A4
Quantized linears (224)	17.33	8.82	121.49	32.98	14.75	8.24	16.32	9.18
Activation quantization (incl. $\beta$ )	–	0.82	–	3.09	–	0.70	–	0.72
lm_head (BF16)	1.19	1.19	9.26	9.26	1.07	1.07	1.10	1.10
Attention	0.19	0.20	0.93	0.94	0.42	0.42	1.73	1.74
Norms / RoPE / KV cache / residual	2.43	2.33	14.03	13.37	1.60	1.64	4.18	4.14

Appendix

Table 11: GPU time (ms) per component for BF16 and DualQuant W4A4, INT4 (

G{=}64

), on Llama-3.1-8B using an RTX 4090.

Component	Prefill ( $b{=}1$ )		Prefill ( $b{=}16$ )		Decode ( $b{=}1$ )		Decode ( $b{=}16$ )
Component	BF16	W4A4	BF16	W4A4	BF16	W4A4	BF16	W4A4
Quantized linear weights	13.96	3.49	13.96	3.49	13.96	3.49	13.96	3.49
Per-group weight scales (BF16)	–	0.22	–	0.22	–	0.22	–	0.22
Embedding + lm_head (BF16)	2.10	2.10	2.10	2.10	2.10	2.10	2.10	2.10
Runtime/allocator overhead	0.14	0.11	0.14	0.11	0.14	0.11	0.14	0.11
Weights subtotal	16.20	5.92	16.20	5.92	16.20	5.92	16.20	5.92

Appendix

Table 12: End-to-end memory breakdown (GB) for BF16 and DualQuant W4A4 on Llama-3.1-8B using an RTX 4090.

Category	Assets and Licenses
Models	Llama-2-7B (Llama 2 Community); Llama-3.1-8B (Llama 3.1 Community); Llama-3.2-1B (Llama 3.2 Community); Qwen2-1.5B, Qwen2.5-7B, Qwen3-0.6B, Qwen3-14B (Apache 2.0)
Datasets	WikiText-2 (CC BY-SA 4.0); C4 (ODC-BY 1.0); MMLU (MIT); HellaSwag (MIT); PIQA (AFL-3.0); ARC-Easy (CC BY-SA 4.0); WinoGrande (CC-BY)

Appendix

Table 13: Licenses of existing assets used in this work.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 381 papers

May 9, 2026

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization

Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference. Existing PTQ methods, such as AWQ and GPTQ, improve how weights are mapped onto a fixed 4-bit grid through scaling, clipping, or error compensation. To further improve accuracy, methods such as OmniQuant and QuIP# uses gradient-assisted algorithms at the cost of hours of quantization time. In this work, we propose AAAC (Activation-Aware Adaptive Codebooks), a lightweight method for 4-bit LLM weight quantization. AAAC replaces the fixed scalar codebook used in standard quantization with two small learned scalar codebooks (64 bytes) per layer. Each group of weights selects the codebook that minimizes activation-weighted reconstruction error, encoding the choice in the unused sign bit of the group's positive scale and adding zero storage overhead. AAAC completes in 3--30 minutes on a single GPU, and adds no memory beyond the model itself. We evaluate against AWQ, GPTQ, IF4, GPTVQ, OmniQuant, SqueezeLLM, and QuIP# across model families. AAAC outperforms baselines at orders-of-magnitude less quantization time.

Beshr IslamBouli, David Jin

Massachusetts Institute of Technology, Cambridge, MA, USA.

LLM Quantization 4-Bit Quantization Weight-Only Quantization Post-Training Quantization Neural Network Quantization

Ask PDF

Oct 5, 2026

SoloQ: Calibration-Free Quantization for Diffusion Language Models

Diffusion large language models dLLMs) have emerged as a promising alternative to autoregressive language models through bidirectional diffusion-based token generation. However, their growing model sizes and high inference costs make efficient deployment challenging: full-sequence denoising repeatedly invokes compute-intensive forward passes, while block-diffusion models additionally introduce a memory-intensive KV-cache. Low-bit weight-activation quantization is therefore attractive, yet existing dLLM post-training quantization methods rely on calibration data despite activation distributions shifting across masking states and denoising steps. We present SoloQ, a calibration-free quantization framework that maps weights and activations into a normalized rotated basis with a predictable marginal distribution, enabling data-independent quantization. SoloQ combines a structured K-RPBH rotation with a lightweight rescaling correction for calibration-free quantization. Its predictable post-rotation distribution supports both distribution-matched codebooks and hardware-native NVFP4. For block-diffusion models, SoloQ further applies commit-time KV-cache quantization to compress persistent states without perturbing the actively denoised block. Across full-sequence dLLMs (LLaDA and Dream) and block-diffusion dLLMs(Fast-dLLM v2 and Nemotron-Labs-Diffusion), SoloQ retains accuracy under 4-bit quantization and outperforms calibration-based baselines on knowledge- and reasoning-intensive benchmarks. With NVFP4, SoloQ reduces peak memory by up to 2.61X and accelerates end-to-end inference by up to 2.24X.

Donghyun Lee, Arkapravo Ghosh, Varun Manjunath +5

University of Southern California · Yale University · Korea University

Diffusion Model Quantization KV-Cache Quantization Post-Training Quantization Neural Network Quantization

Ask PDF

May 13, 2026

Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference

Quantization is essential for efficient large language model (LLM) inference, yet the dequantization step-converting low-bit weights back to high-precision for matrix multiplication has become a critical bottleneck on modern AI accelerators. On architectures with decoupled compute units (e.g., Ascend NPUs), dequantization operations can consume more cycles than the matrix multiplication itself, leaving the high-throughput tensor cores underutilized. This paper presents Multi-Scale Dequant (MSD), a quantization framework that removes weight/KV dequantization from the GEMM critical path. Instead of lifting low-bit weights to BF16 precision, MSD decomposes high-precision BF16 activations into multiple low-precision components, each of which can be multiplied directly with quantized weights via native hardware-accelerated GEMM. This approach shifts the computational paradigm from precision conversion to multi-scale approximation, avoiding INT8-to-BF16 weight conversion before GEMM. We instantiate MSD for two weight formats and derive tight error bounds for each. For INT8 weights (W4A16), two-pass INT8 decomposition achieves near 16 effective bits. For MXFP4 weights (W4A16), two-pass MXFP4 decomposition yields near 6.6 effective bits with error bound 1/64 per block surpassing single-pass MXFP8(5.24 bits) while maintaining the same effective GEMM compute time. We further derive closed-form latency and HBM traffic models showing that MSD avoids the Vector-Cube pipeline stall caused by dequantization and reduces KV cache HBM traffic by up to 2.5 times in attention. Numerical simulations on matrix multiplication and Flash Attention kernels confirm that MSD does not degrade accuracy compared to dequantization baselines, and in many settings achieves lower L2 error.

Lingchao Zheng, Yuwei Fan, Jun Li +5

Huawei

LLM Quantization LLM Inference Acceleration

Ask PDF

Abstract

LLM Quantization

Explore similar work

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization

SoloQ: Calibration-Free Quantization for Diffusion Language Models

Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference