Weight-activation quantization can reduce the memory and compute cost of neural-network inference, but recent methods often require expensive optimization and architecture-specific propagation of basis transforms across layers. We introduce MACaQ, an MXFP4 W4A4 post-training quantization method that treats every linear layer independently. MACaQ applies a per-channel scale and a block-Hadamard transform online to each layer's activations, while absorbing the corresponding inverse transforms into its weights. Retaining these transforms online avoids cross-layer dependencies and enables a simple streaming implementation of the quantization process, in which only one layer must reside in memory at a time. To improve over round-to-nearest quantization, we introduce Flip-Refinement, a Hessian-aware coordinate-descent procedure that greedily moves individual FP4 codes to adjacent representable values. Despite executing the activation transforms at inference time, MACaQ adds only 2.3% end-to-end overhead over pure MXFP4 on Qwen3-32B in vLLM. MACaQ reaches lower perplexity than DuQuant++ and MR-GPTQ, and already matches their best perplexity on Llama-3-8B in 11 times less quantization time. We further apply MACaQ without architecture-specific modifications to an image diffusion model, illustrating the generality of its layer-local design and show quantization of a 122B parameter model in 64 minutes, using only 32.5GB GPU RAM. Code is available at https://github.com/huawei-csl/MACaQ .
Figures & tables
Figure 1: Prior work propagates the activation-side change of basis (labeled P ) outside the linear layer to avoid runtime overhead. This incurs an engineering and quantization-time overhead, where for each layer a suitable absorption target must be found. MACaQ instead applies a lightweight transform online, preserving complete layer locality at negligible (ca. 2%) run-time overhead for sufficiently large layers.
Figure 2: Quantization time vs. WikiText-2 perplexity on Llama-3-8B (OCP MXFP4 W4A4), all measured on the same hardware. MACaQ is run with different number of flips and calibration tokens, the other methods where possible scale by calibration tokens. MACaQ matches the best quality of these strong baselines in 11× less time.
Figure 3: Decode overhead introduced by online rotation and scaling (batch 1, context 512). The overhead decreases with model size and reaches approximately 2.3% for Qwen3-32B. Error bars show the standard error.
Llama-3-8B
Llama-3.2-3B
Llama-3-8B-Inst.
Llama-3.1-8B-Inst.
Qwen3-32B
Qwen3.5-35B-A3B
Method
Wiki2
C4
Wiki2
C4
Wiki2
C4
Wiki2
C4
Wiki2
C4
Wiki2
C4
FP16
6.14
9.46
7.81
11.34
8.31
13.03
7.16
11.41
7.62
12.45
6.55
10.98
QuaRot (RTN)
9.34
15.53
12.40
18.63
11.74
19.21
10.00
16.48
10.19
16.59
6.77
11.38
QuaRot (GPTQ)
7.65
12.62
9.65
15.11
9.62
15.85
8.58
14.49
8.46
13.42
6.79
11.33
FlatQuant
7.21
11.65
9.00
14.87
9.25
15.27
8.10
13.40
8.11
13.28
6.87
11.57
MR-GPTQ
7.13
11.37
8.92
13.51
9.13
14.62
8.14
13.24
8.15
13.27
6.75
11.22
Table 1: OCP MXFP4 W4A4, WikiText2 / C4 perplexity ( ↓ ). Best quantized result per column in bold, second best underlined.
Qwen3-32B
Qwen3.5-35B-A3B
Method
IFEval
MATH500
IFEval
MATH500
Original (FP16)
83.60% ± 0.25
97.78% ± 0.10
85.29% ± 0.25
99.38% ± 0.06
QuaRot (RTN)
78.46% ± 0.23
93.65% ± 0.20
85.41 % ± 0.33
98.78% ± 0.10
QuaRot (GPTQ)
82.35% ± 0.29
96.98% ± 0.19
85.60% ± 0.39
98.60% ± 0.10
MR-GPTQ
82.81% ± 0.27
97.44 % ± 0.16
85.43% ± 0.24
98.86% ± 0.07
DuQuant++ (RTN)
81.57% ± 0.29
96.82% ± 0.21
85.29% ± 0.20
98.58% ± 0.11
Table 2: OCP MXFP4 W4A4. IFEval strict prompt-level accuracy ( ↑ , average over k=10 ; ± across-seed standard error). MATH500 accuracy ( ↑ , average over k=10 ; ± across-seed standard error). Best quantized result per column in bold, second best underlined; also all results within one standard error of the baseline are bolded.
Method
Format
bits/w
PSNR ↑
SSIM ↑
LPIPS ↓
DINO global↑
DINO patch↑
VRAM (GB) ↓
Full Precision
BF16
16.0
-
-
-
-
-
18.5
MACaQ (ours)
MXFP4
4.25
15.04
0.697
0.218
0.882
0.947
13.1
Official NVFP4
NVFP4
4.50
14.85
0.676
0.235
0.878
0.937
13.3
Table 3: FLUX.2-klein-4B : MACaQ vs. the official NVFP4 QAD checkpoint. GenEval2 (800 prompts), 10242 , 6 steps, DiT-only W4A4. PSNR/SSIM/LPIPS and DINOv3 (ViT-L/16) feature cosine are paired against the BF16 output (same prompt and seed). Peak VRAM at batch 1. MACaQ is closer to BF16 on all four fidelity metrics and ties on VRAM.
Figure 4: Comparison of FLUX2-klein-4B under MACaQ MXFP4 vs. the official NVFP4 quantization. MACaQ is more similar to the high precision model; note e.g. the shadow’s orientation in the first image and the empty glasses in the second. Quantitative results in Tab. 3 .
N
B
L
MACaQ
− SR
Marlin
FP16
SR cost
4096
1
1
1.31
1.31
1.31
1.00
< 1%
1
16384
1.75
1.86
1.02
1.00
6.4%
16
1
1.75
1.75
1.75
1.00
< 1%
16
16384
1.50
1.57
1.01
1.00
4.1%
16384
1
1
5.04
5.11
8.26
1.00
1.3%
1
16384
2.01
2.05
1.00
1.00
1.6%
Table 4: Kernel-level speed of one linear layer X⋅W⊤ , W∈RN×N , X∈RB×L×N ( M=B⋅L rows; L=1 is decode). Median as speedup relative to fp16 ( ↑ ; fp16 ≡1.00 ). Standard Error is omitted as it is below 0.02 in all cases. MACaQ is the MXFP4 W4A4 kernel with online per-channel scaling and block-Hadamard rotation; − SR disables both; Marlin is the W4A16 kernel of Frantar et al. (10) . “SR cost” is the relative slowdown from enabling scale+rotation.
Configuration
Wiki2 ↓
Δ
Full Precision (FP16)
6.14
–
RTN
8.28
–
+ Hessian exponent search
8.09
−0.19
+ micro-rotation
7.65
−0.44
+ online second scale
7.08
−0.57
+ flip refinement (= MACaQ)
6.87
−0.21
Table 5: Additive contribution of each MACaQ component on Llama-3-8B (MXFP4 W4A4), WikiText-2 perplexity at 512 Flips and 200’000 calibration tokens.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Llama-3-8B
Qwen3-32B
Method
Prefill
Decode
Prefill
Decode
Batch 1
bf16
38,171
93.2
11,027
22.8
Marlin W4A16
71,760
223.9
29,307
71.8
MACaQ (ours)
70,547
227.9
28,473
69.8
MACaQ − SR
72,852
232.9
28,364
69.4
Appendix
Table 6: End-to-end throughput (median tok/s; 512-token prompt, vLLM graph mode) across batch sizes. Prefill = prompt / TTFT, decode = per-token rate, aggregated over the batch. MACaQ − SR disables the online scale + rotation (speed-only ablation).
Method
Flips
Calib. Blocks
Time (s)
PPL
FP16
–
–
–
6.14
RTN
–
0
2.2
8.29
MACaQ (ours)
0 ∗
0
5.5
7.64
0
1
14.8
7.11
16
2
24.3
7.01
32
4
31.3
6.97
Appendix
Table 7: Quantization cost vs. quality on Llama-3-8B, MXFP4 W4A4. Calib. Blocks is the calibration budget in units of 2048 tokens: for the GPTQ-based baselines one block is exactly one 2048 -token calibration sequence, while MACaQ consumes a flat token stream (its rungs are round decimal token counts, hence the fractional blocks). PPL is WikiText-2 perplexity; FP16 is 6.14. The ∗ configuration uses only rotation and exponent screen.
Figure 5: Hyperparameter sensitivity of MACaQ on Llama3-8B in the 4K token 16 flips configuration. Because the λ curve is flat, we choose a conservative value of 1.0 to prevent possible under-calibration issues.
Figure 6: Qualitative comparison on FLUX.2-klein-4B: BF16 vs. MACaQ (MXFP4 W4A4) vs. the official NVFP4 checkpoint, scaled to 512 pix width. Same seed per prompt. The inset number shows the lpips (lower is better). For a quantitative comparison see Table 3 . (a) Photographic prompts: (1) a photorealistic portrait of an elderly fisherman, weathered face, golden hour light; (2) an astronaut riding a horse on the moon, earth in the background, dramatic; (3) a cozy bookshop interior, warm lamps, stacks of old books, cat sleeping on a chair; (4) a futuristic city skyline at dusk, flying cars, volumetric clouds, highly detailed; (5) a corgi wearing tiny sunglasses on a tropical beach, vibrant colors; (6) a macro photograph of a dewdrop on a spider web at sunrise. (b) GenEval2 prompts: (1) four brown monkeys; (2) a blue kangaroo and a cookie; (3) a stone guitar and a plastic lion; (4) two purple flowers in front of three dogs; (5) a bicycle to the left of a rabbit, and five brown turtles; (6) three purple giraffes to the left of two pink guitars, and two suitcases. Best viewed zoomed in.
Format
Refinement
Wiki2
Pile
Qwen3-4B
F16
–
11.56
7.41
IQ3_S (plain)
none
13.06
8.10
scalar flips
12.31
7.85
block, codes
11.81
7.67
block, codes+scales
11.86
7.68
Appendix
Table 8: Weight-only quantization of MACaQ-transformed Qwen3 models. Lower perplexity is better. imatrix denotes initialization with llama.cpp ’s importance matrix; plain omits it. codes refines codewords and signs only, while codes+scales additionally refines the shared scales. Codes-only refinement was not run at 14B.
Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference. Existing PTQ methods, such as AWQ and GPTQ, improve how weights are mapped onto a fixed 4-bit grid through scaling, clipping, or error compensation. To further improve accuracy, methods such as OmniQuant and QuIP# uses gradient-assisted algorithms at the cost of hours of quantization time. In this work, we propose AAAC (Activation-Aware Adaptive Codebooks), a lightweight method for 4-bit LLM weight quantization. AAAC replaces the fixed scalar codebook used in standard quantization with two small learned scalar codebooks (64 bytes) per layer. Each group of weights selects the codebook that minimizes activation-weighted reconstruction error, encoding the choice in the unused sign bit of the group's positive scale and adding zero storage overhead. AAAC completes in 3--30 minutes on a single GPU, and adds no memory beyond the model itself. We evaluate against AWQ, GPTQ, IF4, GPTVQ, OmniQuant, SqueezeLLM, and QuIP# across model families. AAAC outperforms baselines at orders-of-magnitude less quantization time.
Beshr IslamBouli, David Jin
Massachusetts Institute of Technology, Cambridge, MA, USA.
Low-bit inference on accelerators with native 4-bit support requires quantizing both weights and activations to realize the full throughput gains these formats provide. While post-training weight quantization has been extensively studied, activation quantization remains considerably more challenging and typically requires calibration data. In this work, we introduce DUALQUANT, a dual-scale, calibration-free post-training quantization framework that directly minimizes the weight-reconstruction Mean Squared Error (MSE) and, as an immediate by-product, yields per-channel activation rescalings prior to activation quantization. Our key observation is that the scaling factors obtained by minimizing weight-reconstruction error provide effective activation rescalings, enabling competitive activation quantization without calibration data. We formulate DUALQUANT as a non-differentiable weight-reconstruction problem and develop an efficient solver based on variable splitting and block coordinate descent, where each subproblem admits a closed-form update. DUALQUANT provides a unified algorithm supporting a wide range of formats, including the full MX family and NVFP4. To our knowledge, DUALQUANT is the first dual-scaling method to formally support MX and NV block formats. Across Llama and Qwen models evaluated on WikiText-2, C4, and zero-shot downstream tasks, DUALQUANT matches or exceeds the accuracy of calibration-based SmoothQuant across most models and formats, and substantially outperforms calibration-free baselines, with large gains for MX and NV block formats. We validate these gains with a real packed W4A4 inference path for Llama-3.1-8B on an RTX 4090. The resulting implementation closely matches the accuracy of fake quantization while delivering up to 2.61x memory reduction and up to 2.46x throughput speedup over BF16.
Quantization-aware training (QAT) leverages lower-precision arithmetic to reduce the cost of LLM deployment, but aggressive quantization degrades final model performance. A common remedy is mixed-precision training, in which high precision is assigned to some of the layers to maintain performance while keeping the cost constrained. This approach then requires precision assignments for model layers during training. We provide a new approach, called Q-PACE, consisting of a second-order sensitivity model that predicts the loss increase as a sum of quantization noise MSE weighted by per-layer curvature coefficients. During training, we periodically re-compute these coefficients using perturbations across layers, and re-assign precision. Pretraining and supervised fine-tuning experiments on LLMs of up to 4B parameters show that Q-PACE consistently improves over existing mixed-precision training recipes, and achieves comparable loss at substantially lower total memory budgets. We further find that quantization sensitivity is highly predictable by depth and layer type, and its stability during training allows for infrequent, cheap recalibration.
Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2