Weight-activation quantization can reduce the memory and compute cost of neural-network inference, but recent methods often require expensive optimization and architecture-specific propagation of basis transforms across layers. We introduce MACaQ, an MXFP4 W4A4 post-training quantization method that treats every linear layer independently. MACaQ applies a per-channel scale and a block-Hadamard transform online to each layer's activations, while absorbing the corresponding inverse transforms into its weights. Retaining these transforms online avoids cross-layer dependencies and enables a simple streaming implementation of the quantization process, in which only one layer must reside in memory at a time. To improve over round-to-nearest quantization, we introduce Flip-Refinement, a Hessian-aware coordinate-descent procedure that greedily moves individual FP4 codes to adjacent representable values. Despite executing the activation transforms at inference time, MACaQ adds only 2.3% end-to-end overhead over pure MXFP4 on Qwen3-32B in vLLM. MACaQ reaches lower perplexity than DuQuant++ and MR-GPTQ, and already matches their best perplexity on Llama-3-8B in 11 times less quantization time. We further apply MACaQ without architecture-specific modifications to an image diffusion model, illustrating the generality of its layer-local design and show quantization of a 122B parameter model in 64 minutes, using only 32.5GB GPU RAM. Code is available at https://github.com/huawei-csl/MACaQ .
Figures & tables
Figure 1: Prior work propagates the activation-side change of basis (labeled P ) outside the linear layer to avoid runtime overhead. This incurs an engineering and quantization-time overhead, where for each layer a suitable absorption target must be found. MACaQ instead applies a lightweight transform online, preserving complete layer locality at negligible (ca. 2%) run-time overhead for sufficiently large layers.
Figure 2: Quantization time vs. WikiText-2 perplexity on Llama-3-8B (OCP MXFP4 W4A4), all measured on the same hardware. MACaQ is run with different number of flips and calibration tokens, the other methods where possible scale by calibration tokens. MACaQ matches the best quality of these strong baselines in 11× less time.
Figure 3: Decode overhead introduced by online rotation and scaling (batch 1, context 512). The overhead decreases with model size and reaches approximately 2.3% for Qwen3-32B. Error bars show the standard error.
Llama-3-8B
Llama-3.2-3B
Llama-3-8B-Inst.
Llama-3.1-8B-Inst.
Qwen3-32B
Qwen3.5-35B-A3B
Method
Wiki2
C4
Wiki2
C4
Wiki2
C4
Wiki2
C4
Wiki2
C4
Wiki2
C4
FP16
6.14
9.46
7.81
11.34
8.31
13.03
7.16
11.41
7.62
12.45
6.55
10.98
QuaRot (RTN)
9.34
15.53
12.40
18.63
11.74
19.21
10.00
16.48
10.19
16.59
6.77
11.38
QuaRot (GPTQ)
7.65
12.62
9.65
15.11
9.62
15.85
8.58
14.49
8.46
13.42
6.79
11.33
FlatQuant
7.21
11.65
9.00
14.87
9.25
15.27
8.10
13.40
8.11
13.28
6.87
11.57
MR-GPTQ
7.13
11.37
8.92
13.51
9.13
14.62
8.14
13.24
8.15
13.27
6.75
11.22
Table 1: OCP MXFP4 W4A4, WikiText2 / C4 perplexity ( ↓ ). Best quantized result per column in bold, second best underlined.
Qwen3-32B
Qwen3.5-35B-A3B
Method
IFEval
MATH500
IFEval
MATH500
Original (FP16)
83.60% ± 0.25
97.78% ± 0.10
85.29% ± 0.25
99.38% ± 0.06
QuaRot (RTN)
78.46% ± 0.23
93.65% ± 0.20
85.41 % ± 0.33
98.78% ± 0.10
QuaRot (GPTQ)
82.35% ± 0.29
96.98% ± 0.19
85.60% ± 0.39
98.60% ± 0.10
MR-GPTQ
82.81% ± 0.27
97.44 % ± 0.16
85.43% ± 0.24
98.86% ± 0.07
DuQuant++ (RTN)
81.57% ± 0.29
96.82% ± 0.21
85.29% ± 0.20
98.58% ± 0.11
Table 2: OCP MXFP4 W4A4. IFEval strict prompt-level accuracy ( ↑ , average over k=10 ; ± across-seed standard error). MATH500 accuracy ( ↑ , average over k=10 ; ± across-seed standard error). Best quantized result per column in bold, second best underlined; also all results within one standard error of the baseline are bolded.
Method
Format
bits/w
PSNR ↑
SSIM ↑
LPIPS ↓
DINO global↑
DINO patch↑
VRAM (GB) ↓
Full Precision
BF16
16.0
-
-
-
-
-
18.5
MACaQ (ours)
MXFP4
4.25
15.04
0.697
0.218
0.882
0.947
13.1
Official NVFP4
NVFP4
4.50
14.85
0.676
0.235
0.878
0.937
13.3
Table 3: FLUX.2-klein-4B : MACaQ vs. the official NVFP4 QAD checkpoint. GenEval2 (800 prompts), 10242 , 6 steps, DiT-only W4A4. PSNR/SSIM/LPIPS and DINOv3 (ViT-L/16) feature cosine are paired against the BF16 output (same prompt and seed). Peak VRAM at batch 1. MACaQ is closer to BF16 on all four fidelity metrics and ties on VRAM.
Figure 4: Comparison of FLUX2-klein-4B under MACaQ MXFP4 vs. the official NVFP4 quantization. MACaQ is more similar to the high precision model; note e.g. the shadow’s orientation in the first image and the empty glasses in the second. Quantitative results in Tab. 3 .
N
B
L
MACaQ
− SR
Marlin
FP16
SR cost
4096
1
1
1.31
1.31
1.31
1.00
< 1%
1
16384
1.75
1.86
1.02
1.00
6.4%
16
1
1.75
1.75
1.75
1.00
< 1%
16
16384
1.50
1.57
1.01
1.00
4.1%
16384
1
1
5.04
5.11
8.26
1.00
1.3%
1
16384
2.01
2.05
1.00
1.00
1.6%
Table 4: Kernel-level speed of one linear layer X⋅W⊤ , W∈RN×N , X∈RB×L×N ( M=B⋅L rows; L=1 is decode). Median as speedup relative to fp16 ( ↑ ; fp16 ≡1.00 ). Standard Error is omitted as it is below 0.02 in all cases. MACaQ is the MXFP4 W4A4 kernel with online per-channel scaling and block-Hadamard rotation; − SR disables both; Marlin is the W4A16 kernel of Frantar et al. (10) . “SR cost” is the relative slowdown from enabling scale+rotation.
Configuration
Wiki2 ↓
Δ
Full Precision (FP16)
6.14
–
RTN
8.28
–
+ Hessian exponent search
8.09
−0.19
+ micro-rotation
7.65
−0.44
+ online second scale
7.08
−0.57
+ flip refinement (= MACaQ)
6.87
−0.21
Table 5: Additive contribution of each MACaQ component on Llama-3-8B (MXFP4 W4A4), WikiText-2 perplexity at 512 Flips and 200’000 calibration tokens.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Llama-3-8B
Qwen3-32B
Method
Prefill
Decode
Prefill
Decode
Batch 1
bf16
38,171
93.2
11,027
22.8
Marlin W4A16
71,760
223.9
29,307
71.8
MACaQ (ours)
70,547
227.9
28,473
69.8
MACaQ − SR
72,852
232.9
28,364
69.4
Appendix
Table 6: End-to-end throughput (median tok/s; 512-token prompt, vLLM graph mode) across batch sizes. Prefill = prompt / TTFT, decode = per-token rate, aggregated over the batch. MACaQ − SR disables the online scale + rotation (speed-only ablation).
Method
Flips
Calib. Blocks
Time (s)
PPL
FP16
–
–
–
6.14
RTN
–
0
2.2
8.29
MACaQ (ours)
0 ∗
0
5.5
7.64
0
1
14.8
7.11
16
2
24.3
7.01
32
4
31.3
6.97
Appendix
Table 7: Quantization cost vs. quality on Llama-3-8B, MXFP4 W4A4. Calib. Blocks is the calibration budget in units of 2048 tokens: for the GPTQ-based baselines one block is exactly one 2048 -token calibration sequence, while MACaQ consumes a flat token stream (its rungs are round decimal token counts, hence the fractional blocks). PPL is WikiText-2 perplexity; FP16 is 6.14. The ∗ configuration uses only rotation and exponent screen.
Figure 5: Hyperparameter sensitivity of MACaQ on Llama3-8B in the 4K token 16 flips configuration. Because the λ curve is flat, we choose a conservative value of 1.0 to prevent possible under-calibration issues.
Figure 6: Qualitative comparison on FLUX.2-klein-4B: BF16 vs. MACaQ (MXFP4 W4A4) vs. the official NVFP4 checkpoint, scaled to 512 pix width. Same seed per prompt. The inset number shows the lpips (lower is better). For a quantitative comparison see Table 3 . (a) Photographic prompts: (1) a photorealistic portrait of an elderly fisherman, weathered face, golden hour light; (2) an astronaut riding a horse on the moon, earth in the background, dramatic; (3) a cozy bookshop interior, warm lamps, stacks of old books, cat sleeping on a chair; (4) a futuristic city skyline at dusk, flying cars, volumetric clouds, highly detailed; (5) a corgi wearing tiny sunglasses on a tropical beach, vibrant colors; (6) a macro photograph of a dewdrop on a spider web at sunrise. (b) GenEval2 prompts: (1) four brown monkeys; (2) a blue kangaroo and a cookie; (3) a stone guitar and a plastic lion; (4) two purple flowers in front of three dogs; (5) a bicycle to the left of a rabbit, and five brown turtles; (6) three purple giraffes to the left of two pink guitars, and two suitcases. Best viewed zoomed in.
Format
Refinement
Wiki2
Pile
Qwen3-4B
F16
–
11.56
7.41
IQ3_S (plain)
none
13.06
8.10
scalar flips
12.31
7.85
block, codes
11.81
7.67
block, codes+scales
11.86
7.68
Appendix
Table 8: Weight-only quantization of MACaQ-transformed Qwen3 models. Lower perplexity is better. imatrix denotes initialization with llama.cpp ’s importance matrix; plain omits it. codes refines codewords and signs only, while codes+scales additionally refines the shared scales. Codes-only refinement was not run at 14B.