Low-bit quantization of linear layers is often dominated by a small number of outlier channels. Existing smoothing, rotation, and residual-based methods can mitigate this issue, but may shift the quantization bottleneck to weights or introduce costly online operations. To address these limitations, we propose PRQuant (Permutation Residual Quantization), a training-free framework that combines channel permutation with offline weight residual compensation. PRQuant identifies the scaled-weight columns with the largest quantization errors and permutes them into contiguous tail blocks. This structure allows the corresponding weight residuals to be precomputed entirely offline, while replacing scattered activation gathering with simple contiguous access during inference, yielding a single regular MXFP4 GEMM for compensated computation. Experiments show that PRQuant substantially reduces down-projection reconstruction error, with scaling and residual compensation providing the main numerical gains while permutation enables a hardware-friendly contiguous layout. Comprehensive experiment results on Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507 illustrate that PRQuant achieves up to averagelly 2.6x and 1.8x operator speedup over BF16 respectively, while preserving near plain MXFP4 end-to-end decoding efficiency. Across five downstream benchmarks, PRQuant achieves the best average accuracy among the quantized methods, improving accuracy over MXFP4 by 1.24 and 0.55, respectively.
Figures & tables
Figure 1: Schematic overview of PRQuant: (a) activation-to-weight quantization difficulty migration under scaling; (b) channel permutation, offline weight residual augmentation, and online activation augmentation.
Method
ARC-C
ARC-E
HellaSwag
PIQA
WinoGrande
Average
Qwen3-30B
BF16
95.93
98.77
90.19
92.27
75.92
90.62
MXFP4
94.57
98.58
88.74
90.81
74.03
89.35
ARCQuant
95.25
98.77
87.23
91.57
73.01
89.17
BATQuant
94.24
98.59
88.60
91.62
74.59
89.53
FlatQuant
94.92
98.77
88.73
91.51
75.14
89.81
Table 1: End-to-end accuracy ( ↑ ) on Qwen3-30B and Qwen3-4B using zero-shot evaluation. For quantized configurations, the attention modules use MXFP8 and the FFN/MoE modules use MXFP4. Bold values denote the best result among the quantized configurations for each model. The average is the unweighted mean over the five benchmarks.
Figure 2: Ablation results on the down-projection layers of routed experts of Qwen3-30B. Layer-wise mean y -NRMSE across experts.
Figure 3: Operator-level Speedup ( ↑ ) over BF16 on Ascend 950PR for the down-projection kernels of Qwen3-4B and Qwen3-30B across representative input sizes M . The dashed horizontal line denotes the BF16 baseline ( 1× ).
Method
Quantization Latency (h)
Memory (GiB)
BATQuant
8.5
36.1
FlatQuant
7.4
22.9
MR-GPTQ
9.1
59.2
PRQuant
0.4
4.4
Table 2: Quantization cost ( ↓ ) of Qwen3-30B. Both BATQuant and FlatQuant are implemented using the ModelSlim 2 2 2 https://gitcode.com/Ascend/msmodelslim framework.
Batch
ISL
Qwen3-4B
Qwen3-30B
BF16
MXFP4
PRQuant
ARCQuant
BF16
MXFP4
PRQuant
ARCQuant
1
1024
41.2
30.8
31.8
52.0
65.5
52.8
55.7
62.9
2048
72.2
47.0
46.1
54.1
92.5
65.8
86.8
85.1
4096
147.8
87.4
88.0
93.2
186.2
129.8
161.7
166.5
8
1024
224.9
132.4
132.8
171.2
250.4
174.6
234.6
248.7
2048
346.4
189.6
192.5
222.5
458.8
296.5
411.3
447.9
Table 3: Prefill latency TTFT (ms, ↓ ) of Qwen3-4B and Qwen3-30B on Ascend 950PR under different batch sizes and input sequence lengths (ISL).
Model
Batch
TPOT (ms/token)
Speedup
BF16
MXFP4
PRQuant
ARCQuant
MXFP4
PRQuant
ARCQuant
Qwen3-4B
1
7.4
4.9
5.2
5.3
1.51 ×
1.42 ×
1.40 ×
8
11.0
7.7
7.7
8.2
1.43 ×
1.43 ×
1.34 ×
16
15.3
11.0
11.0
11.8
1.39 ×
1.39 ×
1.30 ×
32
23.9
17.9
17.9
19.0
1.34 ×
1.34 ×
1.26 ×
Qwen3-30B
1
8.2
6.6
6.8
6.9
1.24 ×
1.21 ×
1.19 ×
Table 4: Decode latency TPOT (ms/token, ↓ ) and speedup ( ↑ ) over BF16 of Qwen3-4B and Qwen3-30B on Ascend 950PR with ISL=2048 and OSL=128 .