Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference

Sunghyun Wee; Geunjae Choi; Hyeonjin Kim; Suyoung Kim; Nojun Kwak

cs.LGOct 15, 2026

Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference

Authors: Sunghyun Wee, Geunjae Choi, Hyeonjin Kim, Suyoung Kim, Nojun Kwak

Organizations: Seoul National University · LG Electronics

Abstract

Large Language Models (LLMs) demand efficient inference, where the multiply-accumulate (MAC) operations in linear projections dominate compute and energy cost. Post-training quantization (PTQ) reduces this cost by mapping weights and activations to low-bit integers. Power-of-Two (PoT) weight quantization promises efficient multiplier-free LLM inference via shift-add logic, but suffers significant accuracy degradation at low bitwidths due to its coarse base-2 quantization grid. We introduce Fractional PoT (FPoT), a base- $\sqrt{2}$ (half-octave) fractional PoT grid that achieves practical 4-bit weight quantization while preserving a fully integer shift-add datapath. The method combines (i) the base- $\sqrt{2}$ weight grid with uniform integer activations, (ii) a calibration-time grid-alignment refinement absorbed into the static per-channel scale, and (iii) a lightweight 1-adder $\sqrt{2}$ approximation whose error is absorbed during weight calibration via approximate-grid quantization, yielding a processing element (PE) with significantly fewer gates than a 4-bit integer multiplier. On Llama-3-8B with 4-bit weight and activation quantization, FPoT improves zero-shot commonsense reasoning performance by 8.72% relative to the PoT baseline, while maintaining performance close to the uniform integer baseline, with only a negligible 0.4%p degradation. These results hold across various configurations spanning seven models and three bit-widths. Furthermore, FPoT can be integrated with state-of-the-art LLM PTQ methods, demonstrating framework-agnostic applicability.

Figures & tables

Figure 1: PE architecture: INT4 multiplier, PoT barrel shifter, and FPoT (ours). FPoT adds one adder and a MUX;

\sqrt{2}\approx y+y{\gg}1=1.5

(

6.1\%

error, absorbed by approximate-grid quantization, § 3 ).

Figure 2: (a) MSE vs. shape parameter

k

for

\mathcal{N}(0,1)

weights at 4-bit (

N{=}15

levels): the MSE is minimized at

k^{\star}\approx 1.8

, so the integer choice

k{=}2

is near-optimal. (b, c) Bin-occupancy under the Gaussian proxy. FPoT concentrates levels near the high-density mode, while PoT spreads levels across an excessively wide range, leaving most of them in the sparsely-populated tail. Dashed curves: input density

p_{W}

Figure 3: Accuracy–cost Pareto trade-off for Llama-3-8B. FPoT achieves near-baseline PPL at

{\sim}0.50{\times}

CU cost across all bit-widths. Per-model plots in Appendix E.4 .

Impl.	Value (Err.)	Add.	CU/ ${\times}$	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$
				Exact Grid		Approx Grid		Aware
Exact	1.414 (0%)	—	N/A ‡	7.560	64.7	—	—	—	—
3-adder	1.438 (1.6%)	3	$1.00{\times}$	7.580	64.4	7.569	65.0	7.611	64.3
2-adder	1.375 (2.8%)	2	$0.75{\times}$	7.575	64.5	7.584	64.4	7.584	64.8
1-adder	1.500 (6.1%)	1	$\bm{0.50{\times}}$	7.729	63.4	7.574	64.2	7.625	63.8
$k{=}1$	—	0	$0.38{\times}$	8.806	59.0	—	—	—	—

Table 2:

\sqrt{2}

approximation strategies on Llama-3-8B W4A4KV4 (FPoT; single calibration seed). Exact Grid / Approx Grid (Ours) / Aware: see § 5.3 for definitions. CU/

{\times}

: same naive component model (§ 4.2 ); ‡ exact

\sqrt{2}

has no finite CSD realization. 0-shot 9 in %.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Method	MSE	NBE	MSE ratio
Uniform INT4	0.0128	0.9026	$1.00{\times}$
PoT	0.0411	0.8773	$3.21{\times}$
FPoT k=2	0.0162	0.9364	$\mathbf{1.27{\times}}$
FPoT k=3	0.0368	0.8762	$2.87{\times}$
FPoT k=4	0.0688	0.8115	$5.37{\times}$
Optimal ( $k^{\star}{\approx}1.8$ )	0.0158	0.9417	$1.23{\times}$

Appendix

Table 3: MSE and NBE of 4-bit PoT quantizers applied to

\mathcal{N}(0,1)

weights (

N{=}15

levels: 14 non-zero symmetric + 1 zero, log-domain rounding). MSE ratio is relative to uniform INT4.

Figure 4: Quantization grid comparison: Uniform, PoT (octave), and FPoT (half-octave). The FPoT grid places

2{\times}

more levels within the high-density region of the Gaussian weight distribution.

Figure 5: Empirical bin occupancy for Llama-3-8B (W4A4KV4, aggregated across all projections). The pattern closely matches the Gaussian proxy prediction in Fig. 2 .

Figure 6: Per-projection bin occupancy of RTN-quantized Llama-3-8B weights (W4A4KV4). Rows:

W_{Q}

W_{K}

W_{V}

W_{O}

W_{Gate}

W_{Up}

W_{Down}

. Columns: Uniform, PoT, FPoT. The occupancy pattern is consistent across all seven projections, with FPoT achieving NBE

>0.91

uniformly (0.9167–0.9286), substantially above PoT and uniform INT4.

Figure 7: Bin occupancy before (top) and after (bottom) PTQ preprocessing for Llama-3-8B (W4A4KV4). uniform INT4 NBE rises from 0.8599 to 0.8761 and FPoT from 0.9142 to 0.9271; PoT drops slightly (0.9111

\to

0.8982).

Figure 8: Bin occupancy before (top) and after (bottom) PTQ preprocessing for Llama-2-7B (W4A4KV4). NBE shifts: uniform INT4 0.8662

\to

0.8766; FPoT 0.9177

\to

0.9274.

Figure 9: Bin occupancy before (top) and after (bottom) PTQ preprocessing for Llama-2-13B (W4A4KV4). NBE shifts: uniform INT4 0.8655

\to

0.8766; FPoT 0.9174

\to

0.9275.

Figure 10: Bin occupancy before (top) and after (bottom) PTQ preprocessing for Qwen2.5-1.5B (W4A4KV4). NBE shifts are large (uniform INT4 0.8293

\to

0.8768; FPoT 0.8874

\to

0.9273), reflecting strong raw-weight outlier concentration.

Figure 11: Bin occupancy before (top) and after (bottom) PTQ preprocessing for Qwen2.5-7B (W4A4KV4). NBE shifts: uniform INT4 0.8065

\to

0.8775; FPoT 0.8644

\to

0.9282.

Figure 12: Bin occupancy before (top) and after (bottom) PTQ preprocessing for Qwen2.5-14B (W4A4KV4). NBE shifts: uniform INT4 0.8201

\to

0.8787; FPoT 0.8701

\to

0.9286.

Projection	MACs (M)	% of Total
$W_{Q}$	537	7.7%
$W_{K}$	134	1.9%
$W_{V}$	134	1.9%
$W_{O}$	537	7.7%
Gate	1,879	26.9%
Up	1,879	26.9%

Appendix

Table 4: MAC Breakdown per Token for Llama-3-8B (32 Layers)

FPoT Applied To	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$ (%)
None (non-PoT baseline)	7.444	64.98
$W_{Q}$ only	7.379	65.59
$W_{K}$ only	7.393	64.77
$W_{V}$ only	7.487	65.45
$W_{O}$ only	7.337	65.21
Gate only	7.381	65.45

Appendix

Table 5: Per-projection FPoT weight-only ablation on Llama-3-8B (W4A4KV4, 400 steps, single seed). Each row applies FPoT to the indicated projection(s); all others use uniform INT. Activations use uniform INT4 throughout.

Config	INT	PoT	FPoT	PoT/INT	FPoT/INT
Compute unit only
W4A4	1.00	0.38	0.50	$0.38{\times}$	$0.50{\times}$
W4A8	2.00	0.75	1.00	$0.38{\times}$	$0.50{\times}$
W8A8	4.00	1.75	2.00	$0.44{\times}$	$0.50{\times}$
Full MAC (compute unit + accumulator)
W4A4	1.50	0.88	1.00	$0.58{\times}$	$0.67{\times}$

Appendix

Table 6: Naive component-based cost model for INT, PoT, and FPoT compute units and full MACs (Section 4.2 ). Values are normalized to W4A4 INT compute unit

=1.00{\times}

, using

c_{\mathrm{mux}}=0.5{\times}c_{\mathrm{fa}}

(full derivation in § E.1 ). The accumulator (

C_{\text{ACC}}=W{+}A

) is identical across all three modes and added to obtain the full-MAC cost.

Config	INT	PoT	FPoT	PoT/INT	FPoT/INT
Compute unit only
W4A4	75 (1.00)	18 (0.24)	44 (0.59)	$0.24{\times}$	$0.59{\times}$
W4A8	154 (2.05)	30 (0.40)	87 (1.16)	$0.19{\times}$	$0.57{\times}$
W8A8	391 (5.21)	92 (1.23)	167 (2.23)	$0.24{\times}$	$0.43{\times}$
Full MAC (compute unit + accumulator)
W4A4	119 (1.59)	60 (0.80)	91 (1.21)	$0.50{\times}$	$0.77{\times}$

Appendix

Table 7: Gate counts from Yosys + ABC open-source synthesis (generic-gate library, abc -g gates ), with relative cost in parentheses normalized to W4A4 INT compute unit

=1.00{\times}

(matching Table 6 ). Sign negation and zero-gating reside in the accumulator control path and are excluded from compute-unit counts (Remark 1 , assumption 5).

Figure 13: Accuracy–cost Pareto trade-off: Llama-3-8B.

Figure 14: Accuracy–cost Pareto trade-off: Llama-2-7B.

Figure 15: Accuracy–cost Pareto trade-off: Llama-2-13B.

Figure 16: Accuracy–cost Pareto trade-off: Mistral-7B.

Figure 17: Accuracy–cost Pareto trade-off: Qwen2.5-1.5B.

Figure 18: Accuracy–cost Pareto trade-off: Qwen2.5-7B.

Figure 19: Accuracy–cost Pareto trade-off: Qwen2.5-14B.

Cal. Dataset	$n$	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$ (%)	$\Delta$ PPL
WikiText-2	128	7.559	65.07	—
	256	7.589	64.68	+0.030
	512	7.525	64.80	$-$ 0.034
C4 [ Raffel et al., 2020 ]	128	7.617	64.91	+0.058
	256	7.660	65.01	+0.101
	512	7.647	64.88	+0.088

Appendix

Table 8: Calibration sensitivity analysis on Llama-3-8B (W4A4KV4, FPoT, 400 steps). Rows vary the calibration dataset and number of calibration samples (

n

). The default configuration used throughout the paper is WikiText-2 with

n{=}128

	$k{=}1$		$k{=}2$
$G$	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$
—	8.806	58.99	7.563	64.28
$4$	8.755	59.09	7.545	64.49
$8$	8.771	60.04	7.597	64.45
$16$	8.771	60.08	7.560	64.71
$32$	8.801	59.69	7.568	63.55

Appendix

Table 9: Weight grid-alignment ablation (Llama-3-8B, W4A4KV4, 400 steps, single seed).

G

: number of phase-shift candidates (Eq. ( 4 ));

G{=}

— denotes the plain FPoT baseline. At

k{=}1

, grid-alignment reduces PPL by up to

-0.051

; at

k{=}2

the finer grid already self-corrects, so

|\Delta\text{PPL}|\leq 0.034

. 0-shot 9 values are in %.

$k$	Scope	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$
W4A4KV4 (non-PoT baseline: PPL $=7.44$ )
$k{=}1$	Full	16.91	44.6
$k{=}1$	W-only	8.81	59.0
$k{=}2$	Full	8.21	62.5
$\bm{k{=}2}$	W-only (Ours)	7.56	64.3
W4A8KV8 (non-PoT baseline: PPL $=6.60$ )

Appendix

Table 10: Weight-only vs. Full PoT/FPoT on Llama-3-8B (all 7 projections, single seed). Weight-only: PoT/FPoT weight + uniform INT activation. Full: PoT/FPoT applied to both weight and activation. 0-shot 9 values are in %.

$\alpha$	$\tau$ ( ${=}2^{\alpha/2}$ )	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$ (%)
$\bm{-0.5}$	$\bm{0.841}$	$\bm{7.560}$	$\bm{64.71}$
$-1$	$0.707$	$7.573$	$64.99$
$-2$	$0.500$	$7.565$	$64.99$
$-3$	$0.354$	$7.553$	$65.06$
$-4$	$0.250$	$7.545$	$64.72$
$-6$	$0.125$	$7.557$	$64.54$

Appendix

Table 11: Zero-threshold sensitivity analysis on Llama-3-8B (W4A4KV4, FPoT, 400 steps). The threshold

\tau=2^{\alpha/k}

determines the boundary below which

|w/s|

is mapped to zero. The default

\alpha{=}{-}0.5

corresponds to the log-space midpoint between zero and the smallest PoT level.

Rounding mode	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$ (%)
Log-space NN (default)	$7.560$	$64.71$
Linear-space NN	$7.559$	$64.81$

Appendix

Table 12: Rounding mode comparison on Llama-3-8B (W4A4KV4, FPoT, 400 steps). Log-space rounds to the nearest PoT level measured by relative distance (geometric mean boundary). Linear-space rounds by absolute distance (arithmetic mean boundary).

Preprocessing	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$ (%)
No transform	181.3	37.56
Scale only	9.78	58.79
Fixed Hadamard	8.30	62.64
Learned rotation	7.78	63.30
Learned rotation + scale	7.56	64.28

Appendix

Table 13: Preprocessing component ablation within OSTQuant under FPoT weight quantization on Llama-3-8B W4A4KV4 (400 steps, single seed; cf. Table 1 3-seed mean

7.60_{\pm.02}

for the full-pipeline entry). Activations and KV-cache use uniform INT4 throughout.

Config	Method	ARC-C	ARC-E	Hella	Wino	PIQA	BoolQ	OBQA	Lam	SIQA	AVG
FP (bf16) reference		53.1	77.9	79.2	73.4	81.0	81.3	45.0	75.7	47.1	68.2
W4A4KV4	Uniform	44.8	76.4	57.0	68.2	76.9	77.0	32.2	73.5	46.3	65.0
	PoT	36.4	68.1	53.0	67.2	73.8	67.2	26.6	68.0	42.6	59.0
	FPoT (Ours)	43.9	75.4	56.0	69.2	76.7	78.0	31.2	73.2	45.1	64.2
W4A8KV8	Uniform	49.4	79.7	59.1	74.0	78.5	78.5	34.2	75.3	47.5	67.5
	PoT	44.5	76.5	57.0	72.1	76.2	78.1	31.4	71.8	45.5	64.7

Appendix

Table 14: Per-task results: Llama-3-8B . Single calibration seed (seed 0); FP (bf16) is the unquantized reference; FPoT (Ours) rows are bolded. All scores in %.

Config	Method	ARC-C	ARC-E	Hella	Wino	PIQA	BoolQ	OBQA	Lam	SIQA	AVG
FP (bf16) reference		45.9	74.5	75.9	68.8	78.7	78.0	44.2	73.6	46.0	65.1
W4A4KV4	Uniform	40.6	73.4	54.7	66.9	75.8	74.7	32.4	70.4	44.0	62.2
	PoT	35.8	69.1	52.0	62.8	73.6	68.8	27.2	63.8	41.1	58.1
	FPoT (Ours)	40.4	73.9	53.3	67.3	75.4	73.8	29.8	72.4	44.0	62.1
W4A8KV8	Uniform	42.3	75.6	55.7	68.4	77.0	75.3	30.8	71.6	44.4	63.8
	PoT	39.8	72.4	54.3	66.5	76.2	72.3	28.6	66.7	43.6	61.1

Appendix

Table 15: Per-task results: Llama-2-7B . Single calibration seed (seed 0); FP (bf16) is the unquantized reference; FPoT (Ours) rows are bolded. All scores in %.

Config	Method	ARC-C	ARC-E	Hella	Wino	PIQA	BoolQ	OBQA	Lam	SIQA	AVG
FP (bf16) reference		49.1	77.5	79.4	71.7	80.6	80.7	45.6	76.7	47.2	67.6
W4A4KV4	Uniform	43.2	76.5	58.3	69.6	77.8	77.8	34.0	75.4	46.8	65.5
	PoT	41.4	74.8	56.2	67.8	76.4	75.7	31.0	67.8	42.1	62.3
	FPoT (Ours)	44.0	76.2	57.2	68.7	77.5	78.8	33.8	75.7	44.8	65.2
W4A8KV8	Uniform	45.5	78.1	59.5	72.5	78.6	79.4	33.8	76.6	46.6	66.7
	PoT	46.8	77.4	58.1	69.6	77.7	79.5	32.2	72.1	45.1	65.5

Appendix

Table 16: Per-task results: Llama-2-13B . Single calibration seed (seed 0); FP (bf16) is the unquantized reference; FPoT (Ours) rows are bolded. All scores in %.

Config	Method	ARC-C	ARC-E	Hella	Wino	PIQA	BoolQ	OBQA	Lam	SIQA	AVG
FP (bf16) reference		44.9	71.6	67.8	63.5	76.0	72.8	40.4	61.9	49.0	60.9
W4A4KV4	Uniform	28.1	41.5	29.3	58.0	63.5	51.0	21.2	0.0	35.9	38.7
	PoT	27.1	40.5	29.0	54.2	62.6	53.1	21.8	0.0	35.2	38.1
	FPoT (Ours)	28.0	44.7	29.3	59.0	63.6	56.3	21.2	0.0	35.6	39.9
W4A8KV8	Uniform	29.0	45.5	30.1	62.8	65.4	70.9	23.2	0.0	36.9	43.2
	PoT	28.5	43.9	29.4	61.1	64.4	46.7	20.2	0.0	36.2	39.2

Appendix

Table 17: Per-task results: Qwen2.5-1.5B . Single calibration seed (seed 0); FP (bf16) is the unquantized reference; FPoT (Ours) rows are bolded. All scores in %.

Config	Method	ARC-C	ARC-E	Hella	Wino	PIQA	BoolQ	OBQA	Lam	SIQA	AVG
FP (bf16) reference		51.4	77.7	79.0	73.2	80.0	84.7	47.0	71.6	54.8	68.8
W4A4KV4	Uniform	47.1	78.5	57.2	69.9	77.9	81.5	32.4	68.4	50.3	66.0
	PoT	41.2	74.2	55.2	64.8	76.2	79.1	29.6	61.6	49.4	62.8
	FPoT (Ours)	48.3	79.6	56.5	67.5	76.9	82.3	32.0	69.2	51.9	66.7
W4A8KV8	Uniform	48.0	80.5	59.3	71.6	78.7	83.9	34.6	71.8	51.8	68.0
	PoT	45.2	77.0	57.5	70.9	77.8	81.6	31.8	64.7	50.8	65.6

Appendix

Table 18: Per-task results: Qwen2.5-7B . Single calibration seed (seed 0); FP (bf16) is the unquantized reference; FPoT (Ours) rows are bolded. All scores in %.

Config	Method	ARC-C	ARC-E	Hella	Wino	PIQA	BoolQ	OBQA	Lam	SIQA	AVG
FP (bf16) reference		59.0	79.3	82.9	74.7	82.1	85.4	44.8	74.1	55.5	70.9
W4A4KV4	Uniform	51.6	81.4	60.3	72.1	79.1	83.9	33.6	71.3	50.0	68.4
	PoT	50.2	78.0	57.6	66.6	77.5	80.6	32.4	65.3	47.8	65.7
	FPoT (Ours)	52.4	82.1	59.8	73.2	78.5	84.6	33.6	73.6	52.2	69.2
W4A8KV8	Uniform	56.1	83.0	62.9	75.5	80.6	85.4	35.4	75.4	55.2	71.4
	PoT	54.0	82.8	60.8	73.0	79.8	82.6	31.4	68.0	51.8	68.7

Appendix

Table 19: Per-task results: Qwen2.5-14B . Single calibration seed (seed 0); FP (bf16) is the unquantized reference; FPoT (Ours) rows are bolded. All scores in %.

Config	Method	ARC-C	ARC-E	Hella	Wino	PIQA	BoolQ	OBQA	Lam	SIQA	AVG
FP (bf16) reference		52.2	78.3	80.5	73.6	82.3	82.1	44.2	75.1	45.8	68.2
W4A4KV4	Uniform	45.5	77.7	59.0	69.9	79.2	82.4	30.6	73.4	44.7	66.3
	PoT	44.5	76.7	57.5	68.7	78.8	79.9	29.6	70.5	44.3	65.0
	FPoT (Ours)	48.0	77.8	57.9	70.5	79.2	81.0	31.8	75.0	44.7	66.4
W4A8KV8	Uniform	48.6	79.7	60.0	73.0	79.9	82.3	31.8	74.6	45.4	67.8
	PoT	44.5	77.7	58.8	71.1	79.6	76.7	29.6	71.7	44.9	65.1

Appendix

Table 20: Per-task results: Mistral-7B . Single calibration seed (seed 0); FP (bf16) is the unquantized reference; FPoT (Ours) rows are bolded. All scores in %.

Model	Method	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$
		W4A16		W4A4KV4 †
LLaMA-3-8B	LogART	6.516	68.8	3903	34.4
LLaMA-3-8B	FPoT (Ours)	6.862	66.6	7.60	64.8
LLaMA-2-7B	LogART	5.617	66.8	$\infty$	33.9
LLaMA-2-7B	FPoT (Ours)	5.861	63.5	6.11	61.9

Appendix

Table 21: Comparison with LogART under weight-only (W4A16) and full (W4A4KV4) quantization. All methods are multiplier-free (PoT-based). FPoT W4A4KV4 entries are copied from Table 1 (mean over three calibration seeds); FPoT W4A16 and all LogART entries use a single calibration seed. PPL: WikiText-2 (

\downarrow

). 0-shot 9 : unweighted mean across nine benchmarks (

\uparrow

, %).

Model	Method	Weight	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$	PPL ${\downarrow}$	0-shot ${}^{9}{\uparrow}$
			W4A4KV4		W4A8KV8		W8A8KV8
Llama-3-8B	OSTQuant	Uniform	7.36	65.2	6.60	67.5	6.16	68.2
	OSTQuant	FPoT	7.60	64.8	6.85	67.3	6.58	67.7
	DuQuant	Uniform	7.67	62.9	6.56	67.3	6.17	68.0
	DuQuant	FPoT	7.87	63.1	6.68	66.6	6.51	66.9
Llama-2-7B	OSTQuant	Uniform	5.97	62.1	5.63	63.8	5.48	65.1

Appendix

Table 22: Cross-framework comparison of FPoT weight quantization; only the weight grid differs across methods (Uniform INT vs. FPoT), with activations and KV-cache using uniform INT throughout. OSTQuant (ours): 3-seed mean; see Table 1 . DuQuant : single calibration seed; column-wise weight calibration replaces the default RTN quantizer (see Appendix J ). PPL: WikiText-2 perplexity (

\downarrow

). 0-shot 9 : mean zero-shot accuracy across nine benchmarks (

\uparrow

, %).

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 381 papers

Mar 26, 2026

SliderQuant: Accurate Post-Training Quantization for LLMs

In this paper, we address post-training quantization (PTQ) for large language models (LLMs) from an overlooked perspective: given a pre-trained high-precision LLM, the predominant sequential quantization framework treats different layers equally, but this may be not optimal in challenging bit-width settings. We empirically study the quantization impact of different layers on model accuracy, and observe that: (1) shallow/deep layers are usually more sensitive to quantization than intermediate layers; (2) among shallow/deep layers, the most sensitive one is the first/last layer, which exhibits significantly larger quantization error than others. These empirical observations imply that the quantization design for different layers of LLMs is required on multiple levels instead of a single level shared to all layers. Motivated by this, we propose a new PTQ framework termed Sliding-layer Quantization (SliderQuant) that relies on a simple adaptive sliding quantization concept facilitated by few learnable parameters. The base component of SliderQuant is called inter-layer sliding quantization, which incorporates three types of novel sliding window designs tailored for addressing the varying quantization sensitivity of shallow, intermediate and deep layers. The other component is called intra-layer sliding quantization that leverages an incremental strategy to quantize each window. As a result, SliderQuant has a strong ability to reduce quantization errors across layers. Extensive experiments on basic language generation, zero-shot commonsense reasoning and challenging math and code tasks with various LLMs, including Llama/Llama2/Llama3/Qwen2.5 model families, DeepSeek-R1 distilled models and large MoE models, show that our method outperforms existing PTQ methods (including the latest PTQ methods using rotation transformations) for both weight-only quantization and weight-activation quantization.

Shigeng Wang, Chao Li, Yangyuxuan Kang +3

Intel Labs China · BUPT

LLM Quantization Post-Training Quantization Neural Network Quantization Weight-Only Quantization Activation Quantization

Ask PDF

Jun 5, 2025

FPTQuant: Function-Preserving Transforms for LLM Quantization

Large language models (LLMs) require substantial compute, and thus energy, at inference time. While quantizing weights and activations is effective at improving efficiency, naive quantization of LLMs can significantly degrade performance due to large magnitude outliers. This paper describes FPTQuant, which introduces three novel, lightweight, and expressive function-preserving transforms (FPTs) to facilitate quantization of transformers: (1) a mergeable pre-RoPE transform for queries and keys, (2) a mergeable transform for values, and (3) a cheap, dynamic per-token scaling transform. By leveraging the equivariances and independencies inherent to canonical transformer operation, we designed these FPTs to maintain the model's function while shaping the intermediate activation distributions to be more quantization friendly. FPTQuant requires no custom kernels and adds virtually no overhead during inference. The FPTs are trained both locally to reduce outliers, and end-to-end such that the outputs of the quantized and full-precision models match. FPTQuant enables static INT4 quantization with minimal overhead and shows SOTA speed-up of up to 3.9X over FP. Empirically, FPTQuant has an excellent accuracy-speed trade-off -- it is performing on par or exceeding most prior work and only shows slightly lower accuracy compared to a method that is up to 29% slower.

Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1

Qualcomm AI Research, Amsterdam, Netherlands

LLM Quantization LLM Inference Acceleration Activation Quantization Low-Bit Quantization

Ask PDF

Apr 29, 2026

CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs

Post-training quantization (PTQ) has become an important technique for reducing the inference cost of Large Language Models (LLMs). While recent mixed-precision methods improve ultra-low bit quantization by preserving critical subspaces in high precision, they typically construct these subspaces relying solely on activation statistics. This ignores the fundamental nature of linear operations, where the output perturbation is jointly driven by both activation and weight quantization noise. In this paper, we propose CoQuant, a joint weight-activation subspace projection method. By theoretically modeling the expected output error, CoQuant formulates a closed-form weighted PCA solution that balances activation and weight covariances to select the optimal high-precision subspace. Extensive experiments on Llama-3.2 and Qwen2.5 models show that CoQuant consistently outperforms strong PTQ baselines in both WikiText perplexity and zero-shot common-sense reasoning accuracy. These results demonstrate that joint weight-activation subspace modeling provides a principled and effective direction for low-bit LLM quantization. The source code is available at https://github.com/Zachary5895/CoQuant.

Zhe Ding, Su Pan, Duowei Pan

School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China · Amazon AGI, Seattle, WA, USA

Mixed-Precision Quantization LLM Quantization Activation Quantization Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Ask PDF

Abstract

LLM Quantization

Explore similar work

SliderQuant: Accurate Post-Training Quantization for LLMs

FPTQuant: Function-Preserving Transforms for LLM Quantization

CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs