WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization

Egor Lifar; Semyon Savkin; Or Ordentlich; Yury Polyanskiy

cs.LGMar 5, 2026

WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization

Authors: Egor Lifar, Semyon Savkin, Or Ordentlich, Yury Polyanskiy

Organizations: MIT · Independent Researcher · Hebrew University of Jerusalem

Abstract

This paper considers the problem of converting a given dense linear layer to low precision. The tradeoff between compressed length and output discrepancy is analyzed information theoretically (IT). It is shown that a popular GPTQ algorithm may have an arbitrarily large gap to the IT limit. To alleviate this problem, a novel algorithm, termed ``WaterSIC'', is proposed and is shown to be within a rate gap of 0.255 bits to the IT limit, uniformly over all possible covariance matrices of input activations. The key innovation of WaterSIC's is to allocate different quantization rates to different columns (in-features) of the weight matrix, mimicking the classical IT solution known as "waterfilling". Applying WaterSIC to the Llama and Qwen family of LLMs establishes new state-of-the-art performance for all quantization rates from 1 to 4 bits. Our code is available at https://github.com/egorlifar/watersic.

Figures & tables

Figure 1: WikiText-2 bits-per-byte (BPB) vs. compressed model size (GiB) for a layerwise WaterSIC and finetuned WaterSIC-FT across multiple base models. Lines connect the same model compressed with the same algorithm and with various bit rates. Evaluated at CTX=2048.

Method	Avg. Bitwidth	WikiText-2 PPL $\downarrow$
WaterSIC	1.00	82.44
WaterSIC-FT	1.00	30.47
WaterSIC-FT	1.50	18.50
WaterSIC	1.50	30.73
Huffman-GPTQ	1.52	1000+
Huffman-GPTQ	1.94	86.80

Table 1: Comparison of wikitext2 perplexity results on Llama3.2-1B. In each group of rows WaterSIC-FT achieves the best PPL while having minimal rate. The unquantized model perplexity is 9.76. Evaluated at CTX=2048.

Method (bits)	2.125	2.625	3.125	3.625	4.125
Huffman-GPTQ	13.97	–	10.34	–	9.81
GPTQ	57.51	–	12.77	–	10.10
Huffman-RTN	593.05	–	10.75	–	9.90
RTN	$2\times 10^{10}$	–	16.30	–	10.30
WaterSIC	11.37	10.44	10.03	9.87	9.79
WaterSIC-FT	10.18	9.85	9.73	9.70	9.67

Table 2: Qwen3-8B WikiText-2 perplexity (lower is better) at fixed average bitwidths. Results for Huffman-GPTQ(HPTQ)/GPTQ/Huffman-RTN(HRTN)/RTN are taken from Chen et al. (2026) ; WaterSIC-FT achieves the best perplexity at all rates. The unquantized model perplexity is

9.73

. Evaluated at CTX=2048.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Table 5

Figure 4: Diagonal rescalers statistics. Llama-3.2-1B, various rates. Row is the reduction (in-channel) dimension, so that row rescaler affects one out-channel, and column rescaler affects one in-channel. Calibration at CTX=2048.

Layer	Low-variance input indices
Layer 0, ATTN input	278*
Layer 0, MLP input	64, 146, 509, 1280, 1618, 1938, 2023
Layer 1, ATTN input	146, 735, 762, 841, 894, 1002, 1314, 1334, 1476, 1503, 1619, 2037
Layer 1, MLP input	64, 146, 509, 1228, 2023
Layer 2, ATTN input	1159, 1314, 2023*
Layer 2, MLP input	146, 2023

Appendix

Table 5: Llama-3.2-1B: input features with standard deviation below 1% (asterisk) and 0.1% (no asterisk) of the mean standard deviation among all coordinates. These near-zero coordinates are produced by a diagonal multiplier inside the RMSnorm at the respective layer’s input. Calibration at CTX=2048.

Figure 5: Distribution of actual compression rates per in-channel inside a single layer (left) and over all layers (right) for Qwen3-8B model. Calibration at CTX=2048.

Layer	Matrix	entropy (all matrix entries)	max(col-entropy)	avg(col-entropy)	zstd (bpp)	lzma (bpp)
6	attention.wk	1.963	4.100	1.866	2.037	2.126
	attention.wo	1.975	4.397	1.853	2.016	2.107
	attention.wq	1.994	4.330	1.923	2.061	2.147
	attention.wv	1.979	3.355	1.922	2.049	2.194
	feed_forward.w1	1.993	3.869	1.938	2.045	2.163
	feed_forward.w2	1.986	4.162	1.892	2.094	2.142

Appendix

Table 6: Per-matrix entropy statistics and compression rates (bits/parameter) for Zstandard and LZMA on int8 -packed weights. Calibration at CTX=2048.

Figure 6: Effect of residual stream compensation on Llama-3.2-1B ( $4\,\text{bit}$ ). Residual compensation reduces activation MSE primarily at $w_{o}$ and $w_{2}$ inputs, with improvements propagating to subsequent layers through the residual stream. Computed at CTX=2048.

Figure 7: Effect of activation drift correction on Llama-3.2-1B ( $4\,\text{bit}$ ). Adding Qronos improves most layers but degrades $w_{o}$ inputs in some layers, where softmax amplifies QKV quantization errors. Computed at CTX=2048.

Figure 8: Effect of attention-weighted calibration on Llama-3.2-1B ( $4\,\text{bit}$ ). Joint adaptive mixing with attention-weighted covariances eliminates the $w_{o}$ degradation caused by Qronos. Computed at CTX=2048.

Figure 9: Effect of adaptive mixing on Qwen3-8B ( $4.12\,\text{bit}$ ). Adaptive $\epsilon_{\mathrm{qr}}$ blending stabilizes Qronos in deeper layers where pure drift correction degrades. Computed at CTX=2048.

Figure 10: Full WaterSIC pipeline vs. base WaterSIC on Llama-3.2-1B ( $4\,\text{bit}$ ). All techniques combined yield a consistent reduction in activation MSE across every layer. Computed at CTX=2048.

Figure 11: The figure contains weight histograms for selected layers of Llama-3.2-1B with best-fit Gaussian and Laplace overlays. The table contains the Kolmogorov-Smirnov distance to the best-fit Gaussian and Laplace distributions averaged over the 32 decoder layers of Llama-2-7B. The rightmost column reports the number of layers (out of 32) for which the Gaussian fit is closer than the Laplace fit.

Rate	WaterSIC W2 $\downarrow$	WaterSIC C4 $\downarrow$	WaterSIC-FT W2 $\downarrow$	WaterSIC-FT C4 $\downarrow$
1.00	82.45	643.29	32.85	160.70
1.50	30.73	140.72	19.16	59.97
2.00	16.19	37.73	13.59	27.80
2.50	11.83	18.81	11.34	17.80
3.00	10.57	15.26	10.45	15.06
3.18	10.35	14.73	10.27	14.61

Appendix

Table 7: Llama-3.2-1B WikiText-2 and C4 test perplexity at various rates. WaterSIC is calibrated on WikiText-2 train at CTX=2048; WaterSIC-FT is the same model after post-quantization finetuning on WikiText-2 at CTX=2048. Both evaluated at CTX=2048. Unquantized (BF16): W2 PPL 9.73, C4 PPL 12.77.

Figure 12: Llama-3.2-1B: KL divergence between BF16 and quantized output distributions,

\mathrm{KL}(P_{\mathrm{BF16}}\,\|\,P_{\mathrm{quant}})

, against average bitwidth, for Huffman-GPTQ, WaterSIC, and WaterSIC-FT. The shaded region marks the gap between Huffman-GPTQ and WaterSIC at matched bitwidth. Evaluated at CTX=2048.

Rate	WaterSIC W2 $\downarrow$	WaterSIC C4 $\downarrow$	WaterSIC-FT W2 $\downarrow$	WaterSIC-FT C4 $\downarrow$
1.12	25.59	102.52	14.54	43.17
1.62	14.45	33.96	11.27	24.33
2.12	11.37	18.64	10.18	16.54
2.62	10.44	15.15	9.85	14.50
3.12	10.03	14.11	9.73	13.83
3.62	9.87	13.77	9.70	13.63

Appendix

Table 8: Qwen3-8B WikiText-2 and C4 test perplexity at various rates. WaterSIC is calibrated on WikiText-2 train at CTX=2048; WaterSIC-FT is the same model after post-quantization finetuning on WikiText-2 at CTX=2048. Both evaluated at CTX=2048. Unquantized (BF16): W2 PPL 9.73, C4 PPL 13.30.

Method	Avg. Bitwidth	WikiText-2 PPL
WaterSIC-FT	1.00	18.97
WaterSIC	1.00	47.15
WaterSIC-FT	1.50	10.90
WaterSIC	1.50	15.89
Huffman-GPTQ	1.91	22.79
WaterSIC-FT	2.00	8.06

Appendix

Table 9: Comparison of WikiText-2 perplexity results on Llama-3-8B, evaluated at CTX=2048. WaterSIC-FT is finetuned on WikiText-2 at CTX=2048. In each rate group, WaterSIC-FT achieves the best PPL at minimal rate. The unquantized (BF16) model perplexity is 6.14.

Figure 13: Llama-3-8B: WaterSIC vs. other algorithms. WaterSIC and Huffman-GPTQ report rates as entropy; the other methods use log-cardinality. Evaluated at CTX=2048.

Method	Bitwidth	W2 PPL $\downarrow$	C4 PPL $\downarrow$
BF16	16	5.54	7.10
QuIP	2.01	76.95	98.47
DB-LLM	2.01	12.77	14.82
PV-Tuning	2.01	6.99	8.29
WaterSIC (no FT)	2.00	8.01	13.48
WaterSIC (FT W2)	2.00	7.20	11.69

Appendix

Table 10: Llama-3-8B 2-bit comparison, evaluated at context length 8192. WaterSIC is calibrated on WikiText-2 train at CTX=2048 and finetuned at CTX=8192. Baseline numbers from ( Malinovskii et al., 2024 ) .

Method	Avg. Bitwidth	WikiText-2 PPL
WaterSIC-FT	1.00	12.03
WaterSIC	1.00	22.43
WaterSIC-FT	1.50	7.64
WaterSIC	1.50	8.97
Huffman-GPTQ	1.53	83.23
Huffman-GPTQ	1.95	55.84

Appendix

Table 11: Comparison of WikiText-2 perplexity results on Llama-2-7B, evaluated at CTX=2048. WaterSIC-FT is finetuned on WikiText-2 at CTX=2048. In each rate group, WaterSIC-FT achieves the best PPL at minimal rate. The unquantized (BF16) model perplexity is 5.47.

Figure 14: Llama-2-7B: WaterSIC vs. other algorithms. WaterSIC and Huffman-GPTQ report rates as entropy; the other methods use log-cardinality. Evaluated at CTX=2048.

	Calib. CTX=1024		Calib. CTX=2048
Finetuning set	W2 PPL $\downarrow$	C4 PPL $\downarrow$	W2 PPL $\downarrow$	C4 PPL $\downarrow$
None	6.152	9.853	6.123	9.813
WikiText-2	5.864	9.237	5.864	9.285
C4	5.994	8.737	5.977	8.729
RedPajama	5.986	8.832	5.976	8.818

Appendix

Table 12: Llama-2-7B at 2 bits: effect of calibration context length on WaterSIC. Calibration on WikiText-2 train; finetuning on the full WikiText-2 training split segmented at the evaluation context length 4096. Evaluated at CTX=4096. Unquantized (BF16): W2 PPL 5.12, C4 PPL 6.63.

Method	Bitwidth	W2 PPL $\downarrow$	C4 PPL $\downarrow$
BF16	16	5.12	6.63
AQLM	2.02	6.64	8.56
QuIP#	2.01	6.19	8.16
DB-LLM	2.01	7.23	9.62
YAQA-B (INT2)	2.00	7.45	9.22
PV-Tuning	2.02	5.84	7.62

Appendix

Table 13: Llama-2-7B 2-bit comparison, evaluated at context length 4096. WaterSIC is calibrated on WikiText-2 train at CTX=2048 and finetuned at CTX=4096. Baseline numbers from ( Malinovskii et al., 2024 ) and ( Tseng et al., 2025 ) .

Method	2 bits	4 bits
RTN	$4.6\times 10^{5}$	3.60
QuIP	13.0	3.40
GPTQ	11.9	3.30
AWQ	$1.7\times 10^{6}$	3.30
PB-LLM	11.6	–
OstQuant	–	3.19

Appendix

Table 14: Llama-3-70B WikiText-2 perplexity (

\downarrow

) at CTX=2048 under 2-bit and 4-bit weight-only quantization. The unquantized (BF16) model perplexity is 2.86. RTN, QuIP, GPTQ, AWQ, and PB-LLM results are from Huang et al. (2024) .

	Calibration on WikiText-2		Calibration on RedPajama
Finetuning set	W2 PPL $\downarrow$	C4 PPL $\downarrow$	W2 PPL $\downarrow$	C4 PPL $\downarrow$
None	16.19	37.73	27.43	30.33
WikiText-2	13.59	27.80	12.51	17.73
RedPajama	14.52	23.78	14.52	17.40
C4	14.56	23.53	14.76	17.37

Appendix

Table 15: Llama-3.2-1B at 2 bits: effect of the calibration and finetuning sets on WaterSIC. Calibration on WikiText-2 train or RedPajama at CTX=2048; finetuning at CTX=2048 on the indicated set. Evaluated at CTX=2048. Unquantized (BF16): W2 PPL 9.73, C4 PPL 12.77.

Figure 15: Rate-distortion curves for Llama-3.2-1B calibrated on WikiText-2, evaluated on WikiText-2 test PPL (left) and C4 test PPL (right) for three configurations: no finetuning, finetuning on WikiText-2, and finetuning on RedPajama. Evaluated at CTX=2048.

Rate	No FT	FT W2	FT RP	No FT	FT W2	FT RP
	W2 PPL $\downarrow$			C4 PPL $\downarrow$
1.00	82.45	32.85	41.55	643.29	160.70	88.26
1.50	30.73	19.16	21.80	140.72	59.97	41.84
2.00	16.19	13.59	14.52	37.73	27.80	23.78
2.50	11.83	11.34	11.62	18.81	17.80	17.16
3.00	10.57	10.45	10.54	15.26	15.06	14.91

Appendix

Table 16: Llama-3.2-1B WikiText-2 and C4 test perplexity at various rates. Calibrated on WikiText-2 train at CTX=2048; finetuning at CTX=2048. Unquantized (BF16): W2 PPL 9.73, C4 PPL 12.77.

Rate	Method	ARC-C	ARC-E	HellaSwag	OBQA	PIQA	SocialIQA	Wino
1.50	Huffman-GPTQ	0.2722	0.2597	0.2619	0.2960	0.5185	0.3306	0.5067
	WaterSIC	0.2415	0.3620	0.3127	0.2720	0.5729	0.3501	0.5107
	WaterSIC-FT	0.2577	0.3653	0.3368	0.2680	0.5778	0.3547	0.5328
2.00	Huffman-GPTQ	0.2218	0.2959	0.3030	0.2520	0.5332	0.3465	0.5075
	WaterSIC	0.2679	0.4655	0.4052	0.2920	0.6066	0.3685	0.5328
	WaterSIC-FT	0.2910	0.5143	0.4410	0.3080	0.6333	0.3823	0.5399

Appendix

Table 17: Zero-shot accuracy on Llama-3.2-1B. For each rate and task, we bold the best result among Huffman-GPTQ, WaterSIC, and WaterSIC-FT (higher is better). Quantization calibrated at CTX=2048.

Rate	Method	Wino	MMLU	HSwag	PIQA	SciQ	TQA-MC1	TQA-MC2
16.000	BF16	68.11	73.02	74.90	77.80	95.70	36.35	54.50
4.125	Huffman-GPTQ	67.17	72.28	74.84	77.42	95.60	35.01	53.36
	GPTQ	68.82	71.76	74.22	77.58	95.30	36.35	54.55
	HRTN	67.56	72.15	74.72	76.99	94.20	36.47	56.46
	RTN	67.17	69.71	74.30	75.90	94.50	36.84	55.77
	WaterSIC	70.09	72.77	74.60	76.99	95.90	35.99	53.80

Appendix

Table 18: Zero-shot accuracy on Qwen3-8B. Bold indicates the best value within each rate block for each benchmark (higher is better). Quantization calibrated at CTX=2048.

Neural Network Quantization

Neural network quantization represents weights, activations, and states with fewer bits to reduce memory use, computation, and deployment costs.

See all 404 papers

High-Rate Quantized Matrix Multiplication II
May 13, 2026Or Ordentlich, Yury PolyanskiyLLM Quantization Rate-Distortion Theory
the Hebrew University of Jerusalem, Israel · MIT, USA
GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation
May 31, 2026Shihao Zhang, Rayan SaabLLM Quantization Low-Rank Adaptation
Department of Mathematics, University of California San Diego
GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
Apr 20, 2026Alireza Dadgarnia, Soroush Tabesh, Mahdi Nikdan +4LLM Quantization Post-Training Quantization
ISTA · ETH Zürich · ISTA & Red Hat AI +1

May 13, 2026J/K move · Enter open · S save

High-Rate Quantized Matrix Multiplication II

Or Ordentlich, Yury Polyanskiy

the Hebrew University of Jerusalem, Israel · MIT, USA

LLM Quantization Rate-Distortion Theory Rate-Distortion Optimization Weight-Only Quantization Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

This is the second part of the work investigating quantized matrix multiplication (MatMul). In part I we considered the case of calibration-free quantization, whereas here we discuss the setting where covariance matrix

Σ_X

of the columns of the second factor is available. This setting arises in the ubiquitous task of weight-only post-training quantization of LLMs. Weight-only quantization is related to the problem of weighted mean squared error (WMSE) source coding, whose classical (reverse) waterfilling solution dictates how one should distribute rate between coordinates of the vector. We show how waterfilling can be used to improve practical LLM quantization algorithms (GPTQ), which at present allocate rate equally. A recent scheme (known as ``WaterSIC'') that only uses scalar INT quantizers is analyzed and its high-rate performance is shown to be (a) basis free (i.e., characterized by the determinant of

Σ_X

and, thus, unlike existing schemes, is immune to applying random rotations); and (b) within a multiplicative factor of

\frac{2πe}{12}

(or 0.25 bit/entry) of the information-theoretic distortion limit. GPTQ's performance, in turn, is affected by the choice of basis, but for a random rotation and actual

Σ_X

from Llama-3-8B we find it to be within 0.1 bit (depending on the layer type) of WaterSIC, suggesting that GPTQ with random rotation is also near optimal, at least in the high-rate regime.

Abstract

Neural Network Quantization

Explore similar work

High-Rate Quantized Matrix Multiplication II

GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation

GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling