ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weights

Jonathan Mei; Sang Hyub Kim; Oliver Knitter; Chi Chen; Martin Roetteler

cs.LGSep 29, 2026

ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weights

Authors: Jonathan Mei, Sang Hyub Kim, Oliver Knitter, Chi Chen, Martin Roetteler

Organizations: Applications R&D IonQ College Park, MD 20740, USA

Abstract

We introduce ShamAN-Q, a sub-1-bit post-training quantization method that extends NanoQuant by replacing each its diagonal reconstruction geometry with a tractable dense curvature metric, using a general paradigm popularized by the Shampoo optimizer. For each linear weight, ShamAN-Q fits a Kronecker product to the empirical Fisher information matrix of a small calibration set by Kullback--Leibler minimization, forming a Mahalanobis reconstruction loss from the result. The continuous ADMM updates from NanoQuant become solutions to Sylvester equations, while its discrete projection and deployment format remain unchanged. Because the curvature is local to a given set of weights, ShamAN-Q re-measures the input curvature statistic for each layer immediately before layer factorization, periodically refreshing all statistics on the partially quantized model. ShamAN-Q also redistributes the uniform rank from NanoQuant across layers at the same total number of bits. On Qwen3-Base, ShamAN-Q lowers WikiText-2 perplexity at $\approx$ 1 bpw from 27.56 to 22.96 (0.6B), 19.21 to 16.72 (1.7B), and 14.29 to 13.80 (4B) while matching or improving zero-shot accuracy on the Eleuther LM Evaluation Harness. On 0.6B, ShamAN-Q at $\approx$ 0.8 bpw matches the published perplexity of NanoQuant at $\approx$ 1.0 bpw.

Figures & tables

Figure 1: WikiText-2 perplexity (PPL) (left, lower is better) and zero-shot mean on reasoning tasks in Eleuther LM Evaluation Harness (right, higher is better) vs. Qwen3-Base size at a 1.0 bpw budget. ShamAN-Q matches or improves on NanoQuant in both PPL and zero-shot mean at every size. All points are from Table 1 ; only reported results are plotted.

Figure 2: ShamAN-Q end to end. Phase 1 runs the calibration set through the frozen teacher, fits and shrinks the curvature statistics

(\mathbf{A}_{\gamma},\mathbf{G}_{\gamma})

of every linear layer, and allocates ranks at bit parity. Phase 2 processes decoder blocks in order. The factorized prefix and the untouched full-precision suffix are frozen while block

b

is tuned ( TuneFP ), factorized layer by layer by Kron–LB–ADMM under the transported statistics

(\overline{\mathbf{A}},\overline{\mathbf{G}})

(Figure 3 ), and refined by latent STE. The statistics of the unfactorized layers are refreshed on the partially quantized model after every

\Delta

blocks. Phase 3 distills only the scale vectors against the teacher, with the sign matrices frozen. Flames mark trained parameters, snowflakes frozen ones, and blue blocks denote differences between ShamAN-Q and NanoQuant.

Figure 3: ShamAN-Q pipeline for one layer. Calibration fits and shrinks the curvature statistics, yielding the transported metric

(\overline{\mathbf{A}},\overline{\mathbf{G}})

and diagonal maps

(\widetilde{\mathbf{D}}_{\mathrm{in}},\widetilde{\mathbf{D}}_{\mathrm{out}})

. Preconditioning maps

\mathbf{W}

\widetilde{\mathbf{W}}

. Kron–LB–ADMM returns the continuous weight factors

(\mathcal{U},\mathcal{V})

and scale vectors

(\mathbf{s}_{1},\mathbf{s}_{2})

. The transported statistics alter only the continuous

\mathbf{U}/\mathbf{V}

optimization component; Euclidean consensus leaves SVID, dual updates, scale extraction, and sign extraction unchanged from NanoQuant.

		PPL $\downarrow$					0-shot mean $\uparrow$
bpw	Method	0.6B	1.7B	4B	8B	14B	0.6B	1.7B	4B	8B	14B
$\approx$ 1.0	NanoQuant (paper)	27.56	19.21	14.29	12.47	10.92	–	–	–	0.4894	–
	NanoQuant (repr.)	29.21	18.76	14.86	12.38	11.49	0.387	0.427	0.455	0.474	0.498
	ShamAN-Q	22.96	16.72	13.80	11.82	10.91	0.427	0.448	0.464	0.497	0.506
$\approx$ 0.8	NanoQuant (paper)	33.79	25.31	19.33	14.83	12.88	–	–	–	–	–
	NanoQuant (repr.)	42.22	22.37	17.99	15.63	–	0.387	0.414	0.431	0.443	–

Table 1: WikiText-2 perplexity and zero-shot mean across a suite of reasoning tasks for

<

1 bpw Qwen3-Base models at the same total bits as NanoQuant ( Chong et al., 2026 ) , with both reported and reproduced values. The upper block uses a 1.0 bpw budget, the middle block a 0.8 bpw budget, and the lower block a 0.55 bpw budget. Unreported results are marked with –.

Table 5

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Component	NanoQuant diagonal geometry	ShamAN-Q dense geometry
Curvature state	$\Theta(m+n)$	$\Theta(m^{2}+n^{2})$ statistics and eigensystems
Calibration fit	$\Theta(N(m+n))$	$\Theta(N(m^{2}+n^{2})+m^{3}+n^{3})$ per pass, three passes
Fresh input statistic	–	$\Theta(Nn^{2})$ per shared-input group
Refresh	–	one fit pass over unfactorized layers every $\Delta$ blocks
Rank probe	–	three $K{=}50$ ADMM solves per layer, once
Setup	$\Theta(mn)$	$\Theta(m^{3}+n^{3}+m^{2}n+mn^{2})$

Appendix

Table 5: Per-layer costs for an

m\times n

weight, factor rank

r

, and

N

calibration tokens.

Explore similar work

Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimate
Apr 15, 2026Jaemin Kim, Sungkyun Kim, Junyeol Lee +1Post-Training Quantization Hessian
Seoul National University · Hanyang University
CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs
Apr 29, 2026Zhe Ding, Su Pan, Duowei PanLarge Language Model Quantization Post-Training Quantization
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China · Amazon AGI, Seattle, WA, USA
KronQ: LLM Quantization via Kronecker-Factored Hessian
Jul 8, 2026Donghyun Lee, Yuhang Li, Ruokai Yin +1Large Language Model Quantization Post-Training Quantization
University of Southern California · Yale University

Apr 15, 2026, cs.LGJ/K move · Enter open · S save

Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimate

Jaemin Kim, Sungkyun Kim, Junyeol Lee, Jiwon Seo

Seoul National University · Hanyang University

Post-Training Quantization Hessian Local Curvature

Open paper AskPDF arXiv

Large Language Models (LLMs) are widely used across many domains, but their scale makes deployment challenging. Post-Training Quantization (PTQ) reduces memory footprint without retraining by leveraging a small calibration set. Recent Hessian-based PTQ methods compensate quantization error via cross-channel dependencies, but such approaches degrade at low bit-widths due to noisy curvature estimates from limited calibration data. We propose DASH-Q, a robust PTQ framework using diagonal Hessian approximation and iterative weighted least squares. By discarding noise-prone dependencies, DASH-Q filters sampling noise while prioritizing the preservation of salient feature power. We outperform other PTQ baselines in ultra low-bit regime, improving zero-shot accuracy by 7.01% on average and up to 14.01% over the strongest baselines across five baseline LLM models, while showing robust and stable performance with very small calibration data.

Abstract

Explore similar work

Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimate

CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs

KronQ: LLM Quantization via Kronecker-Factored Hessian