SandwichQuant: Which Parameters Matter Before and After Quantization?

Peng Xia; Junbiao Pang

cs.CVAug 25, 2026

SandwichQuant: Which Parameters Matter Before and After Quantization?

Authors: Peng Xia, Junbiao Pang

Organizations: School of Information Science and Technology, Beijing University of Technology Beijing, China

Abstract

Quantization correction methods usually optimize weights, quantization parameters, or reconstruction objectives, while the underlying parameter subspaces responsible for effective correction remain unclear. In this work, we study quantization correction from a parameter subspace perspective and reveal that correction capability is highly non-uniform across parameter groups. By decomposing trainable parameters into backbone weights, normalization-affine parameters, and quantization parameters, we show that the low-dimensional normalization-affine subspace provides a highly efficient correction direction under matched budgets. Based on this finding, we propose SandwichQuant, a two-stage normalization-affine correction framework that performs adaptation before and after quantization. The pre-stage improves quantization robustness, while the post-stage compensates residual errors after the quantized graph is fixed. Extensive experiments on vision models and large language models demonstrate consistent improvements under various low-bit quantization settings, validating the effectiveness of subspace-aligned correction.

Figures & tables

Panel of Figure 1 — Figure 1: A low-dimensional affine subspace provides high-leverage correction directions before and after full-space optimization. The post-QAT result diagnoses useful residual directions under a fixed protocol; it does not imply greater asymptotic capacity than full-space training.

Figure 2: Matched-budget parameter-subspace comparison on Qwen3-8B under W2A4KV4. All trainable alternatives use exactly the same number of scalar parameters as the normalization-affine subspace

\Phi

and share identical calibration data, optimization steps, and training objectives. (a) Relative WikiText-2 perplexity, C4 perplexity, and Avg6 accuracy, normalized to the frozen quantized baseline (higher is better). (b) Accuracy–perplexity trade-off, where the horizontal axis reports the mean relative reduction in WikiText-2 and C4 perplexity against the frozen baseline. Despite its extremely low dimensionality,

\Phi

achieves the strongest overall recovery and slightly outperforms the gradient-selected TopGradW subspace.

Panel of Figure 3 — Figure 3: The post-PTQ affine component reshapes the functional geometry of quantization error across two regimes. Each subfigure reports the four-state intervention, task-aware recovery metrics, and structured-source analysis for Llama3-8B and Qwen3-8B. These one-sided probes isolate SandwichQuant-Post , the residual-correction component of SandwichQuant . Across weight-only W3A16 and joint W2A4KV4, it reduces task-relevant degradation more consistently than isolated local operator error, indicating that its principal effect emerges through propagation and interaction rather than uniform local reconstruction.

Model	Method	W/A	Val. Acc. (%)	Drop (%)
	Full precision	32/32	71.30	–
	PACT	4/4	64.06	-7.24
	DSQ	4/4	67.36	-3.94
	LSQ	4/4	69.01	-2.29
	LSQ+	4/4	68.21	-3.09
	SandwichQuant-Post + LSQ+	4/4	69.83	-1.47

Table 1: One-sided post-QAT affine adaptation on CIFAR-100 with MobileNetV2. Highlighted rows apply only SandwichQuant-Post after the named QAT checkpoint; they are component controls rather than the complete two-stage SandwichQuant pipeline. Drop denotes the signed accuracy difference from the 71.30% FP32 baseline.

Table 5

	Llama3-8B						Qwen3-8B
Order	W3A16			W2A4KV4			W3A16			W2A4KV4
	Wiki2 $\downarrow$	C4 $\downarrow$	Avg6 $\uparrow$	Wiki2 $\downarrow$	C4 $\downarrow$	Avg6 $\uparrow$	Wiki2 $\downarrow$	C4 $\downarrow$	Avg6 $\uparrow$	Wiki2 $\downarrow$	C4 $\downarrow$	Avg6 $\uparrow$
$\mathrm{PTQ}$	8.33	13.10	68.83	18.07	50.56	46.81	11.29	17.09	71.87	18.71	38.27	49.62
$\mathrm{PTQ}\!\rightarrow\!\Phi_{\mathrm{post}}$	7.89	12.21	70.08	12.83	30.35	52.49	10.60	16.38	72.17	12.73	26.80	56.47
$\Phi_{\mathrm{pre}}\!\rightarrow\!\mathrm{PTQ}$	8.28	12.87	68.89	13.93	37.96	48.72	10.74	16.42	72.31	12.99	27.99	55.28
SandwichQuant	7.87	12.13	70.05	12.31	28.86	53.44	10.51	16.15	72.34	12.03	25.37	58.05

Table 4: Order ablation of normalization-affine adaptation.

\Phi_{\mathrm{pre}}

preconditions PTQ, whereas

\Phi_{\mathrm{post}}

corrects the frozen quantized graph. Wiki2/C4 are perplexities (

\downarrow

) and Avg6 is mean zero-shot accuracy (

\uparrow

). Each affine stage uses the same per-stage budget; SandwichQuant contains two affine stages and rebuilds the PTQ graph from the original dense checkpoint between them.

Model	Method	Perplexity $\downarrow$		Zero-shot Accuracy (%) $\uparrow$						Avg. $\uparrow$
Model	Method	Wiki2	C4	PIQA	ARC-E	ARC-C	HS	WG	BoolQ	Avg. $\uparrow$
	FP16	5.47	7.27	78.9	74.6	46.1	75.9	69.2	77.7	70.4
	RTN	8.22	11.53	76.3	66.5	40.6	68.7	65.8	67.5	64.2
	SandwichQuant (RTN)	7.40	10.29	77.4	68.9	41.9	71.0	66.2	70.9	66.1
	AWQ	6.79	8.93	77.1	69.9	41.8	71.2	67.7	71.5	66.3
	SandwichQuant (AWQ)	6.47	8.55	77.5	70.8	43.1	72.4	67.3	72.9	67.3

Table 5: Weight-only W3A16 results (group size 128). Highlighted rows apply the complete SandwichQuant pipeline to the backend named in parentheses. Avg. is the unweighted mean over the six zero-shot tasks.

\mathrm{ResComp}^{\ast}

uses GPTAQ as its base quantizer. All variants use the same 128 C4 sequences (length 2,048; seed 0); full settings are in Appendix A .

Model	Method	Perplexity $\downarrow$		Zero-shot Accuracy (%) $\uparrow$						Avg. $\uparrow$
Model	Method	Wiki2	C4	PIQA	ARC-E	ARC-C	HS	WG	BoolQ	Avg. $\uparrow$
	FP16	5.47	7.27	78.9	74.6	46.1	75.9	69.2	77.7	70.4
	QuaRot + GPTAQ	11.7	24.8	62.3	45.6	25.6	41.2	54.0	62.4	48.5
	SandwichQuant (QuaRot+GPTAQ)	8.2	14.0	66.8	50.0	31.6	54.9	58.2	64.5	54.3
	QuaRot + $\mathrm{ResComp}^{\ast}$	11.5	23.6	63.6	46.4	24.9	40.8	56.0	61.9	48.9
Llama2-7B	SandwichQuant (QuaRot+ $\mathrm{ResComp}^{\ast}$ )	8.1	14.2	68.1	52.2	30.7	54.8	59.1	65.8	55.1

Table 6: Joint W2A4KV4 results (group size 128). Highlighted rows apply the complete SandwichQuant pipeline to the indicated QuaRot backend. Avg. is the unweighted mean over six zero-shot tasks.

\mathrm{ResComp}^{\ast}

uses GPTAQ as its base quantizer. All variants use the same 128 WikiText-2 sequences (length 2,048; seed 0); full settings are in Appendix A .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Quantization setting	Device	$\|\Phi\|$	Wall-clock time (min)				Total	Peak mem. (GiB)
				Probe PTQ	$\boldsymbol{\Phi}_{\rm pre}$	Deploy PTQ	$\boldsymbol{\Phi}_{\rm post}$	(min)
Llama3-8B	GPTQ W3A16, g128	C600-A	266,240	67.2	1.3	67.2	1.3	137.0	16.1
Llama3-8B	QuaRot+ResComp W2A4KV4, g128	C600-A	266,240	89.0	1.4	89.0	1.4	180.8	25.0
Qwen3-8B	GPTQ W3A16, g128	C500	299,008	55.0	1.6	55.0	1.6	113.2	17.0
Qwen3-8B	QuaRot+ResComp W2A4KV4, g128	C500	299,008	85.0	1.6	85.0	1.6	173.2	25.0

Appendix

Table 7: Representative offline construction cost of SandwichQuant . Wall-clock time is decomposed into the disposable PTQ probe, pre-PTQ affine adaptation, the deployable PTQ rebuild, and post-PTQ affine adaptation. Times are reconstructed from representative single-accelerator C500/C600-A runs and include graph construction and checkpoint I/O but exclude downstream evaluation. The two affine stages update only

|\Phi|

native normalization parameters. The final graph introduces no additional parameters or inference operators relative to its base PTQ graph.

Table 10

Model	Method	Perplexity $\downarrow$		Zero-shot Accuracy (%) $\uparrow$						Avg. $\uparrow$
Model	Method	Wiki2	C4	PIQA	ARC-E	ARC-C	HS	WG	BoolQ	Avg. $\uparrow$
	FP16	5.47	7.27	78.9	74.6	46.1	75.9	69.2	77.7	70.4
	SpinQuant + GPTAQ	11.6	26.5	62.2	42.3	25.5	40.9	54.7	63.4	48.1
	SandwichQuant (SpinQuant+GPTAQ)	8.6	15.2	65.6	52.2	30.1	53.5	57.6	63.1	53.7
	SpinQuant + $\mathrm{ResComp}^{\ast}$	11.1	24.7	62.7	45.4	27.6	42.3	55.0	62.3	49.2
Llama2-7B	SandwichQuant (SpinQuant+ $\mathrm{ResComp}^{\ast}$ )	8.5	15.3	67.1	50.9	27.9	54.0	54.7	63.2	53.0

Appendix

Table 10: SpinQuant-based SandwichQuant results under W2A4KV4. Highlighted rows apply the complete two-stage pipeline with the model-specific learned rotation used by the corresponding SpinQuant baseline. Artifact and rotation identifiers are stored in the run manifests.

Model	Method	Perplexity $\downarrow$		Zero-shot Accuracy (%) $\uparrow$						Avg. $\uparrow$
Model	Method	Wiki2	C4	PIQA	ARC-E	ARC-C	HS	WG	BoolQ	Avg. $\uparrow$
	FP16	5.47	7.27	78.9	74.6	46.1	75.9	69.2	77.7	70.4
	RTN	8.22	11.53	76.3	66.5	40.6	68.7	65.8	67.5	64.2
	RTN + SandwichQuant-Post	7.47	10.42	77.3	68.9	41.7	70.9	66.0	70.8	65.9
	AWQ	6.79	8.93	77.1	69.9	41.8	71.2	67.7	71.5	66.3
	AWQ + SandwichQuant-Post	6.53	8.71	77.2	70.7	43.0	72.4	67.2	72.8	67.2

Appendix

Table 11: One-sided SandwichQuant-Post results under W3A16 (group size 128). Each highlighted row freezes the completed backend in the preceding row and optimizes only

\Phi_{\mathrm{post}}

. These component results are intentionally separated from the full SandwichQuant rows in Table 5 .

Model	Method	Perplexity $\downarrow$		Zero-shot Accuracy (%) $\uparrow$						Avg. $\uparrow$
Model	Method	Wiki2	C4	PIQA	ARC-E	ARC-C	HS	WG	BoolQ	Avg. $\uparrow$
	FP16	5.47	7.27	78.9	74.6	46.1	75.9	69.2	77.7	70.4
	QuaRot + GPTAQ	11.7	24.8	62.3	45.6	25.6	41.2	54.0	62.4	48.5
	QuaRot + GPTAQ + SandwichQuant-Post	8.4	14.6	66.2	49.5	30.5	54.7	57.5	63.3	53.6
	QuaRot + $\mathrm{ResComp}^{\ast}$	11.5	23.6	63.6	46.4	24.9	40.8	56.0	61.9	48.9
Llama2-7B	QuaRot + $\mathrm{ResComp}^{\ast}$ + SandwichQuant-Post	8.3	14.7	66.6	50.2	29.1	54.3	58.2	64.9	53.9

Appendix

Table 12: One-sided SandwichQuant-Post results under joint W2A4KV4 (group size 128). Each highlighted row freezes the completed QuaRot backend and optimizes only

\Phi_{\mathrm{post}}

. Calibration uses 128 WikiText-2 sequences of length 2,048 (seed 0).

Neural Network Quantization

Neural network quantization represents weights, activations, and states with fewer bits to reduce memory use, computation, and deployment costs.

See all 394 papers

Low-Dimensional High-Leverage Subspace Optimization: Beyond Full-Parameter Coupled Training for Neural Network Quantization
Aug 4, 2026Peng Xia, Junbiao Pang, Zheng HuangQuantization-Aware Training Post-Training Quantization
School of Information Science and Technology, Beijing University of Technology, Beijing, China
Optimal Post-Training Quantization Scales and Where to Find Them
Jun 9, 2026Juan Amboage, Pablo Monteagudo-Lago, Ian Colbert +2LLM Quantization Weight-Only Quantization
AMD
Align, Then Correct: Training-Free Two-Stage Low-Rank Compensation for Extremely Quantized Large Language Models
Oct 6, 2026Seobin Song, Geonho Lee, Janghwan Lee +1LLM Quantization Post-Training Quantization
Hanyang University, Seoul, Republic of Korea · Qualcomm AI Research, Qualcomm Korea YH, Seoul, Republic of Korea

Aug 4, 2026J/K move · Enter open · S save

Low-Dimensional High-Leverage Subspace Optimization: Beyond Full-Parameter Coupled Training for Neural Network Quantization

Peng Xia, Junbiao Pang, Zheng Huang

School of Information Science and Technology, Beijing University of Technology, Beijing, China

Quantization-Aware Training Post-Training Quantization Low-Bit Quantization

Open paper AskPDF arXiv

Low-bit quantization suffers severe accuracy degradation on compact networks, rooted in the dominant full-parameter coupled training paradigm that ignores parameter subspace heterogeneity. Their limited feature redundancy leaves little room to absorb quantization errors. Conventional pipelines adopt monolithic optimization: PTQ reconstructs fixed pretrained models without improving inherent quantization friendliness; QAT updates all parameters jointly, suffering from gradient coupling between backbone weights and calibration parameters. In this paper, we identify normalization affine parameters as a low-dimensional high-leverage subspace dominating quantization robustness, and propose Normalization Affine Preconditioning (NAP) for targeted subspace optimization. For PTQ, NAP freezes backbone weights and fine-tunes only affine parameters under the target fake-quantization graph on full-precision models, proactively boosting quantization friendliness before downstream reconstruction. For QAT, we introduce an alternating QAT-NAP schema that decouples feature learning and numerical calibration, breaking the performance ceiling of saturated joint training. Theoretical analysis confirms BN affine parameters fully cancel the channel-wise affine component of quantization distortion, while nonlinear rounding and clipping residuals form the irreducible error boundary; distillation-guided NAP acts as directional flatness optimization, projecting teacher-student logit mismatch onto the restricted subspace. Experiments on ImageNet and CIFAR-100 show NAP recovers severely collapsed low-bit quantization, consistently boosts reconstruction-based PTQ, and outperforms saturated full-parameter QAT with negligible tuning cost. This work reveals the principle of targeted low-dimensional subspace optimization, offering a new perspective beyond full-parameter coupled training for efficient deep learning.

Abstract

Neural Network Quantization

Explore similar work

Low-Dimensional High-Leverage Subspace Optimization: Beyond Full-Parameter Coupled Training for Neural Network Quantization

Optimal Post-Training Quantization Scales and Where to Find Them

Align, Then Correct: Training-Free Two-Stage Low-Rank Compensation for Extremely Quantized Large Language Models