ResCom: Activation Residual Compensation for Large Language Model Quantization

Ning Yang; Bohan Cui; Zhuowen Liu; Yuting Wang

cs.LGOct 25, 2026

ResCom: Activation Residual Compensation for Large Language Model Quantization

Authors: Ning Yang, Bohan Cui, Zhuowen Liu, Yuting Wang

Organizations: Institute of Automation, Chinese Academy of Sciences · Fudan University · The Chinese University of Hong Kong · University of Chinese Academy of Sciences

Abstract

Low-bit weight--activation quantization reduces the memory and computation needed to deploy large language models, but activation rounding can still degrade accuracy after calibration. Reconstruction from rounded inputs cannot distinguish activations within the same quantization bin. We propose ResCom, a low-rank correction that uses the activation rounding residual to compensate for the resulting layer-output error. An output-aware calibration procedure selects residual directions by their contribution to output reconstruction and computes the correction in closed form, without gradient training or updates to the calibrated weights. On Llama-3-8B with 4-bit weights and activations, rank-32 attention corrections reduce WikiText-2 perplexity from 7.708 to 7.147, improve MMLU by 2.89 percentage points, and raise five-shot GSM8K accuracy from 23.96 to 36.16 percent. Evaluations on models from 1B to 70B parameters and on held-out C4 test the method beyond this setting. The evaluated FP32 implementation adds 2.25 percent to 4K prefill time; a separate compact-storage implementation retains a 27.7 percent peak-memory reduction relative to FP16.

Figures & tables

Figure 1: Three reconstruction paths. (a) SERQ applies a selected-channel compensator to rounded activations. (b) ARHQ splits weights before quantization and uses unrounded activations. (c) ResCom instead feeds the activation rounding residual to an output-aware low-rank branch fitted after calibration. Appendices B – C formalize these differences.

Figure 2: ResCom calibration and inference. Calibration pairs each activation rounding residual with the output contribution removed by rounding and fits a low-rank correction. At inference, the branch evaluates that correction for the current token and adds it to the frozen quantized output. The shared layer bias is omitted.

		#Eff.	Train.	MAC	LLaMA-2 7B			LLaMA-2 70B			LLaMA-3 8B
#Bits	Method	bits/wt.	free	overhead	PPL $\downarrow$	0-shot $\uparrow$	MMLU $\uparrow$	PPL $\downarrow$	0-shot $\uparrow$	MMLU $\uparrow$	PPL $\downarrow$	0-shot $\uparrow$	MMLU $\uparrow$
FP16	Full precision †	16	✓	–	5.47	64.09	41.83	3.53	70.54	65.44	6.13	67.16	62.13
W4A4	SmoothQuant †	${\sim}4$	✓	–	1.51e4	35.44	26.04	–	–	–	4.71e5	36.34	25.17
	SmoothQuant, g128 †	4.13	✓	–	7.49	57.15	30.40	–	–	–	17.26	48.97	29.30
	QuaRot †	${\sim}4$	✓	19.8%	6.15	59.53	33.58	–	–	–	8.41	59.12	47.29
	SpinQuant †	${\sim}4$	$\times$	19.8%	6.00	61.00	34.80	–	–	–	8.26	61.75	49.93

Table 1: W4A4 WikiText-2 PPL and task accuracy.

\dagger

: published by SERQ ( Park et al., 2026 ) ; other rows are our evaluations.

\ddagger

: nominal factor cost. Zero-shot averages eight tasks; bold compares paired rows.

Model	Method	WT2	C4	0-shot	MMLU
Mistral-7B v0.3	FP16	5.522	8.480	68.77	59.66
	W4A4	5.77e3	3.69e3	35.32	24.43
	SERQ	5.663	8.244	67.10	56.12
	ResCom	5.617	8.188	67.13	56.73
Gemma-3-1B PT	BF16	14.088	19.980	56.69	24.81
	W4A4	1.68e4	1.36e4	35.50	23.99

Table 2: Cross-family controls and transfer. (a) Unquantized precision is FP16 for Mistral and BF16 for Gemma; W4A4 is direct rounding without an added algorithm. Accuracy is in percent; zero-shot averages eight tasks and MMLU covers 57 subjects. (b) C4 is unseen during WikiText-2 fitting;

\rho_{32}

is the mean layerwise fit reduction, as defined in Appendix A, Eq. ( 18 ). Bold compares each SERQ/ResCom pair.

Figure 3: Perplexity comparisons. (a) Reproduced Llama-3-8B baselines; L 2 QER uses MXINT and QuaRot/SERQ use INT4 simulation. (b) Paired W4A4 reductions across backbones. Table 1 identifies published values.

Backbone	Host	ResCom	Backbone	Host	ResCom
Llama-3-8B	11.192	10.349	Llama-3.2-3B	12.688	12.317
LLaMA-2 7B	7.432	7.315	Llama-3.2-1B	17.227	16.712
LLaMA-2 13B	6.693	6.634

Table 3: Transfer tests. (a) C4 PPL after WikiText-2 calibration, without refitting. (b) WikiText-2 PPL on Llama-3-8B under three host pipelines; only the correction changes within each pair.

			GSM8K 5-shot $\uparrow$		LongBench $\uparrow$
Model	Bits	Method	Flexible	Strict	QMSum	SAMSum	RepoBench-P
Llama-3-8B	FP16	Full precision †	48.07	47.69	23.43	46.26	66.56
	W4A4	L 2 QER †	7.96	7.66	0.09	1.98	5.65
	W4A4	SERQ	24.56	23.96	21.12	42.31	61.03
		ResCom	36.54	36.16	22.96	44.40	65.04
Llama-3.2-3B	FP16	Full precision †	26.08	25.93	23.94	42.98	64.42

Table 4: W4A4 reasoning and long-context generation.

\dagger

: published SERQ values ( Park et al., 2026 ) ; other SERQ rows are our paired evaluations. GSM8K uses five shots; LongBench uses full sets with a 7500-token budget.

Table 8

Figure 4: Repair sites. Orange marks default attention branches; gray marks added feed-forward branches. Residuals

d_{1}

–

d_{4}

follow the quantizers;

Q,K,V,O

and

u,g,d

label the projections.

Backbone	Draws	$\Delta$ PPL
Llama-3-8B	4	$0.5604\pm 0.0059$
LLaMA-2 7B	3	$0.0916\pm 0.0030$
Mistral-7B-v0.3	4	$0.0431\pm 0.0024$

Table 7: Calibration stability and repair scope. (a) Mean

\Delta\mathrm{PPL}=\mathrm{PPL}_{\mathrm{host}}-\mathrm{PPL}_{\mathrm{repaired}}

with sample s.d. across fixed-evaluation calibration draws. (b) Llama-3-8B choices; the channel control matches the default branch’s parameter and factor-MAC budgets.

Panel of Figure 5 — Figure 5: Efficiency and accuracy of Llama-3-8B on B200. (a) 4K prefill latency and WikiText-2 perplexity (top), and cached decoding time per token (bottom), at batch size one. Quantized configurations use FP32 simulation; the FP16 reference uses FP16 computation. (b) MMLU (top) and eight-task zero-shot accuracy (bottom) versus peak allocated memory in a separate compact-storage experiment with 4096 input and 32 generated tokens. Compact quantized weights are dequantized layer by layer for FP32 GEMM.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Activation width	Pipeline	ResCom	change
4 bits	7.708	7.147	$-7.28\%$
6 bits	6.552	6.545	$-0.10\%$
8 bits	6.503	6.502	$-0.01\%$
16 bits	6.499	–	$0$

Appendix

Table 8: Activation-width sweep on Llama-3-8B at fixed quantized weights. The sixteen-bit row is the same quantized host with activation quantization disabled, not the original full-precision model.

Backbone	SERQ	ResCom	change	$\rho_{32}$	SERQ	change	$\rho_{32}$
	Four-bit activations				Eight-bit activations
Llama-3-8B	7.708	7.147	$-7.28$ %	0.294	6.503	$-0.01$ %	0.289
LLaMA-2 7B	5.939	5.847	$-1.55$ %	0.271	5.594	$-0.01$ %	0.265
LLaMA-2 13B	5.227	5.148	$-1.51$ %	0.270	4.977	$-0.01$ %	0.261
LLaMA-3.2 3B	9.177	8.964	$-2.32$ %	0.282	8.185	$-0.02$ %	0.282
LLaMA-3.2 1B	12.461	12.071	$-3.13$ %	0.355	10.411	$-0.02$ %	0.356

Appendix

Table 9: Activation-width sensitivity with SERQ. We retain the weights, repair rank, and attachment sites while changing the activation width. Each relative change is measured against SERQ at the same activation precision.

Pipeline	Repaired layers	$r=0$	$r=16$	$r=32$	$r=64$
SERQ	all linear layers	–	–	7.070	–
SERQ	attention projections	7.715	7.167	7.147	7.128
SERQ	down projection	–	–	7.624	–
SERQ	output projection	–	–	7.672	–
SERQ	gate and up projections	–	–	7.688	–
PrefixQuant	output projection	7.790	–	7.646	7.605

Appendix

Table 10: Attachment-site and rank ablations on Llama-3-8B. Rank zero means a fitted bias only. Dashes denote unmeasured configurations. Site groups differ in parameter count, so they are not equal-budget comparisons.

Backbone	Control	Repaired, mean	$\Delta$ mean	$\Delta$ s.d.	Helps on
Llama-3-8B	7.708	7.147	$+0.5604$	0.0059	4/4
LLaMA-2 7B	5.939	5.847	$+0.0916$	0.0030	3/3
Mistral-7B-v0.3	5.663	5.620	$+0.0431$	0.0024	4/4

Appendix

Table 11: Calibration-draw variability with test windows fixed. Positive

\Delta

is host minus repaired perplexity. Standard deviations describe three or four draws as indicated by the final column; they are not test-set confidence intervals.

2048-token windows	WikiText-2	C4	Mean $\rho_{32}$
32	7.1387	10.3471	0.2934
128	7.1465	10.3493	0.2942
512	7.1374	10.3522	0.2938

Appendix

Table 12: Calibration sample-count sensitivity on Llama-3-8B. Rank 32, attention projections, seed zero; each control reproduces 7.7077.

Backbone	Method	ARC-c	ARC-e	BoolQ	HellaS.	OBQA	PIQA	SIQA	WinoG.	Avg.
LLaMA-2 7B	Full precision †	46.25	74.49	77.74	76.00	44.20	79.05	46.11	68.90	64.09
	SmoothQuant †	27.82	25.97	44.16	26.39	27.20	49.95	34.49	47.51	35.44
	SmoothQuant, g128 †	39.85	65.40	68.59	67.99	37.80	74.48	41.45	61.64	57.15
	QuaRot †	43.00	71.59	72.84	72.30	41.60	77.20	33.06	64.64	59.53
	SpinQuant †	43.60	71.30	73.80	73.20	40.40	76.00	44.10	65.40	61.00
	SERQ	43.26	71.97	75.47	74.05	44.00	77.91	45.04	68.43	62.52

Appendix

Table 13: Zero-shot task details for the principal W4A4 comparisons in Table 1 .

\dagger

marks values published by SERQ ( Park et al., 2026 ) ; the unmarked SERQ and ResCom rows are our paired checkpoint evaluations under the common harness. Averages are formed before rounding.

Backbone	Method	ARC-c	ARC-e	BoolQ	HellaS.	OBQA	PIQA	SIQA	WinoG.	Avg.
Llama-3.2-1B	SERQ	34.30	53.32	60.06	58.35	34.60	69.53	41.25	56.04	50.93
	ResCom	33.11	55.22	59.45	58.84	35.20	69.59	40.84	55.88	51.02
Llama-3.2-3B	SERQ	40.10	67.13	70.70	69.29	37.40	75.63	45.45	66.14	58.98
	ResCom	42.32	67.42	71.22	70.78	40.20	76.77	45.91	66.46	60.14
Mistral-7B-v0.3	SERQ	51.71	78.96	80.92	78.39	46.60	80.85	46.06	73.32	67.10
	ResCom	52.65	78.96	80.83	79.36	45.40	81.12	46.32	72.45	67.13

Appendix

Table 14: Paired W4A4 zero-shot task details for the additional backbones in Tables 1 and 2 . Both rows in each pair are our evaluations of the same checkpoint before and after repair.

Model	Bits	Method	WikiText-2	C4	0-shot	MMLU
Gemma-3-1B-PT	W4A8	SERQ	15.026	19.104	55.81	25.13
		ResCom	15.058	19.138	55.55	25.12
Gemma-3-4B-PT	W4A8	SERQ	10.753	14.901	65.20	55.73
		ResCom	11.128	15.326	65.34	55.52

Appendix

Table 15: Additional-family W4A8 results. Each ResCom row uses the paired SERQ checkpoint and rank-32 attention repair. All task pairs are complete; bold compares each paired result.

			GSM8K 5-shot $\uparrow$		LongBench $\uparrow$
Model	Bits	Method	Flexible	Strict	QMSum	SAMSum	RepoBench-P
Llama-3-8B	W4A8	LLM.int4() †	7.88	4.62	13.29	34.20	44.93
		L 2 QER †	36.24	35.33	7.28	25.81	47.03
	W4A8	SERQ	43.52	43.14	23.28	44.20	65.98
		ResCom	43.37	42.91	23.53	44.45	65.90
Llama-3.2-3B	W4A8	LLM.int4() †	6.76	6.44	15.83	36.42	35.24

Appendix

Table 16: W4A8 generation results.

\dagger

: published comparator values reported by SERQ ( Park et al., 2026 ) ; SERQ rows are our paired checkpoint evaluations. Metrics and prompt settings match the W4A4 study in Table 4 .

Method	Prefill	Time per output token
FP16	209.07	15.89
L 2 QER-A8	2600.15	414.33
L 2 QER-A4	2601.00	416.03
QuaRot	1470.75	78.70
SERQ	1342.32	76.79
ResCom	1372.46	86.11

Appendix

Table 17: Numerical breakdown for Figure 5 . B200, Llama-3-8B, batch one, 4K prompt; milliseconds.

Figure 6: Full memory–accuracy views for MMLU and the eight-task mean, using the measurements summarized in Figure 5 . Each point is labeled with its measured end-to-end latency.

Figure 7: Per-module latency for the layer-0 query, up, and down projections of the six evaluated Llama-3-8B configurations on B200, batch one with 4096 input tokens. The measurements use each checkpoint’s actual inputs and weights and complement the end-to-end results in Figure 5 .

Figure 8: Paired SERQ and ResCom latency across prompt lengths on B200, batch one, with the corresponding WikiText-2 perplexity. Means and sample standard deviations use three interleaved groups. The shaded difference is the measured increment from the fitted rank-32 attention branch.

Projection	Tokens	Eager	Fused	Ratio
q_proj	1	163.8	48.2	3.40
q_proj	16	167.9	52.1	3.22
q_proj	128	167.3	52.3	3.20
q_proj	2048	288.2	164.6	1.75
k_proj	1	160.2	46.8	3.42
k_proj	16	161.6	47.2	3.42

Appendix

Table 18: Fused residual frontend with saved rank-32 factors on B200. Inputs are captured Llama-3-8B layer-0 activations; latency is in microseconds and averaged over three interleaved groups. The fused path also emits packed INT4 codes; main-path GEMM is outside this measurement.

Figure 9: Packed kernel-component measurements for the selected-column path and the two-factor path. The former performs one thin product and the latter performs two. Shapes are

(K,N)

, with input width

K

and output width

N

; BF16 and MXFP4 are the executed dtypes.

Prompt	Path	PPL $\downarrow$	Time (s) $\downarrow$	Peak (GiB) $\downarrow$
128	BF16	6.138	0.531	15.013
	MXFP4	7.865	6.826	8.172
	MXFP4 ResCom	7.733	7.017	8.275
2048	BF16	6.138	0.592	16.568
	MXFP4	7.865	6.854	9.588
	MXFP4 ResCom	7.733	7.186	9.690

Appendix

Table 19: Llama-3-8B native MXFP4 execution on B200, batch one and 32 generated tokens. PPL is full WikiText-2 and shared across prompt lengths; time is the mean end-to-end seconds of three repetitions.

Model	Llama-3.2 1B	Llama-3.2 3B	LLaMA-2 7B	LLaMA-2 13B
Fit time (s)	58	185	544	1353

Appendix

Table 20: One-time rank-32 attention-repair calibration on B200. Times include moment collection, fitting, and diagnostics; original PTQ and task evaluation are excluded.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 388 papers

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
Aug 4, 2026Yu Luo, Bo Dong, Wenhua Cheng +1Mixed-Precision Quantization LLM Quantization
Intel
RDQ: Residual Distribution Quantization for Large Language Models
Jul 11, 2026Prateek SinghLLM Quantization Post-Training Quantization
ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs
May 11, 2026Chayne Thrash, Ali Abbasi, Soheil KolouriRotation-Based Quantization LLM Inference Acceleration
Department of Computer Science Vanderbilt University

Aug 4, 2026J/K move · Enter open · S save

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

Yu Luo, Bo Dong, Wenhua Cheng, Haihao Shen

Intel

Mixed-Precision Quantization LLM Quantization Post-Training Quantization Low-Bit Quantization

Open paper AskPDF arXiv

Serving large language models (LLMs) under diverse deployment constraints requires flexible trade-offs between accuracy, memory footprint, and throughput. However, conventional quantization methods typically require a separate checkpoint for each target bit-width. We introduce Recurrent Residual Quantization (RRQ), a post-training quantization (PTQ) framework that represents weights as a low-bit quantized base together with a sequence of quantized residual corrections, enabling multiple effective precisions from a single checkpoint. Starting from a 2-bit model obtained via post-training quantization (PTQ) or round-to-nearest (RTN), RRQ progressively adds lightweight 2-bit residuals generated via RTN to construct 4-, 6-, and 8-bit representations. The method is calibration-free and avoids joint multi-bit optimization. In our Qwen3-8B setup, the full all-RTN 2-/4-/6-/8-bit package is constructed in 1,293 seconds, 3.3 times faster than the measured MatGPTQ construction. Experiments on six recent LLMs show competitive accuracy at 6 and 8 bits, with model-dependent behavior at 4 bits. The code will be made publicly available upon publication.

Abstract

LLM Quantization

Explore similar work

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

RDQ: Residual Distribution Quantization for Large Language Models

ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs