Preserving Updates in Low-Bit Recurrent State Quantization

Haotian Li; Liran Dong; Kaicheng Xiao; Guoliang Xing

cs.LGOct 11, 2026

Preserving Updates in Low-Bit Recurrent State Quantization

Authors: Haotian Li, Liran Dong, Kaicheng Xiao, Guoliang Xing

Organizations: The Chinese University of Hong Kong

Abstract

Linear attention and recurrent models compress past tokens into a fixed-size state that is updated at every step. However, storing these states in high precision incurs substantial memory and bandwidth costs at large batch sizes, motivating low-bit state quantization. Unlike weights and key--value caches, a recurrent state is requantized after every update, so its quantization errors accumulate across steps. Our analysis shows that these errors are highly coherent across steps, so reducing per-step errors alone is insufficient. This coherence arises mainly from small updates that are lost in rounding. To preserve these updates, we intervene both before and at rounding. Before rounding, we propose Low-Rank Bypass (LRB), which accumulates small updates in a compact low-rank buffer and periodically merges its sufficiently large elements into the state. At rounding, we observe that a state element can even move opposite to its update, and such reversals prove particularly harmful. Therefore, we propose Bounded Directional Rounding (BDR), which rounds reversed elements toward their updates without additional storage. Although BDR enlarges the rounding error, it improves the model, suggesting that the direction of an update matters more than its rounding error at a single step. Across Gated DeltaNet, GLA, Mamba2, and RetNet, our training-free method substantially outperforms INT4, while reducing state memory by up to 5.3 $\times$ and speeding up decoding by up to 1.9 $\times$ relative to FP32 states.

Figures & tables

Model	Decay $\mathbf{A}_{t}$	Update $\mathbf{B}_{t}$
RetNet	$\gamma\mathbf{I}$	$\mathbf{k}_{t}\mathbf{v}_{t}^{\top}$
Mamba2	$\alpha_{t}\mathbf{I}$	$\mathbf{k}_{t}\mathbf{v}_{t}^{\top}$
GLA	$\operatorname{Diag}(\boldsymbol{\alpha}_{t})$	$\mathbf{k}_{t}\mathbf{v}_{t}^{\top}$
GDN	$\alpha_{t}\mathbf{I}$	$\beta_{t}\mathbf{k}_{t}\bigl(\mathbf{v}_{t}-(\mathbf{A}_{t}\mathbf{S}_{t-1})^{\top}\mathbf{k}_{t}\bigr)^{\top}$

Table 1: Decay and update terms of the state recurrence.

Figure 1: Accumulation of per-step errors under 4-bit state quantization. Top: inner products $\langle\mathbf{R}_{\tau},\mathbf{R}_{\tau^{\prime}}\rangle_{F}$ (left), summed over 64-step blocks and normalized by the mean diagonal entry for each layer and head before averaging, and decay weights $a_{\tau:T}\,a_{\tau^{\prime}:T}$ at $T=4$ K (right). Bottom: $\|\mathbf{E}_{T}\|_{F}^{2}$ and its diagonal terms as the context grows, with their ratio $g_{T}$ marked by arrows.

Figure 2: Coherent gains of all per-step errors ( $g_{T}$ ) and of the errors on elements whose update is lost ( $g^{\text{lost}}_{T}$ ) or moved to a different grid point ( $g^{\text{moved}}_{T}$ ). Arrows mark $g^{\text{lost}}_{T}/g^{\text{moved}}_{T}$ .

Figure 3: Overview of our method. LRB accumulates updates in a low-rank bypass and writes them into the quantized state every $m$ steps, and BDR corrects the reversed elements during rounding.

Panel of Figure 4 — Figure 4: Left: three ways rounding distorts an update $d$ , and the correction of BDR. Right: perplexity after removing the rounding errors of each case.

			PG-19 PPL $\downarrow$				WikiText-103 PPL $\downarrow$
Model	Method	Bits	1K	2K	4K	8K	1K	2K	4K	8K
GDN	FP32	32.0	84.78	76.12	70.64	66.84	30.84	26.38	24.85	24.19
	INT4	4.50	96.21	101.08	141.76	220.36	36.10	36.49	48.15	79.76
	INT4+Had	4.50	95.56	96.98	112.52	137.94	35.23	34.69	40.92	53.71
	Ours	5.25	85.91	77.90	74.11	72.28	31.38	27.23	26.27	26.85
GLA	FP32	32.0	64.71	60.89	57.96	55.85	25.06	21.66	20.69	21.57

Table 2: Language modeling quality of 1.3B models on PG-19 and WikiText-103. INT4+Had applies a Hadamard rotation before INT4 quantization. Bits is the average storage per state element over decoding. The configurations of our method are listed in Table 6 .

Model	Method	LMB-PPL $\downarrow$	LMB-Acc $\uparrow$	SciQ $\uparrow$	COPA $\uparrow$	HS $\uparrow$
GDN	FP32	12.53	46.38	90.9	73.0	55.79
	INT4	17.94	39.90	90.3	74.0	55.54
	Ours	12.85	46.01	91.2	73.0	55.84
GLA	FP32	15.56	46.46	80.7	70.0	48.86
	INT4	22.48	38.02	79.1	69.0	48.51
	Ours	15.75	46.05	80.7	70.0	48.86

Table 3: Zero-shot performance of 1.3B models on language modeling and commonsense reasoning benchmarks. LMB: LAMBADA; HS: HellaSwag, reported as length-normalized accuracy.

Model	Method	F-512	F-1K	S-512	S-1K	N-512	N-1K	SQ	TQA	DR	Avg
GDN	FP32	51.2	41.1	42.9	36.7	31.1	31.1	41.0	57.7	21.7	39.4
	INT4	30.1	25.4	33.7	25.5	26.5	24.4	39.0	56.7	16.3	30.8
	Ours	52.5	39.1	41.2	35.7	28.2	29.4	41.7	58.0	20.7	38.5
GLA	FP32	64.9	46.8	44.2	31.3	34.0	30.3	40.3	53.7	20.7	40.7
	INT4	20.4	13.7	36.1	28.2	27.7	25.2	33.7	47.3	17.0	27.7
	Ours	59.9	41.1	41.2	31.6	34.5	30.7	41.0	52.7	24.7	39.7

Table 4: Accuracy

\uparrow

on recall-intensive tasks ( Arora et al., 2024 ) with 1.3B models. F: FDA, S: SWDE, N: NQ, SQ: SQuADv2, TQA: TriviaQA, DR: DROP; 512 and 1K denote the context length.

Figure 5: State memory and end-to-end decoding throughput of our method and FP32 states.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$Q_{25}$	$Q_{50}$	$Q_{75}$	Lost fraction
GDN	0.060	0.207	0.582	68.0%
GLA	0.021	0.072	0.209	83.7%
Mamba2	0.052	0.188	0.543	72.4%
RetNet	0.005	0.017	0.046	96.4%

Appendix

Table 5: Quartiles of

|b|/\Delta_{t}

and fraction of lost updates under INT4 at

T=4

K on WikiText-103.

Model	$k$	$m$	$\epsilon_{L}$	$\epsilon_{B}$	Avg. bits	Peak bits
GDN	1	15	0.50	0.30	5.25	6.00
GLA	1	15	0.50	0.30	5.31	6.07
Mamba2	1	5	0.10	0.25	5.64	6.78
RetNet	1	15	0.70	0.30	5.25	6.00

Appendix

Table 6: Configuration of our method for each model. Avg. bits and Peak bits are the average and peak storage per state element during decoding.

	2K (PPL $\downarrow$ )				8K (PPL $\downarrow$ )
Method	GDN	GLA	Mamba2	RetNet	GDN	GLA	Mamba2	RetNet
FP32	26.38	21.66	26.20	22.38	24.19	21.57	25.19	24.30
INT4	36.49	32.53	31.65	75.61	79.76	66.55	37.43	1193.21
BDR	34.22	32.49	30.85	57.92	63.15	75.61	35.10	707.65
LRB	27.37	22.57	28.00	23.88	27.64	25.91	29.90	31.18
LRB+BDR	27.23	22.36	27.60	23.76	26.85	24.86	28.78	28.80

Appendix

Table 7: Component ablation on WikiText-103 with the configurations in Table 6 .

			PPL $\downarrow$		Bits
Model	$k$	$m$	2K	8K	Avg.	Peak
GDN	0	16	27.34	27.27	5.20	6.00
	1	15	27.23	26.85	5.25
	2	14	27.33	26.97	5.30
	4	12	27.33	27.29	5.39
	8	8	27.52	27.98	5.58

Appendix

Table 8: Effect of the bypass rank

k

on WikiText-103 perplexity. The number of bypass columns

k+m

is fixed at 16 for GDN, GLA, and RetNet and at 6 for Mamba2. Bits is the storage per state element.

		PPL $\downarrow$		Bits
Model	$k+m$	2K	8K	Avg.	Peak
GDN	8	27.99	29.18	4.88	5.25
	12	27.49	27.71	5.06	5.63
	16	27.23	26.85	5.25	6.00
	24	26.94	26.07	5.63	6.76
GLA	8	23.16	29.23	4.94	5.31

Appendix

Table 9: Effect of the number of bypass columns

k+m

on WikiText-103 perplexity, with

k=1

. Bits is the storage per state element.

Figure 6: 8K perplexity on WikiText-103 versus $\epsilon_{L}$ , with the INT4 perplexity noted in each panel.

Figure 7: 8K perplexity on WikiText-103 versus $\epsilon_{B}$ , compared with LRB without BDR.

Low-Bit Quantization

Low-bit quantization reduces neural-network precision to save memory and computation while seeking to preserve task performance.

See all 225 papers

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
Sep 29, 2026Bingchen Yao, Haobo Xu, Haokun Lin +6LLM Quantization LLM Inference Acceleration
Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS +3
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10LLM Quantization LLM Inference Acceleration
UC Berkeley · University of Washington · MIT +2
Low-Discrepancy Dither for Quantized Recurrent State Caches
Sep 30, 2026Snigdha Chandan KhilarLLM Quantization Neural Network Quantization
Independent Researcher

Sep 29, 2026J/K move · Enter open · S save

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

Bingchen Yao, Haobo Xu, Haokun Lin, Yichen Wu+5

Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS+3

LLM Quantization LLM Inference Acceleration Delta-Rule Linear Attention Post-Training Quantization

Open paper AskPDF arXiv

Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates. We discover that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Motivated by these observations, we propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across both long- and short-generation benchmarks show that STEPQuant closely matches FP32-state accuracy under a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%. Our code is available at https://github.com/Dreamer-Toby/STEPQuant.

Abstract

Low-Bit Quantization

Explore similar work

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

Low-Discrepancy Dither for Quantized Recurrent State Caches