Low-Bit GDN Recurrent-State Quantization: Global Gradient-aware Allocation and Pure-Write Sliding Windows

Qinghang Huang; Xuchu Huang; Xunzhao Yin

cs.LGOct 29, 2026

Low-Bit GDN Recurrent-State Quantization: Global Gradient-aware Allocation and Pure-Write Sliding Windows

Authors: Qinghang Huang, Xuchu Huang, Xunzhao Yin

Organizations: Zhejiang University

Abstract

Linear attention stores historical information in a recurrent state of fixed size, providing a compact memory representation for long-context inference. Reducing state precision further lowers per-request storage requirements, but quantized writeback errors enter subsequent recurrences, degrade prediction quality on long-reasoning tasks, and can be accompanied by repetitive generation and excessively long outputs. We propose Global Gradient-aware Allocation (GGA) and the Pure-Write Sliding Window (PW) to address precision allocation and recent-information maintenance. GGA combines candidate quantization residuals with gradient sensitivity to future prediction loss to allocate head bit widths under a model-wide mixed-precision budget. PW independently maintains recent pure writes as FP8 vector pairs, propagates them through the full GDN state transition, and merges them into the low-bit historical state only after they leave the window. Across seven evaluations on Qwen3.5-4B/9B, GGA allocation alone improves over STEPQuant@4 by 19.48/29.84 percentage points. Adding PW-S8 yields seven-task macro-averages of 69.83%/78.63%, improving over STEPQuant@4 by 38.39/41.21 percentage points, respectively. Both precision reallocation and window maintenance improve task scores while mitigating excessively long outputs: at @6, GGA reduces mean generation length, averaged equally across tasks, by 10.45%/14.65% relative to STEPQuant; adding PW-S8 to GGA@4 further reduces mean generation length by 32.32%/36.77%. Comparisons at similar storage budgets further show that using additional capacity to maintain recent writes yields higher task scores and shorter mean outputs than solely increasing the bit width of the accumulated state.

Figures & tables

Figure 1: Head precision versus empirical quantization impact. The horizontal axis shows centers of

0.30\times 10^{-4}

-wide bins of

I_{u}

, with head counts

n

. Both allocations average 4.33 bit/value over all heads; FP16 counts as 16 bits. The empty bin

[3.00,3.30)\times 10^{-4}

and the vertical range 8–14 bits are omitted.

	$\ldots$
934	Count: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10.
935	Okay, I miscounted when I said “that is 9 numbers”.
	$\ldots$
947	Okay, there are 10 divisors.
948	The list is 1, 2, 3, 4, 6, 8, 12, 16, 24, 48 is wrong.
949	Missing divisor:

Figure 2: Paired reasoning excerpts from the same problem. Numbers identify original output lines; ellipses mark omitted intervening lines. Without PW, the model rejects a verified divisor count, rechecks a known factor, and eventually repeats “Wait.” With PW, it retains the count, continues the search, and returns 48.

Figure 3: PW state decomposition and rolling update, with

m=3

shown for illustration. Each FP8 vector pair contributes a rank-one write, maintained separately from the low-bit history. Existing left factors and the history propagate through

A_{t+1}

. After readout, the oldest branch is merged into history before requantization and the new write enters the window; retained right factors remain fixed.

Model	Method	AIME	HMMT	GPQA	HE+	MATH	IFB	LCB	Macro-avg.
4B	FP32	86.67	63.64	70.96	86.89	94.90	62.00	63.75	75.54
	FP16	84.00	61.82	71.97	86.28	95.60	58.00	62.25	74.27
	Naive INT4	0.00	0.00	1.26	0.00	2.40	8.00	0.00	1.67
	STEP@ $4$	0.67	0.00	16.41	78.96	47.30	46.50	30.25	31.44
	GGA@ $4$	30.67	17.58	50.51	78.66	83.60	51.67	43.75	50.92
	STEP@ $4$ +PW-S8	54.67	37.58	60.61	82.62	94.00	56.00	60.25	63.67

Table 1: Measured scores (%) on seven tasks at @

4

and @

6

. For each model, floating-point references are followed by the @

4

and @

6

groups; boldface compares scores among completed quantized settings within each model and nominal-budget group. Scores are averaged across independent samples, and the macro-average weights all tasks equally. HE+, IFB, and LCB denote HumanEval+ base+plus, IFBench prompt strict, and the LiveCodeBench v6 evaluation subset, respectively.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Allocation	INT2	INT4	INT6	INT8	FP16
4B	STEP@ $4$	380	119	158	79	32
4B	GGA@ $4$	369	139	151	77	32
4B	STEP@ $5$	214	185	140	197	32
4B	GGA@ $5$	215	171	165	185	32
4B	STEP@ $6$	0	302	164	270	32
4B	GGA@ $6$	0	293	182	261	32

Appendix

Table 2: Head counts in the precision allocation plans. PW configurations not listed separately use the same bit-width tables as their corresponding configurations without a window.

Model	Allocation	@ $4$	@ $4$ +PW	@ $5$	@ $6$	@ $6$ +PW
4B	STEP	4.758	5.790	5.677	6.573	7.604
	GGA	4.753	5.784	5.678	6.573	7.604
9B	STEP	4.755	5.786	5.677	6.573	7.604
	GGA	4.749	5.780	5.679	6.573	7.604

Appendix

Table 3: Equivalent state bit widths (bit/value) for different allocation and window settings. State encoding, scales, FP16 pivots, and PW queues are included using the fixed array layout. FP32/FP16 floating-point states require 32/16 bit/value, respectively.

Composite impact bin	Heads	STEPQuant	GGA
$[0.30,0.60)$	10	2.00	2.00
$[0.60,0.90)$	7	4.00	2.57
$[0.90,1.20)$	25	2.96	2.40
$[1.20,1.50)$	56	5.14	3.36
$[1.50,1.80)$	81	5.28	3.09
$[1.80,2.10)$	104	4.37	3.13

Appendix

Table 4: Mean state-value bit width under the common composite impact metric. Bin units are

10^{-4}

, and FP16 counts as 16 bits.

Implementation	Task	Version
MathArena	AIME, HMMT	b89f2f0ad64c
math-verify	MATH500	0.9.0
GPQA	GPQA Diamond	56686c06f5e1
EvalPlus	HumanEval+	26d6d00bb1fd
IFBench	IFBench	1c40f0c10d9b
LiveCodeBench	LCB v6 subset	28fef95ea8c9

Appendix

Table 5: Frozen upstream code and grading dependencies for the formal evaluation. Code repositories are identified by the first 12 characters of the commit hash, and grading packages by their version numbers.

Model	Method	AIME	HMMT	GPQA	HE+	MATH	IFB	LCB	Mean
4B	FP32	39.79	53.10	26.62	1.27	15.06	17.81	20.94	24.94
	FP16	47.98	63.50	28.41	1.30	16.46	20.92	22.48	28.72
	Naive INT4	3.74	3.33	3.03	2.18	3.18	2.87	2.64	2.99
	STEP@ $4$	94.21	97.13	84.17	2.06	56.67	27.30	46.06	58.23
	GGA@ $4$	81.36	86.87	48.60	1.41	30.33	16.66	35.46	42.96
	STEP@ $4$ +PW-S8	70.31	81.62	34.76	1.44	19.68	12.85	25.38	35.15

Appendix

Table 6: Mean full output lengths (thousand tokens) by task for 4B/9B at @

4

and @

6

, alongside floating-point state references. Lengths include thinking and final, exclude the input prompt, and retain all stopping types. Mean is the equally weighted average of the seven task means, computed from unrounded values. Task abbreviations follow Table 1 .

Model	Setting	bit/value	Macro-avg.	Mean length
4B	STEP@ $5$	5.677	60.38	37.25
	STEP@ $4$ +PW-S8	5.790	63.67	35.15
	GGA@ $5$	5.678	67.23	32.27
	GGA@ $4$ +PW-S8	5.784	69.83	29.07
9B	STEP@ $5$	5.677	68.46	35.03
	STEP@ $4$ +PW-S8	5.786	70.02	32.39

Appendix

Table 7: Paired comparisons at similar storage budgets. Mean length weights tasks equally and is measured in thousand tokens; scores are seven-task macro-averages (%). Boldface indicates the higher macro-average and shorter mean output within each allocation method.

Model	Method	EOS	100K cap	Repetition	No final	Mean length
4B	FP32	91.81	4.54	3.65	7.54	24.94
	FP16	90.10	6.58	3.32	9.38	28.72
	Naive INT4	100.00	0.00	0.00	11.62	2.99
	STEP@ $4$	53.37	36.66	9.97	45.74	58.23
	GGA@ $4$	79.43	14.18	6.38	19.55	42.96
	STEP@ $4$ +PW-S8	91.12	8.36	0.53	8.88	35.15

Appendix

Table 8: Generation lengths and termination statistics for the floating-point references and the @

4

and @

6

settings. Percentages are aggregated over responses. Mean lengths are in thousands of tokens and are the equally weighted mean of each task’s mean full output length, including thinking and excluding input prompts. EOS, length-cap termination, and repetition stops are mutually exclusive; missing final answers may overlap with them.

Model	Method	EOS	100K cap	Repetition	No final	Mean length
4B	STEP@ $5$	83.38	9.64	6.98	16.42	37.25
	GGA@ $5$	89.87	5.03	5.10	9.94	32.27
9B	STEP@ $5$	89.40	8.03	2.57	10.37	35.03
	GGA@ $5$	95.56	3.29	1.15	4.18	26.78

Appendix

Table 9: Generation lengths and termination statistics for the supplementary @

5

configurations. Percentages are in %, and mean lengths are in thousands of tokens; the accounting convention is the same as in Table 8 .

Model	Method	AIME	HMMT	GPQA	HE+	MATH	IFB	LCB	Mean
4B	STEP@ $5$	69.58	80.97	41.15	1.27	23.02	18.96	25.77	37.25
	STEP@ $4$ +PW-S8	70.31	81.62	34.76	1.44	19.68	12.85	25.38	35.15
	GGA@ $5$	60.99	73.37	33.32	1.27	17.41	14.99	24.56	32.27
	GGA@ $4$ +PW-S8	54.13	67.48	30.22	1.26	15.06	11.25	24.10	29.07
9B	STEP@ $5$	62.23	76.95	40.44	1.05	19.44	14.31	30.79	35.03
	STEP@ $4$ +PW-S8	60.25	71.16	35.54	1.12	16.20	13.97	28.47	32.39

Appendix

Table 10: Mean full output lengths for each task in the supplementary similar-budget comparison (thousands of tokens). The @

5

and @

4

+PW-S8 settings are paired within each allocation method; the accounting convention is the same as in Table 6 .

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 391 papers

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
Sep 29, 2026Bingchen Yao, Haobo Xu, Haokun Lin +6LLM Quantization LLM Inference Acceleration
Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS +3
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10LLM Quantization LLM Inference Acceleration
UC Berkeley · University of Washington · MIT +2
Preserving Updates in Low-Bit Recurrent State Quantization
Oct 11, 2026Haotian Li, Liran Dong, Kaicheng Xiao +1Recurrent-State Quantization Low-Bit Quantization
The Chinese University of Hong Kong

Sep 29, 2026J/K move · Enter open · S save

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

Bingchen Yao, Haobo Xu, Haokun Lin, Yichen Wu+5

Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS+3

LLM Quantization LLM Inference Acceleration Delta-Rule Linear Attention Post-Training Quantization Recurrent-State Quantization

Open paper AskPDF arXiv

Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates. We discover that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Motivated by these observations, we propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across both long- and short-generation benchmarks show that STEPQuant closely matches FP32-state accuracy under a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%. Our code is available at https://github.com/Dreamer-Toby/STEPQuant.

Abstract

LLM Quantization

Explore similar work

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

Preserving Updates in Low-Bit Recurrent State Quantization