Low-Bit GDN Recurrent-State Quantization: Global Gradient-aware Allocation and Pure-Write Sliding Windows

Qinghang Huang; Xuchu Huang; Xunzhao Yin

cs.LGOct 29, 2026

Low-Bit GDN Recurrent-State Quantization: Global Gradient-aware Allocation and Pure-Write Sliding Windows

Authors: Qinghang Huang, Xuchu Huang, Xunzhao Yin

Organizations: Zhejiang University

Abstract

Linear attention stores historical information in a recurrent state of fixed size, providing a compact memory representation for long-context inference. Reducing state precision further lowers per-request storage requirements, but quantized writeback errors enter subsequent recurrences, degrade prediction quality on long-reasoning tasks, and can be accompanied by repetitive generation and excessively long outputs. We propose Global Gradient-aware Allocation (GGA) and the Pure-Write Sliding Window (PW) to address precision allocation and recent-information maintenance. GGA combines candidate quantization residuals with gradient sensitivity to future prediction loss to allocate head bit widths under a model-wide mixed-precision budget. PW independently maintains recent pure writes as FP8 vector pairs, propagates them through the full GDN state transition, and merges them into the low-bit historical state only after they leave the window. Across seven evaluations on Qwen3.5-4B/9B, GGA allocation alone improves over STEPQuant@4 by 19.48/29.84 percentage points. Adding PW-S8 yields seven-task macro-averages of 69.83%/78.63%, improving over STEPQuant@4 by 38.39/41.21 percentage points, respectively. Both precision reallocation and window maintenance improve task scores while mitigating excessively long outputs: at @6, GGA reduces mean generation length, averaged equally across tasks, by 10.45%/14.65% relative to STEPQuant; adding PW-S8 to GGA@4 further reduces mean generation length by 32.32%/36.77%. Comparisons at similar storage budgets further show that using additional capacity to maintain recent writes yields higher task scores and shorter mean outputs than solely increasing the bit width of the accumulated state.

Figures & tables

Figure 1: Head precision versus empirical quantization impact. The horizontal axis shows centers of

0.30\times 10^{-4}

-wide bins of

I_{u}

, with head counts

n

. Both allocations average 4.33 bit/value over all heads; FP16 counts as 16 bits. The empty bin

[3.00,3.30)\times 10^{-4}

and the vertical range 8–14 bits are omitted.

	$\ldots$
934	Count: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10.
935	Okay, I miscounted when I said “that is 9 numbers”.
	$\ldots$
947	Okay, there are 10 divisors.
948	The list is 1, 2, 3, 4, 6, 8, 12, 16, 24, 48 is wrong.
949	Missing divisor:

Figure 2: Paired reasoning excerpts from the same problem. Numbers identify original output lines; ellipses mark omitted intervening lines. Without PW, the model rejects a verified divisor count, rechecks a known factor, and eventually repeats “Wait.” With PW, it retains the count, continues the search, and returns 48.

Figure 3: PW state decomposition and rolling update, with

m=3

shown for illustration. Each FP8 vector pair contributes a rank-one write, maintained separately from the low-bit history. Existing left factors and the history propagate through

A_{t+1}

. After readout, the oldest branch is merged into history before requantization and the new write enters the window; retained right factors remain fixed.

Model	Method	AIME	HMMT	GPQA	HE+	MATH	IFB	LCB	Macro-avg.
4B	FP32	86.67	63.64	70.96	86.89	94.90	62.00	63.75	75.54
	FP16	84.00	61.82	71.97	86.28	95.60	58.00	62.25	74.27
	Naive INT4	0.00	0.00	1.26	0.00	2.40	8.00	0.00	1.67
	STEP@ $4$	0.67	0.00	16.41	78.96	47.30	46.50	30.25	31.44
	GGA@ $4$	30.67	17.58	50.51	78.66	83.60	51.67	43.75	50.92
	STEP@ $4$ +PW-S8	54.67	37.58	60.61	82.62	94.00	56.00	60.25	63.67

Table 1: Measured scores (%) on seven tasks at @

4

and @

6

. For each model, floating-point references are followed by the @

4

and @

6

groups; boldface compares scores among completed quantized settings within each model and nominal-budget group. Scores are averaged across independent samples, and the macro-average weights all tasks equally. HE+, IFB, and LCB denote HumanEval+ base+plus, IFBench prompt strict, and the LiveCodeBench v6 evaluation subset, respectively.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Allocation	INT2	INT4	INT6	INT8	FP16
4B	STEP@ $4$	380	119	158	79	32
4B	GGA@ $4$	369	139	151	77	32
4B	STEP@ $5$	214	185	140	197	32
4B	GGA@ $5$	215	171	165	185	32
4B	STEP@ $6$	0	302	164	270	32
4B	GGA@ $6$	0	293	182	261	32

Appendix

Table 2: Head counts in the precision allocation plans. PW configurations not listed separately use the same bit-width tables as their corresponding configurations without a window.

Model	Allocation	@ $4$	@ $4$ +PW	@ $5$	@ $6$	@ $6$ +PW
4B	STEP	4.758	5.790	5.677	6.573	7.604
	GGA	4.753	5.784	5.678	6.573	7.604
9B	STEP	4.755	5.786	5.677	6.573	7.604
	GGA	4.749	5.780	5.679	6.573	7.604

Appendix

Table 3: Equivalent state bit widths (bit/value) for different allocation and window settings. State encoding, scales, FP16 pivots, and PW queues are included using the fixed array layout. FP32/FP16 floating-point states require 32/16 bit/value, respectively.

Composite impact bin	Heads	STEPQuant	GGA
$[0.30,0.60)$	10	2.00	2.00
$[0.60,0.90)$	7	4.00	2.57
$[0.90,1.20)$	25	2.96	2.40
$[1.20,1.50)$	56	5.14	3.36
$[1.50,1.80)$	81	5.28	3.09
$[1.80,2.10)$	104	4.37	3.13

Appendix

Table 4: Mean state-value bit width under the common composite impact metric. Bin units are

10^{-4}

, and FP16 counts as 16 bits.

Implementation	Task	Version
MathArena	AIME, HMMT	b89f2f0ad64c
math-verify	MATH500	0.9.0
GPQA	GPQA Diamond	56686c06f5e1
EvalPlus	HumanEval+	26d6d00bb1fd
IFBench	IFBench	1c40f0c10d9b
LiveCodeBench	LCB v6 subset	28fef95ea8c9

Appendix

Table 5: Frozen upstream code and grading dependencies for the formal evaluation. Code repositories are identified by the first 12 characters of the commit hash, and grading packages by their version numbers.

Model	Method	AIME	HMMT	GPQA	HE+	MATH	IFB	LCB	Mean
4B	FP32	39.79	53.10	26.62	1.27	15.06	17.81	20.94	24.94
	FP16	47.98	63.50	28.41	1.30	16.46	20.92	22.48	28.72
	Naive INT4	3.74	3.33	3.03	2.18	3.18	2.87	2.64	2.99
	STEP@ $4$	94.21	97.13	84.17	2.06	56.67	27.30	46.06	58.23
	GGA@ $4$	81.36	86.87	48.60	1.41	30.33	16.66	35.46	42.96
	STEP@ $4$ +PW-S8	70.31	81.62	34.76	1.44	19.68	12.85	25.38	35.15

Appendix

Table 6: Mean full output lengths (thousand tokens) by task for 4B/9B at @

4

and @

6

, alongside floating-point state references. Lengths include thinking and final, exclude the input prompt, and retain all stopping types. Mean is the equally weighted average of the seven task means, computed from unrounded values. Task abbreviations follow Table 1 .

Model	Setting	bit/value	Macro-avg.	Mean length
4B	STEP@ $5$	5.677	60.38	37.25
	STEP@ $4$ +PW-S8	5.790	63.67	35.15
	GGA@ $5$	5.678	67.23	32.27
	GGA@ $4$ +PW-S8	5.784	69.83	29.07
9B	STEP@ $5$	5.677	68.46	35.03
	STEP@ $4$ +PW-S8	5.786	70.02	32.39

Appendix

Table 7: Paired comparisons at similar storage budgets. Mean length weights tasks equally and is measured in thousand tokens; scores are seven-task macro-averages (%). Boldface indicates the higher macro-average and shorter mean output within each allocation method.

Model	Method	EOS	100K cap	Repetition	No final	Mean length
4B	FP32	91.81	4.54	3.65	7.54	24.94
	FP16	90.10	6.58	3.32	9.38	28.72
	Naive INT4	100.00	0.00	0.00	11.62	2.99
	STEP@ $4$	53.37	36.66	9.97	45.74	58.23
	GGA@ $4$	79.43	14.18	6.38	19.55	42.96
	STEP@ $4$ +PW-S8	91.12	8.36	0.53	8.88	35.15

Appendix

Table 8: Generation lengths and termination statistics for the floating-point references and the @

4

and @

6

settings. Percentages are aggregated over responses. Mean lengths are in thousands of tokens and are the equally weighted mean of each task’s mean full output length, including thinking and excluding input prompts. EOS, length-cap termination, and repetition stops are mutually exclusive; missing final answers may overlap with them.

Model	Method	EOS	100K cap	Repetition	No final	Mean length
4B	STEP@ $5$	83.38	9.64	6.98	16.42	37.25
	GGA@ $5$	89.87	5.03	5.10	9.94	32.27
9B	STEP@ $5$	89.40	8.03	2.57	10.37	35.03
	GGA@ $5$	95.56	3.29	1.15	4.18	26.78

Appendix

Table 9: Generation lengths and termination statistics for the supplementary @

5

configurations. Percentages are in %, and mean lengths are in thousands of tokens; the accounting convention is the same as in Table 8 .

Model	Method	AIME	HMMT	GPQA	HE+	MATH	IFB	LCB	Mean
4B	STEP@ $5$	69.58	80.97	41.15	1.27	23.02	18.96	25.77	37.25
	STEP@ $4$ +PW-S8	70.31	81.62	34.76	1.44	19.68	12.85	25.38	35.15
	GGA@ $5$	60.99	73.37	33.32	1.27	17.41	14.99	24.56	32.27
	GGA@ $4$ +PW-S8	54.13	67.48	30.22	1.26	15.06	11.25	24.10	29.07
9B	STEP@ $5$	62.23	76.95	40.44	1.05	19.44	14.31	30.79	35.03
	STEP@ $4$ +PW-S8	60.25	71.16	35.54	1.12	16.20	13.97	28.47	32.39

Appendix

Table 10: Mean full output lengths for each task in the supplementary similar-budget comparison (thousands of tokens). The @

5

and @

4

+PW-S8 settings are paired within each allocation method; the accounting convention is the same as in Table 6 .

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 391 papers

Sep 29, 2026

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates. We discover that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Motivated by these observations, we propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across both long- and short-generation benchmarks show that STEPQuant closely matches FP32-state accuracy under a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%. Our code is available at https://github.com/Dreamer-Toby/STEPQuant.

Bingchen Yao, Haobo Xu, Haokun Lin +6

Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS +3

LLM Quantization LLM Inference Acceleration Delta-Rule Linear Attention Post-Training Quantization Recurrent-State Quantization

Ask PDF

Sep 29, 2026

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). Although they compress the context into a fixed-size recurrent state and substantially reduce the cost of long-context processing, repeatedly reading and updating that state remains a major inference bottleneck. Quantization offers a natural way to reduce this cost, but can significantly degrade model quality, due to the accumulation of rounding errors and the presence of outlier rows and columns in the state. To address these challenges, we propose LeapQuant, a training-free method that achieves near-lossless performance under 8-bit recurrent-state quantization. First, to mitigate error accumulation, we propose per-window quantization, which leaps over a window of tokens and quantizes the state only once at its end. Within a window, outputs are computed from the fixed low-bit state together with high-precision buffered updates. Second, to reduce the error introduced by each quantization, LeapQuant retains the state's largest outliers as a few high-precision Compensator Tokens, which share the update path of real tokens. We then smooth the remaining residual before quantization to further reduce the error. Comprehensive experiments across the Qwen, Kimi, and GLM model families show that LeapQuant substantially reduces memory and compute costs during inference. With accuracy comparable to the FP32 baseline, it achieves average speedups of 2.05--3.70

\times

at the kernel level and 1.47

\times

for end-to-end inference on NVIDIA B200, RTX PRO 6000, and RTX 5090 GPUs.

Yi Pan, Haocheng Xi, Kan Zhu +10

UC Berkeley · University of Washington · MIT +2

LLM Quantization LLM Inference Acceleration Delta-Rule Linear Attention Attention Quantization Recurrent-State Quantization Neural Network Quantization

Ask PDF

Oct 11, 2026

Preserving Updates in Low-Bit Recurrent State Quantization

Linear attention and recurrent models compress past tokens into a fixed-size state that is updated at every step. However, storing these states in high precision incurs substantial memory and bandwidth costs at large batch sizes, motivating low-bit state quantization. Unlike weights and key--value caches, a recurrent state is requantized after every update, so its quantization errors accumulate across steps. Our analysis shows that these errors are highly coherent across steps, so reducing per-step errors alone is insufficient. This coherence arises mainly from small updates that are lost in rounding. To preserve these updates, we intervene both before and at rounding. Before rounding, we propose Low-Rank Bypass (LRB), which accumulates small updates in a compact low-rank buffer and periodically merges its sufficiently large elements into the state. At rounding, we observe that a state element can even move opposite to its update, and such reversals prove particularly harmful. Therefore, we propose Bounded Directional Rounding (BDR), which rounds reversed elements toward their updates without additional storage. Although BDR enlarges the rounding error, it improves the model, suggesting that the direction of an update matters more than its rounding error at a single step. Across Gated DeltaNet, GLA, Mamba2, and RetNet, our training-free method substantially outperforms INT4, while reducing state memory by up to 5.3

\times

and speeding up decoding by up to 1.9

\times

relative to FP32 states.

Haotian Li, Liran Dong, Kaicheng Xiao +1

The Chinese University of Hong Kong

Recurrent-State Quantization Low-Bit Quantization Neural Network Quantization Memory-Efficient Inference Linear Attention

Ask PDF

Abstract

LLM Quantization

Explore similar work

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

Preserving Updates in Low-Bit Recurrent State Quantization