LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

Yi Pan; Haocheng Xi; Kan Zhu; Xingyang Li; Yibo Wu; Mayank Mishra; Hongtao Zhang; William X. Zheng

cs.LGSep 29, 2026

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

Authors: Yi Pan, Haocheng Xi, Kan Zhu, Xingyang Li, Yibo Wu, Mayank Mishra, Hongtao Zhang, William X. Zheng, +5 more

Organizations: UC Berkeley · University of Washington · MIT · Perplexity AI · NVIDIA

Abstract

Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). Although they compress the context into a fixed-size recurrent state and substantially reduce the cost of long-context processing, repeatedly reading and updating that state remains a major inference bottleneck. Quantization offers a natural way to reduce this cost, but can significantly degrade model quality, due to the accumulation of rounding errors and the presence of outlier rows and columns in the state. To address these challenges, we propose LeapQuant, a training-free method that achieves near-lossless performance under 8-bit recurrent-state quantization. First, to mitigate error accumulation, we propose per-window quantization, which leaps over a window of tokens and quantizes the state only once at its end. Within a window, outputs are computed from the fixed low-bit state together with high-precision buffered updates. Second, to reduce the error introduced by each quantization, LeapQuant retains the state's largest outliers as a few high-precision Compensator Tokens, which share the update path of real tokens. We then smooth the remaining residual before quantization to further reduce the error. Comprehensive experiments across the Qwen, Kimi, and GLM model families show that LeapQuant substantially reduces memory and compute costs during inference. With accuracy comparable to the FP32 baseline, it achieves average speedups of 2.05--3.70 $\times$ at the kernel level and 1.47 $\times$ for end-to-end inference on NVIDIA B200, RTX PRO 6000, and RTX 5090 GPUs.

Figures & tables

Figure 1: Recurrent-state access is a major cost of LLM serving. (a) Decode-time breakdown for GLM-5.3-Flash-NVFP4 on the B200 GPU at batch size 256. The share spent on linear attention stays roughly constant as context length grows. (b) GPU memory footprint for Qwen3.5-9B with prefix caching, assuming one linear attention state per 1,024 cached tokens.

Figure 2: Per-token versus per-window quantization. Per-token quantization re-quantizes the full state after every update. Per-window quantization holds the quantized boundary state

\hat{S}_{0}

fixed, buffers the

p

updates of the window in higher precision, and reconstructs the state from them; the full state is quantized only once per window, into

\hat{S}_{p}

Figure 3: State MSE over 64K decoded tokens on PG-19 (Qwen3.5-9B); -w quantizes once per window.

Figure 4: Per-window state reconstruction with Compensator Tokens. The FP32 state is split into a low-bit quantized residual, a few higher-precision Compensator Tokens that capture its dominant large-magnitude structure, and the buffered real-token updates of the window. The Compensator Tokens absorb dominant outliers, flattening the residual’s row

\ell_{2}

norms and making it easier to quantize.

8-bit methods
	Qwen3.5-9B				Qwen3.5-35B-A3B				Kimi-Linear-48B-A3B
Method	AIME	GPQA	LCB	MMLU	AIME	GPQA	LCB	MMLU	AIME	GPQA	LCB	MMLU	Avg.
FP32	87.9	81.3	64.1	83.3	91.5	84.7	75.6	85.9	67.5	70.3	41.4	72.4	75.5
BF16	72.1	66.2	49.6	81.0	85.8	79.3	67.2	85.2	64.3	68.1	41.0	64.0	68.7
Ours	87.9	81.8	64.1	83.8	91.0	83.9	76.1	85.8	68.3	69.8	41.3	72.1	75.5
FP8	14.6	34.3	21.4	42.6	29.6	39.9	26.7	56.4	25.6	46.6	16.0	57.0	34.2

Table 1: Downstream performance (%, higher is better). Best scores in each column within the 8-bit, 6-bit, and 4-bit groups are in bold.

Figure 7: Kernel throughput of one linear attention layer.

Figure 8: Decode-step throughput in vLLM at context length 4K. On the RTX PRO 6000, the largest batch size is 256 where 512 does not fit, and Qwen3.5-35B-A3B uses 2K instead of 4K at this batch size.

Figure 9: End-to-end inference throughput on different datasets.

Method	AIME	LCB	Kernel
FP32 per-step	87.9	64.1	1.00 $\times$
BF16 per-step	72.1	49.6	1.64 $\times$
+ Per-window	87.8	62.9	1.71 $\times$
INT8 per-step	7.1	9.2	2.43 $\times$
+ Per-window	82.4	60.6	2.64 $\times$
+ Comp. Tokens	86.6	61.5	2.56 $\times$

Table 2: Ablation of individual LeapQuant components on Qwen3.5-9B.

$p$	AIME	LCB	Kernel
4	86.2	62.8	1.59 $\times$
8	87.1	63.2	2.13 $\times$
16	87.9	64.1	2.52 $\times$
32	87.9	64.2	2.20 $\times$

Table 3: Ablations of window length

p

and Compensator Token count

r

on Qwen3.5-9B.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$D_{t}$	$\tilde{k}_{t}$	$b_{t}$	Record size
Linear Attention	$I$	$k_{t}$	$0$	$d_{k}+d_{v}$
RetNet	$\gamma I$	$k_{t}$	$0$	$d_{k}+d_{v}$
Mamba2	$\alpha_{t}I$	$k_{t}$	$0$	$1+d_{k}+d_{v}$
GLA / RWKV6 / HGRN2	$\Diag(\alpha_{t})$	$k_{t}$	$0$	$2d_{k}+d_{v}$
DeltaNet	$I$	$\beta_{t}k_{t}$	$k_{t}$	$d_{k}+d_{v}$
Gated DeltaNet	$\alpha_{t}I$	$\beta_{t}k_{t}$	$\alpha_{t}k_{t}$	$1+d_{k}+d_{v}$

Appendix

Table 4: Instances of equation 1 . DeltaProduct applies

n

delta-rule sub-steps per token, each of which is an instance of equation 1 . The last column gives the number of values in one buffered update

(d_{i},\tilde{k}_{i},u_{i})

8-bit methods
	Qwen3.5-9B					Qwen3.5-35B-A3B					Kimi-Linear-48B-A3B
Method	GSM	AIME	GPQA	LCB	MMLU	GSM	AIME	GPQA	LCB	MMLU	GSM	AIME	GPQA	LCB	MMLU
FP32	96.1	87.9	81.3	64.1	83.3	96.7	91.5	84.7	75.6	85.9	92.1	67.5	70.3	41.4	72.4
BF16	94.9	72.1	66.2	49.6	81.0	96.6	85.8	79.3	67.2	85.2	92.1	64.3	68.1	41.0	64.0
Ours	96.3	87.9	81.8	64.1	83.8	96.5	91.0	83.9	76.1	85.8	92.0	68.3	69.8	41.3	72.1
FP8 per-tensor	79.2	14.6	34.3	21.4	42.6	86.7	29.6	39.9	26.7	56.4	91.8	25.6	46.6	16.0	57.0

Appendix

Table 5: Full downstream performance (%, higher is better). Best scores in each column within each bit width are in bold.

Figure 10: Decode-step throughput in vLLM on the B200 at context lengths 1K–8K. Each bar stacks the FP32 throughput (lighter) and the gain of LeapQuant (darker).

Figure 11: Decode-step throughput of the two sparse-attention models on the B200 at batch size 256 and long contexts.

Model	Temp.	Top- $p$	Top- $k$	Presence	Thinking	Max output
Qwen3.5-9B	1.0	0.95	20	1.5	on	81,920
Qwen3.5-35B-A3B	1.0	0.95	20	1.5	on	81,920
Kimi-Linear-48B-A3B	1.0	1.0	–	0	–	65,536

Appendix

Table 6: Generation settings of the downstream evaluation.

Explore similar work

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
Sep 29, 2026Bingchen Yao, Haobo Xu, Haokun Lin +6Recurrent State Kimi Delta Attention
Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS +3
HyQuant: Hybrid-Precision Quantization for LLM Attention
Aug 28, 2026Jiatong Ding, Bingxin Xing, Yu Zhang +9Large Language Model Quantization Qantis
Shanghai Jiao Tong University · Xi’an Jiaotong University · Xiamen University +1
DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization
Aug 27, 2026Tao Zhang, Jianchao Tan, Pingwei Sun +6Recurrent State Gated Deltanet
South China University of Technology · Meituan · East China Normal University

Sep 29, 2026, cs.CLJ/K move · Enter open · S save

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

Bingchen Yao, Haobo Xu, Haokun Lin, Yichen Wu+5

Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS+3

Recurrent State Kimi Delta Attention Qantis

Open paper AskPDF arXiv

Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates. We discover that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Motivated by these observations, we propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across both long- and short-generation benchmarks show that STEPQuant closely matches FP32-state accuracy under a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%. Our code is available at https://github.com/Dreamer-Toby/STEPQuant.

Abstract

Explore similar work

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

HyQuant: Hybrid-Precision Quantization for LLM Attention

DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization