LinQuant: Functional Invariances for Low-Precision Linear Recurrences

Neehal Tumma; Philipp Nazari; Rohan Cherukuri; Daniela Rus; T. Konstantin Rusch

cs.LGOct 29, 2026

LinQuant: Functional Invariances for Low-Precision Linear Recurrences

Authors: Neehal Tumma, Philipp Nazari, Rohan Cherukuri, Daniela Rus, T. Konstantin Rusch

Organizations: MIT CSAIL · Max Planck Institute for Intelligent Systems · ETH Zürich · Tübingen AI Center ·

Abstract

Linear recurrences are emerging as practical alternatives to softmax attention in large language models, appearing in both standalone and hybrid architectures. However, despite their rapid emergence, low-precision inference remains much less developed for these models than for softmax attention. During prefill, chunkwise-parallel algorithms cast much of the recurrence as dense matrix multiplications, suggesting a natural path to low-precision execution. Simply quantizing these operations in their native form, however, can be fragile. In this work, we identify functional invariances that preserve the full-precision computation while making the operands more amenable to quantization. Building on this observation, we introduce LinQuant, a framework for low-precision recurrent prefill based on seven such invariances, including rotations, diagonal rescalings, and alternative placements of recurrent decay and write factors. We derive when these transformations remain valid across common recurrence families spanning linear, gated, and delta-rule updates, and show that they provide complementary degrees of freedom for improving low-precision accuracy. Finally, we translate this framework to production by augmenting SSDi8, an existing low-precision prefill pipeline for Mamba-2, with the applicable invariances. LinQuant improves perplexity at matched latency on Mamba-2 1.3B, 2.7B, and 8B and Nemotron-H 8B, demonstrating the practical utility of invariance-based quantization for recurrent models.

Figures & tables

Nr.	Transformation	LA	Mamba-2	GLA	DN	GDN	KDA
(1/7)	${\color[rgb]{0.0313,0.5664,0.6016}\bm{\mathrm{R}}_{QK}}$	$\checkmark$	$\checkmark$	$(\checkmark)$	$\checkmark$	$\checkmark$	$(\checkmark)$
(2/7)	${\color[rgb]{0.8633,0.2227,0.4648}\bm{\mathrm{D}}_{QK}}$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$
(3/7)	${\color[rgb]{0.0313,0.5664,0.6016}\bm{\mathrm{R}}_{V}}$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
(4/7)	${\color[rgb]{0.8633,0.2227,0.4648}\bm{\mathrm{D}}_{V}}$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
(5/7)	$\big({\color[rgb]{0,0.4453,0.6992}(\bm{\mathrm{\Gamma}}^{\rightarrow}_{[i]})^{\eta}},\;{\color[rgb]{0,0.4453,0.6992}(\bm{\mathrm{\Gamma}}^{\rightarrow}_{[i]})^{1-\eta}}\big)$	$\times$	$\checkmark$	$(\checkmark)$	$\times$	$\checkmark$	$(\checkmark)$
(6/7)	$\big({\color[rgb]{0,0.4453,0.6992}\bm{\mathrm{E}}_{[i]}^{\eta}},\;{\color[rgb]{0,0.4453,0.6992}\bm{\mathrm{E}}_{[i]}^{1-\eta}}\big)$	$\times$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$

Table 1: Applicability of the seven invariances introduced in Section 3 to common linear recurrences.

\times

: not applicable.

\checkmark

: applicable.

(\checkmark)

: applicable under constraints.

Panel of Figure 2 — Figure 2: Key-axis transformations at INT6. ( 2(a) ) Recovered excess NLL by decay and write rule. ( 2(b) ) Excess NLL with and without QK normalization ( $\dagger$ ). Panels ( 2(a) , 2(b) ) use $\langle\mathrm{T,C}\rangle$ and $\kappa=1/2$ for ${\color[rgb]{0.8633,0.2227,0.4648}\bm{\mathrm{D}}_{QK}}$ . ( 2(c) ) Recovery under $\langle\mathrm{H,C}\rangle$ and its change under $\langle\mathrm{T,C}\rangle$ , with selected $\kappa$ . ( 2(d) ) $\kappa$ sweep, one curve per model.

Panel of Figure 3 — Figure 3: Value-axis transformations at INT6. ( 3(a) ) Recovered excess NLL under $\langle\mathrm{T,T}\rangle$ . ( 3(b) ) Excess NLL under $\langle\mathrm{T,T}\rangle$ with and without QK normalization ( $\dagger$ ). ( 3(c) ) Recovery under $\langle\mathrm{T,C}\rangle$ and $\langle\mathrm{T,T}\rangle$ . ( 3(d) ) Recovery lost by storing the state rotated instead of canonically under ${\color[rgb]{0.0313,0.5664,0.6016}\bm{\mathrm{R}}_{V}}$ , relative to naive.

Panel of Figure 4 — Figure 4: Absorptions at INT6 under $\langle\mathrm{T,T}\rangle$ . ( 4(a) ) Recovered excess NLL from the selected write-factor allocation ( $\xi$ or $(\sigma,\tau)$ ). ( 4(b) ) NLL increase from placing decay on keys ( $\eta=0$ ) rather than values ( $\eta=1$ ), relative to $\eta=1$ .

Panel of Figure 5 — Figure 5: ( 5(a) ) End-to-end prefill time of LinQuant ( ${\color[rgb]{0.0313,0.5664,0.6016}\bm{\mathrm{R}}_{QK}}$ , value-side decay) relative to the matched SSDi8 setting, production Mamba-2 (A5000, $T=2048$ ). ( 5(b) ) INT8 zoo kernel with ${\color[rgb]{0.0313,0.5664,0.6016}\bm{\mathrm{R}}_{V}}$ and canonical storage relative to naive INT8 (A5000, $B=H=8$ , $T=2048$ , $D_{k}=D_{v}=128$ ). ( 5(c) ) Production recurrence latency with decay on keys relative to values (static scales, 64 heads, batch 8).

			${\color[rgb]{0.0313,0.5664,0.6016}\bm{\mathrm{R}}_{QK}}$	${\color[rgb]{0.0313,0.5664,0.6016}\bm{\mathrm{R}}_{QK}}+{\color[rgb]{0.8633,0.2227,0.4648}\bm{\mathrm{D}}_{QK}}$
Model	FP16	SSDi8	value	value	key
Mamba-2
1.3B	10.420	10.928	10.860	10.849	10.830
2.7B	9.060	9.441	9.379	9.383	9.372
8B	7.250	7.620	7.456	7.435	7.437
Nemotron-H

Table 2: Production WikiText-2 perplexity. SSDi8 uses value-side decay absorption. The combined transformation is evaluated with decay on either operand. More results in Table 17 .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

model	update
Linear attention ( Katharopoulos et al., 2020 )	$\bm{\mathrm{S}}_{t}=\bm{\mathrm{S}}_{t-1}+\bm{v}_{t}\bm{k}_{t}^{\top}$
Mamba-2 ( Dao & Gu, 2024 )	$\bm{\mathrm{S}}_{t}={\color[rgb]{0,0.4453,0.6992}\alpha_{t}}\,\bm{\mathrm{S}}_{t-1}+\bm{v}_{t}\bm{k}_{t}^{\top}$
Gated Linear Attention (GLA) ( Yang et al., 2024b )	$\bm{\mathrm{S}}_{t}=\bm{\mathrm{S}}_{t-1}\,{\color[rgb]{0,0.4453,0.6992}\operatorname{diag}(\bm{\alpha}_{t})}+\bm{v}_{t}\bm{k}_{t}^{\top}$
DeltaNet ( Schlag et al., 2021 ; Yang et al., 2024c )	$\bm{\mathrm{S}}_{t}=\bm{\mathrm{S}}_{t-1}\,\bigl(\bm{\mathrm{I}}-{\color[rgb]{0.9023,0.625,0}\beta_{t}}\bm{k}_{t}\bm{k}_{t}^{\top}\bigr)+{\color[rgb]{0.9023,0.625,0}\beta_{t}}\,\bm{v}_{t}\bm{k}_{t}^{\top}$
Gated DeltaNet ( Yang et al., 2025 )	$\bm{\mathrm{S}}_{t}={\color[rgb]{0,0.4453,0.6992}\alpha_{t}}\,\bm{\mathrm{S}}_{t-1}\,\bigl(\bm{\mathrm{I}}-{\color[rgb]{0.9023,0.625,0}\beta_{t}}\bm{k}_{t}\bm{k}_{t}^{\top}\bigr)+{\color[rgb]{0.9023,0.625,0}\beta_{t}}\,\bm{v}_{t}\bm{k}_{t}^{\top}$
Kimi Delta Attention (KDA) ( Team et al., 2025 )	$\bm{\mathrm{S}}_{t}=\bm{\mathrm{S}}_{t-1}\,{\color[rgb]{0,0.4453,0.6992}\operatorname{diag}(\bm{\alpha}_{t})}\,\bigl(\bm{\mathrm{I}}-{\color[rgb]{0.9023,0.625,0}\beta_{t}}\bm{k}_{t}\bm{k}_{t}^{\top}\bigr)+{\color[rgb]{0.9023,0.625,0}\beta_{t}}\,\bm{v}_{t}\bm{k}_{t}^{\top}$

Appendix

Table 3: State-update rules for recurrent models in the mixer-zoo design space. Colors identify decay and write gating .

Panel of Figure 6 — Figure 6: Response to the exponent $\kappa$ of ${\color[rgb]{0.8633,0.2227,0.4648}\bm{\mathrm{D}}_{QK}}$ , alone (solid) and composed with ${\color[rgb]{0.0313,0.5664,0.6016}\bm{\mathrm{R}}_{QK}}$ (dashed, hollow), per linear cell and regime, INT6-quantized operands, INT8 stored state: percent of the naive excess NLL recovered. At $\kappa=0$ the dashed curve is ${\color[rgb]{0.0313,0.5664,0.6016}\bm{\mathrm{R}}_{QK}}$ alone.

Panel of Figure 7 — Figure 7: Response to the exponent $\kappa$ of ${\color[rgb]{0.8633,0.2227,0.4648}\bm{\mathrm{D}}_{V}}$ , percent of the naive excess NLL recovered: per-channel scales absorb a consistent diagonal rescaling, whereas shared scales allow the exponent to change the error. The full grid $\kappa\in\{\tfrac{1}{4},\tfrac{1}{2},\tfrac{3}{4},1\}$ was run only under coarse scales $\langle\mathrm{T,T}\rangle$ .

Panel of Figure 8 — Figure 8: Full storage-basis control for the 340M zoo. Excess NLL of rotated minus canonical storage under ${\color[rgb]{0.0313,0.5664,0.6016}\bm{\mathrm{R}}_{V}}$ , as a percentage of each model’s naive excess NLL, per scale regime, INT6-quantized operands. Positive values favor canonical storage. Values use rounded per-configuration perplexities.

Panel of Figure 9 — Figure 9: Response to the decay allocation $\eta$ (decay on the key at 0, on the value at 1) for the decaying cells, INT6 operands and INT8 stored state: percent of the naive excess NLL recovered, one curve per scale regime. Placing scalar decay on the keys raises the excess by 21–27 % on M2 $+{\color[rgb]{0.9023,0.625,0}\beta}$ and GDN under $\langle\mathrm{T,T}\rangle$ . Diagonal decay and the per-channel-value regimes stay within a few percent.

Panel of Figure 10 — Figure 10: Write-factor allocation on the six gated linear models, with INT6 operands and INT8 stored state: ${\color[rgb]{0.9023,0.625,0}\bm{\mathrm{B}}_{[i]}^{\xi}}$ multiplies values and ${\color[rgb]{0.9023,0.625,0}\bm{\mathrm{B}}_{[i]}^{1-\xi}}$ multiplies keys. Percent of the naive excess NLL recovered. Each curve shows one scale regime. $\dagger$ marks models without QK normalization.

Panel of Figure 11 — Figure 11: Delta write-factor allocation with INT6 operands and INT8 stored state. Curves vary the Gram exponent $\sigma$ at fixed $\tau$ , which allocates the remaining factor between the transformed write operands: solid $\tau=1$ , dashed $\tau=1/2$ , dotted $\tau=0$ . Percent of the naive excess NLL recovered. Each colour denotes one scale regime.

Figure 12: Write error of DeltaNet 340M against the gate allocation

a

(

\sigma=1-a

\tau=1

;

F_{a}=B^{-a}F

K_{a}=B^{a}K

via the matched solve), with INT6 static per-tensor write scales recalibrated per allocation on the calibration split. Relative Frobenius error of the quantized write against the full-precision

H=F^{\top}K

, quantizing both operands (solid), the corrected values only, or the keys only (dashed). Per-head medians over 256 held-out WikiText-2 chunks at layers 3, 9, 15 and 21; the curve is the median over all heads and the band the interquartile range. Here,

a=0

is value-side absorption and

a=1

is key-side absorption.

Panel of Figure 13 — Figure 13: Percentage of the naive quantizer’s excess NLL recovered per cell and invariance at its selected setting (negative: the invariance increases the loss), shared keys with per-channel values (left) and with per-tensor values (right).

Model	Static write scales	Dynamic write scales
LA	22.7	26.2
Mamba-2	16.1	16.4
GDN	18.4	18.8

Appendix

Table 8: WikiText-2 perplexity for the static and dynamic per-chunk write-scale controls at INT4, with per-head read-key scales and INT8 stored states (340M models).

Method	1.3B	2.7B	8B
WikiText-2 perplexity
FP16	10.42	9.06	7.25
SSDi8 W4A8 (official implementation)	10.923	9.430	7.629
Value-side baseline (our re-implementation)	10.928	9.441	7.620
Prefill latency (ms)
SSDi8 W4A8 (official implementation)	616	1184	2563

Appendix

Table 13: Mamba-2 baseline validation on WikiText-2. Quantized rows use W4A8 projections, value-side static-

(H,D_{v})

absorption, no rotation, and no compensation. Full-model prefill is measured on the A5000 at

B=8

T=2048

Rescaling exponent $\kappa$	1.3B	8B
$1/4$	10.858	7.435
$1/2$	10.849	7.435
$3/4$	10.866	7.441
Implementation	Latency (ms)	Relative latency
Without rescaling	1.498	1.00
Fused rescaling	1.498	1.00

Appendix

Table 14: Production query–key rescaling controls. Top: WikiText-2 perplexity with static value-side decay absorption. Bottom: Mamba-2 1.3B layer latency at

B=1

, relative to the baseline without rescaling. A dash denotes an unreported absolute time.

Transformation and state basis	FP projections
No rotation	10.496
${\color[rgb]{0.0313,0.5664,0.6016}\bm{\mathrm{R}}_{V}}$ , rotated storage	10.533
${\color[rgb]{0.0313,0.5664,0.6016}\bm{\mathrm{R}}_{V}}$ , canonical storage	10.497

Appendix

Table 15: Value-rotation diagnostics on Mamba-2 1.3B: WikiText-2 perplexity with full-precision projections and an INT8 fixed-point state. Canonical storage is evaluated by fake quantization.

Figure 14: Per-layer relative error against the FP16 recurrence output, alongside its magnitude, for three rotation settings with rotated state storage. Value-rotation error grows in later layers with small recurrence outputs. Section F.1 discusses the interaction with RMSNorm.

Panel of Figure 15 — Figure 15: Per-stage Mamba-2 1.3B recurrence latency at $B=8$ , $T=2048$ : the SSDi8 setting (our matched value-side baseline) and key-/value-side configurations with ${\color[rgb]{0.0313,0.5664,0.6016}\bm{\mathrm{R}}_{QK}}$ and static or dynamic ( ′ ) scales.

Panel of Figure 16 — Figure 16: Mamba-2 1.3B recurrence latency against batch size at $T=2048$ (left) and against sequence length at $B=32$ (right) for FP16, the official SSDi8 implementation, and the value- and key-side configurations with static scales and ${\color[rgb]{0.0313,0.5664,0.6016}\bm{\mathrm{R}}_{QK}}$ . SSDi8 and the value-side configuration have similar latency once launch overhead no longer dominates.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 391 papers

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10LLM Quantization LLM Inference Acceleration
UC Berkeley · University of Washington · MIT +2
RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models
Oct 9, 2026Ke Wan, Chen ChenRecurrent Language Models LLM Inference Acceleration
Department of Computer Science, University of Virginia
Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
Aug 4, 2026Yu Luo, Bo Dong, Wenhua Cheng +1Mixed-Precision Quantization LLM Quantization
Intel

Sep 29, 2026J/K move · Enter open · S save

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

Yi Pan, Haocheng Xi, Kan Zhu, Xingyang Li+9

UC Berkeley · University of Washington · MIT+2

LLM Quantization LLM Inference Acceleration Delta-Rule Linear Attention Attention Quantization Recurrent-State Quantization Neural Network Quantization

Open paper AskPDF arXiv

Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). Although they compress the context into a fixed-size recurrent state and substantially reduce the cost of long-context processing, repeatedly reading and updating that state remains a major inference bottleneck. Quantization offers a natural way to reduce this cost, but can significantly degrade model quality, due to the accumulation of rounding errors and the presence of outlier rows and columns in the state. To address these challenges, we propose LeapQuant, a training-free method that achieves near-lossless performance under 8-bit recurrent-state quantization. First, to mitigate error accumulation, we propose per-window quantization, which leaps over a window of tokens and quantizes the state only once at its end. Within a window, outputs are computed from the fixed low-bit state together with high-precision buffered updates. Second, to reduce the error introduced by each quantization, LeapQuant retains the state's largest outliers as a few high-precision Compensator Tokens, which share the update path of real tokens. We then smooth the remaining residual before quantization to further reduce the error. Comprehensive experiments across the Qwen, Kimi, and GLM model families show that LeapQuant substantially reduces memory and compute costs during inference. With accuracy comparable to the FP32 baseline, it achieves average speedups of 2.05--3.70

\times

at the kernel level and 1.47

\times

for end-to-end inference on NVIDIA B200, RTX PRO 6000, and RTX 5090 GPUs.

Abstract

LLM Quantization

Explore similar work

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs