PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States

Patrick Haller; Filipe Laitenberger; Alan Akbik

cs.CLOct 21, 2026

PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States

Authors: Patrick Haller, Filipe Laitenberger, Alan Akbik

Organizations: Humboldt-Universität zu Berlin

Abstract

Linear-attention language models keep their context in a recurrent state of fixed size, where an inference system stores one copy of this state for every sequence it decodes at the same time. Storing the state at lower precision therefore lets more sequences share an accelerator. Existing state quantizers round the state after every generated token and report that four bits lose accuracy unless the bit width is raised or the model is fine-tuned.\par We introduce PACE, a post-hoc four-bit state quantizer. PACE rounds the state only once every few generated tokens, as chunked kernels already do once per chunk during prefill, and keeps the tokens generated in between in a small buffer. Rounding less often reduces the error that accumulates in the state. Additionally, we find that the large entries of the gated-delta state gather in columns, because some channels of the update written at each token remain large across many tokens. PACE therefore shares each quantization scale within one column. It also keeps the heads that retain rounding error longest at eight bits. On four hybrid models from 4B to 35B parameters, PACE with mixed four- and eight-bit heads is competitive with eight-bit DAMP on the three larger models at about two thirds of its memory, PACE with four-bit heads throughout scores higher than Q-Mamba at equal memory, and rounding every few tokens preserves the decoding speedup of the buffer.

Figures & tables

Figure 1: Mean performance of MMLU-Pro, IFEval and RULER-32k against resident bits per value on Qwen3.5-9B. Arrows join per-token and

dP{=}8

rounding variants.

Figure 2: Overview of PACE. (a) Prior state quantizers round the recurrent state after every generated token; PACE rounds it once every

dP

tokens (\scriptsizeA⃝), as prefill already does at each kernel chunk boundary. (b) At each flush, the packed state is dequantized, advanced over the tokens held in the decode buffer (\scriptsizeB⃝), and requantized in channel-aligned scale groups (\scriptsizeC⃝). PACE-4/8 also holds a quarter of the heads in each layer at eight bits, selected once per checkpoint (\scriptsizeD⃝).

Figure 3: Large entries of the recurrent state form columns indexed by the value coordinate. (a) $|S|$ of one head of Qwen3.5-9B on a shared logarithmic scale, in the original model and with the channels of the delta value $u_{t}$ permuted at every token; the head is the median head of the median layer by the value-to-key ratio. (b) Within-group max/RMS against group width, for groups along the key axis (solid) and the value axis (dashed). (c) Ratio of the value-axis to the key-axis statistic at $g{=}128$ , in the original model and with $u_{t}$ or $k_{t}$ permuted at every token.

Figure 4: Mean score over GSM8K, IFEval and RULER-32k against resident bits per value for uniform value-axis states of four to eight bits on the three Qwen3.5 backbones (Qwen3.6-35B-A3B in Appendix A ). Each line is one width, with

dP=1,8,32

from left to right; dotted: full precision; shaded: above the

9.9

bits of DAMP INT8. Right: resident bits per value against

dP

(Eq. 6 ).

Scale structure
configuration	bits	IFEval 4B	IFEval 27B	ppl 4B
full precision	$32.00$	$82.26$	$86.69$	$7.713$
value axis, $g{=}32$	$5.28$	$73.75_{\pm 3.29}$	$75.05_{\pm 3.24}$	$8.158$
Q-Mamba DSQ [ 4 ]	$5.03$	$74.86_{\pm 3.22}$	$73.94_{\pm 3.33}$	$8.320$
key axis, $g{=}32$	$5.28$	$80.59_{\pm 2.53}$	$78.37_{\pm 2.84}$	$7.719$
key axis, $g{=}64$	$5.03$	$80.59_{\pm 2.73}$	$78.93_{\pm 2.78}$	$7.734$

Table 2: Scaling the four-bit state along the key axis and allocating eight bits per head reduce the loss of the uniform value-axis quantizer. Bits are resident bits per value at

dP{=}8

with an int8 decode buffer.

Figure 5: Relative state error against token position on Qwen3.5-4B, one curve per period

dP

. The DAMP panel (right) keeps

16

key channels per head in fp16.

Figure 6: Decoding cost on Qwen3.5-4B (H100 PCIe, 512-token context) against batch size. Left: latency of the recurrent-state update. Right: time per output token. The shading marks the gap to bf16. Labels give the ratio to bf16 (left) and the relative change in TPOT (right) at the ends of each curve.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: State drift on Qwen3.6-35B-A3B, measured as in Figure 5 .

	value/key max/RMS, $g{=}128$				outlier mass on columns
condition	4B	9B	27B	35B-A3B	4B	9B	27B	35B-A3B
original	$1.588$	$1.555$	$1.601$	$1.871$	$82.6$	$78.6$	$83.9$	$87.8$
$u_{t}$ channels permuted per token	$0.776$	$0.760$	$0.808$	$0.963$	$35.5$	$34.5$	$37.1$	$45.4$
$k$ entries permuted per token	$2.265$	$2.220$	$2.261$	$2.580$	$93.1$	$92.3$	$93.9$	$92.7$

Appendix

Table 4: Permuting the channels of the delta value at every token gives key-axis groups the larger max/RMS; permuting the key entries widens the difference. Left: ratio of the value-axis to the key-axis max/RMS at

g{=}128

. Right: share of large entries that fall in the heaviest

5\%

of columns, in percent.

key axis
		IFEval				RULER- $32$ k
$k$ / $v$	bits	4B	9B	27B	35B	9B	27B	35B
full precision	$32.00$	83.2	84.5	86.9	83.4	100.0	100.0	100.0
int $8$ / int $8$	$6.44$	81.5	78.2	79.9	72.8	98.3	95.0	84.3
int $4$ / bf $16$	$6.82$	78.2	82.4	86.5	80.4	100.0	100.0	100.0
bf $16$ / int $4$	$6.44$	13.9	22.7	37.0	12.2	1.7	34.3	0.0

Appendix

Table 5: Rounding the buffered values to four bits reduces accuracy more than rounding the buffered keys. PACE-4/8 state quantizer at

dP{=}8

;

k

v

gives the widths of the buffered keys and values; bits are resident bits per value.

configuration	state	buffer	other	bits/value	Eq. 6
per-token, bf16 state	$24.00$	–	–	$16.00$	$16.00$
decode buffer, bf16 state	$24.75$	$1.14$	$0.08$	$17.32$	$17.28$
periodic INT4, $dP{=}8$	$6.75$	$1.14$	$0.08$	$5.32$	$5.28$

Appendix

Table 6: Recurrent-state memory per sequence on Qwen3.5-4B in MiB, measured in the run of Figure 6 , and the bits per state value it implies, next to the resident bits of Eq. 6 . other holds the buffered decays and kernel scratch space.

Recurrent-State Quantization

See all 9 papers

Preserving Updates in Low-Bit Recurrent State Quantization
Oct 11, 2026Haotian Li, Liran Dong, Kaicheng Xiao +1Recurrent-State Quantization Low-Bit Quantization
The Chinese University of Hong Kong
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10LLM Quantization LLM Inference Acceleration
UC Berkeley · University of Washington · MIT +2
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
Sep 29, 2026Bingchen Yao, Haobo Xu, Haokun Lin +6LLM Quantization LLM Inference Acceleration
Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS +3

Oct 11, 2026J/K move · Enter open · S save

Preserving Updates in Low-Bit Recurrent State Quantization

Haotian Li, Liran Dong, Kaicheng Xiao, Guoliang Xing

The Chinese University of Hong Kong

Recurrent-State Quantization Low-Bit Quantization Neural Network Quantization Memory-Efficient Inference Linear Attention

Open paper AskPDF arXiv

Linear attention and recurrent models compress past tokens into a fixed-size state that is updated at every step. However, storing these states in high precision incurs substantial memory and bandwidth costs at large batch sizes, motivating low-bit state quantization. Unlike weights and key--value caches, a recurrent state is requantized after every update, so its quantization errors accumulate across steps. Our analysis shows that these errors are highly coherent across steps, so reducing per-step errors alone is insufficient. This coherence arises mainly from small updates that are lost in rounding. To preserve these updates, we intervene both before and at rounding. Before rounding, we propose Low-Rank Bypass (LRB), which accumulates small updates in a compact low-rank buffer and periodically merges its sufficiently large elements into the state. At rounding, we observe that a state element can even move opposite to its update, and such reversals prove particularly harmful. Therefore, we propose Bounded Directional Rounding (BDR), which rounds reversed elements toward their updates without additional storage. Although BDR enlarges the rounding error, it improves the model, suggesting that the direction of an update matters more than its rounding error at a single step. Across Gated DeltaNet, GLA, Mamba2, and RetNet, our training-free method substantially outperforms INT4, while reducing state memory by up to 5.3

\times

and speeding up decoding by up to 1.9

\times

relative to FP32 states.

Abstract

Recurrent-State Quantization

Explore similar work

Preserving Updates in Low-Bit Recurrent State Quantization

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization