Optimal Decay Spectra for Linear Recurrences

Yang Cao

cs.LGApr 8, 2026

Optimal Decay Spectra for Linear Recurrences

Authors: Yang Cao

Abstract

Linear recurrent models offer linear-time sequence processing but often suffer from suboptimal long-range memory. We trace this to the decay spectrum: for $N$ channels, random initialization collapses the minimum spectral gap to $O(N^{-2})$ , yielding sub-exponential error $\exp(-Ω(N/\log N))$ ; linear spacing avoids collapse but degrades to $\exp(-O(N/\sqrt{T}))$ , practically algebraic over long contexts. We introduce Position-Adaptive Spectral Tapering (PoST), an architecture-agnostic framework combining two mechanisms: (1) Spectral Reparameterization, which structurally enforces geometrically spaced log-decay rates, proven minimax optimal at rate $O(\exp(-cN/\log T))$ ; and (2) Position-Adaptive Scaling, the provably unique mechanism that eliminates the scale mismatch of static spectra (where only $N\log t/\log T$ of $N$ channels are effective at position $t$ ) by stretching the spectrum to the actual dependency range, sharpening the rate to $O(\exp(-cN/\log t))$ . This scaling natively induces fractional invariance: the impulse response becomes scale-free, with channels interpolating between relative and absolute temporal coordinates. PoST integrates into any diagonal linear recurrence without overhead. We instantiate it across Mamba-2, RWKV-7, Gated DeltaNet, Gated Linear Attention, and RetNet. Pre-training at 180M-440M scales shows consistent zero-shot language modeling improvements, significant long-context retrieval gains for Mamba-2 (MQAR and NIAH), and competitive or improved performance across other architectures. Code: https://github.com/SiLifen/PoST.

Figures & tables

Paradigm	Min. Spectral Gap	Minimax Error (Power-Law Kernel)
Random	$O(N^{-2})$	$\exp\left(-\Omega\left(\frac{N}{\log N}\right)\right)$
Linear	$\Theta(1)$	$\Omega(N^{-\beta})$
Geometric (static)	$O(1/N)$	$O\left(\exp\left(-c\frac{N}{\log T}\right)\right)$
PoST (Ours)	$O(1/N)$	$O\left(\exp\left(-c\frac{N}{\log t}\right)\right)$

Table 1: Theoretical Landscape of Decay Spectra. Approximation bounds for power-law kernels across parameterization strategies. Geometric spacing achieves the optimal spectral shape but leaves the scale fixed to the design length

T

; PoST eliminates this scale mismatch via position-adaptive scaling, sharpening the exponent from

\log T

\log t

	state $=64K$					state $=32K$					state $=16K$
Model	512	1K	2K	4K	Avg	512	1K	2K	4K	Avg	512	1K	2K	4K	Avg
Mamba-2	100.0	96.8	62.2	18.9	69.5	99.2	85.2	41.3	11.6	59.4	99.3	80.6	31.2	5.7	54.2
+PoST	100.0	97.4	68.3	25.1	72.7	99.8	92.1	51.6	13.2	64.2	99.6	87.8	44.1	12.7	61.0
RWKV-7	100.0	100.0	96.1	39.2	83.8	100.0	100.0	80.1	9.5	72.4	100.0	95.2	46.0	10.8	63.0
+PoST	100.0	100.0	98.5	52.9	87.8	100.0	100.0	98.0	28.5	81.6	100.0	99.3	70.9	18.8	72.2
Gated DeltaNet	100.0	100.0	92.0	42.4	83.6	100.0	96.4	56.7	15.9	67.2	99.8	82.7	31.7	7.4	55.4

Table 2: MQAR capacity accuracy (%) at each test length

T

with

K{=}T/4

key–value pairs. All models trained at

T{=}512

; longer lengths are out-of-distribution. GLAPoST and RetNetPoST converge to the same model under the PoST framework. Across almost all settings, PoST outperforms its baseline; the sole exception is GLA at state

=64K

, where the baseline edges ahead (

71.5

vs.

69.7

avg); GLAPoST recovers the lead at state

=32K

and

16K

Model	LAMBADA		HellaSwag	PIQA	ARC-Easy	ARC-Challenge	WinoGrande	OpenBookQA	Avg
	acc $\uparrow$	ppl $\downarrow$	acc ${}_{\text{n}}\uparrow$	acc $\uparrow$	acc $\uparrow$	acc ${}_{\text{n}}\uparrow$	acc $\uparrow$	acc ${}_{\text{n}}\uparrow$
Mamba-2 180M	21.6	145.4	31.1	62.9	50.4	24.7	49.6	30.6	38.7
+PoST	21.5	148.2	31.3	63.2	50.1	24.9	50.6	30.0	38.8
RWKV-7 180M	27.9	69.6	32.1	63.1	49.7	25.7	51.3	29.0	39.8
+PoST	28.3	71.9	32.1	62.9	52.1	25.3	51.8	32.0	40.6
Gated DeltaNet 180M	23.8	94.5	31.9	62.7	49.6	24.2	51.1	30.6	39.1

Table 3: Downstream zero-shot evaluations. PoST achieves consistently better average performance than the baseline across all benchmarks at 180M and 440M scales, indicating that the spectral reparameterization consistently, though modestly, improves general language modeling capabilities.

Figure 1: Empirical timescale distribution

\tau_{k}=e^{-A_{k}}

across trained models. (Left) A kernel density estimate (KDE) over all depths shows Mamba-2 models suffering from a severe minimum gap collapse (density clumping into narrow spikes), while PoST strictly enforces a broad geometric long-tail distribution. (Right) Head timescale allocation within a single representative layer (Layer 12). Mamba-2 models flatten out (allocating many heads to identical timescales), wasting capacity. PoST forms a perfect straight line on the log-scale, empirically proving rigorous geometric spacing.

Figure 2: Layer $\times$ Head heatmap of learned log-timescales $\log\tau_{k}=-A_{k}$ across all layers. Each cell encodes the log-timescale of a single SSM head at its actual model index at a given layer (top to bottom); no manual reordering is applied. Top row (180M): The baseline Mamba-2 heatmap is nearly uniform in color throughout: all heads at every layer collapse to a narrow band of fast timescales, wasting state capacity. The PoST counterpart displays a smooth left-to-right gradient because the Spectral Reparameterization structurally enforces $p_{1}<p_{2}<\cdots<p_{N}$ via a cumulative-softplus construction: the ordering is an intrinsic property of the learned weights, not a visualization artifact. This gradient is consistent across every layer, confirming that geometric spectral order is a global, depth-invariant property. Bottom row (440M): The same contrast holds at larger scale (48 layers, 32 heads). The PoST model preserves a wider dynamic range (larger spread between minimum and maximum $\log\tau_{k}$ ) than the baseline, directly reflecting the broader effective memory horizon predicted by Theorem 4.7 .

Figure 3: Learned Normalization Taper

\alpha_{k}

. We examine the empirical

\alpha_{k}

distributions across all layers of the pre-trained Mamba-2 PoST 180M and 440M models. Rather than fixing

\alpha_{k}

to the linear

(N-k)/(N-1)

blueprint (dashed black line), PoST allows data-dependent parameter adjustments and adaptively recomputes the optimal

\alpha_{k}

for each head to enforce rigid geometric spacing (Proposition 4.16 ). Across both model scales, the empirical average taper strongly aligns closely with the strict linear ideal, revealing that natural language optimization inherently converges to the hierarchically distributed memory blueprint.

	Single-Needle									Multi-Needle
	Single-1			Single-2			Single-3			MultiKey			MultiQuery			MultiValue
Model	1K	2K	4K	1K	2K	4K	1K	2K	4K	1K	2K	4K	1K	2K	4K	1K	2K	4K	Avg
Mamba-2 180M	44.0	5.6	0.2	15.4	3.6	0.8	0.0	0.0	0.0	9.0	6.0	1.6	8.5	5.5	1.4	4.9	4.5	1.7	6.3
+PoST	95.6	47.4	2.0	71.0	13.2	8.4	4.8	0.6	1.8	17.0	16.0	6.4	12.6	12.6	3.1	11.1	12.0	3.5	18.8
RWKV-7 180M	99.6	97.4	62.6	66.8	11.8	12.4	0.0	0.0	0.0	17.8	16.8	8.8	14.1	13.1	3.3	16.2	11.5	7.0	25.5
+PoST	99.8	93.6	57.8	90.2	11.6	5.4	3.0	0.2	0.0	17.8	14.0	5.8	16.1	7.5	0.8	15.3	10.8	3.0	25.1

Table 4: NIAH long-context retrieval results. Single-needle and multi-needle variants evaluated at

T\in\{1\text{K},2\text{K},4\text{K}\}

. PoST significantly improves retrieval for Mamba-2 at both 180M and 440M scales, particularly as context length grows. For Gated DeltaNet, PoST yields a moderate overall improvement. For RWKV-7, whose baseline already achieves strong retrieval, performance remains highly competitive.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: MQAR extrapolation accuracy across equalized per-layer state sizes

\in\{64K,32K,16K\}

(from

d\in\{512,256\}

with varying head count). Each point shows accuracy at

K=T/4

key–value pairs for context length

T

. All models trained at

T=512

; longer lengths are out-of-distribution. Solid lines: PoST variants; dashed: baselines.

Parameter	180M	440M
$d_{\mathrm{model}}$	768	1,024
$d_{\mathrm{inner}}$ ( $=\mathrm{expand}\times d_{\mathrm{model}}$ )	1,536	2,048
Number of layers $L$	24	48
$d_{\mathrm{state}}$	128	128
Head dimension	64	64
Number of heads $h$	24	32

Appendix

Table 5: Mamba-2 model configurations for LM evaluation.

Parameter	Value
Training data	FineWeb-Edu [ 24 ]
Training context length $T_{\mathrm{train}}$	2,048
Tokenizer	Llama-3.1 [ 9 ]
Hardware	$8\times$ H200-SXM
Optimizer	AdamW [ 23 ]
Warmup	1% of total steps

Appendix

Table 6: Training configuration.

	Mamba-2 (Baseline)	Mamba-2 PoST
A initialization	S4D-Real: $\lambda_{k}=-(k{+}1)$	Geometric (Def. 4.4 )
Timescale range at $T_{\mathrm{train}}$	uncontrolled	$[1,T_{\mathrm{train}}]$ (dynamic: $[1,t]$ at position $t$ )
$\Delta_{t}$ initialization	Random (default)	Fixed: $0.05$
Position adaptive	No	Yes

Appendix

Table 7: Mamba-2 initialization comparison.

	RWKV-7 (Baseline)	RWKV-7 PoST
Decay bias init	Power-law + zigzag (official)	Geometric (Def. 4.4 )
$w_{0,k}$ range	$[-5.5,5.5]$ (zigzag)	Eq. 12 (increasing)
Timescale range at $T_{\mathrm{train}}$	uncontrolled (layer-dep.)	$[1,T_{\mathrm{train}}]$ (dynamic: $[1,t]$ at position $t$ )
Taper exponents	—	Cor. 5.1 ; $\alpha_{0}=1$ (slow), $\alpha_{C-1}=0$ (fast)
Position adaptive	No	Yes

Appendix

Table 8: RWKV-7 initialization comparison.

Linear RNNs

Linear recurrent neural networks (RNNs) model sequences with linear state updates, aiming for efficient, parallelizable memory over long contexts.

See all 22 papers

Jul 28, 2026

Raven: High-Recall Sequence Modeling with Sparse Memory Routing

Long-context recall in linear-time sequence models highlights a tradeoff in how they write to memory. State-based linear models, such as state-space models (SSMs) and linear Transformers, write densely, updating the entire state for each newly arrived token, which leads to interference and makes specific past tokens hard to recover. Sliding-window attention (SWA) exhibits the opposite behavior: it writes sparsely by storing explicit token representations, but only within a fixed window, so recall drops once the relevant token is evicted. Interpolating between these models, we introduce Raven, a linear-time sequence model that maintains a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned, input-dependent routing. This lets Raven mitigate SWA's position-based overwriting and hard eviction while reducing interference from dense state updates in SSMs, thereby preserving long-range content much more effectively. Across recall-intensive benchmarks, Raven is competitive with or outperforms prior linear-time baselines, achieving strong long-context recall where both SWA and SSMs sharply degrade. It remains effective when extrapolating to context lengths as large as 16x its training length, with similar gains in hybrid architectures.

Arshia Afzal, Aviv Bick, Eric P. Xing +2

EPFL · Carnegie Mellon University · MBZUAI +1

Memory-Augmented Language Models Long-Context Language Modeling Long-Range Dependency Modeling

Ask PDF

Oct 11, 2026

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ongoing computation and preserve information from an ever-growing history. This coupling has reinforced a prevailing view that the state's memory capacity defines the model's long-context capability. We challenge this presumed ceiling: recurrent computation need not bear the full burden of long-term storage. We introduce NoteBank, a sparse episodic memory that archives selected text before future queries are known, uses the backbone's native features for writing and addressing, and reads retrieved evidence through its native forward computation via replay. This training-free, plug-and-play interface requires no external embedding model, separately trained retriever, or separate neural reader. At one million tokens, it raises Gated DeltaNet's mean accuracy across six RULER retrieval tasks from 1.7% to 70.3%, with no observed significant retrieval degradation from 4K to 1M tokens. This recovery adds

\leqslant

0.025% to analytical FLOPs, with persistent context storage approximately 1/2,600 that of a Transformer's full MHA KV cache at matched width, depth, and precision. Experiments extend from three 1.3B recurrent backbones to Mamba-2-8B and post-trained Qwen3.5-9B, demonstrating gains on LongBench v1/v2 and MemoryAgentBench. Together, these results show that sparse evidence storage and native replay can substantially expand the long-context capability of pretrained linear attention models while retaining the efficiency of recurrent computation.

Li Nanbo, Firas Laakom, Yilan Zhang +10

King Abdullah University of Science and Technology (KAUST)

Long-Context Modeling Memory-Augmented Language Models Linear Attention

Ask PDF

Jul 18, 2026

SpecLA: Efficient Speculative Decoding for Linear-Attention Models

Linear-attention models replace the growing KV cache with recurrent states, but autoregressive decoding still reads, updates, and writes these states one token at a time. Speculative decoding can reduce this cost by verifying several draft tokens in one target pass, yet existing speculative systems are designed for Transformer KV caches. For stateful linear-attention targets, verification must follow recurrent dependencies across chains and branches, acceptance must update only the accepted state trajectory, and the drafter must avoid submitting candidates that waste stateful verification work. This paper presents SpecLA, a speculative decoding runtime for stateful linear-attention models. SpecLA verifies chains and trees with topology-aware kernels, stores compact factors produced during verification to recover accepted states, and uses confidence pruning plus a target-aligned EAGLE-style drafter to feed useful candidates to the verifier. On an NVIDIA H100 with a public GDN-1.3B target, SpecLA achieves up to 1.70x end-to-end speedup over autoregressive decoding.

Zhibin Wang, Xuying Han, Zhaohua Yang +5

State Key Laboratory of Novel Software Technology Nanjing University Nanjing, China · Alibaba Group Hangzhou, China

Speculative Decoding LLM Inference Acceleration Linear Attention

Ask PDF

Abstract

Linear RNNs

Explore similar work

Raven: High-Recall Sequence Modeling with Sparse Memory Routing

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

SpecLA: Efficient Speculative Decoding for Linear-Attention Models