Delta-Rule recurrent models maintain a fixed-size state, enabling O(1) inference memory but potentially becoming unstable under extreme-context extrapolation. By tracking RWKV-7 over sequences of up to 100M tokens, we empirically identify a distinct failure pattern: localized norm explosion atop a relatively sparse substrate, rather than global state saturation. Analysis of the recurrent update suggests that persistent decay keeps weakly updated entries small, whereas uneven injections allow a few channels to accumulate extreme values. Motivated by this diagnosis, we propose State Anomaly Neutralization (SANE), which applies adaptive tanh compression at chunk boundaries while preserving the intra-chunk parallel structure. Within a safe threshold range (3≤α≤5), SANE matches the baseline on 11 short-context reasoning benchmarks with no statistically significant degradation. After a 100M-token prefix, which exceeds the training length by over 24,000×, SANE retains functional reasoning (33.46--35.56) while the baseline encounters numerical overflow. In contrast, overly permissive thresholds (α≥8) remain numerically stable but lose reasoning capability entirely, showing that numerical stabilization alone does not guarantee functional reasoning and revealing a capacity--stability trade-off in state compression.
Figures & tables
Figure 1: Extreme-context degradation of RWKV-7 on mixed_chat . Top: Local perplexity and its 50-token moving average. Bottom: State Frobenius norm, which eventually exhibits runaway growth beyond 1019 .
Figure 2: Spatial evolution of RWKV-7 state absolute values across increasing context lengths. Each heatmap shows the absolute state values (flattened across heads and channels) for all 24 layers (L0–L23). Top-left: Within training length (4,096 tokens)—the state remains largely sparse with values near zero (light blue). Top-right: At 528K tokens—localized hotspots (yellow/orange) begin to emerge in specific heads. Bottom-left: At 1.5M tokens—the hotspots expand into concentrated red regions, indicating uncontrolled growth. Bottom-right: At 10M tokens—massive activation spreads across multiple layers, with black regions marking values that exceed the display scale ( > 100). Crucially, the background remains sparse throughout, confirming that the collapse stems from localized norm explosion rather than overall state saturation.
Figure 3: a: RWKV-SANE-TimeMix internals. The τ branch (green dashed) computes an input-dependent threshold; State Neutralization (green dashed) is applied at chunk boundaries. b: RWKV-SANE architecture. Dashed boxes mark SANE modifications.
Figure 4: Perplexity evolution under extreme-length extrapolation. Left column: Baseline (RWKV-0.4B). Right column: SANE models. Top row: english_math_chat ( ∼ 10M tokens). Bottom row: mixed_chat ( ∼ 100M tokens). Solid lines denote local perplexity at each token position; dashed lines denote cumulative perplexity smoothed over the prefix. The baseline exhibits unbounded growth in cumulative PPL beyond the training length, with local spikes exceeding 7000; its evaluation on mixed_chat encounters numerical overflow (NaN) beyond 20M tokens, so baseline curves are truncated to the finite region, whereas both SANE models remain finite over the entire stream. SANE ( α =5, blue) maintains nearly flat cumulative PPL throughout, while SANE ( α =10, orange) shows a mild cumulative PPL uptrend at 100M tokens, foreshadowing the collapse observed in Table 1 . Extreme outliers are clipped at 4 σ for visualization clarity.
Figure 5: State sparsity preservation in RWKV-SANE ( α =5) across increasing context lengths. Heatmaps show absolute state absolute values (flattened across heads and channels) for all 24 layers. Top-left: 4,096 tokens. Top-right: 528K tokens. Bottom-left: 1.5M tokens. Bottom-right: 9.9M tokens. Crucially, even at ∼ 10M tokens, the state remains highly sparse with values concentrated below 1.0 (light blue), and no head exhibits the uncontrolled red/black hotspots seen in the baseline (Figure 2 ).
Model
Ctx
ASDiv
Carpe
CMATH
Col.M
GaoKao
GSM8k
M500
MAWPS
Min.
SVAMP
Weak
Avg
RWKV-0.4B
0
78.87
35.66
60.33
16.04
31.05
57.16
42.0
89.35
4.04
39.4
28.8
43.88
+SANE ( α =1) p<0.001
0
75.76
32.27
56.83
15.22
31.62
54.51
37.4
87.22
3.68
36.1
26.2
41.53
100M
69.89
31.56
54.33
13.98
29.91
46.25
32.4
84.46
2.21
25.7
20.7
37.40
+SANE ( α =2) p=0.006
0
76.34
33.81
57.83
15.33
32.76
55.27
40.8
88.28
3.31
38.0
27.8
42.68
100M
68.08
31.45
48.0
13.63
30.2
42.15
29.2
84.41
2.57
27.5
16.3
35.77
+SANE ( α =3) p=0.539
0
77.83
34.73
58.33
15.58
34.76
56.71
40.2
88.91
4.78
38.5
29.1
43.58
Table 1: Mathematical reasoning performance under short-context (no prefix, Ctx=0) and ultra-long-context (100M prefix) settings. At 100M tokens, the baseline collapses entirely. Paired t-test p-values (relative to the RWKV-0.4B baseline at Ctx=0, across the 11 benchmarks) are reported as subscripts on the model names.
Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks. Moreover, by learning the initial state of the SDM memory and therefore using it as a parametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.
Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6
Meta FAIR · Inria Paris & ENS-PSL University · University of Tübingen
Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates. We discover that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Motivated by these observations, we propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across both long- and short-generation benchmarks show that STEPQuant closely matches FP32-state accuracy under a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%. Our code is available at https://github.com/Dreamer-Toby/STEPQuant.
Bingchen Yao, Haobo Xu, Haokun Lin +6
Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS +3
Linear attention and state-space models offer constant-memory alternatives to softmax attention, but often struggle with in-context associative recall. The Delta Rule mitigates this by writing each token via one step of online gradient descent. However, its step size relies on a single scalar gate that ignores the feature-wise curvature of the inner objective. We propose Online Scaled DeltaNet (OSDN), which augments the scalar gate with a diagonal preconditioner updated online via hypergradient feedback. Crucially, this right-preconditioning is algebraically equivalent to a per-feature scaling of the write-side key. This equivalence allows OSDN to strictly preserve the hardware-friendly chunkwise parallel pipeline of DeltaNet without incurring high-dimensional state overhead. Theoretically, by exploiting the exact-quadratic structure of the inner regression loss, we establish super-geometric convergence against a right-Newton comparator and prove an algorithm-aligned token-local residual contraction bound. To handle non-stationary contexts, we further introduce Adaptive Preconditioner Forgetting (APF) to dynamically refresh stale calibration. Empirically, OSDN demonstrates strong performance across scales. At the 340M-parameter scale, OSDN improves JRT-style in-context recall by 32% over DeltaNet. Scaling to 1.3B parameters, it achieves a 39% reduction in the recall residual ratio while maintaining parity on general downstream tasks (e.g., perplexity and LongBench) -- demonstrating that our online-preconditioning mechanism effectively transfers and amplifies at the billion-parameter scale.
Chenyu Zhou, Hongpei Li, Yuerou Liu +3
Shanghai Jiao Tong University · Northwestern University · Huazhong University of Science and Technology +1