SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models

Qingwen Lin; Boyan Xu; Xiao Liu; Zhifeng Hao; Ruichu Cai

cs.LGAug 23, 2026

SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models

Authors: Qingwen Lin, Boyan Xu, Xiao Liu, Zhifeng Hao, Ruichu Cai

Organizations: School of Computer Science, Guangdong University of Technology · Yuanshi Intelligence · College of Science, Shantou University

Abstract

Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation. By tracking RWKV-7 over sequences of up to 100M tokens, we empirically identify a distinct failure pattern: localized norm explosion atop a relatively sparse substrate, rather than global state saturation. Analysis of the recurrent update suggests that persistent decay keeps weakly updated entries small, whereas uneven injections allow a few channels to accumulate extreme values. Motivated by this diagnosis, we propose State Anomaly Neutralization (SANE), which applies adaptive $\tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure. Within a safe threshold range ( $3 \le α\le 5$ ), SANE matches the baseline on 11 short-context reasoning benchmarks with no statistically significant degradation. After a 100M-token prefix, which exceeds the training length by over $24{,}000\times$ , SANE retains functional reasoning ( $33.46$ -- $35.56$ ) while the baseline encounters numerical overflow. In contrast, overly permissive thresholds ( $α\ge 8$ ) remain numerically stable but lose reasoning capability entirely, showing that numerical stabilization alone does not guarantee functional reasoning and revealing a capacity--stability trade-off in state compression.

Figures & tables

Figure 1: Extreme-context degradation of RWKV-7 on mixed_chat . Top: Local perplexity and its 50-token moving average. Bottom: State Frobenius norm, which eventually exhibits runaway growth beyond $10^{19}$ .

Figure 2: Spatial evolution of RWKV-7 state absolute values across increasing context lengths. Each heatmap shows the absolute state values (flattened across heads and channels) for all 24 layers (L0–L23). Top-left: Within training length (4,096 tokens)—the state remains largely sparse with values near zero (light blue). Top-right: At 528K tokens—localized hotspots (yellow/orange) begin to emerge in specific heads. Bottom-left: At 1.5M tokens—the hotspots expand into concentrated red regions, indicating uncontrolled growth. Bottom-right: At 10M tokens—massive activation spreads across multiple layers, with black regions marking values that exceed the display scale ( $>$ 100). Crucially, the background remains sparse throughout, confirming that the collapse stems from localized norm explosion rather than overall state saturation.

Figure 3: a: RWKV-SANE-TimeMix internals. The $\tau$ branch (green dashed) computes an input-dependent threshold; State Neutralization (green dashed) is applied at chunk boundaries. b: RWKV-SANE architecture. Dashed boxes mark SANE modifications.

Figure 4: Perplexity evolution under extreme-length extrapolation. Left column: Baseline (RWKV-0.4B). Right column: SANE models. Top row: english_math_chat ( $\sim$ 10M tokens). Bottom row: mixed_chat ( $\sim$ 100M tokens). Solid lines denote local perplexity at each token position; dashed lines denote cumulative perplexity smoothed over the prefix. The baseline exhibits unbounded growth in cumulative PPL beyond the training length, with local spikes exceeding 7000; its evaluation on mixed_chat encounters numerical overflow (NaN) beyond 20M tokens, so baseline curves are truncated to the finite region, whereas both SANE models remain finite over the entire stream. SANE ( $\alpha$ =5, blue) maintains nearly flat cumulative PPL throughout, while SANE ( $\alpha$ =10, orange) shows a mild cumulative PPL uptrend at 100M tokens, foreshadowing the collapse observed in Table 1 . Extreme outliers are clipped at 4 $\sigma$ for visualization clarity.

Figure 5: State sparsity preservation in RWKV-SANE ( $\alpha$ =5) across increasing context lengths. Heatmaps show absolute state absolute values (flattened across heads and channels) for all 24 layers. Top-left: 4,096 tokens. Top-right: 528K tokens. Bottom-left: 1.5M tokens. Bottom-right: 9.9M tokens. Crucially, even at $\sim$ 10M tokens, the state remains highly sparse with values concentrated below 1.0 (light blue), and no head exhibits the uncontrolled red/black hotspots seen in the baseline (Figure 2 ).

Model	Ctx	ASDiv	Carpe	CMATH	Col.M	GaoKao	GSM8k	M500	MAWPS	Min.	SVAMP	Weak	Avg
RWKV-0.4B	0	78.87	35.66	60.33	16.04	31.05	57.16	42.0	89.35	4.04	39.4	28.8	43.88
+SANE ( $\alpha$ =1) ${}_{\text{p<0.001}}$	0	75.76	32.27	56.83	15.22	31.62	54.51	37.4	87.22	3.68	36.1	26.2	41.53
+SANE ( $\alpha$ =1) ${}_{\text{p<0.001}}$	100M	69.89	31.56	54.33	13.98	29.91	46.25	32.4	84.46	2.21	25.7	20.7	37.40
+SANE ( $\alpha$ =2) ${}_{\text{p=0.006}}$	0	76.34	33.81	57.83	15.33	32.76	55.27	40.8	88.28	3.31	38.0	27.8	42.68
+SANE ( $\alpha$ =2) ${}_{\text{p=0.006}}$	100M	68.08	31.45	48.0	13.63	30.2	42.15	29.2	84.41	2.57	27.5	16.3	35.77
+SANE ( $\alpha$ =3) ${}_{\text{p=0.539}}$	0	77.83	34.73	58.33	15.58	34.76	56.71	40.2	88.91	4.78	38.5	29.1	43.58

Table 1: Mathematical reasoning performance under short-context (no prefix, Ctx=0) and ultra-long-context (100M prefix) settings. At 100M tokens, the baseline collapses entirely. Paired t-test p-values (relative to the RWKV-0.4B baseline at Ctx=0, across the 11 benchmarks) are reported as subscripts on the model names.

Figure 6: Average accuracy vs. threshold scale

\alpha

. Blue: Ctx=0; red: 100M; dashed gray: baseline (

43.88

). Shaded region: collapse regime (

\alpha{\geq}8

Recurrent Neural Networks

Recurrent neural networks process sequences by updating a persistent hidden state, enabling temporal modeling, memory, and streaming computation.

See all 307 papers

Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Jul 8, 2026Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6Long-Context Retrieval Memory-Augmented Language Models
Meta FAIR · Inria Paris & ENS-PSL University · University of Tübingen
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
Sep 29, 2026Bingchen Yao, Haobo Xu, Haokun Lin +6LLM Quantization LLM Inference Acceleration
Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS +3
OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
May 13, 2026Chenyu Zhou, Hongpei Li, Yuerou Liu +3Transformer Attention Linear Attention
Shanghai Jiao Tong University · Northwestern University · Huazhong University of Science and Technology +1

Jul 8, 2026J/K move · Enter open · S save

Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy, Matthijs Douze+5

Meta FAIR · Inria Paris & ENS-PSL University · University of Tübingen

Long-Context Retrieval Memory-Augmented Language Models Recurrent Neural Networks Delta-Rule Linear Attention In-Context Learning Gated Delta Networks+1

Open paper AskPDF arXiv

Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks. Moreover, by learning the initial state of the SDM memory and therefore using it as a parametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.

Abstract

Recurrent Neural Networks

Explore similar work

Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention