EMA Is Not All You Need: Mapping the Boundary Between Structure and Content in Recurrent Context

Arth Singh

cs.CLMar 17, 2026

EMA Is Not All You Need: Mapping the Boundary Between Structure and Content in Recurrent Context

Authors: Arth Singh

Organizations: AIM Intelligence National Institute of Technology Agartala

Abstract

What exactly do efficient sequence models gain over simple temporal averaging? We use exponential moving average (EMA) traces, the simplest recurrent context (no gating, no content-based retrieval), as a controlled probe to map the boundary between what fixed-coefficient accumulation can and cannot represent. EMA traces encode temporal structure: a Hebbian architecture with multi-timescale traces achieves 96% of a supervised BiGRU on grammatical role assignment with zero labels, surpassing the supervised model on structure-dependent roles. EMA traces destroy token identity: a 130M-parameter language model using only EMA context reaches C4 perplexity 260 (8x GPT-2), and a predictor ablation (replacing the linear predictor with full softmax attention) yields identical loss, localizing the entire gap to the traces. The traces apply lossy, data-independent compression; by the data processing inequality, no downstream predictor can recover the discarded information. Fixed-coefficient accumulation, whether across time or depth, suffers irreversible information dilution that only learned, input-dependent selection can resolve.

Figures & tables

Panel of Figure 1 — Figure 1: One SPEN block. All temporal context enters through fixed-decay EMA traces (red box). No attention is used. The predictor ablation (§ 7 ) confirms the traces, not any downstream component, limit performance.

Representation	$d$	Within	Transfer	Deep
L0 activation	1024	.795	.415	.326
Traces only	2048	.947	.485	.761
L0 + Traces	3072	.960	.618	.840

Table 1: Probing different components of the same trained SPCN model. Traces encode structural information absent from instantaneous activations.

Model	Within	Transfer	Deep	Labels
SPCN (traces)	.960	.618	.840	None
BiGRU	1.000	.762	1.000	All

Table 2: SPCN achieves 96% of a supervised BiGRU with zero labels during training.

Model	Params	Tokens	C4	Wiki
GPT-2 small	124M	$\sim$ 40B	33	29
SPEN	131M	8B	260	729
Unigram	–	–	963	941
Uniform	–	–	50,257

Table 3: Perplexity comparison. The

8\times

gap reflects both the architectural limitation (data-independent EMA context) and a

5\times

data disadvantage (8B vs.

\sim

40B tokens). The predictor ablation (§ 7 ) shows the architectural component dominates.

Predictor	CE	$\Delta$
Static (linear proj.)	7.61	+0.01
Linear attention	7.57	$-$ 0.03
Softmax attention	7.60	—

Table 4: Predictor ablation. All three produce identical CE when reading from EMA traces. The traces are the bottleneck.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Config	C4	Code	C4 $\Delta$	Code $\Delta$
No PGHU	262	693	–	–
Default	29K	20K	+11K%	+2.8K%
$\eta{=}10^{-4}$ , $\pi_{\max}{=}1$	274	678	+4.5%	$-$ 2.2%
$\eta{=}10^{-5}$ , $\pi_{\max}{=}1$	258	697	$-$ 1.7%	+0.6%

Appendix

Table 5: PGHU hyperparameter sweep.

Domain	Start	End	$\Delta$
Code	647	89	$-$ 86%
Math	544	375	$-$ 31%
Medical	245	246	$-$ 0.4%

Appendix

Table 6: Streaming PPL over 10K tokens. Improvement comes from EMA traces reaching steady state.

Sequence Modeling

Sequence modeling studies how to represent, predict, classify, and generate ordered data such as text, time series, events, and biological sequences.

See all 249 papers

Raven: High-Recall Sequence Modeling with Sparse Memory Routing
Jul 28, 2026Arshia Afzal, Aviv Bick, Eric P. Xing +2Memory-Augmented Language Models Long-Context Language Modeling
EPFL · Carnegie Mellon University · MBZUAI +1
Proteus: Incremental Memory Activation for Long-Context Sequence Modeling
Aug 17, 2026Reza Bayat, Ali Behrouz, Vahab Mirrokni +1Long-Context Language Modeling Memory-Augmented Language Models
Mila · Google
History Replays Itself: Unleashing Long-Context Capability in Linear Attention
Oct 11, 2026Li Nanbo, Firas Laakom, Yilan Zhang +10Long-Context Modeling Memory-Augmented Language Models
King Abdullah University of Science and Technology (KAUST)

Jul 28, 2026J/K move · Enter open · S save

Raven: High-Recall Sequence Modeling with Sparse Memory Routing

Arshia Afzal, Aviv Bick, Eric P. Xing, Volkan Cevher+1

EPFL · Carnegie Mellon University · MBZUAI+1

Memory-Augmented Language Models Long-Context Language Modeling Long-Range Dependency Modeling

Open paper AskPDF arXiv

Long-context recall in linear-time sequence models highlights a tradeoff in how they write to memory. State-based linear models, such as state-space models (SSMs) and linear Transformers, write densely, updating the entire state for each newly arrived token, which leads to interference and makes specific past tokens hard to recover. Sliding-window attention (SWA) exhibits the opposite behavior: it writes sparsely by storing explicit token representations, but only within a fixed window, so recall drops once the relevant token is evicted. Interpolating between these models, we introduce Raven, a linear-time sequence model that maintains a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned, input-dependent routing. This lets Raven mitigate SWA's position-based overwriting and hard eviction while reducing interference from dense state updates in SSMs, thereby preserving long-range content much more effectively. Across recall-intensive benchmarks, Raven is competitive with or outperforms prior linear-time baselines, achieving strong long-context recall where both SWA and SSMs sharply degrade. It remains effective when extrapolating to context lengths as large as 16x its training length, with similar gains in hybrid architectures.

Abstract

Sequence Modeling

Explore similar work

Raven: High-Recall Sequence Modeling with Sparse Memory Routing

Proteus: Incremental Memory Activation for Long-Context Sequence Modeling

History Replays Itself: Unleashing Long-Context Capability in Linear Attention