Long-Range Dependency Modeling

Follow this topic Ask about this topic

Latest papers

27 papers6 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Oct 12, 2026

Limit Theory for Online Expectile Estimation under Short and Long Range Dependence

Expectiles are tail-sensitive distributional summaries with important applications in financial risk measurement and forecasting, yet statistical theory for their online estimation from dependent data remains limited. We study a one-pass, stochastic-gradient algorithm with constant learning rate

γ

for estimating the marginal expectile of a stationary causal process. The recursion admits a unique stationary causal solution and geometrically forgets its initialization. Under short-range dependence, its last iterate is asymptotically Gaussian at the

\sqrtγ

scale, with variance governed by the population curvature and the long-run variance of the expectile score. For long-memory linear processes the scale changes to

γ^{β-1/2}\ell(1/γ)

, with a Gaussian limit arising from a fractional Brownian motion integral. We further derive the joint long-memory limit for multiple expectile levels and characterize its limiting covariance structure. Various numerical experiments corroborate the predicted Gaussian approximations and regime-specific fluctuation scales.

Younghun Lee, Jake Zummo, Wei Biao Wu

Department of Statistics, University of Chicago

Long-Range Dependency Modeling Stochastic Approximation

Oct 9, 2026

The RNN Homunculus: Learning to Gate Long-Term Memory

Learning long-range dependencies with recurrent neural networks (RNNs) is difficult because of the vanishing gradient problem. Gated recurrent neural networks like GRUs and LSTMs address this problem and are commonly explained by their ability to create additive copy paths through time. This explanation is correct, but incomplete: useful RNN gate values must themselves be learned, which is typically done with another RNN. We study this gate-learning problem through carefully controlled sequence classification tasks. Our analysis shows that gated architectures struggle when all training samples require long-range memory and thus equally suffer from the vanishing gradient problem. We intervene by adding short-dependency samples which let gated models learn solutions that capture long-dependency samples, but also generalize across different class labels. When properly learned, gated RNNs match or even exceed the test accuracy of modern architectures like Mamba and xLSTM. Overall, our results highlight gate learning as a central part of learning long-range memory and show how training data can support this process by exposing the same task over shorter dependencies.

Zeyu Fan, Alexandru-Dragos Manolache, Jan van Gemert

Delft University of Technology, Mekelweg 5, 2628 CD Delft, The Netherlands

Long-Range Dependency Modeling Recurrent Neural Networks Sequence Modeling Gated Recurrent Unit Long Short-Term Memory Networks Deep Learning Optimization

Oct 9, 2026

Double-Take: Improving Capacity, Binding, and Selectivity In Long-Context Within Bidirectional Memory

Processing a long sequence does not guarantee that distant information remains usable. As context grows, a fixed-size memory must retain many associations, connect information across positions, and preserve relevant structure without being overwhelmed by everything it observes. These are distinct requirements: a memory may store more information without representing the relationships needed for retrieval, and it may transport those relationships without preserving the ones that matter. Thus, the challenge is not simply extending context length, but constructing a memory that remains usable as context grows. We study this problem in large-chunk test-time training (TTT) and identify two basic failures: single-step writes accumulate interference, limiting capacity, while self-associative writes cannot form cross-position bindings. We introduce Double-Take (DT), which repairs these failures with multi-step unpinned updates and cross-position mixing, and adds bidirectional change-sensitive writes for selective retention. Capacity alone leaves end-to-end recall at the

0.10

chance floor; adding binding raises it to

0.92

. Trained only on

32\,\mathrm{kb}

genomic contexts, DT reaches

0.927

AUROC at

131\,\mathrm{kb}

and

0.856

near

1\,\mathrm{Mb}

, while transferring zero-shot from human to mouse. Together, these results identify capacity, binding, and selectivity as three requirements for usable long-range memory.

Vakul Raghavan, Utkarsh Grover, Mingyang Mao +2

Department of Mathematics and Computer Science, Rutgers University, Newark, NJ, USA · Embodied Robotics and Automation Lab, University of South Florida, Tampa, FL, USA

Long-Context Modeling Memory-Augmented Neural Networks Long-Range Dependency Modeling Associative Memory Sequence Modeling Test-Time Training

Oct 4, 2026

LogSig-SSM: Time-Series Modelling with Multi-Scale Log-Signature Compression for State-Space Models

Time-series data are often sampled irregularly at high frequencies and exhibit long-range dependencies, which makes long-horizon modelling difficult. Continuous-time models such as neural controlled differential equations (NCDEs) and neural rough differential equations (NRDEs) can handle irregular sampling, but they scale poorly to long sequences. Selective state-space models (SSMs) such as Mamba scale linearly with sequence length, but they provide limited recurrent mixing across hidden dimensions within a single block. We propose LogSig-SSM (Log-Signature Compression for State-Space Models), which first compresses long multivariate time series into a shorter sequence of tokens using multi-scale windowed log-signatures, and then processes these tokens with a selective SSM backbone. LogSig-SSM is scalable and robust to irregular sampling, combining log-signature tokens that capture higher-order cross-channel interactions with a selective SSM that models long-range dependencies. The model also admits a continuous-time interpretation as an NCDE/NRDE-style system driven by a log-signature-based input, in which selectivity induces an input-dependent rescaling of the latent dynamics. Across four benchmarks, namely long-sequence classification on UEA, high-frequency physiological regression on PPG-DaLiA, multivariate weather forecasting, and irregularly sampled clinical prediction on PhysioNet Sepsis, LogSig-SSM outperforms or matches strong SSM and continuous-time baselines while training up to

30\times

faster and using up to

37\times

less GPU memory than Mamba on the longest sequences.

Felix Oury, Nicolas Calvo Peiro, Reiko J. Tanaka

Department of Computing Imperial College London · Department of Bioengineering Imperial College London

Irregular Time-Series Modeling Multivariate Time Series Forecasting Selective SSMs Time Series Classification Multiscale Time-Series Modeling Long-Range Dependency Modeling

Sep 30, 2026

Memorizon: Training World Models Beyond Their Context Window

Streaming world models should render a place consistently across repeated visits. Directly supervising such revisits requires training samples that capture both visits, often spanning minutes. Yet dense attention over the full span incurs quadratic costs, making long-span supervision expensive. Memorizon breaks this coupling: long spans are needed for supervision, but not for attention, since the two visits can share a forward pass without including every intervening frame. A training sample covers a span of any length but is scored only on its last

k

chunks. Instead of tokenizing the history before them, each scored chunk retrieves its own top-

K

latents by camera co-visibility, and the union of these requests forms a shared bank. The bank is bounded by

kK

, so the sequence stays bounded however long the span; at the shortest span the recipe is exactly conventional training. Adding the bank raises the cost of a step once; beyond that, a longer span costs little, and going from 100 to 400 s adds 12% to the step time. Against a sliding-window baseline, retrieval raises revisit consistency on every split, and a span long enough to reach the first visit of each return adds a further 24% to 30%, at some cost in image quality; beyond that span, more length no longer helps. Filling the bank from another episode lowers revisit correlation by 83%, so the model uses what it retrieves. Project page: https://tingtingliao.github.io/memorizon

Tingting Liao, Xuezhi Liang, Hao Li +1

IFM, MBZUAI · MBZUAI

World Model Learning World Models Long-Range Dependency Modeling

Sep 29, 2026

Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling

Recurrent neural networks (RNNs) compress the historical context into a memory state of fixed size, thus allowing for constant-time inference. The memory state size is a crucial factor in their performance, as exemplified by the strong performance and resurgence of linear attention, which extends the vector-valued hidden states of ordinary RNNs to matrix-valued hidden states. Crucially, linear attention does so in a parameter-efficient way, in particular by using an outer product of the key and value vectors to write to the matrix-valued hidden state. We generalize this construction and propose triadic linear attention, which writes the triadic outer product of a key, a second key, and a value, into a third-order (i.e., 3D) tensor state, and reads from it by contracting both key axes with two queries. An

E

-dimensional second key thus yields an

E

-fold increase in state size while adding only two projections. Triadic linear attention is compatible with data-dependent forgetting, the delta rule, and chunkwise-parallel training. Applied to Gated DeltaNet and scalar-gated linear attention, triadic linear attention substantially improves long-context language modeling and recall, outperforming alternatives that enlarge the state.

Oliver Sieberling, Bharat Runwal, David Jin +3

Massachusetts Institute of Technology · MIT-IBM Computing Research Lab

Linear Attention Long-Range Dependency Modeling Long-Context Language Modeling

Sep 16, 2026

Learning Fractional-Order Dynamics from a Single Trajectory

Many real-world processes exhibit long-range dependence, where the current state depends on a slowly decaying trace of past states rather than on the most recent state alone. This paper studies system identification for discrete-time fractional-order linear time-invariant systems from a single observed trajectory of length

t

, a setting that captures such non-Markovian dynamics through the Grünwald--Letnikov difference operator. Unlike Markovian systems, fractional-order systems couple estimation across the entire history, making both statistical analysis and practical identification more challenging. We propose Fractional-Order Ordinary-Least-Squares Grid-Search (FO-GS), a simple two-stage estimator that exploits the diagonal structure of the fractional-difference operator to decouple the identification problem row-wise. Under the stability assumption, we establish high-probability, non-asymptotic error bounds for estimating both the fractional order and the system matrix in the heterogeneous setting, with both estimation errors scaling as

\mathcal{O}(t^{-1/2})

. Through experiments, we show that FO-GS outperforms existing baselines in recovering both the fractional order and the underlying system dynamics.

Xiaole Zhang, Ziyi Zhang, Zehao Zhao +4

Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California · Department of Electrical and Computer Engineering, Carnegie Mellon University

Dynamical Systems Long-Range Dependency Modeling System Identification

Sep 8, 2026

Learning Length-Extrapolatable Recurrent Models

Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state credit and propose Credit Stabilization through Time (CST). During backward propagation, CST locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, we specialize CST to each regime. In both settings, CST improves performance beyond the training horizon, with gains observed at up to 128x the training length.

Hanwen Jiang

Adobe Research

Backpropagation Temporal Credit Assignment Recurrent Neural Networks Length Generalization Long-Context Modeling Long-Range Dependency Modeling

Aug 27, 2026

Informational Antilocality and the Locality Bias in LLMs

We consider the ability of transformer-based language models (LLMs) to learn what we call k-antilocal languages, i.e., languages that have no mutual information across any span of

k

contiguous symbols. We construct such languages with increasing

k

, finding that LLMs trained on them achieve comparable cross-entropy loss regardless of antilocality, but converge more slowly on more antilocal languages. Our findings support the idea that non-local dependencies are more difficult to learn, but the evidence for this bias comes from learning speed rather than learning success.

Andrew McInnerney, Shane Storks, Steven Abney +1

University of Michigan · Eastern Michigan University

Long-Range Dependency Modeling Language Modeling Autoregressive Language Modeling

Jul 28, 2026

A Spatial Persistence Gradient in European Warming Consistent with North Atlantic Cold-Blob Influence

Europe is warming faster than the global mean, yet the spatial organisation of this acceleration remains incompletely understood. Using ERA5 reanalysis for 1950--2024 across 28 IPCC AR6 European sub-regions, we identify two connected empirical results. First, the DFA1 Hurst exponent of interannual temperature residuals is strongly and negatively associated with the 1996--2024 warming rate (

r=-0.792

,

p=5.1\times10^{-7}

). High-persistence, mainly Atlantic-proximal regions warm more slowly, whereas low-persistence continental regions warm faster. This relationship is robust to five residualisation schemes, three memory estimators, leave-one-region-out analysis, and five null-test families, including spatial block permutation. It also persists across warming windows (

r=-0.550

for 1981--2024,

-0.792

for 1996--2024, and

-0.875

for 2000--2024), but vanishes under DFA2, indicating that the signal lies in low-frequency interannual-to-decadal persistence rather than trend curvature. The pattern is consistent with, but does not prove, North Atlantic cold-blob and thermohaline influence on European land temperatures. Second, under a strict 2006--2024 holdout, contemporaneous Mediterranean SST reduces mean annual temperature RMSE by 43% (from

0.787

to

0.449,^{\circ}\mathrm{C}

). A causal lag-weight predictor based on prior-year Mediterranean and Atlantic SST also outperforms AR(2) (

0.578

versus

0.695,^{\circ}\mathrm{C}

) and remains informative after removing NAO, AO, and PNA effects. Similar skill from five-year moving-average and exponentially weighted predictors shows that short Mediterranean SST persistence at 1--5-year lags is the key predictive ingredient. Together, the results support a two-regime interpretation: Atlantic-proximal regions exhibit stronger memory and oceanic buffering, while continental interiors show faster warming and weaker interannual persistence.

Mauricio Herrera-Marín, Alex Godoy-Faúndez, Diego Rivera

Research Center on Sustainability and Strategic Resource Management (CISGER), Faculty of Engineering, Universidad del Desarrollo, Avda. Plaza 700, Santiago, Chile.

Climate Science Long-Range Dependency Modeling

Jul 28, 2026

Raven: High-Recall Sequence Modeling with Sparse Memory Routing

Long-context recall in linear-time sequence models highlights a tradeoff in how they write to memory. State-based linear models, such as state-space models (SSMs) and linear Transformers, write densely, updating the entire state for each newly arrived token, which leads to interference and makes specific past tokens hard to recover. Sliding-window attention (SWA) exhibits the opposite behavior: it writes sparsely by storing explicit token representations, but only within a fixed window, so recall drops once the relevant token is evicted. Interpolating between these models, we introduce Raven, a linear-time sequence model that maintains a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned, input-dependent routing. This lets Raven mitigate SWA's position-based overwriting and hard eviction while reducing interference from dense state updates in SSMs, thereby preserving long-range content much more effectively. Across recall-intensive benchmarks, Raven is competitive with or outperforms prior linear-time baselines, achieving strong long-context recall where both SWA and SSMs sharply degrade. It remains effective when extrapolating to context lengths as large as 16x its training length, with similar gains in hybrid architectures.

Arshia Afzal, Aviv Bick, Eric P. Xing +2

EPFL · Carnegie Mellon University · MBZUAI +1

Memory-Augmented Language Models Long-Context Language Modeling Long-Range Dependency Modeling

Jul 28, 2026

A scaling law of contextual persistence in human language

Human language exhibits lawful structure at the level of words (frequency, vocabulary growth) and word pairs (co-occurrence across distance). Here we show that the arrangement of words in sequence -- a central determinant of meaning -- obeys a comparable law. Using large language models as probabilistic probes, we measured the reduction in target perplexity conferred by prior context at distance d beyond that of the same words scrambled; this difference, the contextual persistence function P(d), isolates the influence of arrangement. Across ten corpora spanning six language families and written and spoken modalities, P(d) decayed approximately as 1/d (

P(d) \propto d^{-α}

, mean

α= 1.04

; median

r^2 = 0.96

). The effect vanished in scrambled and synthetic controls, replicated across independent probes, and did not appear in genomic or protein sequences under domain-native models. An exponent near 1 distributes contextual influence approximately uniformly across logarithmic timescales. The results establish a scaling law of contextual persistence in human language.

Elan Barenholtz

Machine Perception and Cognitive Robotics Laboratory, Department of Psychology and Center for Complex Systems, Florida Atlantic University

Long-Range Dependency Modeling Sequence Modeling

Jul 23, 2026

Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory

Long-sequence memory tracking places two opposing demands on a recurrent state: near-lossless retention of stored bindings over long horizons, and active overwriting of stale ones. In our diagnostic suite, the strongest efficient baselines tend to solve only one side well. Continuous-time-parameterized state-space models (SSMs) such as Mamba obtain their discrete recurrence by zero-order-hold discretization of a continuous-time system; we argue that this detour is unnecessary for memory tracking and parameterize the discrete transition directly. Naju (Native Adaptive Junction Unit) factorizes the recurrent update, schematically

x_n = f_n\odot x_{n-1} + i_n\odot(B_n u_n)

, into an explicit discrete pole (a learned forget gate

f_n

), an independent write gain

i_n

, and input-dependent write/read maps. Since the sigmoid pole satisfies

0<f_n<1

, each frozen local coordinate is Schur-stable by construction, and the full time-varying recurrence satisfies a fading-memory/BIBO bound under uniform boundedness assumptions, with no stability regularizer. We formalize the key structural limitation of coupled designs: any non-expansive complementary single-gate recurrence ties the effective retention

r

and write gain

w

through

|r|+w\le 1

, so near-complete retention forces weak writing; decoupling

f_n

from

i_n

removes this constraint. Empirically, Naju is the only evaluated model that remains strong on both retention and overwriting at 4x the training length. Beyond the diagnostic suite, we evaluate Naju on WikiText-103 language modeling, Long Range Arena, and multi-query associative recall. Across these settings, Naju consistently combines strong long-range memory with competitive or superior performance, outperforming the Mamba baselines in the principal comparisons while remaining competitive with the Transformer and preserving linear-time, linear-memory scaling.

Hyuk Lim, Seunghyun Yoon

Korea Institute of Energy Technology (KENTECH)

Persistent Memory for Language Models Long-Context Modeling State Space Models Long-Range Dependency Modeling

Jun 28, 2026

Anti-Collapse Dynamics and the Emergence of Multi-Time-Scale Learning in Recurrent Neural Networks

Long-range learning is hard for recurrent networks trained with stochastic gradient descent, because the influence of a past input fades with the lag

\ell

, and if it fades too fast the dependence cannot be learned from finite data. This fade is captured by an envelope

f(\ell)

. An exponential fade makes the data needed to learn a lag-

\ell

dependence grow exponentially, putting long horizons out of reach; a power-law fade keeps the cost polynomial. We show that the asymptotic decay class of

f(\ell)

is not fixed by the architecture. Instead, it emerges from the coupling between the state dynamics and parameter dynamics, settling into either a collapsed regime (fast, exponential forgetting) or an extended, anti-collapsed regime (slow, power-law forgetting). The intuition is a competition within these coupled dynamics. Training drives the network's effective time scales toward short ones, while rare, heavy-tailed fluctuations of the learning dynamics push a few of them to very long values. The extended regime survives only when these heavy-tailed pushes are strong enough to balance the pull. We make this mathematically precise with a coarse-grained stochastic process and prove exactly when the extended regime exists. A single exponent, the spectral exponent~

β

, then governs both the spread of time scales and how slowly the network forgets. Realizing the regime in practice needs one more ingredient: the joint action of the architecture and the optimizer must be able to hold such a broad spread. A network whose capacity to generate broad time-scale spectra is severely constrained still collapses, even when supplied with strong heavy-tailed forcing. Heavy-tailed fluctuations thus act not as noise to be suppressed, but as the mechanism that sustains long-range learning.

Lorenzo Livi

OPIT – Open Institute of Technology

Heavy-Tailed Noise Recurrent Neural Networks Long-Range Dependency Modeling Neural Network Training Dynamics Sequence Modeling

Jun 28, 2026

How Token Influence Decays with Distance: A Green-Function View of Trained Language Models

We study how the next-token prediction of an autoregressive Transformer language model changes under small perturbations of earlier input token embeddings. Motivated by operator learning and iterative solvers for differential equations, we investigate how the influence of one token on another decays with distance in a trained model. In multilevel methods for differential equations, such as domain decomposition, multigrid, and multilevel preconditioning, one often exploits a separation between strong local interactions and weaker but essential global interactions. The latter correspond to the long tail of the Green's function and are typically handled by a coarse-level operator. Inspired by this perspective, we compute an empirical, distance-resolved gradient profile of token dependencies using autograd. Experiments on trained Pythia models and Qwen2.5-0.5B show that, over the measured distance range, the median Jacobian sensitivity is much better described by a power-law-type decay than by an exponential alternative: the diagonal-normalized profile is well described by

\overline G(r) \approx γ+β(r+1)^{-p}

with exponents

p \approx 0.7

--

0.9

(typically

0.8

--

0.9

). This behavior appears on coherent text from Gutenberg and WikiText-103. Token-shuffling experiments show that the power-law profile persists even when syntax and prediction quality collapse, whereas randomly initialized models do not exhibit it. The slowly decaying long-range sensitivity thus appears to be a learned property of trained autoregressive Transformer operators. These findings suggest that hierarchical or coarse-level mechanisms in language models may be able to exploit the long-tailed sensitivity profiles.

Matthias Brändel, Stephan Köhler, Oliver Rheinbach

Faculty of Mathematics and Computer Science Technische Universität Bergakademie Freiberg, Freiberg, Germany

Transformer Interpretability Autoregressive Language Modeling Next-Token Prediction Long-Context Modeling Long-Range Dependency Modeling

Jun 23, 2026

ATMA: Length-Invariant Language Modeling via Polar Attention and Gated-Delta Compression Memory

Modern large language models based on softmax scaled-dot-product attention are constrained by their training sequence length: as the key-value sequence grows, softmax probability mass can dilute across a wider distribution, inducing activation shift and long-context performance collapse. Moreover, long-context language modeling faces a structural tension: a sliding-window attention core maintains a bounded local representation and low perplexity but is blind to long-range dependencies, while full-context attention preserves global recall but suffers from out-of-distribution perplexity explosion. To resolve these limitations, we introduce ATMA, a hybrid convolutional-attention architecture that integrates a novel three-channel attention mechanism. ATMA factorizes the attention mixing step into: (1) a count-blind, unit-vector direction channel, (2) a bounded magnitude channel driven by the participation ratio of effective matches over an extreme-value-corrected null sink, and (3) a long-term recurrent compression memory optimized via a gated-delta fast-weights rule. Neither the Polar Attention core nor the recurrent memory is sufficient alone; their combination enables monotonic perplexity reduction and high-fidelity long-range retrieval simultaneously. We evaluate ATMA using a 120-run factorial ablation sweep, demonstrating that the combined Polar + memory model maintains induction needle-in-a-haystack retrieval accuracy above 90% out to 64K tokens (32 times the training length of 2K) while its document perplexity improves monotonically, outperforming softmax-based memory baselines which collapse at extreme context lengths. Code: https://github.com/kreasof-ai/atma

Habibullah Akbar

Kreasof AI Jakarta, Indonesia

Hybrid Attention Long-Context Language Modeling Persistent Memory for Language Models Memory-Augmented Language Models Length Generalization Attention Mechanisms+2

Jun 4, 2026

Pretraining Recurrent Networks without Recurrence

Training recurrent neural networks (RNNs) requires assigning credit across long sequences of computations. Standard backpropagation through time (BPTT) addresses this problem poorly: it is sequential in time, limiting parallelism, and suffers from vanishing or exploding gradients, making long-range associations difficult to learn. We propose Supervised Memory Training (SMT), a method for training nonlinear RNNs that sidesteps recurrent credit propagation entirely by reducing RNN training to supervised learning on one-step memory transition labels

(m_t, x_{t+1}) \rightarrow m_{t+1}

. SMT acquires these memory labels by training a Transformer-based encoder on a predictive state objective--retaining only information from the past necessary to predict the future. By decoupling what to remember from how to update memory, SMT enables time-parallel RNN training with a stable

O(1)

length gradient path between any two tokens--without ever unrolling the RNN. We find that SMT outperforms BPTT when pretraining various RNN architectures on tasks like language modeling and pixel sequence modeling. SMT enables nonlinear RNNs to better capture long-range dependencies and train in parallel, potentially unlocking the scaling of models that build temporal abstractions of past experience.

Akarsh Kumar, Phillip Isola

MIT

Recurrent Neural Networks Recurrent Language Models Long-Range Dependency Modeling Sequence Prediction

May 24, 2026

TGFormer: Towards Temporal Graph Transformer with Auto-Correlation Mechanism

The growing interest in Temporal Graph Neural Networks (TGNNs) stems from their ability to model complex dynamics and deliver superior performance. However, TGNNs encounter fundamental challenges in capturing long-term dependencies and identifying periodic patterns. To address these limitations, we propose TGFormer, a novel Transformer architecture specifically designed for temporal graphs. Our model redefines temporal graph learning by establishing a trajectory framework that aligns with time series analysis principles. This approach allows TGFormer to derive node representations through systematic analysis of historical interactions, enabling granular examination of node relationships across sequential timestamps. Building upon stochastic process theory, we develop an auto-correlation mechanism that systematically uncovers periodic dependencies in node interactions. This innovation empowers TGFormer to perform dependency discovery and representation aggregation at sub-interaction levels, demonstrating superior efficiency and accuracy compared to conventional attention mechanisms. Experimental validation across six public benchmarks confirms the effectiveness of our approach, with TGFormer at most achieving 9.35% precision improvement compared to state-of-the-art approaches.

Hongjiang Chen, Pengfei Jiao, Ming Du +4

Hangzhou Dianzi University, School of Cyberspace, Hangzhou, 310018, China · Tianjin University, College of Intelligence and Computing, Tianjin, 300072, China · State Key Laboratory of Systems Medicine for Cancer, Shanghai Cancer Institute, Shanghai, 200127, China

Temporal GNNs Long-Range Dependency Modeling Sequence Modeling

Apr 20, 2026

Sessa: Selective State Space Attention

Modern sequence modeling is dominated by two families: Transformers, whose self-attention can access arbitrary elements of the visible sequence, and structured state-space models, which propagate information through an explicit recurrent state. These mechanisms face different limitations on long contexts: when attention is diffuse, the influence of individual tokens is diluted across the effective support, while recurrent state propagation can lose long-range sensitivity unless information is actively preserved. As a result, both mechanisms face challenges in preserving and selectively retrieving information over long contexts. We propose Sessa, a decoder that places attention inside a recurrent feedback path. This creates many attention-based paths through which past tokens can influence future states, rather than relying on a single attention read or a single recurrent chain. We prove that, under explicit assumptions and matched regimes, Sessa admits power-law memory tails

O(\ell^{-β})

for

0 < β< 1

, with slower decay than in the corresponding Transformer and Mamba-style baselines. We further give an explicit construction that achieves this power-law rate. Under the same assumptions, Sessa is the only model class among those considered that realizes flexible selective retrieval, including profiles whose influence does not decay with distance. Consistent with this theoretical advantage, across matched experiments, Sessa achieves the strongest performance on long-context benchmarks while remaining competitive with Transformer and Mamba-style baselines on short-context language modeling.

Liubomyr Horbatko

State Space Models Long-Context Language Modeling Persistent Memory for Language Models Attention Mechanisms Self-Attention Recurrent Transformers+2

Apr 12, 2026

A Queueing-Theoretic Framework for Dynamic Attack Surfaces: Data-Integrated Risk Analysis and Adaptive Defense

We develop a queueing-theoretic framework to model the temporal evolution of cyber-attack surfaces, where the number of active vulnerabilities is represented as the backlog of a queue. Vulnerabilities arrive as they are discovered or created, and leave the system when they are patched or successfully exploited. Building on this model, we study how automation affects attack and defense dynamics by introducing an AI amplification factor that scales arrival, exploit, and patching rates. Our analysis shows that even symmetric automation can increase the rate of successful exploits. We validate the model using vulnerability data collected from an open source software supply chain and show that it closely matches real-world attack surface dynamics. Empirical results reveal heavy-tailed patching times, which we prove induce long-range dependence in vulnerability backlog and help explain persistent cyber risk. Utilizing our queueing abstraction for the attack surface, we develop a systematic approach for cyber risk mitigation. We formulate the dynamic defense problem as a constrained Markov decision process with resource-budget and switching-cost constraints, and develop a reinforcement learning (RL) algorithm that achieves provably near-optimal regret. Numerical experiments validate the approach and demonstrate that our adaptive RL-based defense policies significantly reduce successful exploits and mitigate heavy-tail queue events. Using trace-driven experiments on the ARVO dataset, we show that the proposed RL-based defense policy reduces the average number of active vulnerabilities in a software supply chain by over 90% compared to existing defense practices, without increasing the overall maintenance budget. Our results allow defenders to quantify cumulative exposure risk under long-range dependent attack dynamics and to design adaptive defense strategies with provable efficiency.

Jihyeon Yun, Abdullah Yasin Etcibasi, Ming Shi +1

University at Buffalo (SUNY), USA.

Long-Range Dependency Modeling Queueing Theory Cybersecurity Autonomous Cyber Defense Software Supply Chain Security

Apr 8, 2026

Optimal Decay Spectra for Linear Recurrences

Linear recurrent models offer linear-time sequence processing but often suffer from suboptimal long-range memory. We trace this to the decay spectrum: for

N

channels, random initialization collapses the minimum spectral gap to

O(N^{-2})

, yielding sub-exponential error

\exp(-Ω(N/\log N))

; linear spacing avoids collapse but degrades to

\exp(-O(N/\sqrt{T}))

, practically algebraic over long contexts. We introduce Position-Adaptive Spectral Tapering (PoST), an architecture-agnostic framework combining two mechanisms: (1) Spectral Reparameterization, which structurally enforces geometrically spaced log-decay rates, proven minimax optimal at rate

O(\exp(-cN/\log T))

; and (2) Position-Adaptive Scaling, the provably unique mechanism that eliminates the scale mismatch of static spectra (where only

N\log t/\log T

of

N

channels are effective at position

t

) by stretching the spectrum to the actual dependency range, sharpening the rate to

O(\exp(-cN/\log t))

. This scaling natively induces fractional invariance: the impulse response becomes scale-free, with channels interpolating between relative and absolute temporal coordinates. PoST integrates into any diagonal linear recurrence without overhead. We instantiate it across Mamba-2, RWKV-7, Gated DeltaNet, Gated Linear Attention, and RetNet. Pre-training at 180M-440M scales shows consistent zero-shot language modeling improvements, significant long-context retrieval gains for Mamba-2 (MQAR and NIAH), and competitive or improved performance across other architectures. Code: https://github.com/SiLifen/PoST.

Yang Cao

Linear RNNs Long-Range Dependency Modeling Long-Context Language Modeling Autoregressive Language Modeling

Mar 30, 2026

On the Mirage of Long-Range Dependency, with an Application to Integer Multiplication

Integer multiplication has long been considered a hard problem for neural networks, with the difficulty widely attributed to the O(n) long-range dependency induced by carry chains. We argue that this diagnosis is wrong: long-range dependency is not an intrinsic property of multiplication, but a mirage produced by the choice of computational spacetime. We formalize the notion of mirage and provide a constructive proof: when two n-bit binary integers are laid out as a 2D outer-product grid, every step of long multiplication collapses into a

3 \times 3

local neighborhood operation. Under this representation, a neural cellular automaton with only 321 learnable parameters achieves perfect length generalization up to

683\times

the training range. Five alternative architectures -- including Transformer (6,625 params), Transformer+RoPE, and Mamba -- all fail under the same representation. We further analyze how partial successes locked the community into an incorrect diagnosis, and argue that any task diagnosed as requiring long-range dependency should first be examined for whether the dependency is intrinsic to the task or induced by the computational spacetime.

Zichao Wei

Saarland University

Long-Range Dependency Modeling Neural Network Generalization

Mar 27, 2026

Entanglement as Memory: Mechanistic Interpretability of Quantum Language Models

Quantum language models have shown competitive performance on sequential tasks, yet whether trained quantum circuits exploit genuinely quantum resources -- or merely embed classical computation in quantum hardware -- remains unknown. Prior work has evaluated these models through endpoint metrics alone, without examining the memory strategies they actually learn internally. We introduce the first mechanistic interpretability study of quantum language models, combining causal gate ablation, entanglement tracking, and density-matrix interchange interventions on a controlled long-range dependency task. We find that single-qubit models are exactly classically simulable and converge to the same geometric strategy as matched classical baselines, while two-qubit models with entangling gates learn a representationally distinct strategy that encodes context in inter-qubit entanglement -- confirmed by three independent causal tests (p < 0.0001, d = 0.89). On real quantum hardware, only the classical geometric strategy survives device noise; the entanglement strategy degrades to chance. These findings open mechanistic interpretability as a tool for the science of quantum language models and reveal a noise-expressivity tradeoff governing which learned strategies survive deployment.

Nathan Roll

Department of Linguistics, Stanford University

Long-Range Dependency Modeling Circuits in Language Models Mechanistic Interpretability Sequence Modeling

Mar 12, 2026

LPC-SM: Local Predictive Coding and Sparse Memory for Long-Context Language Modeling

Most current long-context language models still rely on attention to handle both local interaction and long-range state, which leaves relatively little room to test alternative decompositions of sequence modeling. We propose LPC-SM, a hybrid autoregressive architecture that separates local attention, persistent memory, predictive correction, and run-time control within the same block, and we use Orthogonal Novelty Transport (ONT) to govern slow-memory writes. We evaluate a 158M-parameter model in three stages spanning base language modeling, mathematical continuation, and 4096-token continuation. Removing mHC raises the Stage-A final LM loss from 12.630 to 15.127, while adaptive sparse control improves the Stage-B final LM loss from 12.137 to 10.787 relative to a matched fixed-ratio continuation. The full route remains stable at sequence length 4096, where Stage C ends with final LM loss 11.582 and improves the delayed-identifier diagnostic from 14.396 to 12.031 in key cross-entropy. Taken together, these results show that long-context autoregressive modeling can be organized around a broader division of labor than attention alone.

Keqin Xie

Independent Researcher, Suzhou, China

Long-Context Language Modeling Autoregressive Language Modeling Long-Range Dependency Modeling Memory-Augmented Language Models Persistent Memory for Language Models Predictive Coding

Mar 4, 2026

Small Object Detection in Complex Backgrounds with Multi-Scale Attention and Global Relation Modeling

Small object detection under complex backgrounds remains a challenging task due to severe feature degradation, weak semantic representation, and inaccurate localization caused by downsampling operations and background interference. Existing detection frameworks are mainly designed for general objects and often fail to explicitly address the unique characteristics of small objects, such as limited structural cues and strong sensitivity to localization errors. In this paper, we propose a multi-level feature enhancement and global relation modeling framework tailored for small object detection. Specifically, a Residual Haar Wavelet Downsampling module is introduced to preserve fine-grained structural details by jointly exploiting spatial-domain convolutional features and frequency-domain representations. To enhance global semantic awareness and suppress background noise, a Global Relation Modeling module is employed to capture long-range dependencies at high-level feature stages. Furthermore, a Cross-Scale Hybrid Attention module is designed to establish sparse and aligned interactions across multi-scale features, enabling effective fusion of high-resolution details and high-level semantic information with reduced computational overhead. Finally, a Center-Assisted Loss is incorporated to stabilize training and improve localization accuracy for small objects. Extensive experiments conducted on the large-scale RGBT-Tiny benchmark demonstrate that the proposed method consistently outperforms existing state-of-the-art detectors under both IoU-based and scale-adaptive evaluation metrics. These results validate the effectiveness and robustness of the proposed framework for small object detection in complex environments.

Wenguang Tao, Xiaotian Wang, Tian Yan +2

Unmanned System Research Institute, Northwestern Polytechnical University, Xi’an, China · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China

Long-Range Dependency Modeling Small Object Detection Multi-Scale Attention Multi-Scale Feature Fusion

Feb 28, 2026

Wave-Attractor-Tree: A Hierarchical Binary Tree Reduction Architecture for Efficient Sequence Modeling

Work introduces a hierarchical binary tree-based reduction that replaces standard self-attention. The core idea is to use a recursive Gated Linear Unit merge operation, achieving O(n) total merge operations O(log n) parallel depth O(n d^2) total work and O(n) space complexity. In these experiments, the model significantly outperforms standard Transformers in both convergence speed and accuracy on long-range structural dependencies, specifically where hierarchical inductive bias is critical.

Igor Berezkin

Independent Researcher

Long-Range Dependency Modeling Sequence Modeling Hierarchical Representation Learning

Dec 28, 2024

SegKAN: High-Resolution Medical Image Segmentation with Long-Distance Dependencies

Hepatic vessels in computed tomography scans often suffer from image fragmentation and noise interference, making it difficult to maintain vessel integrity and posing significant challenges for vessel segmentation. To address this issue, we propose an innovative model: SegKAN. First, we improve the conventional embedding module by adopting a novel convolutional network structure for image embedding, which smooths out image noise and prevents issues such as gradient explosion in subsequent stages. Next, we transform the spatial relationships between Patch blocks into temporal relationships to solve the problem of capturing positional relationships between Patch blocks in traditional Vision Transformer models. We conducted experiments on a Hepatic vessel dataset, and compared to the existing state-of-the-art model, the Dice score improved by 1.78%. These results demonstrate that the proposed new structure effectively enhances the segmentation performance of high-resolution extended objects. Code will be available at https://github.com/goblin327/SegKAN

Shengbo Tan, Rundong Xue, Shipeng Luo +7

College of Computer Science and Artificial Intelligence, Southwest Minzu University, Chengdu 610225, China · Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an 710049, China · College of Mechanical and Electrical Engineering, Northeast Forestry University, Harbin 150040, China +4

Image Segmentation Long-Range Dependency Modeling Medical Image Segmentation