cs.LGSep 5, 2026
SaveMemory in Deep Time-Series Models
Organizations: Deakin Applied AI Initiative, Deakin University, Geelong, VIC, Australia
Abstract
Deep learning for time series has progressed through successive architectural paradigms, from recurrent networks and transformers to structured state-space models, retrieval-augmented predictors, foundation models, and tool-using agents. These developments are typically studied in isolation, organized by architecture or modeling era. We argue that they can instead be viewed through a common question of how does a time-series model retain and access information beyond its immediate input? This question is motivated by a fundamental limitation of conventional time-series modeling: information relevant to a prediction may lie far beyond a feasible input window, while compressing history into a fixed-size state can discard information that may become useful later. We formulate this challenge as a memory problem and organize existing time-series methods along a spectrum from internal memory, encoded in parameters and fixed-size states, to external memory that is addressable, retrievable, and increasingly maintained by agents. We then develop a unified taxonomy of memory mechanisms and review three classes of external memory, including explicit modules, retrieval augmentation, and agentic stores, under a common framework for what is retained, how it is written and accessed, and how it persists. A cross-cutting analysis maps these mechanisms to time-series tasks and identifies gaps in both methods and evaluation. We conclude by outlining open problems in building memory systems that can selectively retain, retrieve, revise, and forget information as temporal environments evolve. The result is a framework for studying memory as a first-class dimension of time series modeling, independent of the underlying backbone.
Figures & tables
Figure 1: Evolution of memory mechanisms in time-series modeling from 2015 to 2026. Representative methods are organized along a temporal axis and categorized according to four increasingly explicit memory paradigms.
| Survey | Param. | Explicit | Retrieval | Agentic | Problem view |
|---|---|---|---|---|---|
| Lim and Zohren [74] | ✓ | ( ) | |||
| Wen et al. [138] | ( ) | ||||
| Jin et al. [49] | ( ) | ( ) | |||
| Liang et al. [73] | ( ) | ( ) | ( ) | ||
| Chang et al. [15] | ( ) | ✓ | ( ) | ||
| This survey | ( ) | ✓ | ✓ | ✓ | ✓ |
Table 1: Positioning against representative time series and memory surveys. Coverage of the memory spectrum: ✓ substantial, ( ) partial, blank little/none.
Figure 2: Zero-shot Chronos [ 4 ] error forecasting a real winter peak, vs. input context length (mean of two forecast periods; thin lines show each). Error stays flat under 90 days, then drops sharply once context crosses into the prior winter’s peak ( 275-302 days back), and plateaus past 450 days. The task needs long-range history, but a fixed-size window alone cannot fully exploit it.
Figure 3: Four real-world time-series examples motivating the problems in Section 3 . P1: winter-peaking electricity demand from the Electricity dataset, where a 90-day autumn window misses the previous winter’s peak. P2: monthly tourist arrivals from the Tourism dataset exhibit a strong annual cycle, with the current window closely matching the same period one year earlier. P3: an exchange rate from the Exchange-Rate dataset moving from a long peg to a floating regime, making old peg-era episodes misleading. P4: a product-store sales series from the M5 dataset with 93% zero-sales days, requiring information from related series or higher aggregation levels.
Figure 4: Taxonomy of memory architectures for time-series models. (1) Internal memory retains historical information in dynamic activations, with learned parameters governing how the state is updated. It includes recurrent and structured state-space mechanisms. External memory includes: (2) explicit memory maintains dedicated addressable entries, (3) retrieval memory accesses an external collection, and (4) agentic memory actively manages what is stored and retrieved. The figure also shows the main subclasses within each memory architecture.
| Problem | Internal | Explicit | Retrieval | Agentic |
|---|---|---|---|---|
| P1–Long-horizon dependence | •• | • | •• | • |
| P2–Recurring patterns | • | •• | •• | • |
| P3–Non-stationarity/drift | ( ) | ( ) | • | • |
| P4–Sparse/hierarchical data | • | •• | • | ( ) |
Table 2: Relationship between the four time-series memory problems and memory architectures. •• dominant, • established but less prevalent, and ( ) emerging or underexplored.
Figure 5: Illustration of internal memory in time-series forecasting. The figure shows two approaches to compressing historical observations into a fixed-dimensional internal state: recurrent-state memory, where the state is updated through nonlinear recurrent transitions, and structured state memory, where temporal history is encoded through parameterized state-space dynamics.
Figure 6: Explicit memory for time series. Top: A query window is encoded as and read against a store of individually addressable entries . The read operation weights entry by similarity, and the retrieved mixture is integrated with the query window to produce the prediction. Writing takes one of two forms: appending a new entry, so that the store grows with the observed data, or revising existing entries through the gate , with capacity fixed in advance. Bottom: Illustration of the three explicit-memory forms considered in this section: verbatim, slot, and learned explicit memory.
Figure 7: Retrieval-augmented memory for time series. Top: A query window is encoded as and matched against an external memory store whose entries contain a key , a value , and metadata . For the forecasting task, as an example, the top- relevant entries are retrieved, and their values are integrated with the query to produce the forecast. Bottom: Illustration of the three retrieval-memory forms considered in this section: instance, latent, and knowledge retrieval memory.
| Type | Stored content | Written | Read at | Decay |
|---|---|---|---|---|
| Episodic: “What happened in similar cases before?” | Context, forecast, and error triples | per instance | prediction, reflection | fast |
| Semantic: “What is generally true of this series?” | Regimes, calendar rules, exogenous facts | once evidence accrues | prediction, scoring | medium |
| Procedural: “How should I go about forecasting this?” | Tool trajectories, model-selection playbooks | once evidence accrues | planning, action | slow |
Table 3: Functional memory types instantiated for time series. Decay refers to how quickly an entry loses validity under regime change.
Figure 8: The agentic memory loop for time series, illustrated on forecasting. A write policy distills a forecasting episode into a store partitioned by memory type, a read policy recalls entries to condition prediction and reflection, and the realized outcome is written back once the horizon elapses. The loop itself is task-agnostic; we use forecasting to make the forget policy concrete because the horizon supplies a natural point at which an entry’s validity can be checked. Beyond memory operations in Sections 6 and 7 , an agentic forget policy can demote entries whose validity has lapsed under regime change, which is the mechanism that non-stationarity necessitates.
| System | Type | Memory content | Write policy | Scope | Forget | Task |
|---|---|---|---|---|---|---|
| MemCast [ 123 ] | E, S, P | patterns, wisdom, laws | offline distillation + reflection | per-dataset | conf. decay | Forecasting |
| Cast-R1 [ 124 ] | W | decision-relevant state | RL-learned, multi-turn | per-sequence | none | Forecasting |
| CastFlow [ 98 ] | P | tool-use trajectories | sampled successful paths | per-dataset | none | Forecasting |
| Nexus [ 25 ] | E, S | error signatures, guidelines | backtest calibration | per-dataset | none | Forecasting |
| AlphaCast [ 159 ] | E, S | historical cases, domain knowledge, context | context/case retrieval | per-sequence | none | Forecasting |
| Argos [ 38 ] | S | anomaly rules | multi-agent generate and repair | online | rule revision | Anomaly det. |
Table 4: Representative agentic memory systems for time series, organized by the store each maintains rather than by the agent architecture that surrounds it, which is what separates this account from agent-centric treatments [ 20 , 15 ] . Type : E, S, P, and W denote episodic, semantic, procedural, and working (within-episode) memory. Scope : per-sequence (resets when the interaction ends), per-dataset (distilled offline and frozen at test time), or online. Forget indicates an explicit revision, demotion, or validity-decay policy. Task names the canonical time-series tasks the store serves, where All four denotes forecasting, anomaly detection, classification, and imputation.
| Task | Internal | Explicit | Retrieval | Agentic |
|---|---|---|---|---|
| Forecasting | DeepAR [ 108 ] , DA-RNN [ 102 ] , TreNet [ 75 ] , LSTNet [ 62 ] , SAnD [ 116 ] , Cloudlstm [ 156 ] , P-sLSTM [ 59 ] , FiLM [ 171 ] , TimeMachine [ 1 ] , MambaTS [ 11 ] , ms-Mamba [ 55 ] , Bi-Mamba+ [ 72 ] , SST [ 101 ] Attraos [ 44 ] | MTNet [ 16 ] , MAGL [ 81 ] , PM-MemNet [ 65 ] , MegaCRN [ 48 ] , STanHop [ 142 ] , HAMN [ 66 ] , MEMTS [ 152 ] , TS-Memory [ 86 ] | RATD [ 78 ] , RAFT [ 41 ] , MQ-ReTCNN [ 149 ] , RATSF [ 133 ] , RAF [ 126 ] , TimeRAG [ 148 ] , RAST [ 107 ] , CRAFT [ 54 ] , ReDiTT [ 85 ] , SERAF [ 170 ] , TRACE [ 18 ] , ReTime [ 51 ] , TimeRAF [ 157 ] , TS-RAG [ 97 ] , SpecReTF [ 93 ] GTR [ 12 ] | Cast-R1 [ 124 ] , MemCast [ 123 ] , CastFlow [ 98 ] , Nexus [ 25 ] , AlphaCast [ 159 ] , Agentic-RAG [ 105 ] , TimeSeriesScientist [ 165 ] |
| Classification | LSTM-FCN [ 56 ] , MambaSL [ 52 ] | DPSN [ 122 ] , ShapeNet [ 69 ] , TapNet [ 160 ] , Memory Shapelet [ 132 ] | TRACE [ 18 ] | Agentic-RAG [ 105 ] , ReasonTSC [ 169 ] , ColaCare [ 136 ] |
| Anomaly detection | LSTM-AD [ 89 ] , EncDec-AD [ 88 ] , MAAT [ 109 ] | MEMTO [ 117 ] , MOMEMTO [ 151 ] , MemMambaAD [ 68 ] , Anomaly Trans. [ 145 ] | LLMAD [ 79 ] , RATFM [ 90 ] | Argos [ 38 ] , AnomaMind [ 125 ] , Agentic-RAG [ 105 ] |
| Imputation | GRU-D [ 17 ] , BRITS [ 13 ] | MMNet [ 91 ] , PRIME [ 153 ] | ALER-TI [ 128 ] | Agentic-RAG [ 105 ] |
| Reasoning | open | ChatTS [ 144 ] , ITFormer [ 134 ] | EMERGE [ 173 ] , REALM [ 174 ] | TS-Reasoner [ 150 ] , TimeCopilot [ 28 ] , TimeSeriesScientist [ 165 ] |
| Decision-making | LSTM-DRL [ 175 ] | open | open | Cast-R1 [ 124 ] , MemCast [ 123 ] , CastFlow [ 98 ] |
Table 5: Time-series memory methods by task and mechanism. The symbol open marks task–mechanism combinations the reviewed literature has barely addressed.
| Family | Representative resources | Primary evaluation role |
|---|---|---|
| General | TFB [ 103 ] ; Monash Archive [ 30 ] ; UCR [ 26 ] ; TSB-AD [ 80 ] | Standard forecasting, classification, and anomaly-detection evaluation across heterogeneous datasets. |
| Multivariate | BasicTS+ [ 110 ] | Multivariate forecasting across diverse temporal and cross-variable dependencies; useful for testing shared and cross-variable memory. |
| Spatiotemporal | LargeST [ 82 ] | Large-scale traffic forecasting with long temporal coverage, many sensors, and metadata; useful for testing memory across many spatially related entities. |
| Hierarchical | M5 [ 87 ] | Forecasting across sparse product–store series and multiple aggregation levels. |
| Foundation model | ProbTS [ 158 ] ; Chronos [ 4 ] ; BOOM [ 22 ] | Broad-horizon, held-out/zero-shot, and large-scale evaluation of universal forecasters. |
| Multimodal | Time-MMD [ 77 ] ; CiK [ 140 ] ; Time-IMM [ 14 ] ; TRACE [ 18 ] | Time series paired with text, context, irregular modalities, or cross-modal retrieval targets. |
Table 6: Representative benchmark families relevant to time-series memory. Archival conference/journal papers are prioritized; † denotes an emerging non-archival or workshop benchmark included because no mature archival equivalent currently exists.
| Section | Direction | Guiding research question |
|---|---|---|
| 11.1 | Memory Allocation and Effective Temporal Capacity | How can architectures selectively retain the most informative time-series data, and how should the effective capacity of this recoverable information be quantified? |
| 11.2 | From Information Retrieval to Temporal Retrieval | What makes a historical experience useful beyond surface-level morphological similarity? |
| 11.3 | Memory Validity under Distribution Shift | How should time-series architectures continuously assess whether stored knowledge remains valid under changing conditions? |
| 11.4 | Hierarchical Multimodal Memory | How should time-series architectures organize multimodal information across memory representations while preserving complementary information across modalities? |
| 11.5 | Beyond LLM-Centric Agentic Systems | How should heterogeneous memory representations be coordinated? |
| 11.6 | Memory from Privileged Information | Can privileged information during training shape memory under inference-time constraints? |
Table 7: Guiding research questions for future time-series memory systems.
| Group | Terms |
|---|---|
| Domain (required) | “time series”, “temporal”, “forecasting”, “spatiotemporal”, “sequential data”, “multivariate series”, “anomaly detection”, “imputation”, “time-series classification” |
| Cross-cutting memory | “memory”, “memory-augmented”, “external memory”, “memory network”, “memory module”, “memory bank”, “long-term dependency”, “long-range dependence” |
| Internal | “recurrent”, “LSTM”, “GRU”, “hidden state”, “state space model”, “structured state”, “selective state”, “Mamba”, “linear recurrence” |
| Explicit | “slot memory”, “memory slot”, “addressable memory”, “neural Turing machine”, “differentiable neural computer”, “prototype”, “pattern bank”, “shapelet”, “codebook”, “Hopfield”, “key-value memory” |
| Retrieval | “retrieval-augmented”, “retrieval augmentation”, “RAG”, “nearest neighbour retrieval”, “datastore”, “exemplar retrieval”, “analogue retrieval”, “retrieve and refine”, “in-context examples” |
| Agentic | “agent”, “agentic”, “LLM agent”, “tool use”, “episodic memory”, “semantic memory”, “procedural memory”, “reflection”, “experience replay buffer”, “self-improving” |
Table 8: Search terms. Each mechanism term (right) was conjoined with at least one domain term (top block) and, where a class-specific vocabulary exists, with its class terms. Terms are lower-cased and matched on title, abstract, and keywords.
| ID | Criterion |
|---|---|
| IC1 | Temporal data. The method operates on time series, event sequences, or spatiotemporal streams as its primary modality. |
| IC2 | Identifiable memory mechanism. The work can be placed on all four axes of Section 2.4 : what is retained, how it is written, how it is read, and how long it persists. |
| IC3 | Memory is load-bearing. Retention beyond the immediate input window is a stated mechanism of the contribution, not an incidental property of the backbone. |
| IC4 | Task relevance. The work targets forecasting, classification, anomaly detection, imputation, or the emerging reasoning and decision-making settings of Section 9 . |
| IC5 | Retrievable and in English. Full text available; peer-reviewed, or a preprint meeting EC5. |
| EC1 | Non-temporal modality only. Memory architectures evaluated solely on text, static vision, or tabular data, unless they are an anchor mechanism a surveyed time-series method descends from ( Section S2.1 ). |
Table 9: Inclusion and exclusion criteria. IC = inclusion, EC = exclusion. A work was retained only if it satisfied every inclusion criterion and triggered no exclusion criterion.
More in this topic
Memory-Augmented Neural Networks
Memory-augmented neural networks combine learned computation with explicit or persistent memory for retrieval, long-context reasoning, and adaptation.
See all 91 papers