Progressive Memory Transformer: Memory-Aware Attention for Time-Series

Tord Sture Stangeland; Andreas Köhler; Steffen Mæland; Adín Ramíres Rivera

cs.LGSep 25, 2026

Progressive Memory Transformer: Memory-Aware Attention for Time-Series

Authors: Tord Sture Stangeland, Andreas Köhler, Steffen Mæland, Adín Ramíres Rivera

Organizations: University of Oslo, Department of Informatics, Oslo, Norway · NORSAR, Lillestrøm, Norway · University of Tromsø, Department of Geosciences, Tromsø, Norway · Western Norway University of Applied Sciences, Bergen, Norway

Abstract

Time-series carry structure simultaneously at multiple scales (fine-grained variation, mid-range motifs, and global properties) and downstream tasks operate at correspondingly different scales. Most existing self-supervised learning approaches supervise representations globally via instance-level contrastive losses and limited temporal neighborhood supervision, but do not explicitly exploit the structural hierarchy. We propose a learning framework that explicitly enforces a structural hierarchy across three scales independently: a local objective for token continuity, a mid-range objective for window-level motifs, and a global objective for sequence-level agreement. Realizing this framework requires the backbone to expose a representation at each scale; we introduce Progressive Memory Transformer (PMT), which augments a transformer with writable, window-aligned memory that exposes the mid-range scale alongside the token and sequence-level representations conventional transformers already provide. Across seven UCR/UEA/UCI classification benchmarks, a cue-retention probe, and forecasting benchmarks, PMT learns representations that probe well at the global, mid-range, and local scales---strong low-label classification (1--5% labels), competitive forecasting performance across multiple horizons, and quantitative and qualitative evidence that memory states capture mid-range motifs.

Figures & tables

Figure 1: An illustration of the three levels of representation and how they capture distinct structural patterns. Shaded regions indicate expanding receptive fields. Exposing the intermediate representations (gray background) enables direct supervision at multiple scales, unlike standard global readouts.

Figure 2: PMA information flow ( 1 ).

Dataset	SSL-ECG	CPC	SimCLR	TS2Vec+SoftCLT	TS-TCC+SoftCLT	PMT
	Self-supervised learning (1% labeled)
HAR	60.0 / 54.0	65.4 / 63.8	65.8 / 64.3	91.0 / 91.0	82.9 / 82.8	92.9 / 93.2
Epilepsy	89.3 / 86.0	88.9 / 85.8	88.3 / 84.0	96.3 / 94.1	95.6 / 95.6	96.6 / 94.7
Wafer	93.4 / 76.1	93.5 / 78.4	93.8 / 78.5	95.3 / 88.1	96.5 / 96.5	98.7 / 96.5
FordA	67.9 / 66.2	75.8 / 75.2	55.9 / 55.7	87.1 / 87.1	81.5 / 81.2	88.2 / 88.2
FordB	64.4 / 60.5	66.8 / 65.0	50.9 / 49.8	67.9 / 67.9	74.8 / 74.8	76.3 / 76.2

Table 1: Self-supervised learning results on

1\%

and

5\%

labeled subsets. Each cell shows accuracy / macro-F1.

Method	EOS AUC	EOS Top1	EOS mF1
TS2Vec (+SoftCLT)	0.649	0.638	0.604
Transformer-XL	0.933	0.795	0.790
xLSTM	0.956	0.820	0.817
PMT- $\lambda_{\text{PCL}}=0$	0.983	0.831	0.824
PMT	0.994	0.921	0.920

Table 2: FordA cue detection. EOS-probe averages across seeds.

Figure 3: Cosine similarity matrices for the representations and states between pair-wise signals from HAR. Higher similarity shows that the signals correlate as evidenced by the learned embeddings. The vertical bars denote the different sections of the hybrid wave. The wave number corresponds to the sample index in the test dataset.

Dataset	H	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
		Univariate				Multivariate
		SoftCLT		PMT		SoftCLT		PMT
ETTh1	24	0.045	0.164	0.103	0.257	0.525	0.510	0.546	0.516
	48	0.075	0.212	0.140	0.302	0.580	0.545	0.595	0.549
	168	0.151	0.306	0.184	0.345	0.730	0.634	0.714	0.621
	336	0.171	0.328	0.197	0.361	0.884	0.712	0.834	0.688

Table 3: Forecasting results following SoftCLT evaluation for different horizons (H).

Dataset	Top-1 Acc	mF1	Top-1 Acc	mF1	Top-1 Acc	mF1
	PMT		xLSTM		Transformer-XL
HAR	94.2	94.6	93.5	93.8	93.0	93.3
Epilepsy	96.6	94.7	94.5	91.9	94.5	92.0
Wafer	98.9	96.9	93.4	81.1	95.7	92.0
FordA	88.9	88.9	88.2	88.2	88.3	88.3
FordB	76.5	76.5	75.5	75.4	75.6	75.5

Table 4: Comparison with xLSTM and Transformer-XL (averaged across 1% and 5% labeled data).

Table 5: Ablation of the individual loss components and evaluation of their weighting coefficients, averaged over

1\%

and

5\%

label splits. Higher is better.

Table 6: Evaluation of patch sizes and stride on FordA/B, and ElectricDevices. Training 150 epochs, using the mean 1% and 5% results over 3 seeds.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : An illustration of PMT unrolling several PMA blocks through time (left to right) and hierarchy levels (bottom to top). The sequence is tokenized and encoded through an encoder. Then, each PMA block uses these representations and previous states to produce a new version of both of these inputs. At each level, a given PMA block uses a gating mechanism to control how much of the previous level’s memory state is passed forward or reset (i.e. , uses a reset memory state instead).

Figure B.1 : Forward pass through a stack of

B

PMA blocks: per-window, per-block memory and token updates with adaptive gates and asymmetric attention masking.

Sequence (samples $\to$ tokens)	PMA window ms (seq s)	FA full seq s	Max SR [Hz] $\uparrow$	$\Delta$ FLOPs (PMA/FA)		$\Delta$ mem (PMA/FA)
$512\to 63$	0.8 (0.01)	0.000	48,000	$+95\%$	(0.16/0.08 G)	$0\%$	(26/26 MB)
$1{,}024\to 127$	0.8 (0.01)	0.000	48,000	$+82\%$	(0.30/0.17 G)	$0\%$	(26/26 MB)
$2{,}048\to 255$	0.8 (0.01)	0.000	96,000	$+70\%$	(0.60/0.35 G)	$0\%$	(26/26 MB)
$4{,}096\to 511$	0.8 (0.01)	0.000	96,000	$+53\%$	(1.21/0.80 G)	$0\%$	(28/28 MB)
$8{,}192\to 1{,}023$	0.8 (0.01)	0.000	96,000	$+31\%$	(2.52/1.93 G)	$-36\%$	(28/44 MB)
$16{,}384\to 2{,}047$	0.8 (0.01)	0.001	96,000	$+6\%$	(5.52/5.20 G)	$-68\%$	(30/94 MB)

Appendix

Table C.1 : Streaming PMA: single-window kernel timed; FA run on the full sequence. For long horizons, PMA reduces both FLOPs and peak memory while sustaining 96 kHz real-time with a

20\%

head-room.

Model	Params [M]	GFLOPs	Peak mem [MB]	Latency [ms] $\downarrow$	Tokens/s $\uparrow$
Vanilla Transformer	7.4	692.34	8,485	4,619.13	86,590
FlashAttention (FA)	7.4	692.34	8,336	3,476.55	115,047
PMA (vectorized)	16.3	329.95	12,292	2,962.21	135,022
PMT (full)	20.1	329.99	12,337	5,381.27	74,326

Appendix

Table C.2 : Non-streaming inference (vectorized). Vectorization duplicates overlapped windows for speed, inflating peak memory; the attentive overlap aggregator removes duplicates post-block.

$W$ (tokens)	$S/W$	$O$	$d$	GFLOPs/step	Time share [%] $\downarrow$
1024	0.5	2	320	0.24	3.1
2048	0.5	2	320	0.48	1.7
4096	0.5	2	320	0.96	1.1

Appendix

Table C.3 : Streaming overlap aggregator: isolated component analysis showing minimal overhead.

$L$ (tokens)	$W$	$S/W$	$K$	$d$	GFLOPs	MFLOPs/token
25,000	2,500	0.5	2	320	11.7	0.47
50,000	5,000	0.5	2	320	23.4	0.47
100,000	10,000	0.5	2	320	46.8	0.47

Appendix

Table C.4 : Vectorized overlap aggregator: computational cost scales linearly with sequence length.

Configuration	Aggregator time share [%] $\downarrow$
Streaming ( $W{=}2048$ , $S/W{=}0.5$ , $d{=}320$ )	1.7
Vectorized ( $L{=}100$ k, $W{=}0.1L$ , $S/W{=}0.5$ )	$<$ 1

Appendix

Table C.5 : Aggregator attribution: fraction of end-to-end time attributable to overlap aggregation.

Dataset	# Train	# Test	Length	# Channel	# Class
HAR	7 352	2 947	128	9	6
Epilepsy	9 200	2 300	178	1	2
Wafer	1 000	6 174	152	1	2
FordA	1 320	3 601	500	1	2
FordB	3 636	810	500	1	2
POC	1 800	858	80	1	2

Appendix

Table D.1 : Datasets used for self-supervised classification

	Top-1	mF1	Top-1	mF1
Dataset	PMT (supervised)		Vanilla Transformer (supervised)
HAR	97.8	98.0	97.7	97.9
FordA	91.5	91.5	85.5	85.5
FordB	79.3	79.2	80.5	80.5
Wafer	99.8	99.5	99.5	98.7

Appendix

Table E.1 : Fully supervised results (cross-entropy). Top-1 / macro-F1 (%).

	Top-1	mF1	Top-1	mF1
Model	Supervised (CE)		SSL (5% labels, linear probe)
PMT	97.1	97.4	93.6	93.9
Vanilla Transformer	97.3	97.6	91.9	92.0
Windowed Transformer (with temporal state pass)	96.7	97.0	93.1	93.3
Windowed Transformer (no state pass)	96.8	97.1	93.0	93.2

Appendix

Table E.2 : HAR (50 epochs). Supervised vs. 5% SSL. Top-1 / macro-F1 (%).

Seed 42	Seed 123	Seed 456	Seed 789	Seed 2024	Mean $\pm$ Std
96.13	96.19	96.06	95.99	95.99	$96.07\pm 0.08$

Appendix

Table E.3 : Reset-state robustness (HAR). Supervised training with different random seeds for

M_{r}

; Top-1 accuracy (%).

Window fraction $\rho$	Avg. Top-1 (%)	Avg. mF1 (%)
0.10	92.8	93.0
0.20	92.7	93.0
0.30	92.7	92.9
0.50	92.5	92.6
1.00	92.3	92.5

Appendix

Table E.4 : PMA window-size ablation on HAR (short-run pretraining).

W=\rho K

S=0.5W

n_{m}{=}1

. We report the average over

1\%

and

5\%

label splits. Performance is flat for

\rho\!\in\![0.1,0.3]

and degrades as

W

approaches full context.

Figure E.1 : The cosine similarity matrix between the token (bottom left) and the memory states (bottom right) representations of two signals (number 6 and 12, respectively, from the HAR validation set) of the same class (walking upstairs). The nine channels of the signals are plotted independently (top). The similarities shows that the representations and states correlate to each other between the temporal domain.

Figure E.2 : Cosine similarity matrices of the representations and states for the Epilepsy dataset. Comparison of two sequences from the same class 0 (left) and class 1 (middle), and between two sequences from different classes (right).

Figure E.3 : Cosine similarity matrices for the representations and states between pair-wise signals from HAR. Higher similarity shows that the signals correlate as evidenced by the learned embeddings. The vertical bars denote the different sections of the hybrid wave. The wave number corresponds to the sample index in the validation dataset.

Figure E.4 : Cosine similarity matrices for the representations and states between pair-wise signals from HAR. Higher similarity shows that the signals correlate as evidenced by the learned embeddings. The vertical bars denote the different sections of the hybrid wave. The wave number corresponds to the sample index in the validation dataset.

Figure E.5 : Cosine similarity matrices for the embeddings of PMT and TS2Vec + SoftCLT, from layer 5 and 10 (final), between pair-wise signals from HAR. Higher similarity shows that the signals correlate as evidenced by the learned embeddings. The vertical bars denote the different sections of the hybrid wave. The wave number corresponds to the sample index in the validation dataset. Same samples as Fig. 3 .

Figure E.6 : (Left) Double spliced HAR waveform. (Middle) t-SNE of representations (tokens) of the signal. (Right) t-SNE of memory-states. The splicing is sourced from two unique waveforms from separate activities (standing and laying)

Figure E.7 : (Left) Double spliced HAR waveform. (Middle) PCA of representations (tokens) of the signal. (Right) PCA of memory-states.

Self-Supervised Time Series Representation Learning

Self-supervised time series representation learning uses unlabeled sequences to learn features that transfer to forecasting, classification, and anomaly detection.

See all 137 papers

You Do Not Fully Utilize Transformer's Representation Capacity
Feb 13, 2025Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky +2Representation Collapse Memory-Augmented Language Models
1T-Tech · 2Moscow Institute of Physics and Technology · 3HSE University
Mixture of Chapters: Scaling Learnt Memory in Transformers
Mar 22, 2026Tasmay Pankaj Tibrewal, Pritish Saha, Ankit Meda +2Memory-Augmented Language Models Persistent Memory for Language Models
IIT Kharagpur · Fractal AI
Tensor Memory: Fixed-Size Recurrent State for Long-Horizon Transformers
May 26, 2026Kabir Swain, Sijie Han, Daniel Karl I. Weidele +2Memory-Augmented Neural Networks Memory-Augmented Language Models
Massachusetts Institute of Technology, Cambridge, MA, USA · University of Toronto, Toronto, Canada · IBM Research, Cambridge, MA, USA

Feb 13, 2025J/K move · Enter open · S save

You Do Not Fully Utilize Transformer's Representation Capacity

Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky, Viacheslav Sinii+1

1T-Tech · 2Moscow Institute of Physics and Technology · 3HSE University

Representation Collapse Memory-Augmented Language Models Self-Attention Cross-Layer Attention Transformer Attention Attention Mechanisms

Open paper AskPDF arXiv

In contrast to RNNs, which compress their history into a single hidden state, Transformers can attend to all past tokens directly. However, standard Transformers rely solely on the hidden state from the previous layer to represent the entire context. We show that this design creates pressure toward representation collapse and can degrade performance. To address this issue, we introduce Layer-Integrated Memory (LIMe), a lightweight extension that leverages existing key-value buffers and learns per-head, per-layer routing weights to integrate representations from previous layers. Across language modeling, synthetic reasoning, and deep architectures, LIMe improves perplexity per FLOP in the studied regimes and yields strong gains on synthetic tasks while preserving higher value-vector entropy and token separability. Finally, learned routing weights reveal systematic reuse of local and long-distance features, showing how LIMe enriches attention-time memory without increasing hidden-state size. Code is available at https://github.com/corl-team/lime.

Abstract

Self-Supervised Time Series Representation Learning

Explore similar work

You Do Not Fully Utilize Transformer's Representation Capacity

Mixture of Chapters: Scaling Learnt Memory in Transformers

Tensor Memory: Fixed-Size Recurrent State for Long-Horizon Transformers