Organizations: University of Oslo, Department of Informatics, Oslo, Norway · NORSAR, Lillestrøm, Norway · University of Tromsø, Department of Geosciences, Tromsø, Norway · Western Norway University of Applied Sciences, Bergen, Norway
Time-series carry structure simultaneously at multiple scales (fine-grained variation, mid-range motifs, and global properties) and downstream tasks operate at correspondingly different scales. Most existing self-supervised learning approaches supervise representations globally via instance-level contrastive losses and limited temporal neighborhood supervision, but do not explicitly exploit the structural hierarchy. We propose a learning framework that explicitly enforces a structural hierarchy across three scales independently: a local objective for token continuity, a mid-range objective for window-level motifs, and a global objective for sequence-level agreement. Realizing this framework requires the backbone to expose a representation at each scale; we introduce Progressive Memory Transformer (PMT), which augments a transformer with writable, window-aligned memory that exposes the mid-range scale alongside the token and sequence-level representations conventional transformers already provide. Across seven UCR/UEA/UCI classification benchmarks, a cue-retention probe, and forecasting benchmarks, PMT learns representations that probe well at the global, mid-range, and local scales---strong low-label classification (1--5% labels), competitive forecasting performance across multiple horizons, and quantitative and qualitative evidence that memory states capture mid-range motifs.
Figures & tables
Figure 1: An illustration of the three levels of representation and how they capture distinct structural patterns. Shaded regions indicate expanding receptive fields. Exposing the intermediate representations (gray background) enables direct supervision at multiple scales, unlike standard global readouts.
Figure 2: PMA information flow ( 1 ).
Self-supervised learning (1% labeled)
Dataset
SSL-ECG
CPC
SimCLR
TS2Vec+SoftCLT
TS-TCC+SoftCLT
PMT
HAR
60.0 / 54.0
65.4 / 63.8
65.8 / 64.3
91.0 / 91.0
82.9 / 82.8
92.9 / 93.2
Epilepsy
89.3 / 86.0
88.9 / 85.8
88.3 / 84.0
96.3 / 94.1
95.6 / 95.6
96.6 / 94.7
Wafer
93.4 / 76.1
93.5 / 78.4
93.8 / 78.5
95.3 / 88.1
96.5 / 96.5
98.7 / 96.5
FordA
67.9 / 66.2
75.8 / 75.2
55.9 / 55.7
87.1 / 87.1
81.5 / 81.2
88.2 / 88.2
FordB
64.4 / 60.5
66.8 / 65.0
50.9 / 49.8
67.9 / 67.9
74.8 / 74.8
76.3 / 76.2
Table 1: Self-supervised learning results on 1% and 5% labeled subsets. Each cell shows accuracy / macro-F1.
Method
EOS AUC
EOS Top1
EOS mF1
TS2Vec (+SoftCLT)
0.649
0.638
0.604
Transformer-XL
0.933
0.795
0.790
xLSTM
0.956
0.820
0.817
PMT- λPCL=0
0.983
0.831
0.824
PMT
0.994
0.921
0.920
Table 2: FordA cue detection. EOS-probe averages across seeds.
Figure 3: Cosine similarity matrices for the representations and states between pair-wise signals from HAR. Higher similarity shows that the signals correlate as evidenced by the learned embeddings. The vertical bars denote the different sections of the hybrid wave. The wave number corresponds to the sample index in the test dataset.
Univariate
Multivariate
SoftCLT
PMT
SoftCLT
PMT
Dataset
H
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTh1
24
0.045
0.164
0.103
0.257
0.525
0.510
0.546
0.516
48
0.075
0.212
0.140
0.302
0.580
0.545
0.595
0.549
168
0.151
0.306
0.184
0.345
0.730
0.634
0.714
0.621
336
0.171
0.328
0.197
0.361
0.884
0.712
0.834
0.688
Table 3: Forecasting results following SoftCLT evaluation for different horizons (H).
PMT
xLSTM
Transformer-XL
Dataset
Top-1 Acc
mF1
Top-1 Acc
mF1
Top-1 Acc
mF1
HAR
94.2
94.6
93.5
93.8
93.0
93.3
Epilepsy
96.6
94.7
94.5
91.9
94.5
92.0
Wafer
98.9
96.9
93.4
81.1
95.7
92.0
FordA
88.9
88.9
88.2
88.2
88.3
88.3
FordB
76.5
76.5
75.5
75.4
75.6
75.5
Table 4: Comparison with xLSTM and Transformer-XL (averaged across 1% and 5% labeled data).
Table 5: Ablation of the individual loss components and evaluation of their weighting coefficients, averaged over 1% and 5% label splits. Higher is better.
Table 6: Evaluation of patch sizes and stride on FordA/B, and ElectricDevices. Training 150 epochs, using the mean 1% and 5% results over 3 seeds.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : An illustration of PMT unrolling several PMA blocks through time (left to right) and hierarchy levels (bottom to top). The sequence is tokenized and encoded through an encoder. Then, each PMA block uses these representations and previous states to produce a new version of both of these inputs. At each level, a given PMA block uses a gating mechanism to control how much of the previous level’s memory state is passed forward or reset (i.e. , uses a reset memory state instead).
Figure B.1 : Forward pass through a stack of B PMA blocks: per-window, per-block memory and token updates with adaptive gates and asymmetric attention masking.
Sequence (samples → tokens)
PMA window ms (seq s)
FA full seq s
Max SR [Hz] ↑
Δ FLOPs (PMA/FA)
Δ mem (PMA/FA)
512→63
0.8 (0.01)
0.000
48,000
+95%
(0.16/0.08 G)
0%
(26/26 MB)
1,024→127
0.8 (0.01)
0.000
48,000
+82%
(0.30/0.17 G)
0%
(26/26 MB)
2,048→255
0.8 (0.01)
0.000
96,000
+70%
(0.60/0.35 G)
0%
(26/26 MB)
4,096→511
0.8 (0.01)
0.000
96,000
+53%
(1.21/0.80 G)
0%
(28/28 MB)
8,192→1,023
0.8 (0.01)
0.000
96,000
+31%
(2.52/1.93 G)
−36%
(28/44 MB)
16,384→2,047
0.8 (0.01)
0.001
96,000
+6%
(5.52/5.20 G)
−68%
(30/94 MB)
Appendix
Table C.1 : Streaming PMA: single-window kernel timed; FA run on the full sequence. For long horizons, PMA reduces both FLOPs and peak memory while sustaining 96 kHz real-time with a 20% head-room.
Model
Params [M]
GFLOPs
Peak mem [MB]
Latency [ms] ↓
Tokens/s ↑
Vanilla Transformer
7.4
692.34
8,485
4,619.13
86,590
FlashAttention (FA)
7.4
692.34
8,336
3,476.55
115,047
PMA (vectorized)
16.3
329.95
12,292
2,962.21
135,022
PMT (full)
20.1
329.99
12,337
5,381.27
74,326
Appendix
Table C.2 : Non-streaming inference (vectorized). Vectorization duplicates overlapped windows for speed, inflating peak memory; the attentive overlap aggregator removes duplicates post-block.
Table E.2 : HAR (50 epochs). Supervised vs. 5% SSL. Top-1 / macro-F1 (%).
Seed 42
Seed 123
Seed 456
Seed 789
Seed 2024
Mean ± Std
96.13
96.19
96.06
95.99
95.99
96.07±0.08
Appendix
Table E.3 : Reset-state robustness (HAR). Supervised training with different random seeds for Mr ; Top-1 accuracy (%).
Window fraction ρ
Avg. Top-1 (%)
Avg. mF1 (%)
0.10
92.8
93.0
0.20
92.7
93.0
0.30
92.7
92.9
0.50
92.5
92.6
1.00
92.3
92.5
Appendix
Table E.4 : PMA window-size ablation on HAR (short-run pretraining). W=ρK , S=0.5W , nm=1 . We report the average over 1% and 5% label splits. Performance is flat for ρ∈[0.1,0.3] and degrades as W approaches full context.
Figure E.1 : The cosine similarity matrix between the token (bottom left) and the memory states (bottom right) representations of two signals (number 6 and 12, respectively, from the HAR validation set) of the same class (walking upstairs). The nine channels of the signals are plotted independently (top). The similarities shows that the representations and states correlate to each other between the temporal domain.
Figure E.2 : Cosine similarity matrices of the representations and states for the Epilepsy dataset. Comparison of two sequences from the same class 0 (left) and class 1 (middle), and between two sequences from different classes (right).
Figure E.3 : Cosine similarity matrices for the representations and states between pair-wise signals from HAR. Higher similarity shows that the signals correlate as evidenced by the learned embeddings. The vertical bars denote the different sections of the hybrid wave. The wave number corresponds to the sample index in the validation dataset.
Figure E.4 : Cosine similarity matrices for the representations and states between pair-wise signals from HAR. Higher similarity shows that the signals correlate as evidenced by the learned embeddings. The vertical bars denote the different sections of the hybrid wave. The wave number corresponds to the sample index in the validation dataset.
Figure E.5 : Cosine similarity matrices for the embeddings of PMT and TS2Vec + SoftCLT, from layer 5 and 10 (final), between pair-wise signals from HAR. Higher similarity shows that the signals correlate as evidenced by the learned embeddings. The vertical bars denote the different sections of the hybrid wave. The wave number corresponds to the sample index in the validation dataset. Same samples as Fig. 3 .
Figure E.6 : (Left) Double spliced HAR waveform. (Middle) t-SNE of representations (tokens) of the signal. (Right) t-SNE of memory-states. The splicing is sourced from two unique waveforms from separate activities (standing and laying)
Figure E.7 : (Left) Double spliced HAR waveform. (Middle) PCA of representations (tokens) of the signal. (Right) PCA of memory-states.
Self-supervised time series representation learning uses unlabeled sequences to learn features that transfer to forecasting, classification, and anomaly detection.
In contrast to RNNs, which compress their history into a single hidden state, Transformers can attend to all past tokens directly. However, standard Transformers rely solely on the hidden state from the previous layer to represent the entire context. We show that this design creates pressure toward representation collapse and can degrade performance. To address this issue, we introduce Layer-Integrated Memory (LIMe), a lightweight extension that leverages existing key-value buffers and learns per-head, per-layer routing weights to integrate representations from previous layers. Across language modeling, synthetic reasoning, and deep architectures, LIMe improves perplexity per FLOP in the studied regimes and yields strong gains on synthetic tasks while preserving higher value-vector entropy and token separability. Finally, learned routing weights reveal systematic reuse of local and long-distance features, showing how LIMe enriches attention-time memory without increasing hidden-state size. Code is available at https://github.com/corl-team/lime.
Transformers lack an explicit architectural mechanism for storing and organizing knowledge acquired during training. We introduce learnable sparse memory banks: a set of latent tokens, randomly initialized and trained end-to-end, that transformer layers query via cross-attention to retrieve stored knowledge. To scale memory capacity without prohibitive attention costs, we propose chapter-based routing inspired by Mixture-of-Experts architectures, partitioning the memory bank into chapters and training a router to select relevant subsets per input. This enables scaling to 262K memory tokens while maintaining tractable computation. We evaluate our approach against standard transformers (in iso-FLOP settings) on pre-training and instruction fine-tuning across relevant benchmarks. Our models surpass iso-FLOP baselines suggesting scope for a new axis of scaling, demonstrating that explicit associative memory provides complementary capacity to what is captured implicitly in model parameters. Additionally, we observe improved knowledge retention under continued training, with robustness to forgetting when transitioning between training phases (e.g., pretraining to instruction fine-tuning).
Transformers process images and videos by flattening space and time into long token sequences. While attention and KV caching preserve past features, their memory grows with sequence length and they lack an explicit, persistent spatial state, making long-horizon video understanding and occlusion-sensitive reasoning difficult. We propose Tensor Memory, a lightweight module that augments Transformer blocks with a fixed-size recurrent 3D memory tensor: tokens write into a voxel grid via a differentiable soft write that deposits content as a Gaussian-weighted volume around a predicted continuous 3D location, the memory is updated with an efficient local interaction operator and gated recurrent dynamics, and tokens read back context via continuous sampling with gated residual fusion. Because the memory tensor has a constant size, Tensor Memory decouples state capacity from input length while preserving a spatial inductive bias. We evaluate the module on standard language, image, and video benchmarks and on a controlled toy diagnostic suite designed to isolate when persistent state is beneficial; it integrates with standard Transformer training pipelines and can be attached to or removed from existing blocks without other architectural changes.
Kabir Swain, Sijie Han, Daniel Karl I. Weidele +2
Massachusetts Institute of Technology, Cambridge, MA, USA · University of Toronto, Toronto, Canada · IBM Research, Cambridge, MA, USA