Self-Supervised Time Series Representation Learning

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 333% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 91

Oct 5, 2026cs.CV

T-JEPA: A Temporal Joint-Embedding Predictive Architecture for Learning Better Remote Sensing Representations

Earth observation (EO) data provide rich temporal supervision, yet existing remote sensing foundation models mainly exploit sequential observations through imposing predefined pairwise relations or aggregating holistic reconstruction context. We seek to further exploit the sparse and nonuniform temporal sampling inherent in EO sequences as supervisory signals. To this end, we propose T-JEPA, a temporal joint-embedding predictive architecture that learns time-gap-conditioned latent transitions. A shared single-frame encoder processes each observation, while a temporal predictor estimates the complete target latent field from a masked source latent representation and the actual elapsed time. Across multiple temporal intervals, these predictive constraints organize observed states into structured latent trajectories. Asymmetric metadata injection mitigates shortcut learning, and direct supervision across multiple temporal scales proves more effective than recursively rolling out intermediate states. In parallel, masked pixel reconstruction provides complementary supervision for preserving spatial details. Under matched pre-training data and throughput, T-JEPA achieves leading transfer performance on both static and temporal tasks. Analyses further reveal that T-JEPA learns representations with time-gap-dependent transition predictability and coherent latent dynamics, while maintaining strong cross-period consistency, representation diversity, and semantic discriminability.

Bowen Peng, Li Liu, Yongxiang Liu +3

National University of Defense Technology, Changsha, China

Geospatial Representation Learning Self-Supervised Time Series Representation Learning Joint-Embedding Predictive Architecture

Oct 4, 2026cs.LG

EMG-GPT: Predictive Pretraining on Residual-Quantized EMG Tokens for Hand Pose Estimation

Surface electromyography (sEMG) is a low-power, cost-effective biosignal for hand-pose estimation and gesture classification. In this work, we examine whether self-supervised pretraining on sEMG can yield transferable representations for continuous hand-pose estimation. We introduce EMG-GPT, a causal transformer-based model that operates on discrete sEMG representations from a frozen residual vector quantization (RVQ) tokenizer and learns temporal dynamics through depth-autoregressive future-code prediction. The model combines within-frame integration with causal temporal modeling while preserving the geometry of the pretrained codebook. EMG-GPT shows competitive results in both Regression and Tracking tasks, supporting EMG-only pretraining as a viable approach for learning transferable sEMG representations.

Ettore Magni, Rolandos Alexandros Potamias, Stefanos Zafeiriou +1

Imperial College London

Representation Learning Self-Supervised Time Series Representation Learning Predictive Representation Learning

Sep 30, 2026cs.LG

CORD: Learning Reusable Degradation Representations Across Heterogeneous Physical Systems

Can heterogeneous physical degradation systems benefit from joint pretraining and move beyond system-specific prognostics toward reusable cross-system representation learning? CORD combines type-specific observation interfaces with a shared degradation backbone. Its two self-supervised objectives learn at complementary scales: Intra-Observation Structure Modeling (ISM) captures structure within observations, while Inter-Observation Dynamics Modeling (IDM) captures latent degradation evolution across observation histories. We evaluate CORD under two transfer boundaries: Pretraining-Included System Types, where downstream datasets and held-out units are unseen but their system types are represented during source pretraining, and Pretraining-Excluded System Types, where the entire turbofan-engine type is absent from pretraining. Across bearings, batteries, and cutting tools, CORD (Multi-domain) consistently improves over CORD (Single-domain) under Frozen adaptation, provides further gains under Full FT in most settings, and remains competitive with representative external baselines. Source-pretrained initialization also improves low-label adaptation to the pretraining-excluded engine type. Frozen-representation analysis further shows improved cross-unit lifecycle consistency after multi-domain pretraining. Joint pretraining across heterogeneous physical systems thus produces degradation representations reusable across devices, datasets, and system types.

Haibo Li, Zhiguo Zeng

CentraleSupélec, Université Paris-Saclay

Representation Learning Self-Supervised Time Series Representation Learning Predictive Maintenance Cross-Domain Transfer Learning

Sep 30, 2026cs.LG

WinoTS: Wavelet-based Self-Distillation for Time Series Models

Self-supervised pre-training of time series models is currently dominated by next-token prediction and reconstruction objectives. In continuous-valued domains, these paradigms often waste model capacity on high-frequency, point-wise noise at the expense of learning invariant structure. While invariance-based self-distillation has proven highly effective in computer vision, its application to temporal data remains largely underexplored. Effectively adapting such methods to time series requires carefully designed augmentations: spatial operations like cropping can shift the timing of repeating cycles or distort the signal, while basic jittering may provide limited variation. We introduce Wavelet-based self-distillation for time series (WinoTS), an invariance-based pre-training paradigm designed specifically for temporal signals. At its core, WinoTS leverages time-frequency augmentations to construct multi-scale structural views without distorting underlying signal dynamics. Across extensive evaluations, WinoTS outperforms state-of-the-art baselines in long-term forecasting, cross-domain zero-shot transfer, and unsupervised anomaly detection. Notably, linear probing on frozen WinoTS representations frequently surpasses fully supervised models trained from scratch. Systematic ablations demonstrate that WinoTS is a flexible, architecture-agnostic framework yielding gains across time series backbones, and establish that time-frequency transformations provide a principled alternative to vision-style spatial augmentations.

Noam Major, Kathy Razmadze, Yoli Shavit

Bar-Ilan University, Ramat-Gan, Israel

Self-Supervised Time Series Representation Learning Time Series Data Augmentation Self-Distillation

Sep 29, 2026cs.LG

Longer Records, Broader Invariance: The Hidden Scaling Problem in Longitudinal Contrastive Learning

Longitudinal data are valuable because people change. Yet the objectives used to learn from these data can inadvertently erase that change. In person-level contrastive learning, observations from the same person are treated as positives; as records grow, those positives can span increasingly distant---and increasingly different---behavioral states. More history can therefore produce not only more data, but broader invariance. We show that this distinction is fundamental. We separate \emph{record span}, how much history the learner sees, from \emph{supervision span}, how far across that history positive-pair supervision reaches. Across in-home sensing records spanning up to 2.7 years, broader supervision systematically suppresses recoverable changing-state information, even when the available history is held fixed. At the broadest span, less than 10% of the information recoverable from an untrained encoder remains. Yet keeping positives local is not sufficient: as records grow, even distant states that are never paired become increasingly similar. Explicitly contrasting other observations from the same person reverses this loss without shortening the record, revealing a second route by which longitudinal scale can broaden invariance. Finally, we prospectively reproduce the supervision-span effect in 199 GLOBEM participants. Longitudinal scale therefore presents a choice: more history need not mean more invariance. By controlling what is held invariant as records grow, we can preserve the change that made the longitudinal data valuable in the first place.

Rameen Mahmood, Xuhai "Orson" Xu, Zachary Beattie +2

New York University · Columbia University · Oregon Health & Science University

Contrastive Learning Self-Supervised Time Series Representation Learning Invariant Representation Learning

Sep 28, 2026cs.LG

Addressing Spatial Indistinguishability in Spatiotemporal Prediction via Optimal Transport-Guided Masking

Spatiotemporal prediction aims to learn discriminative representations from correlated temporal signals over spatial structures for accurate future inference. A central challenge is \emph{spatial indistinguishability}: different nodes may share similar historical patterns yet evolve toward divergent futures, severely degrading forecasting performance in real-world sensor networks. Existing embedding-based and graph neural network (GNN)-based approaches can partially detect such ambiguous nodes but rely on historical similarity, struggling to capture \emph{future behavioral divergence}. We propose \textbf{STOT} (\textbf{S}patio\textbf{T}emporal \textbf{O}ptimal \textbf{T}ransport), a self-supervised framework that resolves spatiotemporal ambiguity via structured masking guided by optimal transport. Our key idea treats indistinguishability as a \emph{disambiguation} problem: future states are inferred by exploiting concurrent spatial correlations and their time-varying similarity. We design a similarity-aware metric for dynamic inter-node relationships and an optimal transport-based masking strategy to emphasize ambiguous positions during pre-training. A batch consistency constraint preserves semantic coherence, while a random-walk masking mechanism promotes structured context exploration. Experiments on six real-world datasets show that STOT performs competitively with state-of-the-art baselines on the evaluated benchmarks and improved interpretability through transport-plan visualizations.

Guangyu Wang, Jiawei Tong

Data Science and Artificial Intelligence, Dongbei University of Finance and Economics, Dalian, Liaoning, China · Institute of Systems, Molecular & Integrative Biology, University of Liverpool, Liverpool, United Kingdom

Self-Supervised Time Series Representation Learning Spatiotemporal Forecasting

Sep 24, 2026cs.CV

A Hybrid CNN--State-Space--Attention Backbone with Joint-Embedding Predictive Pretraining for 12-Lead ECG Classification

Automatic 12-lead electrocardiogram (ECG) classification requires representations that jointly capture local waveform morphology, long-range temporal dynamics, and cross-lead dependencies, yet integrating these properties within a single efficient architecture remains challenging. This paper introduces a hybrid CNN-SSM-Attention backbone for 12-lead ECG classification. A convolutional stem performs early waveform tokenization and temporal reduction, mixed state-space and depthwise-convolutional blocks model temporal dynamics and local morphology, and a late self-attention stage enables global token interaction at reduced resolution. To improve transfer from unlabeled data, we further develop an ECG-oriented Joint-Embedding Predictive Pretraining (JEPA) framework. Unlike ViT-based JEPA methods that mask patch tokens before the encoder, the proposed method samples span masks at the latent temporal resolution and projects them back to the waveform domain, then predicts clean latent targets from a momentum encoder without waveform reconstruction. Experiments on CPSC2018, Chapman-Shaoxing, and PTB-XL, with pretraining on approximately 350K unlabeled CODE-15 recordings, show that the proposed backbone provides strong supervised baselines under a compact parameter budget. JEPA pretraining further improves transfer, particularly in reduced-label settings and under both full fine-tuning and LoRA-based adaptation. Code: https://github.com/yakoubbazi/Hybrid_ECG_Jepa

Yakoub Bazi, Sarah Aljuhani, Mohamad M. Al Rahhal +2

Computer Engineering Department, College of Computer and Information Sciences, King Saud University, Riyadh, 11543, Saudi Arabia · Applied Computer Science Department, College of Applied Computer Science, King Saud University, Riyadh, 11543, Saudi Arabia

Electrocardiography Self-Supervised Time Series Representation Learning Joint-Embedding Predictive Architecture Electrocardiogram Classification

Sep 23, 2026cs.LG

Transferable Evidence Reconstruction for Longitudinal Glucose Representations

Long physiological recordings contain many routine measurements, while predictive information is often concentrated in rare events, sustained burden, and recurring temporal patterns. Masked autoencoding recovers measurements; contrastive learning aligns views. We study self-supervision that explicitly prioritizes structured signal evidence. We introduce transferable evidence reconstruction (TER), which constructs evidence from unlabeled recordings, fits a fresh low-capacity reader on one recording group, and requires that reader to recover the same evidence in another group without refitting. Differentiating through this cross-group test learns representations with transferable evidence-decoding rules; the evidence guides self-supervision but is not used as a downstream feature. For continuous glucose monitoring (CGM), an observation-aware daily encoder and clock-aware multi-day memory bind glucose level and change to recorded time while organizing up to seven days of history. On the 14-task leaderboard, TER improves the strongest prior overall PR-AUC/ROC-AUC/Macro-F1 scores by 5.51/4.43/2.80 percentage points and sets a new best metric on 12/14 tasks. These leaderboard gains are 2.0-2.9 times the respective gaps between the two strongest baselines. With public pretraining data, folds, and the linear probe matched, TER outperforms our GlucoFM reproduction by 6.09/5.52/2.72 points. Target-reader ablations, same-history controls, and cross-person readouts support the combination of structured evidence, cross-group reader fitting, and learned multi-day organization.

Tian Zhou, Bingqing Peng, Linxiao Yang +6

Ant Group · Independent Researcher

Representation Learning Self-Supervised Time Series Representation Learning Wearable Health Monitoring

Sep 22, 2026cs.LG

EMGBlend: Heterogeneity-Aware Self-Supervised Pretraining for Gesture and Force Decoding

Public surface electromyography (EMG) datasets vary widely in electrode layout, channel count, frequency support, and size. Simply mixing them for pretraining can misalign channel semantics, introduce spectral targets that some devices cannot observe, and let large or high-channel-count datasets dominate learning. We introduce EMGBlend, a self-supervised framework designed around these differences. It combines shared channel patches with geometry-aware attention, restricts spectral targets to each recording's supported frequency band, and balances exposure across data sources. We pretrain a 109M-parameter model on 11 public EMG sources and evaluate it on gesture recognition, continuous-force regression, and contact classification. EMGBlend consistently outperforms matched random initialization and waveform reconstruction controls. Fixed-budget source controls show that multi-source pretraining improves gesture recognition and remains competitive for force decoding. Ablations confirm that geometry, band-aware targets, and source balancing each contribute to transfer, although cross-person NinaPro force estimation remains difficult. Overall, EMGBlend shows how heterogeneous EMG datasets can be combined through explicit mechanism design rather than simple concatenation. Code is available at https://github.com/tamanano/EMGBlend

Yuwei Jia, Cheng Zhong, Jinyang Yu +1

Beijing University of Posts and Telecommunications, China · Dexwise, China · Shenzhen University, China

EMG-Based Gesture Recognition Self-Supervised Pre-Training Self-Supervised Time Series Representation Learning Transfer Learning

Sep 17, 2026cs.AI

FreqCondNorm: Towards Cross-domain Predictive Maintenance through a Frequency-Conditioned Transformer Foundation Model

Deep learning predictive maintenance models suffer from poor transferability across machines and operating conditions, especially when labelled data are scarce and signals span five orders of magnitude in sampling frequency (1 Hz to ~100 kHz). We propose FreqCondNorm, a Transformer-based architecture that introduces a FiLM-style frequency-conditioned normalization layer to unify heterogeneous time-series within a single model. The architecture is pretrained on five public predictive maintenance datasets (CWRU, MFPT, UOC18, PRONOSTIA, CMAPSS) using masked auto-encoding and contrastive learning with balanced domain sampling. On fault diagnosis, the model achieves 99.2% accuracy on CWRU (+6.4 pp over CNN) and 82.1% zero-shot accuracy on MFPT, demonstrating strong transfer across sampling frequencies. However, the approach does not improve remaining useful life prediction, suggesting a mismatch between pretraining and RUL objectives that warrants future investigation.

Zaynab Raounak, Camille LHermine, Zhiguo Zeng

Laboratoire Génie Industriel, CentraleSupélec, Université Paris-Saclay, France

Multi-Source Domain Adaptation Self-Supervised Time Series Representation Learning Predictive Maintenance Bearing Fault Diagnosis Time-Series Foundation Models

Sep 15, 2026cs.LG

Distributed JEPA: A Self-Supervised Framework for Energy Forecasting

Traditional energy forecasting solutions rely on task-specific supervision and energy asset representations, limiting transferability and the ability to capture general temporal dynamics across heterogeneous assets. We address this by proposing a distributed Joint Embedding Predictive Architecture (JEPA) for self-supervised learning from heterogeneous energy time-series. The framework predicts latent representations of masked temporal segments while integrating temporal observations and contextual information within a shared embedding space. To prevent representation collapse, training combines a latent-space predictive objective with covariance and temporal variance regularization. The evaluation was conducted on energy consumption and generation datasets under data-degradation scenarios and compared with a Transformer forecasting baseline. The learned representations remained stable (cosine similarity

\approx 0.98

; effective rank 185-235). JEPA achieved performance comparable to a Transformer on building energy data, higher

R^2

in 3/5 consumer clusters, and outperformed the baseline on 9/10 unseen PVs (

R^2

=0.73-0.88 vs. <0.45), while showing greater robustness to missing data.

Liana Toderean, Tudor Cioara, Vasilis Michalakopoulos +3

Distributed Systems Research Laboratory, Computer Science Department, Technical University of Cluj-Napoca, G. Barițiu 26-28, 400027 Cluj-Napoca, Romania · Decision Support Systems Laboratory, School of Electrical & Computer Engineering, National Technical University of Athens, Ir. Politechniou 9, 157 73 Athens, Greece

Self-Supervised Time Series Representation Learning Joint-Embedding Predictive Architecture Time Series Forecasting

Sep 12, 2026cs.AI

MANAS-2: Constrained Reconstruction for EEG Foundation Models

Masked reconstruction is widely used for EEG foundation models, but optimizing reconstruction on low-SNR waveforms does not necessarily produce the most useful latent representation. We introduce MANAS-2, a new EEG foundation model that combines a Raw-Band Hybrid (RBH) masked autoencoder with Constrained Reconstruction (ConRec), a physics-motivated regularizer. RBH jointly reconstructs temporal waveform patches and compact spectral-band targets, while ConRec acts only on the temporal decoder output, penalizing differences in RMS energy between adjacent short windows of the reconstructed waveform. ConRec is intended to shape the encoder by biasing it toward the organization of oscillatory-envelope information. Across seven held-out EEG datasets, adding ConRec to an otherwise identical RBH model increases frozen ridge recovery of six-band spectral power from mean R^2=0.860 to 0.906 and recovery of inter-patch band-energy dynamics from R^2=0.283 to 0.354, while temporal waveform information remains highly recoverable from the frozen latents. Applied to a temporal-only masked autoencoder, ConRec also improves frozen downstream transfer and frequency-dependent latent geometry despite receiving no spectral targets: i.e., the effects of ConRec are architecture-independent. MANAS-2 also outperforms leading EEG Foundation Models on most downstream knowledge-transfer tasks. From the effects of ConRec, we see that a physically motivated constraint imposed through the decoder can make for a more spectrally organized and transferable latent space. MANAS-2 therefore provides a new EEG foundation model built around constrained reconstruction as a mechanism for shaping representation--rather than reconstruction--quality.

Arvasu Kulkarni, Aditya Ray Mishra, Jeet Bandhu Lahiri +5

Mannas AI · Indian Institute of Technology, Mandi

Electroencephalography Masked Autoencoders Self-Supervised Time Series Representation Learning EEG Foundation Models

Sep 7, 2026cs.AI

DGCPath: Distribution-Aware Generative Contrastive Framework for Self-supervised Path Representation Learning -- Extended Version

Due to the proliferation of vehicle trajectory data enabled by advanced sensing technologies, path representation learning has become a pivotal task in intelligent transportation systems. Although existing self-supervised approaches have achieved promising performance, their dependence on deterministic contrastive learning paradigms and handcrafted view augmentation strategies inherently restricts their cross-scenario generalization capabilities. To address these limitations, we present DGCPath, an innovative Distribution-aware Generative Contrastive learning framework for Path representation. This framework establishes a synergistic connection between generative modeling and distributional contrastive learning, enabling the acquisition of robust and transferable feature embeddings. Specifically, our framework incorporates: (1) a diffusion-based view generator that autonomously produces semantically coherent yet diverse trajectory views from Gaussian noise; (2) a variational contrastive mechanism that enforces latent feature alignment at the distribution level, transcending conventional instance-wise consistency; and (3) a novel generative cross-supervision module that reinforces view-level consistency through cross-view reconstruction learning. Comprehensive evaluations on three real-world trajectory datasets demonstrate that DGCPath outperforms state-of-the-art baselines on two distinct downstream tasks, validating its enhanced generalization capability and representation effectiveness.

Sean Bin Yang, Hao Miao, Zongyi Xu +5

Aalborg University, Denmark · The Hong Kong Polytechnic University, Hong Kong · Chongqing Univeristy of Posts and Telecommunications, China +1

Contrastive Learning Representation Learning Self-Supervised Time Series Representation Learning Trajectory Representation Learning

Sep 7, 2026cs.LG

REFINE: Trajectory Representation Learning via Closed-Loop Transcription -- Extended Version

Trajectory representation learning underpins a wide range of trajectory analytics tasks; however, most existing self-supervised approaches, whether discriminative or generative, adopt an open-loop paradigm, relying on fixed data augmentations or random masking without feedback, which limits their ability to generalize and scale. We propose REFINE, a simple yet effective Representation lEarning Framework vIa closed-loop traNscription rEfinement for trajectory data. Drawing upon feedback control theory, REFINE tightly couples road-network-aware generative reconstruction with feedback-driven contrastive learning, enabling the model to capture fine-grained local movement semantics and global spatio-temporal dependencies without manually designed augmentation views. We further provide a control-theoretic analysis that establishes convergence guarantees for the proposed closed-loop optimization. Extensive experiments on four real-world datasets demonstrate that REFINE consistently outperforms state-of-the-art methods across multiple downstream tasks while remaining computationally efficient and scalable. This paper is an extended version of REFINE: Trajectory Representation Learning via Closed-Loop Transcription, to appear in KDD 2026.

Sean Bin Yang, Ying Sun, Jilin Hu +5

Aalborg University Aalborg, Denmark · Chongqing Universtiy of Posts and Telecommunications Chongqing, China · East China Normal University Shanghai, China +1

Contrastive Learning Self-Supervised Time Series Representation Learning Trajectory Representation Learning

Aug 20, 2026cs.LG

CLaST: Context-aware Contrastive VAE for Probabilistic Time Series Forecasting

Probabilistic forecasting models are widely used for time series forecasting in domains such as energy systems, finance, medicine, and transportation. In recent years, deep generative models have shown strong results on probabilistic forecasting, yet many conventional approaches struggle to capture internal temporal dependencies, leading to latent representations with limited expressive power. To address this limitation, we propose \textit{CLaST}, a VAE framework for probabilistic multivariate time series forecasting. Unlike existing generative models, CLaST learns embeddings that preserve contextual similarity between observations through our contrastive loss function. Experiments across nine widely adopted benchmarks demonstrate that CLaST consistently surpasses strong baseline methods. In short-term forecasting tasks, our approach achieves improvements of up to

16.4\%

in CRPS and

14.4\%

in NMAE over the second-best method. Furthermore, in long-term prediction CLaST attains superior overall performance, exceeding the second-best method by up to

48.6\%

and

25.1\%

in CRPS and NMAE, respectively.

Alexander Marusov, Dmitry Anikin, Petr Sokerin +3

Applied AI Institute Moscow, Russia

Variational Autoencoders Multivariate Time Series Forecasting Self-Supervised Time Series Representation Learning Time Series Representation Learning Probabilistic Time Series Forecasting

Aug 13, 2026cs.LG

The Impact of Temporal Context Length and Encoding Strategies on Self-Supervised ECG Representation Learning

Self-supervised electrocardiogram (ECG) models are often trained on a few seconds of ECG signal and, increasingly, on discretized token sequences. It remains unclear whether these choices sacrifice information needed for rhythm inference and longitudinal consistency in real-world ambulatory recordings. We present a controlled study on the Icentia11k single-lead dataset that varies (i) the input horizon (16 seconds, 1 minute, 5 minutes, and 10 minutes) and (ii) the front-end representation (continuous convolutional patch embeddings vs. fixed vector-quantized tokens), while holding the Transformer backbone and training protocol constant. Representations are assessed by downstream abnormal rhythm detection and by patient-level retrieval that probes cross-session stability. Our results show that increasing temporal context beyond 16-second snapshots yields stronger transfer and higher retrieval accuracy, with the strongest performance achieved by the 5- and 10-minute models, indicating improved capture of slow-varying rhythm dynamics and individual-specific structure. Across all evaluated horizons, continuous patch embeddings outperform discretized tokens, suggesting that quantization can discard clinically relevant waveform detail. These findings motivate ECG foundation models that emphasize extended context and continuous encoders for clinical prediction and similarity-based applications. Our code and pretrained models are publicly available at https://github.com/muha-0/ecg-ssl-representation-learning.

Ahmed Sameh, Ramzi Al-Sharawi, Yogatheesan Varatharajah

Computer Science & Engineering, University of Minnesota Twin Cities, Minneapolis, MN 55455. · Robotics, University of Minnesota Twin Cities, Minneapolis, MN 55455.

Self-Supervised Time Series Representation Learning Electrocardiogram Foundation Models Long-Context Modeling Electrocardiogram Classification

Aug 11, 2026cs.CV

Capturing Uncertainty in Human Motion for Representation Learning in Soccer

This paper presents a self-supervised representation learning framework for understanding 3D skeleton-based human motion in soccer, using future motion prediction as the learning objective. Since human motion is inherently uncertain, accounting for multiple plausible futures is essential for capturing the underlying motion dynamics and learning effective representations. To this end, we introduce a conditioning module for motion prediction that models a probabilistic distribution over discretized future motions in 3D Euclidean space, learning multimodality with explicit supervision from future trajectories. Experiments on large-scale soccer player tracking data show that our approach substantially improves motion prediction accuracy. Moreover, the learned representations effectively transfer to multiple soccer downstream applications, demonstrating strong cross-task generalization.

Yizhou Xu, Lars Bretzner, Tiesheng Wang +1

KTH Royal Institute of Technology · EA Sports TRACAB

Self-Supervised Time Series Representation Learning Sports Video Analysis Predictive Representation Learning Human Motion Prediction Multimodal Trajectory Prediction Trajectory Prediction

Aug 6, 2026cs.LG

Is Self-Pretraining really useful to improve diagnosis in medical Time Series?

Inspired by recent evidence that transformer architectures benefit from Self-PreTraining (SPT) on long-context benchmarks, we investigate whether similar gains extend to multimodal, multivariate, and even simple univariate medical time series. Our objective is to assess the impact of SPT on the performance and scalability of transformer-based models across diverse medical applications, particularly under limited data conditions. We evaluate transformer architectures on three representative medical time-series tasks: rehabilitation robotics (Camargo dataset), stress detection (Non-EEG Stress), and Parkinson's disease detection (Gait Parkinson's Disease). Models are trained either from scratch or through SPT using four masking-based objectives designed to promote temporal and cross-modal representation learning, and we systematically vary model depth to examine how capacity interacts with pre-training benefits. Across datasets and configurations, SPT consistently improves classification accuracy by 0-6 percentage points depending on masking strategy, dataset and architecture, with gains observed not only in multivariate settings but also when models are restricted to simple univariate inputs. The improvements increase for deeper models that can better exploit the enriched temporal representations learned during pre-training. These findings indicate that SPT is a simple and general strategy that enhances transformer performance on medical time-series tasks without requiring task-specific architectural changes, supporting its potential to improve robustness and accuracy in data-limited clinical settings.

Omar Coser, Antonio Orvieto, Paolo Soda +1

Unit of Artificial Intelligence & Computer Systems, Universit`a Campus Bio-Medico di Roma, Via ´Alvaro del Portillo, 21, Rome, 00128, Italy. · Unit of Advanced Robotics and Human-Centered Technologies, Universit`a Campus Bio-Medico di Roma, Via ´Alvaro del Portillo, 21, Rome, 00128, Italy. · Max Planck Institute for Intelligent Systems +2

Time Series Classification Self-Supervised Time Series Representation Learning Multimodal Pretraining Time Series Transformers

Aug 6, 2026cs.LG

Spectral Aliasing Pretext: A novel task for Self-Supervised fault diagnosis in rotating machinery

Deep learning is a new way for machinery fault diagnosis but requires extensive labeled data, a scarce resource in industrial settings. We propose Spectral Aliasing Pretext (SAP), a self-supervised learning method that pretrains models on unlabeled vibration data by exploiting spectral aliasing. We deliberately undersample signals to create folded spectrum, then train a Transformer to reconstruct the original unfolded spectrum. This pretext task forces the model to learn frequency-domain invariants characteristic of mechanical faults, without potentially destructive augmentations. Experiments on the CWRU dataset show that SAP learns stable and highly discriminative representations. In a linear probing setting, SAP quickly achieves very high classification performance with only a small fraction of labeled data and low variance. In contrast, full fine-tuning, including fully supervised training, does not lead to more stable or better results. Overall, these findings suggest that SAP combined with linear probing can be more effective and reliable than fully supervised training for fault diagnosis with limited labeled data.

Victor Gialis, Maxime Metz, David Esteve +1

LASPI, Univ. Jean Monnet, Roanne, 42300, France · Pellenc ST, Applied Research Group, Pertuis, 84120, France · LabCom Aioly, Artificial Intelligence and Optics Laboratory, Montpellier, 34196, France +1

Self-Supervised Time Series Representation Learning Fault Diagnosis

Jul 31, 2026eess.SP

EEG-JEPA: Structured Latent Prediction for EEG Foundation Models

Electroencephalography (EEG) foundation models aim to learn reusable representations from large-scale unlabeled recordings. A common pretraining strategy is masked waveform reconstruction, but applying supervision directly to noisy EEG may encourage models to recover predictable background activity, acquisition effects, and artifacts rather than neural structure that transfers across tasks. This raises a central question: what should an EEG foundation model predict to learn transferable representations? We introduce EEG-JEPA a structured latent-prediction framework for EEG foundation modeling. Rather than reconstructing masked voltage samples, a masked context encoder and predictor infer contextual latent states produced by an exponential-moving-average target encoder that observes the complete input. EEG-JEPA organizes target design along three complementary dimensions: target content specifies what representation is predicted, target support specifies where prediction occurs over structured electrode--time regions through Neurotopology-Aware Multi-scale Electrode-Temporal Masking (N-MET), and target depth specifies at which encoder layers supervision is applied. Together, these designs shift EEG pretraining from recovering missing measurements to inferring latent states from structured electrode--time context. We evaluate EEG-JEPA through controlled objective comparisons, frozen multitask transfer, and full fine-tuning. Under the same backbone, pretraining corpus, and training duration, EEG-JEPA improves the 14-task frozen macro balanced accuracy from 40.49% to 50.42% over CBraMod-style masked waveform reconstruction. Multi-source continuation further raises this result to 52.94%, the highest average among the EEG foundation models evaluated on EEG-FM-Bench. Under protocol-matched full fine-tuning, EEG-JEPA also improves the nine-task average balanced accuracy from 68.98% to 70.65%.

Jinhao Li, Zhiyuan Ma, Xueqiao Han +8

Tsinghua Laboratory of Brain and Intelligence, Tsinghua University · School of Basic Medical Sciences, Tsinghua Medicine, Tsinghua University · School of Biomedical Engineering, Tsinghua Medicine, Tsinghua University +2

Self-Supervised Time Series Representation Learning Joint-Embedding Predictive Architecture Predictive Representation Learning EEG Foundation Models

Jul 29, 2026cs.LG

Rethinking EEG-Based Disease Diagnosis: Decoupling Instance Representation Learning from Subject-Level Supervision

EEG-based disease diagnosis requires one prediction per subject, yet common pipelines segment recordings into short instances, inherit the subject label for every instance, and train instance-level classifiers. This assumes that all instances provide equally reliable diagnostic evidence. Multiple instance learning (MIL) avoids inherited labels by treating each subject as a bag. However, EEG datasets contain far fewer subjects than instances, which can limit the quality of the representations learned by end-to-end MIL. We propose BridgeMIL, a two-stage framework that decouples instance representation learning from subject-level supervision. Stage 1 pretrains the encoder without inherited instance labels by aligning temporally nearby windows and independently sampled within-subject sub-bags. Variance and covariance regularization prevent collapse and reduce redundancy without negative pairs. Stage 2 transfers the encoder to an attention-based MIL aggregator, applies supervision only to subject predictions, and limits representation drift through feature retention. Across three EEG disease datasets and five representative backbones, BridgeMIL attains the highest mean accuracy in 14 of 15 dataset-backbone settings and an overall mean accuracy of 76.57%, 4.28 percentage points higher than the strongest baseline. Further analyses reveal substantial variation in inherited-label reliability across instances, greater performance sensitivity to subject scarcity than to instance scarcity, and a more structured representation space with distinct subject-wise clusters and improved separation between diagnostic classes. Together, these findings underscore the importance of aligning supervision with the subject-level prediction objective while learning from abundant EEG instances without assigning disease labels to individual instances.

Zhiyuan Ma, Zeyuan Li, Zhiyi Lu +7

School of Biomedical Engineering, Tsinghua Medicine, Tsinghua University · Department of Biomedical Engineering, Southern University of Science and Technology

Electroencephalography Self-Supervised Time Series Representation Learning Multiple Instance Learning

Jul 26, 2026cs.LG

Neonatal Hypoxic-ischaemic Encephalopathy Classification from the EEG and HRV Signals Using a Conformer based Masked Autoencoder

In this paper, we propose the MAEConformer, a novel self-supervised learning framework that combines the Conformer architecture with the Masked Autoencoder (MAE) paradigm for large-scale representation learning from unlabelled electroencephalography (EEG) and heart rate variability (HRV) signals. By integrating convolutional operations with Transformer-based self-attention, MAEConformer effectively captures both local temporal patterns and long-range contextual dependencies in physiological time series. To enhance reconstruction fidelity and representation quality, a multi-resolution short-time Fourier transform (MR-STFT) loss is incorporated alongside the reconstruction objective, enabling the model to jointly learn temporal and spectral characteristics across multiple scales. Modality-specific EEG and HRV MAEConformer models were pretrained on 6,030h and 4,868h of unlabelled recordings, respectively, and subsequently transferred to expert-annotated downstream tasks. Experimental results demonstrate that the learned representations provide strong transferability and data efficiency. In EEG-based hypoxic ischemic encephalopathy (HIE) severity classification, the pretrained MAE-EEG model achieved test AUCs of 97.19% and 96.56% for binary and four-class classification tasks, respectively, outperforming a range of state-of-the-art supervised and self-supervised baselines. On the HRV-based HIE severity classification task, MAE-HRV achieved a test AUC of 82.42%, surpassing both self-supervised Transformer-based and supervised convolutional baselines. These findings demonstrate the effectiveness of MAEConformer for learning robust and transferable representations across multiple physiological modalities.

Shuwen Yu, William P Marnane, Geraldine B. Boylan +1

Department of Electrical & Electronic Engineering, School of Engineering and Architecture, University College Cork, Cork, Ireland · INFANT Research Centre, University College Cork, Cork, Ireland · Pediatrics and Child Health, University College Cork, Cork, Ireland

Electroencephalography Masked Autoencoders Hybrid CNN-Transformer Architectures Self-Supervised Time Series Representation Learning

Jul 21, 2026cs.CV

STS-NET: Spatio-Temporal Stress Network for Self-Supervised Crop Stress Detection using Satellite Image Time Series

Early and accurate detection of crop stress is essential to improve agricultural productivity and ensure global food security. However, collecting a large labeled crop stress dataset is a challenging task. To address this challenge, we introduce a novel spatial-temporal stress network (STS-NET), built on a self-supervised 3D-convolutional autoencoder (3D-CAE), designed to utilize Satellite Image Time Series (SITS) data for crop stress detection. STS-NET exploits four vegetation indices: Normalized Difference Vegetation Index (NDVI), Normalized Difference Vegetation Index (GNDVI), Red-Edge Chlorophyll Index (RECI) and Normalized Difference Red-Edge Index (NDRE) obtained from high resolution Planetscope imagery to capture spatiotemporal stress patterns. The model is trained on our BSPT (Barnala Spatial-Temporal) dataset and evaluated on a real-world sugarcane dataset collected over a year from a 2.5-acre test plot located in Lakhimpur-Kheri (LK) district in Uttar Pradesh in India. STS-NET achieved a precision of 97. 98% for water stress, 85.08% for nitrogen stress, and 83.47% for combined stress. The results demonstrate the potential of STS-NET in effectively detecting stress in sugarcane crops with minimal reliance on labeled data. Furthermore, STS-NET can serve as a robust feature extractor for simpler models.

Pradeep Dalal, Rajiv Ranjan, Sushil Ghildiyal +2

Department of Computer Science and Engineering, Indian Institute of Technology Ropar, Rupnagar 140001, India · Department of Robotics and Autonomous Systems, Plaksha University, Mohali 140306, India

Agricultural Remote Sensing Self-Supervised Time Series Representation Learning 3D CNNs Convolutional Neural Networks

Jul 17, 2026eess.SP

Joint-Embedding Predictive Architecture for Sensor-based Activity Recognition

Sensor-based human activity recognition (HAR) has achieved significant progressed in fully supervised learning settings. However, these supervised learning models rely on large amount of labeled data, which require labor-intensive collection and meticulous annotation. To address these challenges, this paper proposes a Joint Embedding Predictive Architecture framework tailored for sensor-based HAR, designed to learn robust and generalizable representations from unlabeled datasets. The proposed framework features an encoder designed to explicitly model both the fine-grained local temporal representations within individual window and the long-term temporal sequence of adjacent windows. Furthermore, we introduce an improved Variance-Invariance-Covariance Regularization (VICReg) objective function that incorporates computationally lightweight norm term to stabilize the JEPA pre-training phase. This term balances variance, invariance and covariance constraints to prevent representation collapse. The proposed HAR-JEPA framework is evaluated using two benchmark continuously performed activity datasets. The results show that high-quality representations are successfully learned by the proposed framework. Furthermore, the representations learned by HAR-JEPA demonstrates superior generalization on minority, high variance transitional activities such as sit-to-stand and sit-to-lie where supervised learning tend to overfit due to limited support.

Mohd Halim Mohd Noor, Abdulrahman M. A. Baraka

School of Computer Sciences, Universiti Sains Malaysia, 11800 Pulau Pinang, Malaysia · Artificial Intelligence Research Center, Ajman University, Ajman, United Arab Emirates · Al-Quds Open University, Ramallah, Palestine

Human Activity Recognition Self-Supervised Time Series Representation Learning Joint-Embedding Predictive Architecture Invariant Representation Learning

Jul 13, 2026cs.LG

DiffEEG: A Self-Supervised Denoising Diffusion Model for Learning EEG Generic Representations

Deep learning for EEG-based seizure detection faces critical challenges: severe annotation scarcity and extreme class imbalance, where ictal events comprise less than 10% of clinical recordings. We present DiffEEG, a 9.6M-parameter self-supervised foundation model that addresses both limitations through denoising diffusion pre-training and reinforcement learning (RL)-based fine-tuning. Pre-trained on 1.3M unlabeled segments from the Temple University Hospital Seizure Corpus (TUHSZ), DiffEEG learns generic neural representations via a 1D U-Net with multi-head self-attention. For downstream adaptation, a reinforced decision layer employs policy gradient optimization to directly maximize F1-score, prioritizing sensitivity to rare seizure events over overall accuracy. Under strict patient-wise evaluation (279 patients, Leave-One-Fold-Out), DiffEEG achieves 61% accuracy and 59% F1 for 4-class seizure subtyping, and 81% accuracy with 85% weighted F1 for binary detection, maintaining clinically viable seizure recall (59%) despite extreme imbalance (6.7% prevalence). Segment-level evaluation establishes an upper bound of 97.6% accuracy, confirming strong architectural capacity. DiffEEG demonstrates that diffusion-based pre-training combined with metric-aware reinforcement learning enables clinically deployable seizure monitoring with minimal labeled data requirements.

Abdulkader Helwan, Lina Abou-Abbas, Hussein El Amouri +2

Department of Electrical and Computer Engineering, Lebanese American University, Byblos, Lebanon · Institute of Applied Artificial Intelligence, T´ELUQ University, Montreal, Canada

Electroencephalography EEG Seizure Detection Self-Supervised Time Series Representation Learning EEG Foundation Models Diffusion Models

Jul 11, 2026cs.CV

PhenoEmbed: Self-Supervised Multispectral UAV Time-Series Embeddings for Individual Tree Crown Phenology

Tree crowns are a challenging target for resilient AI because they are not static objects: their spectral response, internal texture, translucency, and apparent boundaries change substantially across the growing season. We develop PhenoEmbed, a self-supervised crown-centric temporal embedding model trained with contrastive and masked reconstruction objectives on HeideBench, an 18-date UAV multispectral time-series benchmark for forest crown phenology in D{ö}lauer Heide. The model treats seasonal crown dynamics as phenological appearance change driven by leaf emergence, canopy closure, senescence, and leaf-off conditions. Segmented tree crown polygons are retained as object anchors to extract aligned crown-centered crops through time, allowing one 256-dimensional vector summarizing seasonal crown appearance to be learned per tree. On 5,885 crop-safe crowns, the exported embeddings show structured low-dimensional organization, with the first two principal components explaining 25.1% of variance and nearest-neighbor retrieval producing a median top-1 cosine similarity of 0.946. Compared with handcrafted temporal features and a learned mean-pooling baseline, PhenoEmbed yields substantially more compact nearest-neighbor structure, while ablations show that the contrastive loss, masked reconstruction loss, and explicit seasonal time features each affect the structure of the learned embedding space. These results support PhenoEmbed as a reusable forest crown representation learner and motivate future downstream tests of whether such features improve tree-level models under seasonal change.

Taimur Khan

Helmholtz Centre for Environmental Research - UFZ, Community Ecology, Theodor-Lieser-Str. 4, 06120 Halle, Germany

Plant Phenotyping Self-Supervised Time Series Representation Learning UAV Remote Sensing

Jul 8, 2026cs.LG

FMMVCC: Fuzzy Mamba-based Multi-View Contrastive Clustering for Univariate Time Series

In many realistic scenarios, large volumes of time series data are generated with limited or expensive annotations. This limitation makes supervised learning methods difficult to apply and leads to the use of unsupervised approaches capable of discovering meaningful structures directly from raw data. Clustering therefore plays a crucial role in organizing time series into groups that share similar temporal patterns, enabling exploratory analysis and downstream tasks without requiring manual labeling. However, existing deep clustering methods often struggle to capture long-range temporal dependencies or rely on architectures with high computational cost. This paper introduces FMMVCC, a Mamba-based deep clustering framework for time series that leverages state space sequence modeling to efficiently learn temporal representations with linear complexity. Additionally, it utilizes multi-view self-supervised learning with temporal masking and augmentations. Experimental evaluation in 15 benchmark datasets proves that FMMVCC consistently outperforms state-of-the-art baselines, achieving the best overall performance in 29 of 60 total metric evaluations and the highest average rank in all tested scenarios.

Donato Cerciello, Leonardo Schiavo, Angel Panizo-LLedot +2

Departamento de Sistemas Inform´aticos, Universidad Polit´ecnica de Madrid Madrid, Spain

Self-Supervised Time Series Representation Learning Clustering Unsupervised Clustering Multi-View Clustering Time Series Clustering

Jul 7, 2026cs.LG

STST-JEPA: Shallow-Target Spatio-Temporal Joint Embedding Prediction Architecture For EEG Self-Supervised Learning

Brain age - the age inferred from a physiological recording - is an emerging biomarker whose deviation from chronological age tracks neurological and psychiatric burden, and EEG is an attractive substrate for it because it is cheap, portable, and temporally rich. Yet EEG brain-age models must contend with cross-site montage heterogeneity, small labelled cohorts, and dominant subject-level non-stationarity, and few EEG foundation models have been shown to deliver competitive age regression across the full pediatric to older adult range in which such a biomarker would actually be deployed. We introduce STST-JEPA, a self-supervised transformer for resting-state and task EEG, pretrained on 47,703 sessions spanning ages 5-81 from the brain.space and Healthy Brain Network (HBN) corpora. The model combines a latent-prediction objective - predicting masked-token representations against an EMA-of-tokenizer target - with an auxiliary signal-reconstruction term, applied to 30-second multi-channel windows under spatiotemporal block masks. A lightweight attentive probe trained on frozen pretrained embeddings achieves a best held-out-validation mean absolute error of 3.06 years (r = 0.924) for age regression on 3,367 sessions, against a predict-the-mean baseline of approximately 10 years MAE. With light task-specific finetuning of the model's final layers, the same pretrained encoder achieves rank-1 placements - with the model's native 30-second windows - on the public NeuralBench x brain.space EEG leaderboard for sex classification (balanced accuracy 0.911), age prediction (r = 0.749), and psychopathology composite regression (r = 0.215). We further show that the model's age-prediction residual is negatively correlated with cognitive efficiency over several tasks we examined.

Roy Segal, Yoni Svechinsky, Tomer Fekete

Brain Age Prediction Self-Supervised Time Series Representation Learning Joint-Embedding Predictive Architecture EEG Foundation Models

Jul 5, 2026cs.LG

Masked Generative-Contrastive Representation Learning for Cross-Dataset EEG-Based Emotion Recognition

Self-supervised learning (SSL) shows strong potential for cross-dataset transfer by improving feature representation and generalization. However, its application to EEG-based emotion recognition remains largely unexplored. Existing SSL methods struggle to capture the intricate spatiotemporal dependencies of EEG signals under varying channel configurations, extract fine-grained representations resilient to noise, and derive global features that generalize well across subjects. To address these challenges, we propose Masked Generative-Contrastive Representation Learning (MGCRL), a novel SSL framework specifically designed for EEG-based emotion recognition. Built upon a region-aware spatiotemporal encoder, MGCRL integrates generative and contrastive learning to achieve both fine-grained and global discriminative representations for cross-dataset generalization. MGCRL introduces three key designs: 1) a spatiotemporal encoder that incorporates region-based graph convolution to capture localized spatial and functional relationships, enhancing region-specific feature learning and mitigating the impact of varying EEG channel configurations across datasets; 2) a generative learning mechanism based on the joint embedding predictive architecture (JEPA) that utilizes masked features to capture noise robustness fine-grained representations, improving the model's capability to characterize subtle emotional states; and 3) a contrastive learning strategy that leverages masked and original features to learn temporally stable and cross-subject-invariant representations across the same stimuli, boosting emotion discrimination and cross-subject generalization. Under these designs, MGCRL exhibits remarkable ability to learn universal representation. Extensive experiments involving pretraining on the large FACED dataset and fine-tuning on multiple SEED-series datasets demonstrate the effectiveness of MGCRL.

Huqin Weng, Jiayang Huang, Yimin Wen +3

Contrastive Learning Self-Supervised Learning Self-Supervised Time Series Representation Learning EEG Emotion Recognition Cross-Dataset Generalization

Jul 3, 2026eess.SP

MorphologyFM: A Foundation Model for Morphology-Aware Representation Learning from ECG and Pulse Oximetry Waveforms

Foundation models have recently emerged as a powerful paradigm for learning transferable representations from large scale biomedical data, yet existing approaches for physiological waveforms primarily optimize reconstruction or forecasting objectives that do not explicitly preserve clinically meaningful waveform morphology. Electrocardiograms (ECGs) and pulse oximetry (SpO2) waveforms encode rich cardiovascular and hemodynamic information through their morphological structure. In this work, we introduce MorphologyFM, a multimodal foundation model pretrained on paired ECG and SpO2 waveforms from the MIMIC critical care database using a morphology aware self supervised learning objective. MorphologyFM combines morphology guided masking, cross modal representation learning, and contrastive latent alignment to learn representations that capture clinically relevant physiological structure without requiring manual annotations. We evaluate MorphologyFM across multiple downstream prediction tasks, including arrhythmia classification, hypoxemia prediction, mortality prediction, and length of stay estimation, demonstrating consistent improvements over representative self supervised learning methods, including Masked Autoencoders (MAE), contrastive learning, Barlow Twins, and Joint Embedding Predictive Architectures (JEPA). Furthermore, we show that jointly modeling ECG and SpO2 waveforms produces more transferable representations than single modality pretraining. Our results establish waveform morphology as a powerful inductive bias for self supervised physiological representation learning and introduce MorphologyFM as a general purpose foundation model for continuous physiological monitoring.

Saiyang Feng, Yuanyun Zhang, Shi Li

University of the Chinese Academy of Sciences · Columbia University

Representation Learning Cross-Modal Representation Learning Electrocardiography Self-Supervised Time Series Representation Learning Electrocardiogram Foundation Models Time-Series Foundation Models