Self-Supervised Time Series Representation Learning

Ask about this topic

Overview

Self-supervised time series representation learning studies how to learn useful features from sequences without relying on extensive labels, which are often costly or unavailable. Common approaches include contrastive learning between related views, masked reconstruction, generative prediction, and joint-embedding predictive architectures (JEPAs), implemented with convolutional networks, Transformers, or state-space models. These representations are tested across tasks such as activity recognition, forecasting, anomaly detection, and clinical analysis of ECG, EEG, and wearable signals, often using frozen linear probes, fine-tuning, and cross-dataset transfer. A central open question is which objectives and inductive biases help each task: gains vary across datasets and architectures, while invariance to nuisance variation can also erase meaningful changes over time.

Latest papers

CardsList

146 papers23 papers in the last four weeks, up 77% on the four weeks before. 0.1% of all new papers.

Apr 4, 2026

Can Natural Image Autoencoders Compactly Tokenize fMRI Volumes for Long-Range Dynamics Modeling?

Modeling long-range spatiotemporal dynamics in functional Magnetic Resonance Imaging (fMRI) remains a key challenge due to the high dimensionality of the four-dimensional signals. Prior voxel-based models, although demonstrating excellent performance and interpretation capabilities, are constrained by prohibitive memory demands and thus can only capture limited temporal windows. To address this, we propose TABLeT (Two-dimensionally Autoencoded Brain Latent Transformer), a novel approach that tokenizes fMRI volumes using a pre-trained 2D natural image autoencoder. Each 3D fMRI volume is compressed into a compact set of continuous tokens, enabling long-sequence modeling with a simple Transformer encoder with limited VRAM. Across large-scale benchmarks including the UK-Biobank (UKB), Human Connectome Project (HCP), and ADHD-200 datasets, TABLeT outperforms existing models in multiple tasks, while demonstrating substantial gains in computational and memory efficiency over the state-of-the-art voxel-based method given the same input. Furthermore, we develop a self-supervised masked token modeling approach to pre-train TABLeT, which improves the model's performance for various downstream tasks. Our findings suggest a promising approach for scalable and interpretable spatiotemporal modeling of brain activity. Our code is available at https://github.com/beotborry/TABLeT.

Peter Yongho Kim, Juhyeon Park, Jungwoo Park +4

ECE, Seoul National University · IPAI, Seoul National University · BCS, Seoul National University +2

fMRI Representation LearningLong-Context ModelingMedical Image Representation LearningSelf-Supervised Time Series Representation Learning
AskPDF
Apr 3, 2026

A General Framework for Generative Self-supervised Learning in Non-invasive Estimation of Physiological Parameters Using Photoplethysmography

Aligning physiological parameter labels with large-scale photoplethysmographic (PPG) data for deep learning is challenging and resource-intensive. While self-supervised representation learning (SSRL) can handle limited annotated data, the challenge lies in learning robust shared representations from vast unlabeled data and integrating contextual cues to learn distinctive representations. To alleviate these challenges, a generative SSRL framework TS2TC is proposed to utilize the temporal, spectrogram, and temporal-spectrogram mixed domains to explore and incorporate the unique features of PPG for universal and noninvasive physiological parameter estimation. A pretext task named Cross-Temporal Fusion Generative Anchor (CTFGA) is designed, modeling temporal dependencies and reconstructing independent segments at a coarse level to provide robust global feature extraction and local contextual representation. The framework includes sub-signals from PPG with diverse frequency scales and order derivatives reflecting hemodynamics to facilitate learning shared representations at varying semantic levels. Secondly, a cognitive-inspired dual-process transfer (DPT) strategy is formulated, consisting of prior-dependent autonomous processes and posterior observation reasoning processes, to leverage the independent and integrated advantages of shared and specific representations. TS2TC introduces a bilinear temporal-spectrogram fusion method in the mixed domain, aligning latent representations from different domains and establishing fine-grained contextual interactions across multiple sources of information. Extensive experiments on physiological parameter estimation tasks showed that the joint performance of CTFGA and DPT outperforms standard generative learning significantly. TS2TC achieved an average 2.49% improvement in RMSE over state-of-the-art estimation methods with only 10% training data.

Zexing Zhang, Huimin Lu, Songzhe Ma +4

School of Computer Science and Engineering, Changchun University of Technology, Jilin, Changchun, 130102, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Jilin, Changchun, 130012, China · Smart Health Joint Innovation Laboratory for the New Generation of AI, Jilin, Changchun, 130102, China

Self-Supervised Time Series Representation LearningPhotoplethysmographyCross-Modal Representation Learning
AskPDF
Mar 31, 2026

Temporal Contrastive Transformer for Financial Crime Detection: Self-Supervised Sequence Embeddings via Predictive Contrastive Coding

We introduce the Temporal Contrastive Transformer (TCT), a representation learning framework designed to capture contextual temporal dynamics in sequences of financial transactions. The model is trained using a self-supervised contrastive objective to produce embeddings that encode behavioral patterns over time, with the goal of supporting downstream fraud detection tasks. We evaluate TCT in a realistic setting by using the learned embeddings as input features to a gradient boosting classifier. Experimental results show that embeddings alone achieve meaningful predictive performance (AUC 0.8644), indicating that the model captures non-trivial temporal structure. However, when combined with domain-engineered features, no measurable improvement is observed over the baseline (AUC 0.9205 vs. 0.9245), suggesting that the learned representations largely overlap with existing feature abstractions. These findings position TCT as a promising representation learning approach that captures relevant behavioral signal, while highlighting the challenges of achieving additive value over strong domain features. The results reflect an intermediate stage in the development of temporal representation learning for financial crime detection and motivate further research on model architecture, training objectives, and integration strategies. At this early stage, achieving performance comparable to a strong feature-engineered baseline is itself a meaningful outcome, indicating that learned representations approximate domain-specific features without manual engineering. While not yet production-ready, these results point to a promising direction for reducing reliance on feature engineering in financial crime detection.

Danny Butvinik, Yonit Marcus, Nitzan Tal +1

NICE Actimize

Self-Supervised Time Series Representation LearningPredictive Representation LearningFinancial Fraud DetectionTime Series TransformersContrastive Learning
AskPDF
Mar 27, 2026

Foundation Model for Cardiac Time Series via Masked Latent Attention

Electrocardiograms (ECGs) are among the most widely available clinical signals and play a central role in cardiovascular diagnosis. While recent foundation models (FMs) have shown promise for learning transferable ECG representations, most existing pretraining approaches treat leads as independent channels and fail to explicitly leverage their strong structural redundancy. We introduce the latent attention masked autoencoder (LAMAE) FM that directly exploits this structure by learning cross-lead connection mechanisms during self-supervised pretraining. Our approach models higher-order interactions across leads through latent attention, enabling permutation-invariant aggregation and adaptive weighting of lead-specific representations. We provide empirical evidence on the Mimic-IV-ECG database that leveraging the cross-lead connection constitutes an effective form of structural supervision, improving representation quality and transferability. Our method shows strong performance in predicting ICD-10 codes, outperforming independent-lead masked modeling and alignment-based baselines.

Moritz Vandenhirtz, Samuel Ruipérez-Campillo, Simon Böhi +6

Department of Computer Science, ETH Zurich, Switzerland · Department of Biomedical Engineering, University of Basel, Switzerland

Electrocardiogram Representation LearningElectrocardiogram Foundation ModelsSelf-Supervised Time Series Representation LearningElectrocardiographyMasked AutoencodersElectrocardiogram Classification
AskPDF
Mar 25, 2026

Contrastive Learning Boosts Deterministic and Generative Models for Weather Data

Weather data, comprising multiple variables, poses significant challenges due to its high dimensionality and multimodal nature. Creating low-dimensional embeddings requires compressing this data into a compact, shared latent space. This compression is required to improve the efficiency and performance of downstream tasks, such as forecasting or extreme-weather detection. Self-supervised learning, particularly contrastive learning, offers a way to generate low-dimensional, robust embeddings from unlabelled data, enabling downstream tasks when labelled data is scarce. Despite initial exploration of contrastive learning in weather data, particularly with the ERA5 dataset, the current literature does not extensively examine its benefits relative to alternative compression methods, notably autoencoders. Moreover, current work on contrastive learning does not investigate how these models can incorporate sparse data, which is more common in real-world data collection. It is critical to explore and understand how contrastive learning contributes to creating more robust embeddings for sparse weather data, thereby improving performance on downstream tasks. Our work extensively explores contrastive learning on the ERA5 dataset, aligning sparse samples with complete ones via a contrastive loss term to create SPARse-data augmented conTRAstive spatiotemporal embeddings (SPARTA). We introduce a temporally aware batch sampling strategy and a cycle-consistency loss to improve the structure of the latent space. Furthermore, we propose a novel graph neural network fusion technique to inject domain-specific physical knowledge. Ultimately, our results demonstrate that contrastive learning is a feasible and advantageous compression method for sparse geoscience data, thereby enhancing performance in downstream tasks.

Nathan Bailey

Supervised by Dr Sibo Cheng, Faculty of Engineering, Imperial College London and Dr Cesar Quilodran-Casas, The Grantham Institute for Climate Change, Faculty of Natural Sciences, Imperial College London.

Contrastive LearningGeospatial Representation LearningSelf-Supervised Time Series Representation Learning
AskPDF
Mar 19, 2026

LuMamba: Latent Unified Mamba for Electrode Topology-Invariant and Efficient EEG Modeling

Electroencephalography (EEG) enables non-invasive monitoring of brain activity across clinical and neurotechnology applications, yet building foundation models for EEG remains challenging due to differing electrode topologies and computational scalability, as Transformer architectures incur quadratic sequence complexity. As a joint solution, we propose LuMamba (Latent Unified Mamba), a self-supervised framework combining topology-invariant encodings with linear-complexity state-space modeling, using LUNA's learned-query cross-attention mechanism for channel unification, and FEMBA's bidirectional Mamba blocks for efficient temporal modeling. Within this architecture, we provide the first systematic investigation of the Latent-Euclidean Joint-Embedding Predictive Architecture (LeJEPA) for biosignal learning. Pre-trained on over 21,000 hours of unlabeled EEG from the TUEG corpus, LuMamba is evaluated on five downstream tasks spanning abnormality detection, artifact recognition, and mental condition classification across electrode configurations ranging from 16 to 26 channels. In the pre-training objective, masked reconstruction alone yields structured but less generalizable representations, while LeJEPA alone produces diffuse embeddings; combining both objectives achieves the most robust performance. With only 4.6M parameters, LuMamba attains 80.99% balanced accuracy on TUAB and achieves state-of-art performance on Alzheimer's detection (0.97 AUPR), while requiring 377x fewer FLOPS than state-of-art models at equivalent sequence lengths and scaling to 12x longer sequences before reaching typical GPU memory limits. Code is available at https://github.com/pulp-bio/biofoundation.

Danaé Broustail, Anna Tegon, Thorir Mar Ingolfsson +2

Integrated Systems Laboratory, ETH Zürich, Zürich, Switzerland · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · DEI, University of Bologna, Bologna, Italy

EEG Foundation ModelsElectroencephalographySelf-Supervised Time Series Representation LearningMambaJoint-Embedding Predictive Architecture
AskPDF
Mar 18, 2026

BoundAD: Boundary-Aware Negative Generation for Time Series Anomaly Detection

Contrastive learning methods for time series anomaly detection (TSAD) heavily depend on the quality of negative sample construction. However, existing strategies based on random perturbations or pseudo-anomaly injection often struggle to simultaneously preserve temporal semantic consistency and provide effective decision-boundary supervision. Most existing methods rely on prior anomaly injection, while overlooking the potential of generating hard negatives near the data manifold boundary directly from normal samples themselves. To address this issue, we propose a reconstruction-driven boundary negative generation framework that automatically constructs hard negatives through the reconstruction process of normal samples. Specifically, the method first employs a reconstruction network to capture normal temporal patterns, and then introduces a reinforcement learning strategy to adaptively adjust the optimization update magnitude according to the current reconstruction state. In this way, boundary-shifted samples close to the normal data manifold can be induced along the reconstruction trajectory and further used for subsequent contrastive representation learning. Unlike existing methods that depend on explicit anomaly injection, the proposed framework does not require predefined anomaly patterns, but instead mines more challenging boundary negatives from the model's own learning dynamics. Experimental results show that the proposed method effectively improves anomaly representation learning and achieves competitive detection performance on the current dataset.

Xiancheng Wang, Lin Wang, Zhibo Zhang +2

School of Ocean Engineering, Harbin Institute of Technology, West Wenhua Road, Weihai, 264209, Shandong, China. · Technical Center, Bogie Development Department, CRRC Qingdao Sifang Locomotive and Rolling Stock Co., Ltd., Jinhong East Road, Qingdao, 266111, Shandong, China.

Time-Series Anomaly DetectionUnsupervised Anomaly DetectionHard Negative MiningSynthetic Anomaly GenerationSelf-Supervised Time Series Representation Learning
AskPDF
Mar 17, 2026

Laya: A LeJEPA Approach to EEG via Latent Prediction over Reconstruction

Electroencephalography (EEG) is a widely used tool for studying brain function, with applications in clinical neuroscience, diagnosis, and brain-computer interfaces (BCIs). Recent EEG foundation models trained on large unlabeled corpora aim to learn transferable representations, but their effectiveness remains unclear; reported improvements over smaller task-specific models are often modest, sensitive to downstream adaptation and fine-tuning strategies, and limited under linear probing. We hypothesize that one contributing factor is the reliance on signal reconstruction as the primary self-supervised learning (SSL) objective, which biases representations toward high-variance artifacts rather than task-relevant neural structure. To address this limitation, we explore an SSL paradigm based on Joint Embedding Predictive Architectures (JEPA), which learn by predicting latent representations instead of reconstructing raw signals. We introduce Laya, the first EEG foundation model based on LeJEPA. We show that latent prediction yields representations that encode semantic structure in EEG: Laya embeddings track clinically meaningful state changes such as seizure onset, are resilient to noise, and achieve the strongest mean clinical accuracy under frozen linear probing, with particular gains on tasks where relevant neural patterns are subtle and easily obscured by artifacts. Controlled ablations against matched MAE variants confirm that the choice of pretraining objective, rather than architecture or data, is the primary driver of these gains.

Saarang Panchavati, Uddhav Panchavati, Hiroki Nariai +2

Department of Medical Informatics UCLA Los Angeles, USA · Department of Cognitive Science UCSD San Diego, USA · David Geffen School of Medicine UCLA Mattel Children’s Hospital Los Angeles, USA

EEG Foundation ModelsSelf-Supervised Time Series Representation LearningPredictive Representation LearningJoint-Embedding Predictive Architecture
AskPDF
Mar 13, 2026

Representation Learning for Spatiotemporal Physical Systems

Machine learning approaches to spatiotemporal physical systems have primarily focused on next-frame prediction, with the goal of learning an accurate emulator for the system's evolution in time. However, these emulators are computationally expensive to train and are subject to performance pitfalls, such as compounding errors during autoregressive rollout. In this work, we take a different perspective and look at scientific tasks further downstream of predicting the next frame, such as estimation of a system's governing physical parameters. Accuracy on these tasks offers a uniquely quantifiable glimpse into the physical relevance of the representations of these models. We evaluate the effectiveness of general-purpose self-supervised methods in learning physics-grounded representations that are useful for downstream scientific tasks. Surprisingly, we find that not all methods designed for physical modeling outperform generic self-supervised learning methods on these tasks, and methods that learn in the latent space (e.g., joint embedding predictive architectures, or JEPAs) outperform those optimizing pixel-level prediction objectives. Code is available at https://github.com/helenqu/physical-representation-learning.

Helen Qu, Rudy Morel, Michael McCabe +4

Flatiron Institute · Princeton University · Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM +1

Representation LearningScientific MLSelf-Supervised Time Series Representation LearningJoint-Embedding Predictive Architecture
AskPDF
Mar 12, 2026

Multi-Station WiFi CSI Sensing Framework Robust to Station-wise Feature Missingness and Limited Labeled Data

We propose a WiFi Channel State Information (CSI) sensing framework for multi-station deployments that addresses two fundamental challenges in practical CSI sensing: station-wise feature missingness and limited labeled data. Feature missingness is commonly handled by resampling unevenly spaced CSI measurements or by reconstructing missing samples, while label scarcity is mitigated by data augmentation or self-supervised representation learning. However, these techniques are typically developed in isolation and do not jointly address long-term, structured station unavailability together with label scarcity. To bridge this gap, we explicitly incorporate station unavailability into both representation learning and downstream model training. Specifically, we adapt cross-modal self-supervised learning (CroSSL), a representation learning framework originally designed for time-series sensory data, to multi-station CSI sensing in order to learn representations that are inherently invariant to station-wise feature missingness from unlabeled data. Furthermore, we introduce Station-wise Masking Augmentation (SMA) during downstream model training, which exposes the model to realistic station unavailability patterns under limited labeled data. Our experiments show that neither missingness-invariant pre-training nor station-wise augmentation alone is sufficient; their combination is essential to achieve robust performance under both station-wise feature missingness and label scarcity. The proposed framework provides a practical and robust foundation for multi-station WiFi CSI sensing in real-world deployments.

Keita Kayano, Takayuki Nishio, Daiki Yoda +2

School of Engineering, Institute of Science Tokyo, 2-12-1 Ookayama, Tokyo, 152-8550, Japan · Wireless Systems R&D Department, Infrastructure Systems R&D Center, Corporate Laboratory, Toshiba Corporation, 1, Komukai Toshiba-cho, Saiwai-ku, Kawasaki 212-8582, JAPAN

Wi-Fi SensingLearning with Missing DataSelf-Supervised Time Series Representation LearningSelf-Supervised Learning
AskPDF
Mar 11, 2026

Bio-Inspired Self-Supervised Learning for Wrist-worn Accelerometer Data

Wearable accelerometers enable large-scale health monitoring, yet learning robust human-activity representations has been constrained by scarce labeled data. While self-supervised learning offers a remedy, existing methods treat sensor streams as unstructured time series, overlooking the underlying biological structure of human movement, a factor we argue is critical for effective Human Activity Recognition (HAR). We introduce a novel tokenization strategy grounded in the submovement theory of motor control, which posits that continuous wrist motion is composed of elementary basis functions called submovements. We define our token as the movement segment, a computationally tractable unit of motion composed of a finite sequence of submovements. By pretraining a Transformer encoder via masked reconstruction of these tokens, we shift the learning focus from local waveform morphology to high-level structural and temporal organization. Pretrained on the NHANES corpus (approximately 28k hours; 11k participants), our representations outperform strong wearable SSL baselines across six subject-disjoint HAR benchmarks. Code and pretrained weights are available at https://prithvitarale.github.io/biopm-site/.

Prithviraj Tarale, Kiet Chu, Abhishek Varghese +4

College of Information and Computer Sciences, University of Massachusetts, Amherst, United States · Stevens Institute of Technology, Hoboken, United States · Google Health, Seattle, United States +1

Human Activity RecognitionSelf-Supervised Time Series Representation LearningMotion Representation LearningWearable Sensing
AskPDF
Mar 9, 2026

Echo2ECG: Enhancing ECG Representations with Cardiac Morphology from Multi-View Echos

Electrocardiography (ECG) is a low-cost, widely used modality for diagnosing electrical abnormalities like atrial fibrillation by capturing the heart's electrical activity. However, it cannot directly measure cardiac morphological phenotypes, such as left ventricular ejection fraction (LVEF), which typically require echocardiography (Echo). Predicting these phenotypes from ECG would enable early, accessible health screening. Existing self-supervised methods suffer from a representational mismatch by aligning ECGs to single-view Echos, which only capture local, spatially restricted anatomical snapshots. To address this, we propose Echo2ECG, a multimodal self-supervised learning framework that enriches ECG representations with the heart's morphological structure captured in multi-view Echos. We evaluate Echo2ECG as an ECG feature extractor on two clinically relevant tasks that fundamentally require morphological information: (1) classification of structural cardiac phenotypes across three datasets, and (2) retrieval of Echo studies with similar morphological characteristics using ECG queries. Our extracted ECG representations consistently outperform those of state-of-the-art unimodal and multimodal baselines across both tasks, despite being 18x smaller than the largest baseline. These results demonstrate that Echo2ECG is a robust, powerful ECG feature extractor. Our code is accessible at https://github.com/michelleespranita/Echo2ECG.

Michelle Espranita Liman, Özgün Turgut, Alexander Müller +3

Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Germany · Department of Cardiology, TUM University Hospital, Germany · Department of Computing, Imperial College London, UK +1

Electrocardiogram Representation LearningMultimodal LearningElectrocardiogram ClassificationSelf-Supervised Time Series Representation LearningCross-Modal Retrieval
AskPDF
Mar 2, 2026

UTICA: Multi-Objective Self-Distllation Foundation Model Pretraining for Time Series Classification

Self-supervised foundation models have achieved remarkable success across domains, including time series. However, the potential of non-contrastive methods, a paradigm that has driven significant advances in computer vision, remains underexplored for time series. In this work, we adapt DINOv2-style self-distillation to pretrain a time series foundation model, building on the Mantis tokenizer and transformer encoder architecture as our backbone. Through a student-teacher framework, our method Utica learns representations that capture both temporal invariance via augmented crops and fine-grained local structure via patch masking. Our approach achieves state-of-the-art classification performance on both UCR and UEA benchmarks. These results suggest that non-contrastive methods are a promising and complementary pretraining strategy for time series foundation models.

Yessin Moakher, Youssef Attia El Hili, Vasilii Feofanov

Ecole Polytechnique, France · Huawei Noah’s Ark Lab

Self-Supervised Time Series Representation LearningTime-Series Foundation ModelsTime Series ClassificationSelf-Supervised LearningSelf-Distillation
AskPDF
Feb 28, 2026

Interpretable Cross-Network Attention for Resting-State fMRI Representation Learning

Understanding how large-scale functional brain networks reorganize during cognitive decline remains a central challenge in neuroimaging. While recent self-supervised models have shown promise for learning representations from resting-state fMRI, their internal mechanisms are difficult to interpret, limiting mechanistic insight. We propose BrainInterNet, a network-aware self-supervised framework based on masked reconstruction with cross-attention that explicitly models inter-network dependencies in rs-fMRI. By selectively masking predefined functional networks and reconstructing them from remaining context, our approach enables direct quantification of network predictability and interpretable analysis of cross-network interactions. We train BrainInterNet on multi-cohort fMRI data (from the ABCD, HCP Development, HCP Young Adults, and HCP Aging datasets) and evaluate on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, in total comprising 5,582 recordings. Our method reveals systematic alterations in the brain's network interactions under AD, including in the default mode, limbic, and attention networks. In parallel, the learned representations support accurate Alzheimer's-spectrum classification and yield a compact summary marker that tracks disease severity longitudinally. Together, these results demonstrate that network-guided masked modeling with cross-attention provides an interpretable and effective framework for characterizing functional reorganization in neurodegeneration.

Karanpartap Singh, Adam Turnbull, Mohammad Abbasi +3

Stanford University, Stanford, CA 94305

Brain Network AnalysisFunctional ConnectivitySelf-Supervised Time Series Representation LearningNeural Network InterpretabilityAlzheimer's Disease ClassificationCross-Attention
AskPDF
Feb 26, 2026

RhythmBERT: A Self-Supervised Language Model Based on Latent Representations of ECG Waveforms for Heart Disease Detection

Electrocardiogram (ECG) analysis is crucial for diagnosing heart disease, but most self-supervised learning methods treat ECG as a generic time series, overlooking physiologic semantics and rhythm-level structure. Existing contrastive methods utilize augmentations that distort morphology, whereas generative approaches employ fixed-window segmentation, which misaligns cardiac cycles. To address these limitations, we propose RhythmBERT, a generative ECG language model that considers ECG as a language paradigm by encoding P, QRS, and T segments into symbolic tokens via autoencoder-based latent representations. These discrete tokens capture rhythm semantics, while complementary continuous embeddings retain fine-grained morphology, enabling a unified view of waveform structure and rhythm. RhythmBERT is pretrained on approximately 800,000 unlabeled ECG recordings with a masked prediction objective, allowing it to learn contextual representations in a label-efficient manner. Evaluations show that despite using only a single lead, RhythmBERT achieves comparable or superior performance to strong 12-lead baselines. This generalization extends from prevalent conditions such as atrial fibrillation to clinically challenging cases such as subtle ST-T abnormalities and myocardial infarction. Our results suggest that considering ECG as structured language offers a scalable and physiologically aligned pathway for advancing cardiac analysis.

Xin Wang, Burcu Ozek, Aruna Mohan +3

Virginia Tech, VA, USA · Walkky LLC, Boston, MA, USA · Northeastern University, Boston, MA, USA +1

Electrocardiogram Representation LearningElectrocardiogram Foundation ModelsElectrocardiogram InterpretationElectrocardiogram ClassificationSelf-Supervised Time Series Representation LearningSelf-Supervised Pre-Training
AskPDF
Feb 23, 2026

A general extractor for coherent patterns across time-frequency measurements

Scientific discovery increasingly depends on finding meaningful patterns within volumes of sensor data too large for manual inspection. Existing time-frequency event extractors are often engineered or trained on manually labeled examples for each instrument, which can be laborious and difficult to adapt to new environments. To address this, we introduce TokEye, a reference-free method that learns a general criterion for coherent structure from heterogeneous fusion measurements. A model developed under this method, without retraining, can extract signals from various other fields including marine bioacoustics and wideband radar. It recovers 82.5% of expert-annotated stellarator signals, detects 78% of dolphin whistles, and identifies radar events with 81.5% average precision across six benchmarks, while processing full experimental spectrograms in under one second on a GPU. This transfer across unrelated sensors and physical systems reveals common exploitable structure in time-frequency measurements and establishes a scalable foundation for automated scientific signal discovery.

Nathaniel Chen, Kouroche Bouchiat, Peter Steiner +7

Time-Frequency AnalysisSelf-Supervised Time Series Representation Learning
AskPDF
Feb 22, 2026

LWM-Temporal: Sparse Spatio-Temporal Attention for Wireless Channel Representation Learning

LWM-Temporal is a new member of the Large Wireless Models (LWM) family that targets the spatiotemporal nature of wireless channels. Designed as a task-agnostic foundation model, LWM-Temporal learns universal channel embeddings that capture mobility-induced evolution and are reusable across various downstream tasks. To achieve this objective, LWM-Temporal operates in the angle-delay-time domain and introduces Sparse Spatio-Temporal Attention (SSTA), a propagation-aligned attention mechanism that restricts interactions to physically plausible neighborhoods, reducing attention complexity by an order of magnitude while preserving geometry-consistent dependencies. LWM-Temporal is pretrained in a self-supervised manner using a physics-informed masking curriculum that emulates realistic occlusions, pilot sparsity, and measurement impairments. Experimental results on channel prediction across multiple mobility regimes show consistent improvements over strong baselines, particularly under long horizons and limited fine-tuning data, highlighting the importance of geometry-aware architectures and geometry-consistent pretraining for learning transferable spatiotemporal wireless representations.

Sadjad Alikhani, Akshay Malhotra, Shahab Hamidi-Rad +1

Spatio-Temporal AttentionWireless CommunicationsWireless Foundation ModelsRepresentation LearningSparse AttentionSelf-Supervised Time Series Representation Learning
AskPDF
Feb 18, 2026

BrainRVQ: A High-Fidelity EEG Foundation Model via Dual-Domain Residual Quantization and Hierarchical Autoregression

Developing foundation models for electroencephalography (EEG) remains challenging due to the signal's low signal-to-noise ratio and complex spectro-temporal non-stationarity. Existing approaches often overlook the hierarchical latent structure inherent in neural dynamics, leading to suboptimal reconstruction of fine-grained information. In this work, we propose BrainRVQ, a general-purpose EEG foundation model pre-trained on a large-scale corpus of clinical EEG data. Unlike standard masked modeling, BrainRVQ features a Dual-Domain Residual Vector Quantization (DD-RVQ) tokenizer that disentangles temporal waveforms and spectral patterns into hierarchical discrete codes. We further introduce a hierarchical autoregressive pre-training objective that learns to reconstruct these codes in a coarse-to-fine manner, utilizing an importance-guided curriculum masking strategy to prioritize information-rich neural events over background noise. Extensive experiments across 8 diverse downstream datasets demonstrate that BrainRVQ consistently outperforms state-of-the-art baselines, validating its effectiveness in learning robust and generalizable neural representations. Our code and model weights are available:https://github.com/keqicmz/BrainRVQ

Mingzhe Cui, Tao Chen, Yang Jiao +4

State Key Laboratory of Industrial Control Technology, Zhejiang University, Hangzhou, China · Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy · Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China

EEG Foundation ModelsSelf-Supervised Time Series Representation LearningResidual VQAutoregressive Modeling
AskPDF
Feb 7, 2026

SleepMaMi: A Universal Sleep Foundation Model for Integrating Macro- and Micro-structures

While the shift toward unified foundation models has revolutionized many deep learning domains, sleep medicine remains largely restricted to task-specific models that focus on localized micro-structure features. These approaches often neglect the rich, multi-modal context of Polysomnography (PSG) and fail to capture the global macro-structure of a full night's sleep. To address this, we introduce SleepMaMi , a Sleep Foundation Model engineered to master both hour-long sleep architectures and fine-grained signal morphologies. Our framework utilizes a hierarchical dual-encoder design: a Macro-Encoder to model full-night temporal dependencies and a Micro-Encoder to capture short-term characteristics from biosignals. Macro-Encoder is trained via Demographic-Guided Contrastive Learning, which aligns overnight sleep patterns with objective subject metadata, such as age, sex and BMI to refine global representations. Micro-Encoder is optimized via a hybrid Masked Autoencoder (MAE) and multi-modal contrastive objective. Pre-trained on a massive corpus of >>20,000 PSG recordings (158K hours),SleepMaMi outperforms or matches state-of-the-art existing foundation models across a diverse suite of downstream tasks, demonstrating superior generalizability and label-efficient adaptation for clinical sleep analysis.

Keondo Park, Younghoon Na, Yourim Choi +3

Graduate School of Data Science, Seoul National University, Seoul, South Korea · Department of Biomedical Sciences, Seoul National University College of Medicine, Seoul, Republic of Korea · Obstructive Upper Airway Research (OUaR) Laboratory, Department of Pharmacology, Seoul National University College of Medicine, Seoul, Republic of Korea +1

Self-Supervised Time Series Representation Learning
AskPDF
Feb 1, 2026

PaAno: Patch-Based Representation Learning for Time-Series Anomaly Detection

Although recent studies on time-series anomaly detection have increasingly adopted ever-larger neural network architectures such as transformers and foundation models, they incur high computational costs and memory usage, making them impractical for real-time and resource-constrained scenarios. Moreover, they often fail to demonstrate significant performance gains over simpler methods under rigorous evaluation protocols. In this study, we propose Patch-based representation learning for time-series Anomaly detection (PaAno), a lightweight yet effective method for fast and efficient time-series anomaly detection. PaAno extracts short temporal patches from time-series training data and uses a 1D convolutional neural network to embed each patch into a vector representation. The model is trained using a combination of triplet loss and pretext loss to ensure the embeddings capture informative temporal patterns from input patches. During inference, the anomaly score at each time step is computed by comparing the embeddings of its surrounding patches to those of normal patches extracted from the training time-series. Evaluated on the TSB-AD benchmark, PaAno achieved state-of-the-art performance, significantly outperforming existing methods, including those based on heavy architectures, on both univariate and multivariate time-series anomaly detection across various range-wise and point-wise performance measures.

Jinju Park, Seokho Kang

Department of Industrial Engineering, Sungkyunkwan University

Self-Supervised Time Series Representation LearningTime Series Representation LearningTime-Series Anomaly Detection
AskPDF
Sep 22, 2025

SingLEM: Single-Channel Large EEG Model

Current deep learning models for electroencephalography (EEG) are often task-specific and depend on large labeled datasets, limiting their adaptability. Although EEG foundation models seek broader applicability, many still rely on predefined multi-channel inputs, electrode-layout assumptions, or model-specific channel handling. To address these limitations, we introduce the Single-Channel Large EEG Model (SingLEM), a self-supervised foundation model whose hybrid convolutional--Transformer encoder maps each channel independently to a reusable representation capturing local and long-range temporal structure. These representations can be used individually or combined through channel-wise feature concatenation. We assembled 71 public EEG datasets comprising approximately 9,200 subjects and 357,000 single-channel hours. For leakage-controlled evaluation, downstream results were obtained with a model pretrained on 68 datasets after excluding the three source datasets underlying the six tasks. A model pretrained on all 71 datasets is provided for general reuse. Across six motor imagery and cognitive tasks under strict leave-one-subject-out (LOSO) evaluation, the leakage-controlled model with concatenated representations and a support vector machine (SVM) classifier achieved the best overall performance among the compared pretrained and classical feature-based methods. Additional classifier and subject-adapted analyses supported the robustness of its representations. These findings support single-channel self-supervised learning as a montage-flexible foundation for reusable EEG feature extraction and electrode-level spatial analysis. The source code and pretrained models are available at https://github.com/ttlabtuat/SingLEM.

Jamiyan Sukhbaatar, Satoshi Imamura, Ibuki Inoue +5

Department of Electronic and Information Engineering, Tokyo University of Agriculture and Technology, Koganei-Shi 184–8588, Japan · Department of Electronics and Communication Engineering, National University of Mongolia, Ulaanbaatar, 14200 Mongolia

Representation LearningSelf-Supervised Time Series Representation LearningEEG DecodingEEG Foundation Models
AskPDF
Aug 28, 2025

EEGDM: Learning EEG Representation with Latent Diffusion Model

Recent advances in self-supervised learning for EEG representation have largely relied on masked reconstruction, where models are trained to recover randomly masked signal segments. While effective at modeling local dependencies, the training objective of masked reconstruction does not compel the model to capture global generative constraints essential for characterizing neural activity. To address this limitation, we propose EEGDM, a novel self-supervised framework that leverages latent diffusion models to generate EEG signals as an objective. Unlike masked reconstruction, diffusion-based generation progressively denoises signals from noise to realism, compelling the model to capture holistic temporal patterns and cross-channel relationships. Specifically, EEGDM incorporates an EEG encoder that distills raw signals and their channel augmentations into a compact representation, which serves as conditional information to guide the diffusion denoising process, thereby enabling the encoder and diffusion model to be jointly optimized through the generative objective. This design endows EEGDM with a compact latent space, which not only offers ample control over the generative process but also can be leveraged for downstream tasks. Experimental results show that EEGDM (1) reconstructs high-quality EEG signals, (2) learns robust representations, and (3) achieves competitive performance across diverse downstream tasks, thus exploring a new direction for self-supervised EEG representation learning.

Shaocong Wang, Tong Liu, Yihan Li +6

Tsinghua University · Tianjin University

ElectroencephalographyEEG Signal GenerationLatent Diffusion ModelsSelf-Supervised Time Series Representation LearningTime Series Generation
AskPDF
Aug 13, 2025

EEGDM: Label-Efficient EEG Representation Learning with Generative Diffusion Model

Electroencephalography (EEG) is a critical tool for monitoring brain activity and diagnosing neurological disorders such as epilepsy. However, learning meaningful representations from raw EEG signals remains challenging due to limited annotations, substantial inter-subject variability, and complex temporal dynamics. Recent EEG foundation models (FMs) have demonstrated promising performance through transformer-based architectures and large-scale self-supervised pretraining, yet they often incur substantial computational costs and exhibit diminishing returns with increasing model and dataset scale, limiting their practicality in clinical settings. To address these challenges, we propose EEGDM, a diffusion-based EEG representation learning framework. Specifically, EEGDM introduces a Structured State-Space Model for Diffusion Pretraining (SSMDP) that effectively captures long-range temporal dependencies through generative diffusion training on unlabeled EEG data. The representations are subsequently leveraged for downstream tasks via our Latent Fusion Module (LFM), which integrates multi-layer latent features of SSMDP. We evaluate EEGDM on three EEG benchmarks spanning EEG event classification (TUEV), seizure detection (CHB-MIT), and seizure classification (IIIC). Compared with existing state-of-the-art methods, including EEG FMs, EEGDM achieves competitive performance across diverse tasks and datasets, exceeding existing methods in most cases while requiring substantially fewer samples for both pretraining and downstream adaptation. These results demonstrate that diffusion-based learning can unlock more discriminative EEG representations, with direct implications for epilepsy diagnosis and management. Our source code and pretrained checkpoints are publicly available at: https://github.com/jhpuah/EEGDM.

Jia Hong Puah, Sim Kuan Goh, Ziwei Zhang +6

School of Artificial Intelligence and Robotics, Xiamen University Malaysia · School of Energy and Chemical Engineering, Xiamen University Malaysia · Department of Biomedical Engineering, Universiti Malaya +3

ElectroencephalographyEEG Seizure DetectionRepresentation LearningSelf-Supervised Time Series Representation LearningDiffusion ModelsData-Efficient Learning+1
AskPDF
Jul 22, 2025

FISHER: A Foundation Model for Multi-Modal Industrial Signal Comprehensive Representation

Industrial signal analysis is hindered by severe data heterogeneity, which we characterize as the M5 problem. Existing solutions rely on specialized models that lack robustness and scalability, while large-scale pre-training has rarely been investigated in this area. In this work, we derive a prioritized roadmap for the M5 problem and propose FISHER, a Foundation model for multi-modal Industrial Signal compreHEnsive Representation. To address the foremost multi-sampling-rate problem, FISHER utilizes a novel sub-band modeling approach that treats sampling rate increments as concatenated sub-band information, enabling the adaptive usage of full signal bandwidth without resampling. FISHER is pre-trained by teacher-student self-distillation over external audio and music data. We also establish the RMIS benchmark, comprising 19 datasets across four modalities. In the experiment, FISHER outperforms 24 state-of-the-art series encoders (up to 2B) with much smaller sizes (up to 16x), showcasing groundbreaking diagnostic accuracy and remarkable versatility. We further demonstrate that 1) seamless adaptation to variable sampling rates is the key to generalization 2) audio and music data provide better temporal variability, which is essential for pre-training. Both FISHER and RMIS are open-sourced.

Pingyi Fan, Anbai Jiang, Shuwei Zhang +9

Department of Electronic Engineering, Tsinghua University, Beijing, 100084, P. R. China · Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, 200240, P. R. China · Huakong AI Plus Company Limited, Beijing, 100084, P. R. China +1

Representation LearningSelf-Supervised Time Series Representation LearningTime-Series Foundation ModelsMultiscale Time-Series Modeling
AskPDF
Feb 21, 2025

Mantis: Lightweight Foundation Model for Time Series Classification

While foundation models have revolutionized various domains, their application to time series classification remains rather under-explored, with existing literature predominantly focused on forecasting. To bridge this gap, we introduce Mantis, a transformer-based foundation model pre-trained exclusively on synthetic data via self-supervised contrastive learning. We demonstrate that effective tokenization is critical to unlocking the full potential of transformers, proposing a novel token generator unit. Furthermore, we introduce an enhanced test-time methodology that bridges the performance gap between Mantis and strong specialized approaches by leveraging intermediate-layer representations, self-ensembling, and cross-model embedding fusion. Extensive experiments demonstrate that Mantis establishes a new state-of-the-art, outperforming existing foundation models across four diverse dataset collections covering various application domains.

Vasilii Feofanov, Songkang Wen, Shifeng Xie +10

1. Huawei Noah’s Ark Lab · 3. Paris Cit´e University · 4. Helmholtz Munich +3

Time Series ClassificationSelf-Supervised Time Series Representation LearningTime Series TokenizationTime-Series Foundation Models
AskPDF
Feb 7, 2025

Stochastic Optimal Control for Continuous-Time fMRI Representation Learning

Learning robust representations from functional magnetic resonance imaging (fMRI) is fundamentally challenged by the temporal irregularity and noise inherent in data from heterogeneous sources. Existing self-supervised learning (SSL) methods often discard critical temporal information by discretizing or averaging fMRI signals. To address this, we introduce a novel framework that reframes SSL as a Stochastic Optimal Control (SOC) problem. Our approach models brain activity as continuous-time latent dynamics, learning a robust representation of brain dynamics by optimizing a control policy that is agnostic to the temporal irregularity. This SOC framework naturally unifies masked autoencoding (MAE) and joint-embedding prediction (JEPA) to extract compact, control-derived representations. Furthermore, a simulation-free inference strategy ensures computational efficiency and scalability for large-scale fMRI datasets. Our model demonstrates state-of-the-art performance across diverse downstream applications, highlighting the potential of the SOC-based continuous-time representation learning framework.

Joonhyeong Park, Byoungwoo Park, Chang-Bae Bang +4

KAIST · Yonsei University · EverEx +1

Latent Dynamics ModelingSelf-Supervised Time Series Representation LearningfMRI Representation LearningPredictive Representation LearningMedical Image Representation Learning
AskPDF