Time series such as recordings of human motion data often encode structure across multiple temporal scales, and when a latent state determining this sequence evolves, this structure needs to survive in the representation space for downstream use. Yet the currently prevalent attention-based encoders paired with sequence-level pooling have made single-vector, pointwise comparison the default mode of inference, discarding the trajectory a sequence traces through the representation space. We propose recovering this geometric structure directly, without relying on forecasting-based objectives, by training an encoder model (SemTAC) with a temporal-neighborhood contrastive loss: representations that satisfy local consistency everywhere must give rise to a coherent global structure, letting inference happen at temporal scales beyond those seen during training. We evaluate SemTAC against four baselines on a five-step motion sequence discrimination task: the same architecture trained with a classification or a standard contrastive objective, and two established self-supervised time-series methods, TS-TCC and TS2Vec. At the full sequence length all five are broadly competitive, but on incomplete, truncated sequences SemTAC holds a large advantage: a median F1 of 0.77 one step before the sequence completes, against 0.27-0.70 for the four baselines. A hyperparameter-matched ablation attributes this effect specifically to the temporal-contrastive objective. SemTAC's representation trajectories are also the only ones among these five to exhibit interpretable geometric structures, exceeding a no-structure chance baseline on a displacement-smoothness metric which we introduce. This is evidence that its objective, not classification accuracy alone, produces genuinely temporally-ordered trajectories.
Figures & tables
Figure 1: Training our model with temporal contrasts induces a structure in the latent space that mirrors the temporal structure of the data. (a) shows the representation space when trained with purely discrimination-centric objective, (b) shows the original data structure (see Fig. 2 ), and (c) is the latent space created by our approach.
Figure 2: Generated motion sequences with divergent structure. Both sequence types begin identically (jump, step), then diverge toward a preferential lateralization before performing completely unrelated motions. Step Right/Left labels are 60/40% corrupted (only 60% match the stated direction); Wave Right/Left labels are 80/20% corrupted.
Distance
TS2Vec †
TS-TCC †
SemTAC
Self-similarity ‡
Classification ‡
Fréchet
0.81±0.04 *
0.55±0.03 *
0.87±0.12
0.83±0.15
0.95±0.06 *
DTW
0.91±0.02 *
0.82±0.03 *
0.86±0.06
0.89±0.07
0.95±0.06 *
Table 1: F1-scores (mean ± std) per approach, using Fréchet or DTW as the latent sequence distance.
Distance
TS2Vec †
TS-TCC †
SemTAC
Self-similarity ‡
Classification ‡
Fréchet
0.40±0.19 *
0.51±0.03 *
0.68±0.21
0.37±0.28 *
0.37±0.32 *
DTW
0.70±0.05 *
0.47±0.12 *
0.72±0.19
0.64±0.15 *
0.63±0.25
Table 2: F1-scores (mean ± std) per approach, using Fréchet or DTW as the latent sequence distance.
Metric
TS2Vec †
TS-TCC †
SemTAC
Self-similarity ‡
Classification ‡
Smoothness
−0.57±0.02 *
−0.53±0.01 *
−0.26±0.30
−0.58±0.12 *
−0.58±0.17 *
Order Rec.
0.67±0.04 *
0.48±0.04 *
0.79±0.10
0.66±0.15 *
0.73±0.11
(Exp. Var.)
0.06±0.01
0.20±0.04
0.30±0.23
0.26±0.27
0.24±0.18
Table 3: Temporal Encoding results (mean ± std). Beyond the two metrics described above, we report the explained variability (Exp. Var.) of the Order Recoverability metric as validation.
Metric
TS2Vec †
TS-TCC †
SemTAC
Self-similarity ‡
Classification ‡
Silhouette
Fréchet
0.05±0.02 *
0.02±0.01 *
0.13±0.13
0.09±0.06 *
0.29±0.15 *
DTW
0.10±0.01 *
0.09±0.02 *
0.20±0.11
0.15±0.07 *
0.35±0.10 *
kNN-Consist.
Fréchet
0.87±0.02 *
0.71±0.04 *
0.95±0.07
0.91±0.08 *
1.00±0.01 *
DTW
0.90±0.02 *
0.79±0.04 *
0.94±0.06
0.91±0.06 *
1.00±0.01 *
Table 4: Sequence Branching metrics (mean ± std). underlined : second best per distance.
TS2Vec †
TS-TCC †
SemTAC
Self-similarity ‡
Classification ‡
0.98±0.00 *
0.86±0.02 *
0.85±0.14
0.80±0.23
0.92±0.07 *
Table 5: F1-scores (mean ± std) for single motion classification.
Figure 3: PHATE visualization of the latent space of an example run from each type of model. Points represent actual data points from the test split, while the solid poly-lines represent the prototypes calculated from the validation data (red for left sequence, blue for right sequence). The numbers denote the step positions for each of the sequence prototypes.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Motion type
# Accepted instances
A person jumps.
1637
A person does push-ups.
994
A person raises their right arm and waves.
1561
A person raises their left arm and waves.
2132
A person claps.
1837
A person does a squat.
1589
Appendix
Table 6: Amount of accepted instances after dataset filtering using the method from Rodrigues et al. (2026) .
SemTAC vs. …
Fréchet (p-value)
DTW (p-value)
TS-TCC
+0.96* (2.9e -5)
+0.50 (5.7e -2)
TS2Vec
+0.51* (3.2e -2)
-0.63* (1.2e -2)
Self-similarity
+0.20 (1.7e -1)
-0.26 (1.3e -1)
Classification
-0.56* (4.9e -3)
-0.72* (4.0e -4)
Appendix
Table 7: Statistical comparison of SemTAC’s sequence type classification at full length vs all other model families using the Mann Whitney U test. Results are reported as rrb (p-value), and significant results are marked with a *. The p-values presented are the corrected values based on the Holm-Bonferroni correction method. Comparisons are made separately for each distance method used for inference.
SemTAC vs. …
Fréchet (p-value)
DTW (p-value)
TS-TCC
+0.79* (7.0e -4)
+0.88* (2.0e -4)
TS2Vec
+0.78* (7.0e -4)
+0.74* (1.4e -3)
Self-similarity
+0.77* (8.7e -7)
+0.68* (1.8e -5)
Classification
+0.75* (2.0e -4)
+0.29 (1.0e -1)
Appendix
Table 8: Statistical comparison of SemTAC’s sequence type classification at length 4 vs all other model families using Mann Whitney U test. Results are reported as rrb (p-value), and significant results are marked with a *. The p-values presented are the corrected values based on the Holm-Bonferroni correction method. Comparisons are made separately for each distance method used for inference.
Sequence Branching
Silhouette
kNN-Consistency
SemTAC vs.
Fréchet
DTW
Fréchet
DTW
TS-TCC
+0.98* (1.5e -5)
+0.94* (4.0e -5)
+0.94* (3.9e -5)
+0.94* (5.0e -5)
TS2Vec
+0.81* (3.0e -4)
+0.92* (4.0e -5)
+0.81* (2.6e -4)
+0.72* (1.3e -3)
Self-similarity
+0.29* (4.3e -2)
+0.36* (1.5e -2)
+0.37* (1.1e -2)
+0.36* (1.2e -2)
Classification
-0.76* (6.3e -5)
-0.83* (1.7e -5)
-0.91* (1.7e -6)
-0.92* (1.5e -6)
Appendix
Table 9: Statistical comparison of SemTAC’s metrics related to sequence branching vs all other model families using Mann Whitney U test. Results are reported as rrb (p-value), and significant results are marked with a *. The p-values presented are the corrected values based on the Holm-Bonferroni correction method. Comparisons are made separately for each distance method used for inference.
Temporal Encoding
SemTAC vs.
Order Recoverability*
Smoothness
TS-TCC
+1.00* (1.5e -5)
+1.00* (9.9e -6)
TS2Vec
+0.70* (3.0e -3)
+1.00* (9.9e -6)
Self-similarity
+0.51* (2.1e -3)
+0.90* (5.2e -9)
Classification
+0.30 (9.2e -2)
+0.74* (7.5e -5)
Appendix
Table 10: Statistical comparison of SemTAC’s metrics related to temporal encoding vs all other model families using Mann Whitney U test. Results are reported as rrb (p-value), and significant results are marked with a *. The p-values presented are the corrected values based on the Holm-Bonferroni correction method. * Order recoverability results are further related to the portion of variance explained by the linear projection to make sure that high recoverability values do not stem from a coincidental infinitesimal dimension.
Video representation learning has seen tremendous progress in recent years. This has been driven by many factors, including the scale of training and the success of self-supervised models trained on next-frame prediction. While these factors have pushed the boundaries of what video models can do, they also introduce their own set of limitations. First, scaling video models can reach prohibitive costs, with recent models needing hundreds of years of video to be trained. Second, learning to predict the next frame (or embedding of the frame) naturally focuses on spatial information and as a result, video models still struggle with temporal understanding. In this paper we propose a novel approach that uses motion as a modality to alleviate both of these core issues. Given the motion in a video in the form of point tracks, we mask some of the tracks and use a masked autoencoder to reconstruct the missing tracks. This allows us to learn a representation in a self-supervised manner which we call TIME (Temporally-Informed Motion Embedding), and is focused on capturing temporal information. Also, as motion is inherently appearance-invariant, TIME needs far fewer examples to generalize well. As a result, without bells and whistles, on temporal tasks TIME performs on par with state-of-the-art models, using up to 4 orders of magnitude less training data. For general tasks, TIME can be used in combination with existing representations, and we observe that it leads to a significant improvement for V-JEPA 2, RVM and VideoMAE on standard benchmarks such as SSV2, EgoExo4D and Diving48. These results point to a new promising video paradigm for both more temporally-aware as well as more scalable models.
Mantas Skackauskas, Xinyue Hao, Laura Sevilla-Lara
Self-supervised learning for time-series representation aims to reduce reliance on labeled data while maintaining strong downstream performance, yet many existing approaches incur high computational costs or rely on assumptions that do not hold across diverse temporal dynamics. In this work, we introduce Divide and Contrast (Di-COT), an unsupervised framework that avoids data augmentation and multiple encoder passes by contrasting informative substructures within a window rather than individual timesteps. Di-COT stochastically partitions each window into a small number of overlapping sub-blocks per iteration, enabling efficient and meaningful contrast while mitigating false positives during temporal transitions. To further improve scalability, we adopt a contrastive objective whose computation depends on the batch size and the number of sub-blocks, making loss computation independent of sequence length. Extensive experiments on six large-scale real-world datasets, as well as the UCR and UEA benchmarks, demonstrate that Di-COT learns semantically structured and transferable representations, achieving state-of-the-art performance on classification, clustering, kNN, and cross-dataset transfer, while substantially reducing training time. The source code is publicly available at https://github.com/sfi-norwai/Di-COT.
Abdul-Kazeem Shamba, Kerstin Bach, Gavin Taylor
Department of Computer Science, Norwegian University of Science and Technology, Norway · Department of Computer Science, United States Naval Academy, USA.
Text-to-motion generation must produce motions that are semantically correct, temporally coherent, and physically plausible. A natural approach is to first project motion data into a structured semantic space and then train a generative model within that space. Such a paradigm has been highly successful in image generation through Representation Autoencoders (RAEs), where a frozen self-supervised encoder provides semantic features for diffusion or flow models to learn from. However, direct transfer of such a paradigm to motion space using Motion-JEPA as the frozen encoder fails dramatically. We diagnose this failure geometrically and identify two motion-specific bottlenecks: (1) the JEPA feature space is spectrally ill-conditioned, making the Gaussian-to-data transport unstable; and (2) even with a well-conditioned spectrum, flow residuals tend to align with decoder-sensitive directions, where small latent errors are amplified into large motion artifacts after decoding. Based on these insights, we propose MoRAE. MoRAE addresses the two bottlenecks separately. A compact bottleneck distills the structured JEPA representation while removing weak and redundant directions, bringing the latent spectrum into a transport-stable regime. Motion-coupled training then aligns the retained latent geometry with the decoder, making characteristic flow errors less costly after decoding. With this flow-friendly latent, a standard non-autoregressive Flow-Matching DiT achieves state-of-the-art performance.
Yifei Zhu, Mingyi Shi, Yangyang Cai +3
1Tohoku University, Sendai, Japan · 2The University of Hong Kong, Hong Kong