Local Temporal Contrasts on Human Motion Data Induce a Latent Geometry

Lourenço A. Rodrigues; Felix Putze

cs.LGOct 15, 2026

Local Temporal Contrasts on Human Motion Data Induce a Latent Geometry

Authors: Lourenço A. Rodrigues, Felix Putze

Organizations: Cognitive Systems Lab FB3, Department of Mathematics and Computer Science, University of Bremen Bremen, 28359 Bremen, Germany

Abstract

Time series such as recordings of human motion data often encode structure across multiple temporal scales, and when a latent state determining this sequence evolves, this structure needs to survive in the representation space for downstream use. Yet the currently prevalent attention-based encoders paired with sequence-level pooling have made single-vector, pointwise comparison the default mode of inference, discarding the trajectory a sequence traces through the representation space. We propose recovering this geometric structure directly, without relying on forecasting-based objectives, by training an encoder model (SemTAC) with a temporal-neighborhood contrastive loss: representations that satisfy local consistency everywhere must give rise to a coherent global structure, letting inference happen at temporal scales beyond those seen during training. We evaluate SemTAC against four baselines on a five-step motion sequence discrimination task: the same architecture trained with a classification or a standard contrastive objective, and two established self-supervised time-series methods, TS-TCC and TS2Vec. At the full sequence length all five are broadly competitive, but on incomplete, truncated sequences SemTAC holds a large advantage: a median F1 of 0.77 one step before the sequence completes, against 0.27-0.70 for the four baselines. A hyperparameter-matched ablation attributes this effect specifically to the temporal-contrastive objective. SemTAC's representation trajectories are also the only ones among these five to exhibit interpretable geometric structures, exceeding a no-structure chance baseline on a displacement-smoothness metric which we introduce. This is evidence that its objective, not classification accuracy alone, produces genuinely temporally-ordered trajectories.

Figures & tables

Figure 1: Training our model with temporal contrasts induces a structure in the latent space that mirrors the temporal structure of the data. (a) shows the representation space when trained with purely discrimination-centric objective, (b) shows the original data structure (see Fig. 2 ), and (c) is the latent space created by our approach.

Figure 2: Generated motion sequences with divergent structure. Both sequence types begin identically (jump, step), then diverge toward a preferential lateralization before performing completely unrelated motions. Step Right/Left labels are 60/40% corrupted (only 60% match the stated direction); Wave Right/Left labels are 80/20% corrupted.

Distance	TS2Vec †	TS-TCC †	SemTAC	Self-similarity ‡	Classification ‡
Fréchet	$0.81\pm 0.04$ *	$0.55\pm 0.03$ *	$0.87\pm 0.12$	$0.83\pm 0.15$	$\mathbf{0.95\pm 0.06}$ *
DTW	$0.91\pm 0.02$ *	$0.82\pm 0.03$ *	$0.86\pm 0.06$	$0.89\pm 0.07$	$\mathbf{0.95\pm 0.06}$ *

Table 1: F1-scores (mean

\pm

std) per approach, using Fréchet or DTW as the latent sequence distance.

Distance	TS2Vec †	TS-TCC †	SemTAC	Self-similarity ‡	Classification ‡
Fréchet	$0.40\pm 0.19$ *	$0.51\pm 0.03$ *	$\mathbf{0.68\pm 0.21}$	$0.37\pm 0.28$ *	$0.37\pm 0.32$ *
DTW	$0.70\pm 0.05$ *	$0.47\pm 0.12$ *	$\mathbf{0.72\pm 0.19}$	$0.64\pm 0.15$ *	$0.63\pm 0.25$

Table 2: F1-scores (mean

\pm

std) per approach, using Fréchet or DTW as the latent sequence distance.

Metric	TS2Vec †	TS-TCC †	SemTAC	Self-similarity ‡	Classification ‡
Smoothness	$-0.57\pm 0.02$ *	$-0.53\pm 0.01$ *	$\mathbf{-0.26\pm 0.30}$	$-0.58\pm 0.12$ *	$-0.58\pm 0.17$ *
Order Rec.	$0.67\pm 0.04$ *	$0.48\pm 0.04$ *	$\mathbf{0.79\pm 0.10}$	$0.66\pm 0.15$ *	$0.73\pm 0.11$
(Exp. Var.)	$0.06\pm 0.01$	$0.20\pm 0.04$	$0.30\pm 0.23$	$0.26\pm 0.27$	$0.24\pm 0.18$

Table 3: Temporal Encoding results (mean

\pm

std). Beyond the two metrics described above, we report the explained variability (Exp. Var.) of the Order Recoverability metric as validation.

Metric		TS2Vec †	TS-TCC †	SemTAC	Self-similarity ‡	Classification ‡
Silhouette
	Fréchet	$0.05\pm 0.02$ *	$0.02\pm 0.01$ *	$\underline{0.13\pm 0.13}$	$0.09\pm 0.06$ *	$\mathbf{0.29\pm 0.15}$ *
	DTW	$0.10\pm 0.01$ *	$0.09\pm 0.02$ *	$\underline{0.20\pm 0.11}$	$0.15\pm 0.07$ *	$\mathbf{0.35\pm 0.10}$ *
kNN-Consist.
	Fréchet	$0.87\pm 0.02$ *	$0.71\pm 0.04$ *	$\underline{0.95\pm 0.07}$	$0.91\pm 0.08$ *	$\mathbf{1.00\pm 0.01}$ *
	DTW	$0.90\pm 0.02$ *	$0.79\pm 0.04$ *	$\underline{0.94\pm 0.06}$	$0.91\pm 0.06$ *	$\mathbf{1.00\pm 0.01}$ *

Table 4: Sequence Branching metrics (mean

\pm

std). underlined : second best per distance.

TS2Vec †	TS-TCC †	SemTAC	Self-similarity ‡	Classification ‡
$\mathbf{0.98\pm 0.00}$ *	$0.86\pm 0.02$ *	$0.85\pm 0.14$	$0.80\pm 0.23$	$0.92\pm 0.07$ *

Table 5: F1-scores (mean

\pm

std) for single motion classification.

Figure 3: PHATE visualization of the latent space of an example run from each type of model. Points represent actual data points from the test split, while the solid poly-lines represent the prototypes calculated from the validation data (red for left sequence, blue for right sequence). The numbers denote the step positions for each of the sequence prototypes.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Motion type	# Accepted instances
A person jumps.	1637
A person does push-ups.	994
A person raises their right arm and waves.	1561
A person raises their left arm and waves.	2132
A person claps.	1837
A person does a squat.	1589

Appendix

Table 6: Amount of accepted instances after dataset filtering using the method from Rodrigues et al. (2026) .

SemTAC vs. …	Fréchet (p-value)	DTW (p-value)
TS-TCC	+0.96* $\left(2.9\text{e -}5\right)$	+0.50 $\left(5.7\text{e -}2\right)$
TS2Vec	+0.51* $\left(3.2\text{e -}2\right)$	-0.63* $\left(1.2\text{e -}2\right)$
Self-similarity	+0.20 $\left(1.7\text{e -}1\right)$	-0.26 $\left(1.3\text{e -}1\right)$
Classification	-0.56* $\left(4.9\text{e -}3\right)$	-0.72* $\left(4.0\text{e -}4\right)$

Appendix

Table 7: Statistical comparison of SemTAC’s sequence type classification at full length vs all other model families using the Mann Whitney U test. Results are reported as

r_{rb}

(p-value), and significant results are marked with a *. The p-values presented are the corrected values based on the Holm-Bonferroni correction method. Comparisons are made separately for each distance method used for inference.

SemTAC vs. …	Fréchet (p-value)	DTW (p-value)
TS-TCC	+0.79* $\left(7.0\text{e -}4\right)$	+0.88* $\left(2.0\text{e -}4\right)$
TS2Vec	+0.78* $\left(7.0\text{e -}4\right)$	+0.74* $\left(1.4\text{e -}3\right)$
Self-similarity	+0.77* $\left(8.7\text{e -}7\right)$	+0.68* $\left(1.8\text{e -}5\right)$
Classification	+0.75* $\left(2.0\text{e -}4\right)$	+0.29 $\left(1.0\text{e -}1\right)$

Appendix

Table 8: Statistical comparison of SemTAC’s sequence type classification at length 4 vs all other model families using Mann Whitney U test. Results are reported as

r_{rb}

	Sequence Branching
	Silhouette		kNN-Consistency
SemTAC vs.	Fréchet	DTW	Fréchet	DTW
TS-TCC	+0.98* $\left(1.5\text{e -}5\right)$	+0.94* $\left(4.0\text{e -}5\right)$	+0.94* $\left(3.9\text{e -}5\right)$	+0.94* $\left(5.0\text{e -}5\right)$
TS2Vec	+0.81* $\left(3.0\text{e -}4\right)$	+0.92* $\left(4.0\text{e -}5\right)$	+0.81* $\left(2.6\text{e -}4\right)$	+0.72* $\left(1.3\text{e -}3\right)$
Self-similarity	+0.29* $\left(4.3\text{e -}2\right)$	+0.36* $\left(1.5\text{e -}2\right)$	+0.37* $\left(1.1\text{e -}2\right)$	+0.36* $\left(1.2\text{e -}2\right)$
Classification	-0.76* $\left(6.3\text{e -}5\right)$	-0.83* $\left(1.7\text{e -}5\right)$	-0.91* $\left(1.7\text{e -}6\right)$	-0.92* $\left(1.5\text{e -}6\right)$

Appendix

Table 9: Statistical comparison of SemTAC’s metrics related to sequence branching vs all other model families using Mann Whitney U test. Results are reported as

r_{rb}

	Temporal Encoding
SemTAC vs.	Order Recoverability*	Smoothness
TS-TCC	+1.00* $\left(1.5\text{e -}5\right)$	+1.00* $\left(9.9\text{e -}6\right)$
TS2Vec	+0.70* $\left(3.0\text{e -}3\right)$	+1.00* $\left(9.9\text{e -}6\right)$
Self-similarity	+0.51* $\left(2.1\text{e -}3\right)$	+0.90* $\left(5.2\text{e -}9\right)$
Classification	+0.30 $\left(9.2\text{e -}2\right)$	+0.74* $\left(7.5\text{e -}5\right)$

Appendix

Table 10: Statistical comparison of SemTAC’s metrics related to temporal encoding vs all other model families using Mann Whitney U test. Results are reported as

r_{rb}

(p-value), and significant results are marked with a *. The p-values presented are the corrected values based on the Holm-Bonferroni correction method. * Order recoverability results are further related to the portion of variance explained by the linear projection to make sure that high recoverability values do not stem from a coincidental infinitesimal dimension.

Temporal Representation Learning

See all 28 papers

The TIME Machine: On The Power of Motion for Efficient Perception
May 21, 2026Mantas Skackauskas, Xinyue Hao, Laura Sevilla-LaraRepresentation Learning Video Representation Learning
School of Informatics University of Edinburgh
Divide and Contrast: Learning Robust Temporal Features without Augmentation
May 20, 2026Abdul-Kazeem Shamba, Kerstin Bach, Gavin TaylorContrastive Learning Self-Supervised Learning
Department of Computer Science, Norwegian University of Science and Technology, Norway · Department of Computer Science, United States Naval Academy, USA.
MoRAE: Flow-Friendly Self-Supervised Latents for Text-to-Motion Generation
Jul 31, 2026Yifei Zhu, Mingyi Shi, Yangyang Cai +3Self-Supervised Learning Text-to-Motion Generation
1Tohoku University, Sendai, Japan · 2The University of Hong Kong, Hong Kong

May 21, 2026J/K move · Enter open · S save

The TIME Machine: On The Power of Motion for Efficient Perception

Mantas Skackauskas, Xinyue Hao, Laura Sevilla-Lara

School of Informatics University of Edinburgh

Representation Learning Video Representation Learning Motion Representation Learning Self-Supervised Visual Representation Learning Temporal Video Understanding Temporal Representation Learning

Open paper AskPDF arXiv

Video representation learning has seen tremendous progress in recent years. This has been driven by many factors, including the scale of training and the success of self-supervised models trained on next-frame prediction. While these factors have pushed the boundaries of what video models can do, they also introduce their own set of limitations. First, scaling video models can reach prohibitive costs, with recent models needing hundreds of years of video to be trained. Second, learning to predict the next frame (or embedding of the frame) naturally focuses on spatial information and as a result, video models still struggle with temporal understanding. In this paper we propose a novel approach that uses motion as a modality to alleviate both of these core issues. Given the motion in a video in the form of point tracks, we mask some of the tracks and use a masked autoencoder to reconstruct the missing tracks. This allows us to learn a representation in a self-supervised manner which we call TIME (Temporally-Informed Motion Embedding), and is focused on capturing temporal information. Also, as motion is inherently appearance-invariant, TIME needs far fewer examples to generalize well. As a result, without bells and whistles, on temporal tasks TIME performs on par with state-of-the-art models, using up to 4 orders of magnitude less training data. For general tasks, TIME can be used in combination with existing representations, and we observe that it leads to a significant improvement for V-JEPA 2, RVM and VideoMAE on standard benchmarks such as SSV2, EgoExo4D and Diving48. These results point to a new promising video paradigm for both more temporally-aware as well as more scalable models.

Abstract

Temporal Representation Learning

Explore similar work

The TIME Machine: On The Power of Motion for Efficient Perception

Divide and Contrast: Learning Robust Temporal Features without Augmentation

MoRAE: Flow-Friendly Self-Supervised Latents for Text-to-Motion Generation