Recurrent Temporal Restricted Boltzmann Machines as Unsupervised Encoders for Biomechanical Movement Sequences

Wafi Sharif; Nicholas LaHaye

cs.LGOct 17, 2026

Recurrent Temporal Restricted Boltzmann Machines as Unsupervised Encoders for Biomechanical Movement Sequences

Authors: Wafi Sharif, Nicholas LaHaye

Organizations: Queens High School for the Sciences · Jet Propulsion Laboratory, California Institute of Technology

Abstract

Biomechanical movement recordings are continuous, high-dimensional time series, and analyzing them without manually labeling them requires representations that capture how movements evolve over time. Most unsupervised representation-learning pipelines, however, treat each observation frame independently and discard this temporal structure, while energy-based models that could capture it have not extensively been adapted to biomechanical signals. So, this paper implements the Recurrent Temporal Restricted Boltzmann Machine (RTRBM), a recurrent energy-based model, into an unsupervised encoder that maps windows of movement data to fixed-length embeddings, and tests what its recurrent structure contributes. We adapt the model to continuous-valued signals, stack it into a deep architecture, and integrate it into the SIT-FUSE segmentation pipeline. We evaluate the embeddings on simulated musculoskeletal reaching and on two real gait datasets, using clustering, reconstruction, and walking-speed classification of unseen subjects, and compare them against non-recurrent and non-probabilistic baselines, including a control that removes the recurrent connection. Clustering the embeddings recovers movement structure tied to progress toward the target without any labels, which also transfers to unseen episodes. On gait data, the embeddings outperform PCA and autoencoder baselines under a nonlinear probe. However, removing the recurrent connection does not yield a measurable loss on the reaching task, and simple per-channel summary statistics classify walking speed at least as well as the embeddings. The model is therefore a working, verified encoder for SIT-FUSE, while the evidence for a benefit of recurrence is mixed, and the learned embeddings did not outperform simple summary statistics on gait.

Figures & tables

Panel of Figure 1 — Figure 1: Three ways to add memory to an RBM. (a) A standard RBM has no temporal connection; each frame is modeled independently. (b) The RTRBM used in this work reuses its own hidden units as the recurrent signal: $h_{t-1}$ sets the hidden bias at $t$ through a single learned weight matrix $W^{\prime}$ (red). (c) The RNN-RBM keeps a separate recurrent state $r_{t}$ , updated from the current frame and the previous $r$ , which then sets the RBM’s bias one step ahead (teal): more expressive, but with a second set of parameters to train and coordinate.

Panel of Figure 2 — Figure 2: The full RTDBN pipeline for a single window of length $T$ : each frame $v_{t}$ passes through the RTRBM layer, whose recurrent connection carries information from preceding frames; then, the resulting hidden sequence is mean-pooled into one embedding per window, which $k$ -means then assigns to a movement-phase cluster.

Figure 3: (a) Training MSE divided by total window variance (mean $\pm$ SD over ten seeds) for the fixed-variance Gaussian (red) and learned-variance Gaussian (green) RTRBMs; the dashed line marks epoch 15, when $\sigma$ is unfrozen. The Bernoulli model is omitted from (a) because its MSE is measured against binary samples. (b) Median per-channel held-out $R^{2}$ for all three variants (bars: mean $\pm$ SD; dots: individual seeds).

Figure 4: PCA of 64-dimensional RTDBN embeddings for 11,220 windows (seed 1), colored by $k$ -means cluster ( $k=10$ ). PC1 and PC2 account for 31.1% and 12.5% of variance.

Figure 5: Eight randomly drawn example windows from each of the three largest clusters (rows; seed 1), cluster sizes 2,174, 1,370, and 1,344, across four observation-derived channels (columns): reach-error norm, joint-speed norm, muscle-activation norm, and tip displacement from the start of the window. Thin lines are individual windows, while the bold line is the per-cell mean.

Configuration	Test MSE	MSE $/(Tn_{v})$	Median $R^{2}$	Silhouette	$H_{\text{norm}}$
$T=10$ , $n_{h}=64$	276 $\pm$ 14	0.345	0.610	0.148	0.976
$T=20$ , $n_{h}=64$	533 $\pm$ 26	0.333	0.610	0.153	0.978
$T=30$ , $n_{h}=64$	807 $\pm$ 65	0.336	0.602	0.143	0.969
$T=40$ , $n_{h}=64$	1034 $\pm$ 50	0.323	0.608	0.140	0.975
$T=20$ , $n_{h}=32$	835 $\pm$ 49	0.522	0.549	0.212	0.974
$T=20$ , $n_{h}=64$	533 $\pm$ 26	0.333	0.610	0.153	0.978

Table 1: Ablation results (five seeds, mean; learning rates selected at

T=20

n_{h}=64

and not re-tuned). Test MSE is summed over the window (

\pm

SD over seeds); MSE

/(Tn_{v})

divides it by

T\times 80

to make window lengths comparable;

R^{2}

is the median per-channel held-out value. All configurations produce 10 populated clusters.

	Canonical ( $\mathbf{W}^{\prime}$ free)	$\mathbf{W}^{\prime}=0$
Test MSE	$539\pm 27$	$538\pm 31$
Median channel $R^{2}$ (test)	$0.608\pm 0.012$	$0.615\pm 0.018$
Trainval silhouette	$0.154\pm 0.013$	$0.149\pm 0.010$
Test silhouette	$0.170\pm 0.018$	$0.174\pm 0.018$
Probe $R^{2}$ : phase	$0.786\pm 0.044$	$0.779\pm 0.054$
Probe $R^{2}$ : reach trend	$0.365\pm 0.037$	$0.376\pm 0.043$

Table 2:

\mathbf{W}^{\prime}=0

ablation (ten seeds, mean

\pm

SD; models paired by seed). Identical data, split, and training schedule to the canonical model; only

\mathbf{W}^{\prime}

is zeroed and frozen. Probe columns are test

R^{2}

of a ridge probe on the embedding.

Figure 6: Silhouette score (blue, left) and inertia (orange, right) versus $k$ . Silhouette declines overall. Inertia decreases smoothly with no sharp elbow; the red line marks $k=10$ , the value used throughout the rest of the paper.

Model	Test MSE	Median $R^{2}$	Silhouette	Phase	Speed	Reach trend
RTRBM, canonical	$539.5\pm 26.8$	$0.608\pm 0.012$	$0.154\pm 0.013$	$0.786\pm 0.044$	$0.790\pm 0.088$	$0.365\pm 0.037$
RTRBM, $\mathbf{W}^{\prime}=0$	$538.2\pm 31.2$	$0.615\pm 0.018$	$0.149\pm 0.010$	$0.779\pm 0.054$	$0.825\pm 0.057$	$0.376\pm 0.043$
PCA, flattened windows	$478.8\pm 14.4$	$0.714\pm 0.020$	$0.088\pm 0.002$	$0.808\pm 0.030$	$0.329\pm 0.079$	$0.660\pm 0.033$
PCA, frame-wise	$0.0\pm 0.0$	$1.000\pm 0.000$	$0.098\pm 0.003$	$0.805\pm 0.033$	$0.345\pm 0.069$	$0.540\pm 0.047$
Frame-wise autoencoder	$2.3\pm 0.5$	$0.998\pm 0.000$	$0.077\pm 0.007$	$0.803\pm 0.032$	$0.327\pm 0.081$	$0.531\pm 0.049$
Causal GRU autoencoder	$2.0\pm 0.4$	$0.999\pm 0.000$	$0.080\pm 0.008$	$0.801\pm 0.032$	$0.326\pm 0.081$	$0.531\pm 0.056$

Table 3: RTRBM variants and non-RBM baselines on the held-out reach windows (ten seeds, mean

\pm

s.d.). MSE is summed over the 20 frames and 80 channels of a window and averaged over windows (the RBM numbers use sampled binary hidden states; the baselines are deterministic). Silhouette is computed on the train and validation episodes (

k=10

). Probe columns are held-out

R^{2}

of a ridge probe on the embedding.

	Probe	RTRBM	RTRBM $\mathbf{W}^{\prime}=0$	PCA frame	PCA window	MLP-AE	GRU-AE	Stats
Cohort 1	linear	0.514	0.470	0.504	0.465	0.503	0.497	0.730
(57 subj.)	random forest	0.577	0.515	0.456	0.515	0.463	0.466	0.649
Cohort 2	linear	0.529	0.518	0.437	0.455	0.446	0.500	0.567
(14 subj.)	random forest	0.610	0.591	0.413	0.559	0.498	0.479	0.649

Table 4: Walking-speed accuracy on unseen subjects (mean over ten subject-level splits; chance

=0.333

). RTRBM: learned-variance Gaussian RTRBM;

\mathbf{W}^{\prime}=0

removes the recurrence; Stats: per-channel mean, standard deviation, minimum and maximum of the window (no training). Bold: best in row.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Reach-error magnitude by $k$ -means cluster, all 10 clusters (seed 1; 8 example windows per cluster, thin lines; mean, bold line).

Cluster	$n$	Reach err.	Trend	Joint spd.	Muscle eff.	Tip disp.
0	2,174	0.143	$-$ 0.077	0.069	1.535	0.164
1	1,194	0.142	$-$ 0.032	0.071	1.502	0.172
2	1,207	0.577	$-$ 0.122	0.082	1.555	0.224
3	700	0.496	$-$ 0.157	0.084	1.666	0.328
4	713	0.226	$+$ 0.004	0.076	1.500	0.202
5	935	0.430	$-$ 0.022	0.070	1.570	0.205

Appendix

Table 5: Per-cluster kinematic summary, all 10 clusters. Trend is the change in reach-error norm from the start to the end of the window (negative = closing distance to target). Joint speed, muscle effort, and tip displacement are window-mean norms.

Motion Representation Learning

Motion representation learning develops structured encodings of movement for recognition, prediction, generation, and control.

See all 166 papers

Oct 15, 2026

Local Temporal Contrasts on Human Motion Data Induce a Latent Geometry

Time series such as recordings of human motion data often encode structure across multiple temporal scales, and when a latent state determining this sequence evolves, this structure needs to survive in the representation space for downstream use. Yet the currently prevalent attention-based encoders paired with sequence-level pooling have made single-vector, pointwise comparison the default mode of inference, discarding the trajectory a sequence traces through the representation space. We propose recovering this geometric structure directly, without relying on forecasting-based objectives, by training an encoder model (SemTAC) with a temporal-neighborhood contrastive loss: representations that satisfy local consistency everywhere must give rise to a coherent global structure, letting inference happen at temporal scales beyond those seen during training. We evaluate SemTAC against four baselines on a five-step motion sequence discrimination task: the same architecture trained with a classification or a standard contrastive objective, and two established self-supervised time-series methods, TS-TCC and TS2Vec. At the full sequence length all five are broadly competitive, but on incomplete, truncated sequences SemTAC holds a large advantage: a median F1 of 0.77 one step before the sequence completes, against 0.27-0.70 for the four baselines. A hyperparameter-matched ablation attributes this effect specifically to the temporal-contrastive objective. SemTAC's representation trajectories are also the only ones among these five to exhibit interpretable geometric structures, exceeding a no-structure chance baseline on a displacement-smoothness metric which we introduce. This is evidence that its objective, not classification accuracy alone, produces genuinely temporally-ordered trajectories.

Lourenço A. Rodrigues, Felix Putze

Cognitive Systems Lab FB3, Department of Mathematics and Computer Science, University of Bremen Bremen, 28359 Bremen, Germany

Temporal Representation Learning Self-Supervised Time Series Representation Learning Motion Representation Learning Contrastive Learning Geometric Representation Learning

Ask PDF

Oct 8, 2026

PMTRM: Pseudo-Memory Temporal Re-encoding Module for Embodied Policy Learning

Robotic manipulation often contains repeated motions whose local observations look similar at different phases. When these phases require different actions, a policy that relies mainly on the current observation may repeat completed motions or switch phases at the wrong time. To address this phase ambiguity, we present the Pseudo-Memory Temporal Re-encoding Module (PMTRM), a lightweight plug-in module with only 7.61M parameters that encodes a bounded history of executed states and actions into a latent sequence for existing policies. To help distinguish phases, a temporal heterogeneity objective penalizes positive similarity between distant positions in this sequence, while anchor and reconstruction losses preserve information needed for action prediction. The reconstruction decoder is used only during training, leaving the temporal re-encoder to supply history to the policy at inference. We train the module progressively on synthetic sequences and robot data, then jointly with the policy, using temporal masking to accommodate partial histories. This integration retains the original action head and action space and adds auxiliary losses to the original policy loss. Experiments with multiple policy backbones in simulation and on a real robot show improved task success on tasks with phase ambiguity, with little additional computation.

Changchuan Yang, Haoxuan Xu, Wenbo Chen +5

Zhejiang University · The Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou) +2

Robot Policy Learning Robotic Manipulation Trajectory Representation Learning

Ask PDF

Oct 16, 2026

RTSM: Adapting Recursive Models to Multivariate Time-Series Classification

Accurate multivariate time-series classification requires extracting temporal features and integrating information across a sequence. Weight-tied refinement offers a way to repeatedly process these features without allocating new parameters at each stage. Applying this principle to time series requires choosing both a temporal representation and a latent-state organization suited to classification. We introduce the Recursive Time-Series Model (RTSM), which combines a learned convolutional temporal tokenizer, single-state weight-tied refinement, and a pooled classification readout. Across all 30 UEA datasets, RTSM achieves 72.81% mean accuracy under validation-based configuration and checkpoint selection, the highest observed mean among the 17 evaluated deep-learning models and 1.60 percentage points above MambaSL. Ablations on eight datasets favor the proposed tokenizer over a simple convolutional embedding and the single-state model over the tested two-state variants. The complete classifier reaches 86.10% mean accuracy versus 84.79% for a separately trained tokenizer-only classifier. These findings identify temporal encoding and state organization as key design choices for adapting recursive architectures to time series. The resulting classifier combines competitive accuracy with a shared refinement module whose parameter count is independent of computational depth.

Georg Tenzing, Anna Tegon, Luca Benini +1

Integrated Systems Laboratory, ETH Zürich, Switzerland

Time Series Classification Multivariate Time Series Classification Recurrent Neural Networks

Ask PDF

Abstract

Motion Representation Learning

Explore similar work

Local Temporal Contrasts on Human Motion Data Induce a Latent Geometry

PMTRM: Pseudo-Memory Temporal Re-encoding Module for Embodied Policy Learning

RTSM: Adapting Recursive Models to Multivariate Time-Series Classification