Objects move, cameras pan, and weather drifts with the wind: much of what changes in sequential data is motion. Motion changes where things are, not what they are, and is therefore a symmetry of the data. Yet standard recurrent networks are not equivariant to it: their memory stays put while the input moves on. Flow equivariant recurrent networks first brought this symmetry into sequence models, for motions of constant velocity, or flows, by enumerating a bank of velocities. We extend this idea in three directions: from simple recurrent networks to convolutional LSTMs, from flows to motions whose velocity changes in time, and from enumerating velocities to estimating them. Treating a sequence as a field over time, we show that motion becomes a gauge symmetry: a global symmetry made local in time. Built on this view, our motion equivariant convolutional LSTM (MELSTM) uses its velocity model as the gauge field and is provably equivariant to every motion. Across future-frame prediction, classification of digits visible only through their motion, and action recognition under camera shake, MELSTM outperforms a convolutional LSTM and its flow equivariant counterpart with the same number of parameters: it is more accurate, generalizes to longer horizons and to unseen motions, and carries a fraction of the flow equivariant model's state.
Figures & tables
Figure 1: Hidden-state transport under RNN, Flow Equivariant RNN (FERNN), and Motion Equivariant RNN (MERNN) models. The first row shows the transformed input, ϕ▹Xt in the flow column and ψ▹Xt in the motion column. Here an MNIST digit undergoes two simultaneous changes: a geometric transformation along the group orbit (the motion, which differs across columns) and an intrinsic appearance change transverse to the group action (identical across columns). The second row shows the hidden state of an untrained leaky accumulator, ht+1=λht+Xt ( λ=0.92 ), which can be viewed as a convolutional RNN, ht+1=W⋆ht+U⋆Xt , with kernels W(x)=λδ(x) and U(x)=δ(x) . The third row shows FERNN’s hidden state, which is lifted to velocity copies g∈{−3,…,3}2 and evolves as ht+1g=λg⋅htg+Xt ; we show the copy of maximum energy, i.e., the copy with the largest ∥htg∥22 . The fourth row shows MERNN’s hidden state, ht+1=λTvtht+Xt , where Tvt is the transport operator and vt is the digit’s velocity at time t . Matched transport stacks evidence into a sharp digit at the true location, whereas mismatched transport leaves a decaying ghost trail along the residual motion.
Figure 2: Training efficiency and test error. Right : test MSE over the trained horizon and at t=100 ; mean ± s.d. over five seeds.
Figure 3: Rollout stability and context-boundary sensitivity. Left : median rollout MSE. Middle : MSE at t=16 of sequences whose last velocity change came 1 – 3 frames before the context boundary (boundary velocity age), relative to settled sequences (age ≥4 ). Right : the age- 1 group over the rollout; the FELSTM penalty persists through mid-rollout before gradually decaying.
Figure 4: Motion generalization sensitivity. Each model normalized by its own training error ( 1.0 = no degradation).
Figure 5: Motion-Only MNIST: Training efficiency and test accuracy. Right: five-seed mean ± s.d. of the best validation and the test accuracy and of the median time per epoch. Dotted lines: chance ( 10% ).
Figure 6: Motion generalization sensitivity in classification. Relative to their own training accuracy (far lower for LSTM), LSTM and FELSTM respond alike, except at speed 1 , where LSTM improves because its 3×3 recurrence can move its memory with the digit.
Figure 7: KTH: training and generalization to camera shake. Left: validation accuracy under the training shake. Right: test accuracy under shakes of amplitude A and frequency f ( A=0 : static camera); mean ± s.d. over three seeds.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Length generalization beyond the trained horizon. Two held-out sequences; rows are the ground truth, LSTM, FELSTM, and MELSTM. The dashed box marks the 25 frames seen in training ( 15 context +10 predicted); the rest is extrapolation, run out to t=140 , past the 85 -step horizon scored in Figure 3 . LSTM collapses almost immediately, FELSTM smears into high-frequency artifacts late in the rollout, and MELSTM stays sharp throughout.
Figure 9: The channel-handover penalty across the seven motion regimes. Top: kinematics of the seven evaluation regimes and the training baseline. The unpredictability ρ collapses the switching rate r and the jump magnitude ∣Δv∣ into a single number; net context displacement falls as ρ rises, because the trajectories become diffusive. Bottom: FELSTM’s architectural penalty (MSE normalised by the LSTM baseline) against ρ(1−ρ) . Because FELSTM allocates separate channels to discrete velocities, a velocity change forces a handover that strands accumulated temporal history; we model its cost as the product of the disruption ( ρ ) and the predictive value of what is stranded ( 1−ρ ). The penalty is negligible up to ρ(1−ρ)≈0.1 , including under pure chaos ( churn_max , ρ→1 ), where the stranded history is worthless, and grows steeply above it. random_walk (frequent tiny jitters) and churn_half (less frequent but far larger jumps) are kinematically opposite, yet sit at nearly symmetric points of the parabola and show nearly equal penalties.
Figure 10: Slot specialisation. Hidden-state evolution for a validation sequence (top) and a training sequence (bottom). Within each block the first two rows show the channel-averaged states Hˉ1,Hˉ2 of the K=2 slots and the third the corresponding frame; column titles give t , in red once the decoder takes over. Each slot localises onto one digit and tracks it, and this is the template that Eq. ( 97 ) correlates against the incoming frame.
Figure 11: Velocity accuracy (one training run). The estimator is parameter-free and receives no gradient, so the rise from ≈66% at epoch 5 to above 99.5% by epoch 8 reflects the hidden-state template improving, not the estimator being fitted. Final values are means over the last five epochs. In the same run, the validation loss stays on a plateau near 7×10−2 until about epoch 6 and then falls by an order of magnitude while the accuracy climbs: the model leaves the plateau when the slots lock onto their digits. The five seeds of Figure 2 leave it at different times, which smooths their mean.
Figure 12: Stochastic frame-pair velocities. In training, MELSTM uses frame-pair velocities with a probability that decays to 0 by epoch 25 ; validation uses slot tracking only. Losses (MSE + L1) are means over three seeds with the min–max range shaded; faint traces are recorded every 25 batches and circles once per epoch (Appendix B.1.2 ). Table: test MSE over the trained horizon and at t=100 ; mean ± s.d.
Figure 13: Frame-pair curriculum. As Figure 12 , but training uses frame-pair velocities on the first steps of the context, a window that shrinks from the whole context to none by epoch 25 .
Figure 14: Hidden states on Motion-Only MNIST. Channel-averaged state of each MELSTM slot (a) and of the FELSTM copies at the velocities the digit takes during the context (b); the bottom rows show the input frames. The digit’s outline is drawn only for the reader: the frames themselves are pure noise. MELSTM’s slot s1 follows the digit and accumulates its shape, while s0 follows the background; because the digit’s velocity keeps changing, no FELSTM copy accumulates the shape for long.
Figure 15: Example KTH clip with and without camera shake. All 16 frames of a handwaving clip from the test set. Top: original clip (static camera). Bottom: the same clip with the camera shaking horizontally, with A=8 px and P=8 frames (1.6 Hz).
Equivariant neural networks encode transformations such as rotations, translations, and permutations so model outputs change consistently with their inputs.
Department of Computer Science, Dartmouth College, Hanover, NH 03755, USA · Departments of Computer Science & Neuroscience and Biomedical Engineering, Aalto University, Espoo, Finland