cs.LGOct 20, 2026

Motion Equivariant Convolutional LSTM: When Motion Becomes a Gauge Symmetry

Authors: Vahid Hosseinzadeh, T. Anderson Keller, Michael Puthawala

Organizations: Department of Mathematics & Statistics, South Dakota State University · The Kempner Institute at Harvard University · Cusp AI

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Objects move, cameras pan, and weather drifts with the wind: much of what changes in sequential data is motion. Motion changes where things are, not what they are, and is therefore a symmetry of the data. Yet standard recurrent networks are not equivariant to it: their memory stays put while the input moves on. Flow equivariant recurrent networks first brought this symmetry into sequence models, for motions of constant velocity, or flows, by enumerating a bank of velocities. We extend this idea in three directions: from simple recurrent networks to convolutional LSTMs, from flows to motions whose velocity changes in time, and from enumerating velocities to estimating them. Treating a sequence as a field over time, we show that motion becomes a gauge symmetry: a global symmetry made local in time. Built on this view, our motion equivariant convolutional LSTM (MELSTM) uses its velocity model as the gauge field and is provably equivariant to every motion. Across future-frame prediction, classification of digits visible only through their motion, and action recognition under camera shake, MELSTM outperforms a convolutional LSTM and its flow equivariant counterpart with the same number of parameters: it is more accurate, generalizes to longer horizons and to unseen motions, and carries a fraction of the flow equivariant model's state.

Figures & tables

Figure 1: Hidden-state transport under RNN, Flow Equivariant RNN (FERNN), and Motion Equivariant RNN (MERNN) models. The first row shows the transformed input, ϕ▹Xt\phi\triangleright{\cal{X}}_{t} in the flow column and ψ▹Xt\psi\triangleright{\cal{X}}_{t} in the motion column. Here an MNIST digit undergoes two simultaneous changes: a geometric transformation along the group orbit (the motion, which differs across columns) and an intrinsic appearance change transverse to the group action (identical across columns). The second row shows the hidden state of an untrained leaky accumulator, ht+1=λ ht+Xth_{t+1}=\lambda\,h_{t}+{\cal{X}}_{t} ( λ=0.92\lambda=0.92 ), which can be viewed as a convolutional RNN, ht+1=W⋆ht+U⋆Xth_{t+1}=W\star h_{t}+U\star{\cal{X}}_{t} , with kernels W(x)=λδ(x)W(x)=\lambda\delta(x) and U(x)=δ(x)U(x)=\delta(x) . The third row shows FERNN’s hidden state, which is lifted to velocity copies g∈{−3,…,3}2g\in\{-3,\dots,3\}^{2} and evolves as ht+1g=λ g⋅htg+Xth^{g}_{t+1}=\lambda\,g\cdot h^{g}_{t}+{\cal{X}}_{t} ; we show the copy of maximum energy, i.e., the copy with the largest ∥htg∥22\|h_{t}^{g}\|_{2}^{2} . The fourth row shows MERNN’s hidden state, ht+1=λTvt ht+Xth_{t+1}=\lambda T_{v_{t}}\,h_{t}+{\cal{X}}_{t} , where TvtT_{v_{t}} is the transport operator and vtv_{t} is the digit’s velocity at time tt . Matched transport stacks evidence into a sharp digit at the true location, whereas mismatched transport leaves a decaying ghost trail along the residual motion.
Figure 2: Training efficiency and test error. Right : test MSE over the trained horizon and at t=100t=100 ; mean ±\pm s.d. over five seeds.
Figure 3: Rollout stability and context-boundary sensitivity. Left : median rollout MSE. Middle : MSE at t=16t=16 of sequences whose last velocity change came 11 – 33 frames before the context boundary (boundary velocity age), relative to settled sequences (age ≥4\geq 4 ). Right : the age- 11 group over the rollout; the FELSTM penalty persists through mid-rollout before gradually decaying.
Figure 4: Motion generalization sensitivity. Each model normalized by its own training error ( 1.01.0 = no degradation).
Figure 5: Motion-Only MNIST: Training efficiency and test accuracy. Right: five-seed mean ±\pm s.d. of the best validation and the test accuracy and of the median time per epoch. Dotted lines: chance ( 10%10\% ).
Figure 6: Motion generalization sensitivity in classification. Relative to their own training accuracy (far lower for LSTM), LSTM and FELSTM respond alike, except at speed 11 , where LSTM improves because its 3×33\times 3 recurrence can move its memory with the digit.
Figure 7: KTH: training and generalization to camera shake. Left: validation accuracy under the training shake. Right: test accuracy under shakes of amplitude AA and frequency ff ( A=0A=0 : static camera); mean ±\pm s.d. over three seeds.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Length generalization beyond the trained horizon. Two held-out sequences; rows are the ground truth, LSTM, FELSTM, and MELSTM. The dashed box marks the 2525 frames seen in training ( 1515 context ++ 1010 predicted); the rest is extrapolation, run out to t=140t=140 , past the 8585 -step horizon scored in Figure 3 . LSTM collapses almost immediately, FELSTM smears into high-frequency artifacts late in the rollout, and MELSTM stays sharp throughout.
Figure 9: The channel-handover penalty across the seven motion regimes. Top: kinematics of the seven evaluation regimes and the training baseline. The unpredictability ρ\rho collapses the switching rate rr and the jump magnitude ∣Δv∣|\Delta v| into a single number; net context displacement falls as ρ\rho rises, because the trajectories become diffusive. Bottom: FELSTM’s architectural penalty (MSE normalised by the LSTM baseline) against ρ(1−ρ)\rho(1-\rho) . Because FELSTM allocates separate channels to discrete velocities, a velocity change forces a handover that strands accumulated temporal history; we model its cost as the product of the disruption ( ρ\rho ) and the predictive value of what is stranded ( 1−ρ1-\rho ). The penalty is negligible up to ρ(1−ρ)≈0.1\rho(1-\rho)\approx 0.1 , including under pure chaos ( churn_max , ρ→1\rho\to 1 ), where the stranded history is worthless, and grows steeply above it. random_walk (frequent tiny jitters) and churn_half (less frequent but far larger jumps) are kinematically opposite, yet sit at nearly symmetric points of the parabola and show nearly equal penalties.
Figure 10: Slot specialisation. Hidden-state evolution for a validation sequence (top) and a training sequence (bottom). Within each block the first two rows show the channel-averaged states Hˉ1,Hˉ2\bar{\cal{H}}^{1},\bar{\cal{H}}^{2} of the K=2K=2 slots and the third the corresponding frame; column titles give tt , in red once the decoder takes over. Each slot localises onto one digit and tracks it, and this is the template that Eq. ( 97 ) correlates against the incoming frame.
Figure 11: Velocity accuracy (one training run). The estimator is parameter-free and receives no gradient, so the rise from ≈66%\approx 66\% at epoch 55 to above 99.5%99.5\% by epoch 88 reflects the hidden-state template improving, not the estimator being fitted. Final values are means over the last five epochs. In the same run, the validation loss stays on a plateau near 7×10−27\times 10^{-2} until about epoch 66 and then falls by an order of magnitude while the accuracy climbs: the model leaves the plateau when the slots lock onto their digits. The five seeds of Figure 2 leave it at different times, which smooths their mean.
Figure 12: Stochastic frame-pair velocities. In training, MELSTM uses frame-pair velocities with a probability that decays to 00 by epoch 2525 ; validation uses slot tracking only. Losses (MSE +{}+{} L1) are means over three seeds with the min–max range shaded; faint traces are recorded every 2525 batches and circles once per epoch (Appendix B.1.2 ). Table: test MSE over the trained horizon and at t=100t=100 ; mean ±\pm s.d.
Figure 13: Frame-pair curriculum. As Figure 12 , but training uses frame-pair velocities on the first steps of the context, a window that shrinks from the whole context to none by epoch 2525 .
Figure 14: Hidden states on Motion-Only MNIST. Channel-averaged state of each MELSTM slot (a) and of the FELSTM copies at the velocities the digit takes during the context (b); the bottom rows show the input frames. The digit’s outline is drawn only for the reader: the frames themselves are pure noise. MELSTM’s slot s1s_{1} follows the digit and accumulates its shape, while s0s_{0} follows the background; because the digit’s velocity keeps changing, no FELSTM copy accumulates the shape for long.
Figure 15: Example KTH clip with and without camera shake. All 16 frames of a handwaving clip from the test set. Top: original clip (static camera). Bottom: the same clip with the camera shaking horizontally, with A=8A=8 px and P=8P=8 frames (1.6 Hz).
More in this topic

Equivariant Neural Networks

Equivariant neural networks encode transformations such as rotations, translations, and permutations so model outputs change consistently with their inputs.

See all 157 papers

Explore similar work

CardsList
Oct 12, 2026

When Left Is Right? Camera-Motion Understanding beyond Screen

Can a vision-language model tell left from right? Modern VLMs solve visual puzzles and follow hour-long videos, yet when asked whether the camera moved left or right, most do no better than a coin flip. Adding camera-motion data or geometric representations has been tried, separately, and neither closes the gap. We argue that camera-motion understanding needs two conditions at once: the representation must couple off-screen camera motion with on-screen flow dynamics, and it must be interpreted by the language model rather than trained as a standalone pose or depth estimator. We propose CameraVLM, which pairs a motion-state representation with state-language alignment. Camera tokens capture how the camera moves off screen, flow tokens capture how the subject and background move on screen, and a two-stage training strategy keeps language in the loop so that both are learned together with camera-motion semantics. With only 9B parameters, CameraVLM outperforms all open-source and camera-motion-specific models by a wide margin across three benchmarks and more than 20 baselines, and matches or surpasses GPT-6 Astra as well as human viewers in a 48-participant study. Project page: https://3dagentworld.github.io/CameraVLM/

Chong Cheng, Ran Zhang, Xinyuan Li +3

The Hong Kong University of Science and Technology (Guangzhou) · Tencent · LIX, Ecole Polytechnique, IP Paris

Camera Motion UnderstandingVideo-Language ModelsTemporal Video UnderstandingVision-Language AlignmentVLM Evaluation
AskPDF
Feb 20, 2026

Latent Equivariant Operators for Robust Object Recognition: Promises and Challenges

Despite the successes of deep learning in computer vision, difficulties persist in recognizing objects that have undergone group-symmetric transformations rarely seen during training\unicodex2013\unicode{x2013}for example objects seen in unusual poses, scales, positions, or combinations thereof. Equivariant neural networks are a solution to the problem of generalizing across symmetric transformations, but require knowledge of transformations a priori. An alternative family of architectures proposes to learn equivariant operators in a latent space, from examples of symmetric transformations. Here, using simple datasets of rotated and translated noisy MNIST, we illustrate how such architectures can successfully be harnessed for out-of-distribution classification, thus overcoming the limitations of both traditional and equivariant networks. While conceptually enticing, we discuss challenges ahead on the path of scaling these architectures to more complex datasets. Our code is available at https://github.com/BRAIN-Aalto/equivariant_operator.

Minh Dinh, Stéphane Deny

Department of Computer Science, Dartmouth College, Hanover, NH 03755, USA · Departments of Computer Science & Neuroscience and Biomedical Engineering, Aalto University, Espoo, Finland

Equivariant Neural NetworksImage ClassificationOperator LearningOOD Generalization
AskPDF
Jun 2, 2026

Exact equivariance, kept through training, buys zero-shot generalisation across the symmetry group

A latent world model built from an equivariant encoder and predictor inherits a provable symmetry of its training loss: when the dynamics carries a group GG acting on latents by an orthogonal representation ρ(g)ρ(g), the one-step prediction relMSE is exactly invariant across the whole group, so fitting a restricted slice of orientations mathematically determines it on the entire orbit. The symmetry survives a real Muon/AdamW+EMA+VICReg run -- composed residual ∼10−6\sim 10^{-6} after training, under any optimiser (intrinsic Vector-Neuron/e3nn parametrisation) -- and one-step error is flat across the group (5-seed medians: equivariant ×1.00\times 1.00 vs a higher-capacity non-equivariant baseline ×12.7\times 12.7 in 2D, ×17.2\times 17.2 in 3D), while that baseline fits the slice but breaks out-of-distribution. The flatness is not a synthetic artefact: on real-robot DROID end-effector trajectories the equivariant model stays flat across the orbit (×1.000\times 1.000, rotation residual 1.5×10−161.5\times 10^{-16}) while a 4.5×4.5\times-larger baseline degrades ×11\times 11. One caution is load-bearing: flatness is necessary, not sufficient -- the theorem transports the in-distribution error level unchanged but does not lower it (3D relMSE ≈0.43\approx 0.43): across-group error is constant, not low. The same isometry lifts to a closed-loop corollary: under a matching equivariant planner the control error is invariant across the group -- float-floor-exact in 2D/SO(2), statistically flat in 3D/SE(3). Stress-tested against Sutton's Bitter Lesson (augmentation, scale, soft-equivariance), each closes at most the across-group task metric, never the float-floor exactness. This is the generalisation-side foundation of a certified-world-models programme (arXiv:2606.13092, 2606.24945, 2606.24946): flatness transports competence, and the trust bounds built on it are downstream products.

Hongbo Wang

Department of Mathematics, Stony Brook University, Stony Brook, NY 11794, USA

World Models for RoboticsWorld ModelsEquivariant Neural NetworksLatent World ModelsEquivariant Representation LearningStructure-Preserving Neural Networks
AskPDF