Estimating the motion of deformable bodies in soft robotics from proprioceptive sensing is challenging due to partial observability, as internal observations provide only indirect access to global motion, contact, and shape evolution. In such settings, instantaneous observations are insufficient, and accurate estimation requires temporally consistent latent representations. This paper studies deformable motion estimation under proprioceptive partial observability from a latent dynamics perspective. We argue that a key requirement for long-horizon estimation is latent temporal consistency, where latent states remain predictive of future observations. To this end, we present DeepSoMo, a learning-based framework that enforces predictive structure in latent dynamics through a self-supervised latent prediction constraint. We evaluate the approach in a canonical deformable dynamics setting using high-fidelity finite-element simulation as a controlled source of ground truth. Results demonstrate consistent improvements in trajectory estimation, orientation recovery, shape reconstruction, and contact inference over architecture-matched recurrent baselines.
Figures & tables
Fig. 1: DeepSoMo estimates deformable motion states from proprioceptive visual observations over time. Top: Sequences of internally observed deformation images at different time steps, serving as the only sensory input. A fixed shape prior is used as a reference for deformable shape reconstruction. Bottom: Motion state estimation results in a canonical deformable dynamics scenario, including global pose, shape, and contact evolution. Finite-element-method (FEM) simulation is used solely to generate ground-truth data for evaluation (green). Compared to an architecture-matched recurrent baseline (red), DeepSoMo (purple) produces temporally consistent estimates that remain closer to the ground truth over long horizons, illustrating the effect of enforcing predictive consistency in latent dynamics.
Fig. 2: Overall framework of DeepSoMo. Given a sequence of proprioceptive visual observations It , the framework first encodes each observation into a latent representation ft . A recurrent latent dynamics model aggregates these representations over time to estimate a compact latent state lt under partial observability. To enforce latent temporal consistency , a self-supervised latent prediction module regularizes the latent dynamics by encouraging lt to remain predictive of future observations, producing a predictive latent state ht . The resulting latent representation is then decoded at each time step into deformable object motion states, including translation, rotation, shape, and contact.
Fig. 3: (a) Snapshots of a deformable sphere rolling over terrain, illustrating the local body-frame coordinate system used for motion estimation. Translation and orientation are estimated as incremental changes expressed in the body frame attached to the object, and global pose is reconstructed by composing these local estimates over time. (b) Examples of procedurally generated terrain surfaces used in simulation. Terrains are generated by adding Gaussian noise to 20×20 m mesh grids, providing controlled variation in elevation and local surface orientation for training and evaluation.
Fig. 4: Self-supervised latent prediction for enforcing temporal consistency. Given encoded observation features {ft} , the recurrent latent dynamics model produces latent states {lt} . To enforce latent temporal consistency , a self-supervised prediction constraint encourages the latent state at time t to remain predictive of future observations in the latent feature space. Specifically, lt is aligned with the next-step encoded feature ft+1 , and a cascaded predictive latent ht is further aligned with ft+2 . This constraint regularizes latent dynamics and mitigates error accumulation over long horizons.
Fig. 5: Qualitative deformable motion estimation under proprioceptive partial observability. Results from a representative unseen test simulation. Top : Body-frame incremental translation and orientation estimates over time, comparing ground truth (blue) and DeepSoMo predictions (orange). Translation components are shown as x , y , and z in centimeters, and orientation components are shown as Euler angles ϕ , θ , and ψ in degrees. Bottom : Deformable shape reconstruction with predicted contact regions (green) overlaid at selected time steps. For this sequence, DeepSoMo achieves an NTE of 4.73% , ATE of 23.38cm , and ARE of 8.84∘ , illustrating temporally consistent motion estimation and contact-aware shape reconstruction from internal deformation observations.