Estimating the motion of deformable bodies in soft robotics from proprioceptive sensing is challenging due to partial observability, as internal observations provide only indirect access to global motion, contact, and shape evolution. In such settings, instantaneous observations are insufficient, and accurate estimation requires temporally consistent latent representations. This paper studies deformable motion estimation under proprioceptive partial observability from a latent dynamics perspective. We argue that a key requirement for long-horizon estimation is latent temporal consistency, where latent states remain predictive of future observations. To this end, we present DeepSoMo, a learning-based framework that enforces predictive structure in latent dynamics through a self-supervised latent prediction constraint. We evaluate the approach in a canonical deformable dynamics setting using high-fidelity finite-element simulation as a controlled source of ground truth. Results demonstrate consistent improvements in trajectory estimation, orientation recovery, shape reconstruction, and contact inference over architecture-matched recurrent baselines.
Figures & tables
Fig. 1: DeepSoMo estimates deformable motion states from proprioceptive visual observations over time. Top: Sequences of internally observed deformation images at different time steps, serving as the only sensory input. A fixed shape prior is used as a reference for deformable shape reconstruction. Bottom: Motion state estimation results in a canonical deformable dynamics scenario, including global pose, shape, and contact evolution. Finite-element-method (FEM) simulation is used solely to generate ground-truth data for evaluation (green). Compared to an architecture-matched recurrent baseline (red), DeepSoMo (purple) produces temporally consistent estimates that remain closer to the ground truth over long horizons, illustrating the effect of enforcing predictive consistency in latent dynamics.
Fig. 2: Overall framework of DeepSoMo. Given a sequence of proprioceptive visual observations It , the framework first encodes each observation into a latent representation ft . A recurrent latent dynamics model aggregates these representations over time to estimate a compact latent state lt under partial observability. To enforce latent temporal consistency , a self-supervised latent prediction module regularizes the latent dynamics by encouraging lt to remain predictive of future observations, producing a predictive latent state ht . The resulting latent representation is then decoded at each time step into deformable object motion states, including translation, rotation, shape, and contact.
Fig. 3: (a) Snapshots of a deformable sphere rolling over terrain, illustrating the local body-frame coordinate system used for motion estimation. Translation and orientation are estimated as incremental changes expressed in the body frame attached to the object, and global pose is reconstructed by composing these local estimates over time. (b) Examples of procedurally generated terrain surfaces used in simulation. Terrains are generated by adding Gaussian noise to 20×20 m mesh grids, providing controlled variation in elevation and local surface orientation for training and evaluation.
Fig. 4: Self-supervised latent prediction for enforcing temporal consistency. Given encoded observation features {ft} , the recurrent latent dynamics model produces latent states {lt} . To enforce latent temporal consistency , a self-supervised prediction constraint encourages the latent state at time t to remain predictive of future observations in the latent feature space. Specifically, lt is aligned with the next-step encoded feature ft+1 , and a cascaded predictive latent ht is further aligned with ft+2 . This constraint regularizes latent dynamics and mitigates error accumulation over long horizons.
Fig. 5: Qualitative deformable motion estimation under proprioceptive partial observability. Results from a representative unseen test simulation. Top : Body-frame incremental translation and orientation estimates over time, comparing ground truth (blue) and DeepSoMo predictions (orange). Translation components are shown as x , y , and z in centimeters, and orientation components are shown as Euler angles ϕ , θ , and ψ in degrees. Bottom : Deformable shape reconstruction with predicted contact regions (green) overlaid at selected time steps. For this sequence, DeepSoMo achieves an NTE of 4.73% , ATE of 23.38cm , and ARE of 8.84∘ , illustrating temporally consistent motion estimation and contact-aware shape reconstruction from internal deformation observations.
Manipulating deformable objects (DOs) is challenging due to their high-dimensional state space, underactuated dynamics, and partial observability. In this paper, we propose cRVAE, a lightweight conditional recurrent variational autoencoder that estimates the full DO state from only partial corner-node observations during inference. The resulting model is used as the forward model in a receding-horizon optimal control framework for obstacle-aware collaborative DO manipulation. In simulation on rope and fabric, cRVAE estimates the full DO state from the available corner-node measurements alone, matching the accuracy of a parameter-identified XPBD model. At inference it uses no physical parameters as model inputs and performs no online parameter identification. It also runs approximately 350 times faster on the rope and over 1500 times faster on the fabric per forward pass, keeping horizon-based planning within the 100 ms control budget where XPBD exceeds it already at short horizons. Full-shape estimation from corner sensing at in-loop speed is what makes the model deployable on hardware, which we demonstrate on a Unitree Go2 robot.
Kosar Behnia, Ville Kyrki, Gokhan Alcan
Faculty of Engineering and Natural Sciences, Tampere Univ., Finland. · Electrical Engineering and Automation Department, Aalto Univ., Finland.
Surgical tissue retraction requires effective manipulation planning under partial and noisy perception. We study state estimation for deformable tissue retraction, where only sparse observations of the tissue surface are available at decision time. We propose a learned state estimator that reconstructs the full deformable mesh state from 40 noisy vertex observations. The estimator combines a multilayer perceptron with a low-dimensional PCA latent representation and is trained using geometry-aware regularization that encourages smooth and physically plausible deformations. We evaluate the approach in a 2D deformable sheet simulation using single-step and multi-step retraction planning. Results show that the learned estimator achieves 98.1% of oracle performance in multi-step retraction while supporting efficient inference. These results demonstrate that learned, geometry-regularized state estimation can support effective deformable manipulation under realistic perception constraints.
Estimating deformable object states remains a fundamental challenge in robotics and simulation. We propose a novel factor graph-based framework for probabilistic mesh state estimation of deformable objects. The method directly updates a tetrahedral mesh, a rich and physically-grounded representation of an environment, by combining physics priors, noisy sensor measurements, and temporal smoothness constraints within a unified probabilistic formulation. The estimation problem is posed as a nonlinear least-squares optimization and solved using Levenberg-Marquardt. Ex vivo central-airway obstruction experiments and simulations on deforming cube models demonstrate reliable and accurate reconstruction under both rigid motion and deformation, highlighting the potential of this probabilistic approach for principled, measurement-driven mesh state estimation in deformable object reconstruction.
Lidia Al-Zogbi, Fangjie Li, Samuel Tobin +13
Department of Electrical and Computer Engineering, Tufts University, Medford, MA 02155, USA