Accurate Open-Loop Control of a Soft Continuum Robot Using Visually Learned Latent Dynamics
Authors: Henrik Krauss, Johann Licher, Naoya Takeishi, Annika Raatz, Takehisa Yairi
Organizations: Department of Advanced Interdisciplinary Studies, The University of Tokyo, Tokyo, 153-8904, Japan. · Institute of Assembly Technology and Robotics, Leibniz University Hannover, 30823 Germany. · Research Center for Advanced Science and Technology, The University of Tokyo, Tokyo, 153-8904, Japan.
This work addresses open-loop control of a soft continuum robot (SCR) from video-learned latent dynamics. Visual Oscillator Networks (VONs) from previous work are used, which provide mechanically interpretable 2D oscillator latents through an attention broadcast decoder (ABCD). Open-loop, single-shooting optimal control is performed in latent space to track image-specified waypoints without camera feedback. An interactive SCR live simulator enables design of static, dynamic, and extrapolated targets and maps them to model-specific latent waypoints. On a two-segment pneumatic SCR, Koopman, MLP, and oscillator dynamics, each with and without ABCD, are evaluated on setpoint and dynamic trajectories. ABCD-based models consistently reduce image-space tracking error. The VON and ABCD-based Koopman models attain the lowest mean squared errors (MSEs). Ablation and simulation stress tests confirm that training and architecture choices support open-loop performance, including static holding, stable extrapolated equilibria, and relaxation to rest. To the best of our knowledge, this is the first demonstration of reliable long-horizon open-loop control of a physical soft robot using interpretable latent dynamics models learned solely from video.
Figures & tables
Fig. 1 : Overview of this study’s approach: an SCR live simulator is employed to generate target states for open-loop optimal control of the physical SCR. The feasibility of this method is validated across diverse trajectories.
Name
n
T
K
wQ
wQ˙
wQk
wQ˙k
wQf
wQ˙f
Setp. Normal
9
100
2
1
0.002
0
0
0
0
Setp. Extrap.
6
100
2
1
0.002
0
0
0
0
Dyn. Normal
5
200
8
1
0
0
0
0
0.002
Dyn. Fast
3
150
8
1
0
0
0
0
0.002
Dyn. Upswing
3
150
3
0
0
1
0.002
0
0
Dyn. Long
5
1000
22
1
0
0
0
0
0.002
TABLE I : Trajectory types and control parameters. n = number of trajectories tested.
Fig. 2 : (a) Model performances in open-loop control of real hardware over various trajectory types (image MSEs). (b) Mean absolute error (MAE) of input pressure prediction from open-loop setpoint control in simulation, compared to setpoints in the dataset, over multi-step prediction errors (50 samples each), including ablations.
Fig. 3 : (a) Hardware examples for trajectories evaluated for open-loop optimal control. The real trajectories shown are based on the VON model. All target trajectories are accurately achieved, except the upswing, where an additional plot highlights a deviation between set and measured pressure inputs. (b) Predicted and real latent trajectories and input pressures for the fast dynamic trajectory and upswing trajectory (c), depicted on the left.
Fig. 4 : Simulation stress tests. (a) Static holding. (b) Ramp-up to extrapolated pressures, with VON excitation vs. stiffness forces. (c) In-dataset range harmonic excitation then release. Top shows image MSEs and bottom final decoded observations.
Latent world models improve sample efficiency in continuous control by optimizing policies over imagined latent trajectories, but common neural transitions offer limited direct control over modal persistence and error accumulation in long rollouts. We propose Koopman Dreamer, a Dreamer-style world model with a spectrally constrained deterministic latent dynamics core. Its Koopman-inspired backbone uses two-dimensional rotation--scaling blocks with bounded radii to represent damping, rotation, and near-periodic modes. Linear and low-rank bilinear action terms capture global and state-dependent control effects, while stochastic-state modulation supplies local correction information. To reduce the mismatch between posterior-conditioned training and prior-only imagination, the model combines posterior-conditioned EMA teacher targets with one-step consistency, multi-step rollout, and open-loop observation-prediction objectives. We further derive a multi-step rollout-error bound that separates amplification by the spectral backbone and bilinear interaction from the additive effects of stochastic-state mismatch and modeling residuals, clarifying the trade-off between error attenuation and long-term information retention. Experimental results on proprioceptive continuous-control tasks from the DeepMind Control Suite and UAV-LiDAR autonomous navigation demonstrate that Koopman Dreamer improves the stability of long-horizon latent rollouts and achieves stronger closed-loop control performance on tasks that rely on high-quality multi-step imagination.
Jiaqi Li, Xinglong Zhang, Haibin Xie +3
College of Intelligence Science and Technology, National University of Defense Technology, Changsha, China
Tendon-Driven Continuum Robots (TDCRs) pose significant modeling and control challenges due to complex nonlinearities, such as frictional hysteresis and transmission compliance. This paper proposes a differentiable learning framework that integrates high-fidelity dynamics modeling with robust neural control. We develop a GRU-based dynamics model featuring bidirectional multi-channel connectivity and residual prediction to effectively suppress compounding errors during long-horizon auto-regressive prediction. By treating this model as a gradient bridge, an end-to-end neural control policy is optimized through backpropagation, allowing it to implicitly internalize compensation for intricate nonlinearities. Experimental validation on a physical three-section TDCR demonstrates that our framework achieves accurate tracking and superior robustness against unseen payloads, outperforming Jacobian-based methods by eliminating self-excited oscillations. For implementation details and source code, please refer to https://github.com/ZiqingZou/ContinuumControl.
Ziqing Zou, Ke Qiu, Fei Wang +3
Department of Control Science and Engineering, Zhejiang University, Hangzhou, China
In this paper, we design feedback control laws for soft robots modelled using the Cosserat rod, which is spatially discretised using the Piecewise Constant Strain (PCS) approach. The PCS approach transforms the nonlinear PDEs describing the Cosserat rod to a system of nonlinear ODEs. This simplification results in a model describing soft robots which is similar to the serial rigid-link manipulators. We design feedback control laws for the quasi-static PCS model by using external wrenches as control inputs. The control laws are designed based on state-feedback linearisation in strain and task spaces. An extensive set of numerical results demonstrates the performance of the control laws for end-effector trajectory tracking and shape control of soft robots.
Srishti Siddharth, Domenico Campolo, Ravi Banavar
Centre for Systems and Control Indian Institute of Technology Bombay Mumbai, India · School of Mechanical and Aerospace Engineering Nanyang Technological University Singapore