Continuum robots possess high flexibility and redundancy, making them well suited for safe interaction in complex environments, yet their continuous deformation and nonlinear dynamics pose fundamental challenges to perception, modeling, and control. Existing vision-based control approaches often rely on end-to-end learning, achieving shape regulation without explicit awareness of robot geometry or its interaction with the environment. Here, we introduce a shape-interpretable visual self-modeling framework for continuum robots that enables geometry-aware control. Robot shapes are encoded from multi-view planar images using a Bezier-curve representation, transforming visual observations into a compact and physically meaningful shape space that uniquely characterizes the robot's three-dimensional configuration. Based on this representation, neural ordinary differential equations are employed to self-model both shape and end-effector dynamics directly from data, enabling hybrid shape-position control without analytical models or dense body markers. The explicit geometric structure of the learned shape space allows the robot to reason about its body and surroundings, supporting environment-aware behaviors such as obstacle avoidance and self-motion while maintaining end-effector objectives. Experiments on a cable-driven continuum robot demonstrate accurate shape-position regulation and tracking, with shape errors within 1.56% of image resolution and end-effector errors within 2% of robot length, as well as robust performance in constrained environments. By elevating visual shape representations from two-dimensional observations to an interpretable three-dimensional self-model, this work establishes a principled alternative to vision-based end-to-end control and advances autonomous, geometry-aware manipulation for continuum robots.
Figures & tables
Fig. 1: Conceptual illustration of the bio-inspired self-modeling mechanism. (A) Self-learning mechanism in biological systems. An elephant controls the motion of its trunk through muscular actuation and acquires corresponding trunk shape feedback via binocular vision, thereby learning the relationship between muscle activation and trunk morphology. Based on this learned internal model, the elephant can anticipate the shape changes induced by its own motor commands and accordingly control the trunk to perform complex manipulation tasks. (B) The continuum robot is actuated using arbitrary temporal input signals, while image data of the robot are recorded at each time instant from two different views. The image data are parameterized and encoded together with the actuation signals to form a dataset for neural network training. After training, the continuum robot can exploit the learned network to autonomously predict the visual shape dynamics from different views under given input temporal signals.
Fig. 2: Shape-interpretable visual self-modeling and geometry-aware control framework for continuum robots. (A) The robot shape is represented using a piecewise Bézier curve, which captures global shape characteristics with a small number of control points. (B) The robot region is first extracted using image processing techniques, and the skeleton curve is obtained via morphological operations. The skeleton is then fitted by Bézier curves, and the resulting control points are used as shape parameters, enabling an interpretable shape encoding. (C) Position NODE and shape NODE models are trained using the collected shape, position, and actuation data to achieve shape-position self-modeling. (D) When an environmental obstacle enters a predefined warning distance, the obstacle avoidance mechanism is activated. A repulsive escape velocity is applied at the point on the robot body closest to the obstacle, and this local escape velocity is mapped to a global shape variation, thereby realizing obstacle avoidance through shape control. (E) The trained position NODE and shape NODE are used to estimate the position Jacobian and shape Jacobian, respectively. A hybrid controller is then constructed by integrating the position controller, shape controller, and obstacle avoidance controller. (F) The proposed control framework is validated on a three-segment continuum robot and is capable of performing shape-position regulation, shape-position tracking, obstacle-aware regulation, and self-motion tasks.
Fig. 3: Experimental results of continuum robot shape-position regulation using the proposed method. (A) Experimental snapshots from two different views during the process in which the proposed method regulates the robot to the reference shape and end-effector position. (B) The robot end-effector moves from the initial position to the reference position. (C) Convergence of the shape-state regulation error to zero. (D) Convergence of the end-effector position regulation error to zero. (E) Four different shape-position regulation tasks are all successfully accomplished.
Fig. 4: Experimental results of simultaneous shape-position tracking control of a continuum robot using the proposed method. (A) Experimental snapshots from two different views during tracking of the ∞ -shaped end-effector trajectory. (B) Profile of the robot end-effector trajectory during tracking of the ∞ -shaped trajectory. (C) Experimental snapshots from two different views during tracking of the 8-shaped end-effector trajectory. (D) Profile of the robot end-effector trajectory during tracking of the 8-shaped trajectory. (E) Distribution of shape tracking errors from different views. (F) Distribution of tracking errors along different coordinate axes.
Fig. 5: Experimental results of shape-position control of a continuum robot in an environment with obstacle using the proposed method. (A) Experimental snapshots from two different views during shape-position regulation in the obstacle environment. (B) Evolution of the robot end-effector position during the shape-position regulation task in the presence of obstacle. (C) Minimum visual distance between the robot body and the obstacle. (D) Shape-state regulation error during the shape-position regulation task in the obstacle environment. (E) End-effector position regulation error during the shape-position regulation task in the obstacle environment. (F) The robot avoids a dynamic obstacle by shape control while keeping the end-effector position unchanged.
Fig. 6: Comparative results of shape regulation. (A) Snapshots of the robot. Our method ensures that the robot is regulated to the reference shape in two different views, whereas the DVIK method [ 22 ] , being a planar shape control approach, can only guarantee that the robot in View 1 reaches the reference shape. In View 2, a clear discrepancy exists between the reference shape and the actual robot shape when using the DVIK method. (B) Shape-state error of our method. (C) Shape-state error of the DVIK method. (D) Statistical error results for 12 different shape regulation tasks. The proposed method achieves nearly identical error distributions in the two views, and these errors are consistently smaller than those of the DVIK method in View 1. Moreover, since the DVIK method performs planar shape regulation, its error in View 2 is significantly larger than that in View 1. (E) Shape regulation results in the environment with an obstacle. The proposed method can simultaneously exploit interpretable shape state information during shape regulation to improve the controller and achieve obstacle avoidance, whereas the DVIK method fails to avoid the obstacle, resulting in collisions between the robot and the obstacle.
In this paper, we propose a novel vision-based control algorithm for regulating the whole body shape of extensible multisection soft continuum manipulators. Contrary to existing vision-based control algorithms in the literature that regulate the robot's end effector pose, our proposed control algorithm regulates the robot's whole body configuration, enabling us to leverage its kinematic redundancy. Additionally, our model-based 2.5D shape visual servoing provides globally stable asymptotic convergence in the robot's 3D workspace compared to the closest works in the literature that report local minima. Unlike existing visual servoing algorithms in the literature, our approach does not require information from proprioceptive sensors, making it suitable for continuum manipulators without such capabilities. Instead, robot state is estimated from images acquired by an external camera that observes the robot's whole body shape and is also utilized to close the shape control loop. Traditionally, visual servoing schemes require an image of the robot at its reference pose to generate the reference features. In this work, we utilize an inverse kinematics solver to generate reference features for the desired robot configuration and do not require images of the robot at the reference. Experiments are performed on a multisection continuum manipulator demonstrating the controller's capability to regulate the robot's whole body shape while precisely positioning the robot's end effector. Results validate our controller's ability to regulate the shape of continuum robots while demonstrating a smooth transient response and a steady-state error within 1 mm. Proof-of-concept object manipulation experiments including stacking, pouring, and pulling tasks are performed to demonstrate our controller's applicability.
Abhinav Gandhi, Shou-Shan Chiang, Cagdas D. Onal +1
Robotics Engineering Department, Worcester Polytechnic Institute 100 Institute Rd, Worcester, MA 01609, USA
Continuum robots enable smooth shape morphing and safe interaction in confined environments. However, most existing systems are task-specific and depend on external sensing infrastructure, limiting their adaptability and real-world deployment. This paper presents a self-contained modular continuum robotic platform that combines mechanical reconfigurability with onboard pose estimation. The robot is constructed from interchangeable continuum joints with analytically precomputed stiffness, allowing rapid assembly and direct programming of the robot shape. Proprioceptive sensing is achieved using magnetic sensors and a modular learning-based framework, where a single model is trained per joint and reused across configurations. The system is experimentally validated in real world, demonstrating self-sensing capabilities and adaptation without external tracking.
Predicting the shape of tendon driven continuum robots (TDCRs) at steady state from actuation remains challenging due to continuous deformation, complex tendon routing, compliance, friction, and fabrication variability. In this paper, we address this problem as kinematic self modeling conditioned on action. We present a lightweight 3D printed TDCR hardware platform and an RGB-D data collection pipeline with multiple cameras, and we learn a point cloud flow matching model that maps motor actuation states to the robot's settled 3D geometry. The model is trained from randomly sampled quasi static configurations and evaluated on test motor commands within the same TDCR design family and actuation range. We compare against prior 3D deformable object and robot self modeling approaches in both MuJoCo simulation and real hardware experiments. Experiments on simulated 2-, 3-, and 5-module TDCRs and real 2- and 3-module robots show improved shape prediction accuracy under CD and EMD metrics. We further show in simulation that the same conditional formulation generalizes to tip payload as a conditioning input, enabling payload conditioned steady-state shape prediction. These results demonstrate a data driven self modeling framework for quasi static TDCR geometry prediction.
Jiong Lin, Jinchen Ruan, Hod Lipson
Department of Mechanical Engineering, Columbia University, New York, NY, USA