Organizations: Instituto de Investigación en Ingeniería de Aragón (I3A), University of Zaragoza, Spain · DEI, University of Bologna, Italy · Institut Pascal, Clermont Auvergne INP, France
In robotics, shape control (or shape servoing) refers to the automatic manipulation of a deformable object to achieve a desired shape. It involves a feedback loop that exploits the relation between robot motion and changes in the measured shape. This relation, known as the interaction Jacobian or, in this context, the deformation Jacobian, depends on the object's mechanical properties, geometry, and boundary conditions (e.g., where the object is grasped). Because these factors are often difficult to model accurately, the deformation Jacobian can be estimated locally from observed shape changes produced by commanded robot motions, without requiring a fine-tuned mechanical model. This article is the first of a two-part tutorial on data-driven Jacobian-based shape control of deformable objects. It introduces the central elements of the approach: the formulation of the shape-control problem, shape representation, estimation and online update of the deformation Jacobian, and a standard control law from the literature. It then assembles these elements into a complete shape-servoing loop, derives a local stability condition for this loop, and illustrates why some target shapes cannot be reached. Part II covers more advanced topics. Throughout both parts, a simulated planar example in which two robot arms manipulate an elastic rod observed by a camera illustrates the concepts.
Figures & tables
Fig. 1: Illustration of a shape-servoing loop and of the setup (plant) used as the running example in this tutorial. Left: shape-servoing loop, with the measured shape obtained from the camera image and the shape error converted into robot motion using a local deformation model represented by the deformation Jacobian. The Jacobian is estimated from the applied robot motions and the corresponding measured shape changes. Right: running-example setup, in which two robot arms deform an elastic rod x (blue), observed by a stationary camera, toward a target shape x∗ (dashed). For simplicity, in the running example, the grippers and the rod move in a plane parallel to the image plane of the camera, although this tutorial’s formulation generalizes to three-dimensional actuation. The arrows at the grippers indicate the six in-plane actuation degrees of freedom [{\color[rgb]{1,0,0}x_{L}},{\color[rgb]{0,1,0}y_{L}},{\color[rgb]{0,0,1}\theta_{L}},{\color[rgb]{1,0,0}x_{R}},{\color[rgb]{0,1,0}y_{R}},{\color[rgb]{0,0,1}\theta_{R}}]^{\top} . Bottom: schematic definition of shape servoing, with an illustration of the object shape approaching the target over time t .
Section
Main content
Section II
Shape representation and shape error with respect to the target, the deformation Jacobian relating robot motion to shape change, and the control law mapping shape error to robot motion.
Section III
Estimation of the initial Jacobian from exploratory motions, online adaptation from closed-loop data, and model-based alternatives.
Section IV
Integration of the components into a complete feedback loop, characterization of reachable target shapes, and conditions for local decrease of the shape error.
TABLE I: Contents of Part I
Fig. 2: Discrete-time shape-servoing loop of the running example, with the sections and equations defining each block. Perception extracts the image coordinates sk of the tracked points from the camera measurement yk , and the controller computes the command uk from the shape error ek . The operators Δtz−1 and 1−z−1 , where z−1 denotes a one-iteration delay, compute the latest robot-state increment Δqk−1 and feature change Δsk . These are used by the online update ( 7 ) to correct the Jacobian estimate, initialized from the offline estimate ( 6 ) J^0 (Fig. 3 ). Right: current features sk (yellow), target features s∗ , and shape error ek in the image; the error arrows are also shown on the rod in the scene.
Fig. 3: Offline Jacobian estimation in the running example. Left: a small robot perturbation Δqc produces a measured feature change Δsc . Center: the C=24 action–response pairs are stacked as the columns of ΔQ and ΔS . Right: least-squares regression gives the initial Jacobian estimate J^0 .
Fig. 4: Targets with different closed-loop outcomes in the running example. Top: rod configurations from k=0 (light grey) to k=90 (blue, with yellow feature points) for two target shapes. Left: a target recorded at another gripper configuration, which is reached from the tested initial state. Right: a modified target obtained by displacing three feature points, for which the controller settles with a nonzero residual. Dashed curves with grey rings indicate the target shapes, and red arrows denote the residual feature error. Bottom: normalized error histories for the two cases.
Fig. 5: Closed-loop behavior for the arch target of Fig. 1 . Top: rod configurations (blue, with yellow feature points) and dashed target at the indicated iterations k , selected at approximately equal intervals of travel in feature space; lighter curves show two preceding iterations. Bottom, from left to right: normalized error with online Jacobian updates and with J^0 held fixed ( α=0.05 ); commanded robot velocities ( α=0.05 ; yL and yR coincide by symmetry); and normalized error for three control gains (solid), each with its nominal decay (1−α)k (dotted, same shade).
Case
Sufficient local condition
Stability and convergence
n≤m , maximum rank
rank(J)=n and Sym(JJ^+)≻0
e=0 is locally exponentially stable and e→0 . For n=m , full row rank corresponds to the nonsingular square case.
n>m , maximum rank
rank(J)=m and Sym(J^+J)≻0
The controllable closed-loop modes converge exponentially, and e approaches a residual in Null(J^⊤) . For exact Jacobian knowledge ( J^=J ), this becomes the local least-squares residual in Null(J⊤) and is zero when e∈Range(J) .
Rank deficient
rank(J)<min(n,m) , with Sym(JJ^+) positive definite when restricted to Range(J) for n≤m , or Sym(J^+J) positive definite when restricted to Range(J⊤) for n>m
Local exponential convergence is guaranteed only on the remaining controllable subspace. Error components outside Range(J) cannot be reduced locally; rank loss in J^ may remove additional corrective directions.
TABLE II: Local Stability and Convergence with an Estimated Deformation Jacobian
Most research on the manipulation of deformable objects focuses on lightweight systems with negligible mechanical response, effectively restricting attention to quasi-static regimes. This assumption excludes a broad class of practically relevant objects, such as hoses, pipes, and wiring harnesses, whose dynamics cannot be ignored during manipulation. In this work, we address this limitation by introducing a closed-loop control architecture that explicitly accounts for object dynamics and recasts manipulation as a shape-regulation problem. Control is achieved by modulating forces and torques applied at multiple fixed points along the object. This approach builds on three methodological contributions: a fully dynamic model of linear deformable objects based on discrete strain parameterizations; an extension of the notion of actuation coordinates to SE(3), yielding a structured and inherently underactuated control architecture; and nonlinear feedback strategies providing explicit conditions for steady-state convergence to desired configurations. Extensive simulations on representative manipulation tasks demonstrate the performance gains enabled by the proposed modelbased formulation. We finally validate the approach experimentally through a real-time closed-loop implementation with online shape estimation, confirming its practical feasibility and effectiveness
Daniel Feliu-Talegon, Cosimo Della Santina
Department of Cognitive Robotics, Delft University of Technology, 2628 CN Delft, The Netherlands · Institute of Robotics and Mechatronics, German Aerospace Center (DLR), 82234 Oberpfaffenhofen, Germany
Precise shape control of Deformable Linear Objects (DLOs) is crucial in robotic applications such as industrial and medical fields. However, existing methods face challenges in handling complex large deformation tasks, especially those involving opposite curvatures, and lack efficiency and precision. To address this, we propose a two-stage framework combining Reinforcement Learning (RL) and online visual servoing. In the large-deformation stage, a model-based reinforcement learning approach using an ensemble of dynamics models is introduced to significantly improve sample efficiency. Additionally, we design a self-curriculum goal generation mechanism that dynamically selects intermediate-difficulty goals with high diversity through imagined evaluations, thereby optimizing the policy learning process. In the small-deformation stage, a Jacobian-based visual servo controller is deployed to ensure high-precision convergence. Simulation results show that the proposed method enables efficient policy learning and significantly outperforms mainstream baselines in shape control success rate and precision. Furthermore, the framework effectively transfers the policy trained in simulation to real-world tasks with zero-shot adaptation. It successfully completes all 30 cases with diverse initial and target shapes across DLOs of different sizes and materials. The project website is available at: https://anonymous.4open.science/w/sc-mbrl-dlo-EB48/
Zhaowei Liang, Song Wang, Zhao Jin +2
All authors are with the Department of Mechanical Engineering, Tsinghua University, Beijing, China.
Ropes, cables, and other deformable linear objects appear in tasks from untangling to cable routing and suturing, yet controlling their shape remains a challenge in robot manipulation. We study model-based shape control in a general setting: the object lies unfixated on a support surface and two arms may grasp and move it anywhere along its length. Because each arm chooses a grasp point, direction, and magnitude, the joint action space is combinatorially large, and the dynamics model's per-prediction cost bounds how much of it a planner can search. We present ForwardDLO, a recurrent latent dynamics model for this unfixated bimanual setting that predicts per-segment displacements grounded in the observed rope state at every step. Our model reaches accuracy comparable to more expensive baselines while containing no explicit segment-to-segment operations, which makes batched evaluation of candidate actions cheap. On open-loop prediction of real rope motion it reaches the lowest error of the learned models we evaluate, 13% below the strongest baseline. Within a fixed time budget it scores 8 to 22 times more candidate actions than models of comparable accuracy while matching them in real-world shape matching; and on a simulated routing task at a 30Hz control rate, this throughput converts into 98% task success versus at most 30% for the baselines at their own budgets. We release the model, code, and a dataset of 2.42 million simulated and 14,107 real rope transitions at https://anonymous.4open.science/r/ForwardDLO/
Tim Missal, Berk Guler, Lucas Domingues +3
Technical University of Darmstadt · Honda Research Institute Europe GmbH · School of Electrical and Computer Engineering, Universidade Estadual de Campinas (UNICAMP), Brazil +4