cs.ROFeb 26, 2026
SaveA Perspective on Open Challenges in Deformable Object Manipulation
Organizations: Department of Computer Science, University of York, Heslington, York, YO10 5DD, United Kingdom
Abstract
Deformable object manipulation (DOM) represents a critical challenge in robotics, with applications spanning healthcare, manufacturing, food processing, and beyond. Unlike rigid objects, deformable objects exhibit infinite dimensionality, dynamic shape changes, and complex interactions with their environment, posing significant hurdles for perception, modeling, and control. This paper reviews the state of the art in DOM, focusing on key challenges such as occlusion handling, task generalization, and scalable, real-time solutions. It highlights advancements in multimodal perception systems, including the integration of multi-camera setups, active vision, and tactile sensing, which collectively address occlusion and improve adaptability in unstructured environments. Cutting-edge developments in physically informed reinforcement learning (RL) and differentiable simulations are explored, showcasing their impact on efficiency, precision, and scalability. The review also emphasizes the potential of simulated expert demonstrations and generative neural networks to standardize task specifications and bridge the simulation-to-reality gap. Finally, future directions are proposed, including the adoption of graph neural networks for high-level decision-making and the creation of comprehensive datasets to enhance DOM's real-world applicability. By addressing these challenges, DOM research can pave the way for versatile robotic systems capable of handling diverse and dynamic tasks with deformable objects.
Figures & tables
| Category | Advantages | Disadvantages | Representative Literature |
| Visual perception | • Effective prediction of deformable object states. • Accurate object representation (affordance, keypoints, point clouds). • High performance for specific tasks. • Improved precision via advanced sensing. | • Struggles with unseen scenarios. • High computational and sensory demands. • Limited generalization across tasks. • Expensive hardware requirements. | • Learning Foresightful Dense Visual Affordance for Deformable Object Manipulation [ 103 ] • Learning Visual-Based Deformable Object Rearrangement with Local Graph Neural Networks [ 16 ] • DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects [ 96 ] |
| Tactile perception | • Excels where vision fails (e.g., occlusion, force-sensitive tasks). • Enables precise force and position control for delicate interactions. • Enhances adaptability with integration into learning frameworks. • Combines multimodal perception for richer object understanding. | • Highly task-specific, requiring customization for new applications. • Dependence on specialized tactile sensors increases cost and complexity. • Computational demands for real-time performance and multimodal processing. • Limited generalization across diverse tasks or environments. | • Precise Robotic Needle-Threading with Tactile Perception and Reinforcement Learning [ 113 ] • VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects [ 101 ] • Interaction Control for Tool Manipulation on Deformable Objects Using Tactile Feedback [ 118 ] |
| Interactive perception | • Enhances understanding of deformable objects through active engagement. • Improves performance via dynamic adjustments and interactive feedback. • Expands manipulation capabilities using novel methods (e.g., assistive tools, markerless systems). • Offers better modeling and data richness compared to static perception. | • High computational and hardware demands. • Task-specific approaches limit generalization. • Dependence on complex setups (e.g., multi-view systems, assistive tools). • Struggles with unseen objects or scenarios without extensive retraining. | • Interactive Perception for Deformable Object Manipulation [ 100 ] • Enhancing Deformable Object Manipulation by Using Interactive Perception and Assistive Tools [ 123 ] • HMDO: Markerless Multi-view Hand Manipulation Capture with Deformable Objects [ 104 ] |
Table 1: Overview of recent literature in perception for deformable object manipulation
Figure 1: Illustration of cloth modeling using a mass-spring system: a triangular mesh structure representing the cloth is depicted, with a magnified view showing the individual mass points and spring connections, highlighting the deformation mechanics and the use of Hooke’s law (bottom centre).
Figure 2: The undeformed cube (left) represents the reference configuration, while the deformed cube (right) illustrates the effect of compressive forces acting along the horizontal sides (grey arrows). The deformation is described by the deformation gradient ( ), where , mapping the reference configuration ( ) to the deformed configuration ( ). The strain in the material is captured by the strain tensor , with indicating compression along the horizontal axis. The stress tensor ( ) satisfies the equilibrium condition under static loading, and the material behavior follows Hooke’s law: . The top and bottom faces remain undeformed due to fixed boundary conditions ( ).
| Category | Advantages | Disadvantages | Literature |
| Mass-Spring-Systems | • Flexible for modeling deformable objects. • Accounts for material variations like stiffness and knots. • Supports real-world tasks (e.g., string handling, dual-arm coordination). • Learning integration improves adaptability. • Ensures grasp stability in deformable object manipulation. • Easy to implement in robotic systems. | • High computational complexity for real-time applications. • Simplified models may miss nonlinear deformations. • Requires pre-computation and parameter tuning. • Limited representation for complex materials (e.g., anisotropy). • Needs extensive training data. | • Tabata et al. ( [ 92 ] ) • Shi et al. ( [ 81 ] ) • Zaidi et al. ( [ 116 ] ) |
| Position-Based-Dynamics | • Efficient, stable simulation for deformable objects. • Realistic modeling for ropes and DLOs. • Suitable for real-world tasks (e.g., suturing, obstacle avoidance). • Supports optimization via differentiable frameworks. • Validated in practical scenarios. | • Limited to linear or rope-like objects, reducing generalization. • Computational challenges for real-time tasks. • Relies on accurate parameter tuning. • Sensitive to parameter estimation errors. | • Yu et al. [ 111 ] • Liu et al. [ 49 ] |
| Continuum Mechanics | • High accuracy for modeling deformations. • Provides physics-based realism. • Enables blind manipulation using force sensing. • Effective for grasping tasks with deformable objects. • Validated through simulation and real-world applications. | • Computationally expensive, limiting real-time use. • Requires precise material properties and meshing. • Difficult to scale for complex objects. • Limited generalization to other deformable manipulation scenarios beyond the specific tasks studied. | • Sanchez et al. [ 76 ] • Huang et al. [ 35 ] |
Table 2: Overview of recent literature in modelling for deformable object manipulation
Figure 3: Shooting control diagram demonstrating forward dynamics and optimization. The constrained variables are in the action and control space (dashed rectangle). Diagram is based on [ 85 ]
Figure 4: State trajectory exploration: The constrained variables lie within the state space (dashed rectangle), with actions determined based on state trajectories and inverse dynamics. This is a common framework in Model-Predictive-Control, a thorough explanation can be found here [ 95 ]
Figure 5: A basic closed-loop control framework based on visual features. A thorough description can be found here [ 88 ]
Figure 6: RL pipeline: First, the configurations of the DO are encoded into a latent space. Next, the RL agent generates manipulation actions by mapping the current and goal latent representations, enabling continuous interaction with the environment until the task is successfully completed. A thorough introduction to these concepts can be found in [ 91 ]
| Planning and Control | Advantages | Disadvantages | Literature |
| Analytical Approaches | • High precision and predictability. • Robust, systematic frameworks. • Tailored, reproducible solutions. • Scalable to variations. | • Poor adaptability to uncertainties. • High computational cost. • Relies on accurate models. • Complex development. | • [ 121 ] • [ 15 ] • [ 12 ] |
| Reinforcement Learning | • Adaptive and robust. • Excels in dynamic scenarios. • Reduces need for explicit models. • Handles complex, high-dimensional tasks. | • High training costs. • Needs large, diverse data. • Struggles with sim-to-real gaps. • Risk of overfitting. | • [ 26 ] • [ 108 ] • [ 24 ] |
| Imitation Learning | • Adapts to deformable dynamics. • Requires fewer data. • Task-specific optimization. • Sim2Real benefits. | • Needs high-quality demos. • Limited generalization. • Computationally heavy. • Risk of overfitting. | • [ 12 ] • [ 24 ] • [ 102 ] |
Table 3: Overview of recent literature in planning and Control for deformable object manipulation
Figure 7: Illustration links techniques and approaches that show great promise in DOM to face the key challenges outlined at the beginning of this review.