A Perspective on Open Challenges in Deformable Object Manipulation

Ryan Paul McKennaa; John Oyekan

cs.ROFeb 26, 2026

A Perspective on Open Challenges in Deformable Object Manipulation

Authors: Ryan Paul McKennaa, John Oyekan

Organizations: Department of Computer Science, University of York, Heslington, York, YO10 5DD, United Kingdom

Abstract

Deformable object manipulation (DOM) represents a critical challenge in robotics, with applications spanning healthcare, manufacturing, food processing, and beyond. Unlike rigid objects, deformable objects exhibit infinite dimensionality, dynamic shape changes, and complex interactions with their environment, posing significant hurdles for perception, modeling, and control. This paper reviews the state of the art in DOM, focusing on key challenges such as occlusion handling, task generalization, and scalable, real-time solutions. It highlights advancements in multimodal perception systems, including the integration of multi-camera setups, active vision, and tactile sensing, which collectively address occlusion and improve adaptability in unstructured environments. Cutting-edge developments in physically informed reinforcement learning (RL) and differentiable simulations are explored, showcasing their impact on efficiency, precision, and scalability. The review also emphasizes the potential of simulated expert demonstrations and generative neural networks to standardize task specifications and bridge the simulation-to-reality gap. Finally, future directions are proposed, including the adoption of graph neural networks for high-level decision-making and the creation of comprehensive datasets to enhance DOM's real-world applicability. By addressing these challenges, DOM research can pave the way for versatile robotic systems capable of handling diverse and dynamic tasks with deformable objects.

Figures & tables

Category	Advantages	Disadvantages	Representative Literature
Visual perception	• Effective prediction of deformable object states. • Accurate object representation (affordance, keypoints, point clouds). • High performance for specific tasks. • Improved precision via advanced sensing.	• Struggles with unseen scenarios. • High computational and sensory demands. • Limited generalization across tasks. • Expensive hardware requirements.	• Learning Foresightful Dense Visual Affordance for Deformable Object Manipulation [ 103 ] • Learning Visual-Based Deformable Object Rearrangement with Local Graph Neural Networks [ 16 ] • DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects [ 96 ]
Tactile perception	• Excels where vision fails (e.g., occlusion, force-sensitive tasks). • Enables precise force and position control for delicate interactions. • Enhances adaptability with integration into learning frameworks. • Combines multimodal perception for richer object understanding.	• Highly task-specific, requiring customization for new applications. • Dependence on specialized tactile sensors increases cost and complexity. • Computational demands for real-time performance and multimodal processing. • Limited generalization across diverse tasks or environments.	• Precise Robotic Needle-Threading with Tactile Perception and Reinforcement Learning [ 113 ] • VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects [ 101 ] • Interaction Control for Tool Manipulation on Deformable Objects Using Tactile Feedback [ 118 ]
Interactive perception	• Enhances understanding of deformable objects through active engagement. • Improves performance via dynamic adjustments and interactive feedback. • Expands manipulation capabilities using novel methods (e.g., assistive tools, markerless systems). • Offers better modeling and data richness compared to static perception.	• High computational and hardware demands. • Task-specific approaches limit generalization. • Dependence on complex setups (e.g., multi-view systems, assistive tools). • Struggles with unseen objects or scenarios without extensive retraining.	• Interactive Perception for Deformable Object Manipulation [ 100 ] • Enhancing Deformable Object Manipulation by Using Interactive Perception and Assistive Tools [ 123 ] • HMDO: Markerless Multi-view Hand Manipulation Capture with Deformable Objects [ 104 ]

Table 1: Overview of recent literature in perception for deformable object manipulation

Figure 1: Illustration of cloth modeling using a mass-spring system: a triangular mesh structure representing the cloth is depicted, with a magnified view showing the individual mass points and spring connections, highlighting the deformation mechanics and the use of Hooke’s law (bottom centre).

Figure 2: The undeformed cube (left) represents the reference configuration, while the deformed cube (right) illustrates the effect of compressive forces acting along the horizontal sides (grey arrows). The deformation is described by the deformation gradient ( $\mathbf{F}$ ), where $\mathbf{F}=\frac{\partial\mathbf{x}}{\partial\mathbf{X}}$ , mapping the reference configuration ( $\mathbf{X}$ ) to the deformed configuration ( $\mathbf{x}$ ). The strain in the material is captured by the strain tensor $\boldsymbol{\varepsilon}=\frac{1}{2}\left(\nabla\mathbf{u}+(\nabla\mathbf{u})^{\top}\right)$ , with $\varepsilon_{xx}<0$ indicating compression along the horizontal axis. The stress tensor ( $\boldsymbol{\sigma}$ ) satisfies the equilibrium condition $\nabla\cdot\boldsymbol{\sigma}=0$ under static loading, and the material behavior follows Hooke’s law: $\boldsymbol{\sigma}=\mathbf{C}:\boldsymbol{\varepsilon}$ . The top and bottom faces remain undeformed due to fixed boundary conditions ( $u_{y}=0$ ).

Category	Advantages	Disadvantages	Literature
Mass-Spring-Systems	• Flexible for modeling deformable objects. • Accounts for material variations like stiffness and knots. • Supports real-world tasks (e.g., string handling, dual-arm coordination). • Learning integration improves adaptability. • Ensures grasp stability in deformable object manipulation. • Easy to implement in robotic systems.	• High computational complexity for real-time applications. • Simplified models may miss nonlinear deformations. • Requires pre-computation and parameter tuning. • Limited representation for complex materials (e.g., anisotropy). • Needs extensive training data.	• Tabata et al. ( [ 92 ] ) • Shi et al. ( [ 81 ] ) • Zaidi et al. ( [ 116 ] )
Position-Based-Dynamics	• Efficient, stable simulation for deformable objects. • Realistic modeling for ropes and DLOs. • Suitable for real-world tasks (e.g., suturing, obstacle avoidance). • Supports optimization via differentiable frameworks. • Validated in practical scenarios.	• Limited to linear or rope-like objects, reducing generalization. • Computational challenges for real-time tasks. • Relies on accurate parameter tuning. • Sensitive to parameter estimation errors.	• Yu et al. [ 111 ] • Liu et al. [ 49 ]
Continuum Mechanics	• High accuracy for modeling deformations. • Provides physics-based realism. • Enables blind manipulation using force sensing. • Effective for grasping tasks with deformable objects. • Validated through simulation and real-world applications.	• Computationally expensive, limiting real-time use. • Requires precise material properties and meshing. • Difficult to scale for complex objects. • Limited generalization to other deformable manipulation scenarios beyond the specific tasks studied.	• Sanchez et al. [ 76 ] • Huang et al. [ 35 ]

Table 2: Overview of recent literature in modelling for deformable object manipulation

Figure 3: Shooting control diagram demonstrating forward dynamics and optimization. The constrained variables are in the action and control space (dashed rectangle). Diagram is based on [ 85 ]

Figure 4: State trajectory exploration: The constrained variables lie within the state space (dashed rectangle), with actions determined based on state trajectories and inverse dynamics. This is a common framework in Model-Predictive-Control, a thorough explanation can be found here [ 95 ]

Figure 5: A basic closed-loop control framework based on visual features. A thorough description can be found here [ 88 ]

Figure 6: RL pipeline: First, the configurations of the DO are encoded into a latent space. Next, the RL agent generates manipulation actions by mapping the current and goal latent representations, enabling continuous interaction with the environment until the task is successfully completed. A thorough introduction to these concepts can be found in [ 91 ]

Planning and Control	Advantages	Disadvantages	Literature
Analytical Approaches	• High precision and predictability. • Robust, systematic frameworks. • Tailored, reproducible solutions. • Scalable to variations.	• Poor adaptability to uncertainties. • High computational cost. • Relies on accurate models. • Complex development.	• [ 121 ] • [ 15 ] • [ 12 ]
Reinforcement Learning	• Adaptive and robust. • Excels in dynamic scenarios. • Reduces need for explicit models. • Handles complex, high-dimensional tasks.	• High training costs. • Needs large, diverse data. • Struggles with sim-to-real gaps. • Risk of overfitting.	• [ 26 ] • [ 108 ] • [ 24 ]
Imitation Learning	• Adapts to deformable dynamics. • Requires fewer data. • Task-specific optimization. • Sim2Real benefits.	• Needs high-quality demos. • Limited generalization. • Computationally heavy. • Risk of overfitting.	• [ 12 ] • [ 24 ] • [ 102 ]

Table 3: Overview of recent literature in planning and Control for deformable object manipulation

Figure 7: Illustration links techniques and approaches that show great promise in DOM to face the key challenges outlined at the beginning of this review.

Explore similar work

Mar 14, 2026

Exploration-assisted Bottleneck Transition Toward Robust and Data-efficient Deformable Object Manipulation

Imitation learning has demonstrated impressive results in robotic manipulation but fails under out-of-distribution (OOD) states. This limitation is particularly critical in Deformable Object Manipulation (DOM), where the near-infinite possible configurations render comprehensive data collection infeasible. Although several methods address OOD states, they typically require exhaustive data or highly precise perception. Such requirements are often impractical for DOM owing to its inherent complexities, including self-occlusion. To address the OOD problem in DOM, we propose a novel framework, Exploration-assisted Bottleneck Transition for Deformable Object Manipulation (ExBot), which addresses the OOD challenge through two key advantages. First, we introduce bottleneck states, standardized configurations that serve as starting points for task execution. This enables the reconceptualization of OOD challenges as the problem of transitioning diverse initial states to these bottleneck states, significantly reducing demonstration requirements. Second, to account for imperfect perception, we partition the OOD state space based on recognizability and employ dual action primitives. This approach enables ExBot to manipulate even unrecognizable states without requiring accurate perception. By concentrating demonstrations around bottleneck states and leveraging exploration to alter perceptual conditions, ExBot achieves both data efficiency and robustness to severe OOD scenarios. Real-world experiments on rope and cloth manipulation demonstrate successful task completion from diverse OOD states, including severe self-occlusions.

Yujiro Onishi, Ryo Takizawa, Yoshiyuki Ohmura +1

Mechano-Informatics Department, Graduate School of Information Science and Technology, The University of Tokyo, Tokyo 113-8656, Japan.

Deformable Object Manipulation Generalization in Robotic Manipulation Robot Imitation Learning Data-Efficient Robot Learning

Ask PDF

Mar 3, 2026

RL-Based Coverage Path Planning for Deformable Objects on 3D Surfaces

Currently, manipulation tasks for deformable objects often focus on activities like folding clothes, handling ropes, and manipulating bags. However, research on contact-rich tasks involving deformable objects remains relatively underdeveloped. When humans use cloth or sponges to wipe surfaces, they rely on both vision and tactile feedback. Yet, current algorithms still face challenges with issues like occlusion, while research on tactile perception for manipulation is still evolving. Tasks such as covering surfaces with deformable objects demand not only perception but also precise robotic manipulation. To address this, we propose a method that leverages efficient and accessible simulators for task execution. Specifically, we train a reinforcement learning agent in a simulator to manipulate deformable objects for surface wiping tasks. We simplify the state representation of object surfaces using harmonic UV mapping, process contact feedback from the simulator on 2D feature maps, and use scaled grouped convolutions (SGCNN) to extract features efficiently. The agent then outputs actions in a reduced-dimensional action space to generate coverage paths. Experiments demonstrate that our method outperforms previous approaches in key metrics, including total path length and coverage area. We deploy these paths on a Kinova Gen3 manipulator to perform wiping experiments on the back of a torso model, validating the feasibility of our approach.

Yuhang Zhang, Jinming Ma, Feng Wu

School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui, China · Xiaomi Robotics Lab, Beijing, China

Coverage Path Planning Deformable Object Manipulation Contact-Rich Robotic Manipulation Robotic RL Robotic Tool Use

Ask PDF

Apr 24, 2026

LeHome: A Simulation Environment for Deformable Object Manipulation in Household Scenarios

Household environments present one of the most common, impactful yet challenging application domains for robotics. Within household scenarios, manipulating deformable objects is particularly difficult, both in simulation and real-world execution, due to varied categories and shapes, complex dynamics, and diverse material properties, as well as the lack of reliable deformable-object support in existing simulations. We introduce LeHome, a comprehensive simulation environment designed for deformable object manipulation in household scenarios. LeHome covers a wide spectrum of deformable objects, such as garments and food items, offering high-fidelity dynamics and realistic interactions that existing simulators struggle to simulate accurately. Moreover, LeHome supports multiple robotic embodiments and emphasizes low-cost robots as a core focus, enabling end-to-end evaluation of household tasks on resource-constrained hardware. By bridging the gap between realistic deformable object simulation and practical robotic platforms, LeHome provides a scalable testbed for advancing household robotics. Webpage: https://lehome-web.github.io/ .

Zeyi Li, Yushi Yang, Shawn Xie +15

Institute of Automation, Chinese Academy of Sciences. · Peking University. · Lightwheel. +1

Robotics Simulation Deformable Object Manipulation Robotics Robot Manipulation Benchmarks Deformable Object Simulation

Ask PDF