Manipulating deformable objects toward desired 3D configurations is a fundamental capability for robots operating in unstructured physical environments. However, as the range of possible initial and target configurations grows, goal-conditioned deformable manipulation requires a robot to infer increasingly diverse manipulation behaviors in addition to precise continuous actions. We introduce DefFlow, a framework for goal-conditioned manipulation of deformable objects toward a broad range of 3D configurations. DefFlow learns from demonstrations by jointly predicting the goal-conditioned 3D point flow of the robot and deformable object. We show that this formulation better captures both the manipulation decisions, such as when and where to grasp, release, or re-grasp, and the precise continuous motions required for goal reaching, rather than predicting only the robot flow. Across rope, granular-media, and cloth manipulation settings with diverse initial and target configurations, DefFlow substantially outperforms goal-conditioned action-prediction and robot-flow-only baselines on success rates and goal-distance metrics. We further demonstrate deployment of our framework on a real robot using real point-cloud observations and unseen goal configurations.
Figures & tables
Fig. 2 : DefFlow architecture. Given the scene (including the robot) and a goal point cloud, DefFlow processes all points in a unified 3D space using a PTv3 backbone with learned type-embeddings, which is passed through an MLP to get 3D scene flow. The robot flows are then used to recover executable robot actions.
Fig. 3 : Simulation benchmarking environments. This figure shows the distribution of tasks in each environment. As can be seen in (a), each task differs not only in the deformed shape, orientation, and position of both start and goal configurations, but also in the action trajectory required to reach the goal. Some require grasping the middle of the rope, others an end or a point in between. Some require only one grasp, while others require multiple grasps; some may require dragging forward, to the left, or in a zigzag pattern, or swishing. For (b), each task involves a different target shape and requires a different combination of paddle sweeps, varying in number, angle, and extent. Some target shapes require the sweeps to be performed in a specific order. In (c), tasks differ not only in the cloth’s starting position and orientation and its goal configuration, but also significantly in the actions required, as some might require grasping the middle of an edge and simply draping the cloth over the bunny, while others require a corner grasp, a slight swish, and then draping. All three environments share large scene configuration spaces and combinatorial action spaces, much broader than those explored previously [ 4 , 18 , 5 , 6 ] .
Fig. 4 : Quantitative results on RopeEnv . Lower is better for L2 error, higher for SR.
GranularEnv
ClothEnv
Method
CD ↓
EMD ↓
3D IoU ↑
L2 (cm) ↓
Succ@3cm ↑
GC-DP3
1.050.18
2.410.54
55.667.05
6.606.10
25%
GC-DP3-PTv3
0.990.18
2.270.79
58.447.23
6.587.05
22.9%
Robot-Flow
0.950.15
2.020.54
57.807.03
6.066.81
32.3%
Rigid Robot-Flow
0.920.15
1.900.45
59.007.76
6.026.79
32.3%
GC-PointBERT
0.920.17
2.070.57
58.787.76
–
–
TABLE I : Quantitative results on GranularEnv and ClothEnv . CD, EMD, and L2 are reported in cm, with means and standard deviations over episodes; 3D IoU is shown ×102 .
Fig. 5 : Qualitative Results on rope manipulation. In (a) the robot needs to grasp the rope, lift and swish back in a precise manner to reach the exact shape. As can be seen, DefFlow predicts the precise grasp+swishing, while the baselines either get the grasp incorrect and collide with the rope, or do not predict the precise swish resulting in the rope not reaching the exact shape. In (b) we showcase two settings with the same goal but different initial configurations, one from the same demo as the goal and the other from a separate demo, while in (c) we show settings with the same starting state but another goal from a separate demo. As can be seen, DefFlow also generalizes to these unseen settings by predicting different action trajectories as in (c) and taking different decisions in (b) where for one it grasps and lifts while for the second it first grasps and moves the rope to align it and then re-grasps and lifts.
Fig. 6 : Qualitative Results on Granular Shaping. We show qualitative results on unseen goal shapes in (a). We additionally show results on unseen letter shapes in (b). Each goal shape is its own unique design requiring different sweep actions to accomplish in different orders and numbers.
Fig. 7 : Toy cube-stacking experiment setting and results. The three columns show the top-down view of the distribution and the perturbations as well as the density of the demonstrations, with the corresponding success rate bar plots below them