Manipulating deformable objects toward desired 3D configurations is a fundamental capability for robots operating in unstructured physical environments. However, as the range of possible initial and target configurations grows, goal-conditioned deformable manipulation requires a robot to infer increasingly diverse manipulation behaviors in addition to precise continuous actions. We introduce DefFlow, a framework for goal-conditioned manipulation of deformable objects toward a broad range of 3D configurations. DefFlow learns from demonstrations by jointly predicting the goal-conditioned 3D point flow of the robot and deformable object. We show that this formulation better captures both the manipulation decisions, such as when and where to grasp, release, or re-grasp, and the precise continuous motions required for goal reaching, rather than predicting only the robot flow. Across rope, granular-media, and cloth manipulation settings with diverse initial and target configurations, DefFlow substantially outperforms goal-conditioned action-prediction and robot-flow-only baselines on success rates and goal-distance metrics. We further demonstrate deployment of our framework on a real robot using real point-cloud observations and unseen goal configurations.
Figures & tables
Fig. 2 : DefFlow architecture. Given the scene (including the robot) and a goal point cloud, DefFlow processes all points in a unified 3D space using a PTv3 backbone with learned type-embeddings, which is passed through an MLP to get 3D scene flow. The robot flows are then used to recover executable robot actions.
Fig. 3 : Simulation benchmarking environments. This figure shows the distribution of tasks in each environment. As can be seen in (a), each task differs not only in the deformed shape, orientation, and position of both start and goal configurations, but also in the action trajectory required to reach the goal. Some require grasping the middle of the rope, others an end or a point in between. Some require only one grasp, while others require multiple grasps; some may require dragging forward, to the left, or in a zigzag pattern, or swishing. For (b), each task involves a different target shape and requires a different combination of paddle sweeps, varying in number, angle, and extent. Some target shapes require the sweeps to be performed in a specific order. In (c), tasks differ not only in the cloth’s starting position and orientation and its goal configuration, but also significantly in the actions required, as some might require grasping the middle of an edge and simply draping the cloth over the bunny, while others require a corner grasp, a slight swish, and then draping. All three environments share large scene configuration spaces and combinatorial action spaces, much broader than those explored previously [ 4 , 18 , 5 , 6 ] .
Fig. 4 : Quantitative results on RopeEnv . Lower is better for L2 error, higher for SR.
GranularEnv
ClothEnv
Method
CD ↓
EMD ↓
3D IoU ↑
L2 (cm) ↓
Succ@3cm ↑
GC-DP3
1.050.18
2.410.54
55.667.05
6.606.10
25%
GC-DP3-PTv3
0.990.18
2.270.79
58.447.23
6.587.05
22.9%
Robot-Flow
0.950.15
2.020.54
57.807.03
6.066.81
32.3%
Rigid Robot-Flow
0.920.15
1.900.45
59.007.76
6.026.79
32.3%
GC-PointBERT
0.920.17
2.070.57
58.787.76
–
–
TABLE I : Quantitative results on GranularEnv and ClothEnv . CD, EMD, and L2 are reported in cm, with means and standard deviations over episodes; 3D IoU is shown ×102 .
Fig. 5 : Qualitative Results on rope manipulation. In (a) the robot needs to grasp the rope, lift and swish back in a precise manner to reach the exact shape. As can be seen, DefFlow predicts the precise grasp+swishing, while the baselines either get the grasp incorrect and collide with the rope, or do not predict the precise swish resulting in the rope not reaching the exact shape. In (b) we showcase two settings with the same goal but different initial configurations, one from the same demo as the goal and the other from a separate demo, while in (c) we show settings with the same starting state but another goal from a separate demo. As can be seen, DefFlow also generalizes to these unseen settings by predicting different action trajectories as in (c) and taking different decisions in (b) where for one it grasps and lifts while for the second it first grasps and moves the rope to align it and then re-grasps and lifts.
Fig. 6 : Qualitative Results on Granular Shaping. We show qualitative results on unseen goal shapes in (a). We additionally show results on unseen letter shapes in (b). Each goal shape is its own unique design requiring different sweep actions to accomplish in different orders and numbers.
Fig. 7 : Toy cube-stacking experiment setting and results. The three columns show the top-down view of the distribution and the perturbations as well as the density of the demonstrations, with the corresponding success rate bar plots below them
Cross-embodiment data have become central to training robotic foundation models. To leverage such heterogeneous data, we focus on flow-based object manipulation, where robot flows (robot velocity fields) serve as embodiment-agnostic motion representations. Previous studies do not formulate robot flows as dense velocity fields, but as displacements of sparse keypoints, even though dense velocity fields better match the continuous-time nature of motions. To address this, we propose Flow as Flow, a framework that models robot flows as probability flows based on a flow matching formulation. By naturally modeling such velocity fields within this formulation, our method achieves efficient and high-quality robot flow generation. Across standard benchmarks, our method outperforms representative baseline methods on standard metrics, while achieving approximately 24× faster generation than standard flow matching. Furthermore, through real-world experiments evaluating 9 methods with 260 trials per method across 13 manipulation tasks, we show that our method achieves a higher average success rate than the baseline methods.
Deformable object manipulation (DOM) is challenging due to high-dimensional, partially observable states that evolve through long-horizon, topology-changing interactions with multiple valid manipulation modes. We introduce Instant-Fold, an in-context imitation learning framework for DOM. Given a single human demonstration, our policy infers and executes diverse manipulation modes directly from the demonstration, including variations in spatial execution and ordering, without requiring gradient updates. Our approach first learns deformation-aware visual representations via temporal contrastive pretraining, after which a flow-matching transformer policy conditioned on the demonstration predicts actions to execute the intended manipulation mode. Trained entirely in simulation, Instant-Fold generalizes across diverse folding modes and transfers zero-shot to real-world settings without additional data collection or finetuning. Videos are available at https://instant-fold.github.io.
Planning and acting in 3D environments is a fundamental capability for robotic manipulation in the real world. Although prior work has explored predictive flow planners to guide 3D manipulation, existing approaches often rely on modular pipelines stacking multiple submodels, resulting in high computational overhead and limited real-time performance. To address these challenges, we introduce RoboFlow4D, a lightweight flow world model that unifies perception and planning by estimating temporal motion in physical 3D space. As an end-to-end framework, RoboFlow4D directly predicts multi-frame 3D flows from visual observations and textual instructions, providing explicit flow-based planning to guide action generation. This design allows seamless integration with general action policies, forming an efficient observation-planning-execution closed loop. Through slow-fast collaboration between flow prediction and action control, RoboFlow4D enables real-time and resource-efficient manipulation. Extensive experiments in both simulation and real-world settings demonstrate that RoboFlow4D consistently improves manipulation success rates and computational efficiency, advancing flow-guided planning for embodied intelligence.
Sixu Lin, Junliang Chen, Huaiyuan Xu +8
School of Data Science, The Chinese University of Hong Kong (Shenzhen) · The Hong Kong Polytechnic University · Shenzhen Loop Area Institute +1