Paired video-action demonstrations enable autonomous surgical behavior, but such data is scarce: robots perform roughly 1% of surgeries, while video-only data is abundant. Learning 3D object flow offers an embodiment-agnostic way to utilize video data, but flow alone specifies how an object should move, not where and when the tool should engage it, a distinction that is critical in surgery. We introduce SurgFlow, a framework that learns 3D Object-Centric Contact Flow from stereo surgical video without action labels. For each object point, it predicts a future 3D trajectory and contact scores. We extract targets via 3D tracking and tool-object proximity, train a flow matching generator to predict them, and use predicted contact to trigger grasp and release while optimizing end effector motion from flow. On the da Vinci Research Kit (dVRK), SurgFlow succeeds in 37 of 39 stage evaluations across tissue retraction, bimanual reveal, needle pickup, and handover, outperforming baselines trained on equal data with or without action labels. Zero-shot transfer to a humanoid-based laparoscopic robot achieves 85% and 70% average success under similar and novel camera viewpoints, respectively.
Figures & tables
Fig. 1 : SurgFlow extracts object motion and tool–object contact from stereo video demonstrations without robot action labels. Contact flow directly guides bimanual surgical manipulation tasks, with experiments on two robot platforms.
Data
Learned Representation
Action Interface
New platform
Method
Action labels
Representation
Explicit contact
Motion
Grasping
Requires data
BC/IL [ 1 , 2 , 16 ]
✓
Implicit
✗
Policy
Policy
✓
Video generation [ 9 , 17 ]
✓
Synthetic frames
✗
IDM
IDM
✓
2D flow policy [ 10 , 12 ]
✓
2D point tracks
✗
Policy
Policy
✓
3D flow policy [ 18 , 19 ]
✓
3D keypoint flow
✗
Policy
Policy
✓
Embodiment flow [ 20 ]
✗
3D robot-point flow
✗
Optimization
Learned open/close
✗
TABLE I : Structural comparison of representative methods, as originally proposed. Methods are compared on whether they require action labels, their learned representation and whether it models contact explicitly, how end effector motion and gripper actions are realized, and whether deploying on a new robot platform requires new data.
Fig. 2 : Overview of SurgFlow. (a) Tool–object contact and 3D object flow are extracted from stereo video and combined into an object-centric contact flow representation. (b) Point cloud features are concatenated with query features to condition a flow matching generator that predicts future contact flow. (c) Contact flow supports bimanual manipulation of rigid and deformable objects, from sustained contact during tissue retraction to coordinated grasp and release during needle handover.
Fig. 3 : Bimanual needle handover with SurgFlow. Predicted contact flow (top) guides execution (bottom) through grasping, reorientation, right grasp, and left release. Contact scores remain high at the left grasp during reorientation, then rise at the right before fading on the left to guide the transfer between the two tools. Colors encode contact scores and horizon. Solid arrows show execution; dashed arrows show replanning from updated observations.
Method
Single Reveal
Bimanual reveal
Needle pickup
Needle handover
DP3
9/10
8/10
8/10
4/8
Object Flow(w/o contact)
9/10
4/10
6/10
—
Tool Flow
9/10
7/10
8/10
0/8
SurgFlow+IDM
10/10
10/10
9/10
4/9
SurgFlow
10/10
10/10
9/10
8/9
TABLE II : Task success on the dVRK. Cells report successful trials over attempts from shared randomized initial configurations; handover is evaluated only on trials with successful pickup. Best success rates are shown in bold, including ties; “—” denotes an inapplicable setting.
Grasp point error (mm), median [IQR]
Comparison with SurgFlow
Method
Single Reveal
Bimanual Reveal
Needle Pickup
Needle Handover
All
Δ [95% CI]
pHolm
Random
34.3 [31.1–41.8]
16.6 [14.7–22.4]
15.3 [10.7–22.2]
5.1 [3.8–9.7]
15.9 [9.7–24.3]
13.4 [8.9–18.7]
3.6×10−12
VLM
20.3 [15.9–27.1]
8.2 [6.5–9.7]
15.9 [10.9–18.1]
9.8 [2.3–18.4]
13.7 [7.5–18.4]
9.1 [5.3–14.6]
2.9×10−9
Max Flow
6.7 [4.2–10.4]
10.9 [9.5–14.5]
18.3 [17.4–19.2]
4.6 [1.7–6.2]
10.4 [5.1–16.5]
5.9 [3.1–8.6]
3.5×10−7
Tool Flow
6.8 [4.8–13.0]
5.7 [5.2–6.8]
2.4 [1.7–3.2]
2.2 [1.3–3.1]
4.6 [2.4–6.1]
1.3 [0.5–1.8]
0.007
SurgFlow
6.7 [6.6–7.5]
3.7 [2.8–3.8]
1.3 [0.9–1.6]
2.6 [0.7–3.3]
3.3 [1.4–4.5]
—
—
TABLE III: Grasp point accuracy on 40 decision states. Errors are reported as median [IQR] in millimeters. Δ reports the median paired improvement of SurgFlow over each baseline with a bootstrap 95% confidence interval, and pHolm denotes the Holm corrected paired Wilcoxon p value.
Fig. 4 : Grasp predictions for needle pickup, handover, bimanual reveal, and tissue retraction (columns) from Max Flow, Tool Flow, VLM, and SurgFlow (rows). White circles mark expert annotations; red crosses mark predictions.
Fig. 5 : Experimental platforms: (a) dVRK and (b) LapSurgie, with two endoscope viewpoints for cross platform transfer evaluation. Blue outlines indicate the endoscope cameras.
Task
SurgFlow Similar view
SurgFlow Extreme view
w/o aug. Extreme view
DP3
Bimanual reveal
7/10
5/10
1/10
0/10
Ex-vivo retraction
10/10
9/10
3/10
0/10
TABLE IV : Task success on LapSurgie, reported as successful trials out of ten attempts. Ex-vivo retraction uses one arm; reveal uses both arms. The augmentation ablation uses SurgFlow trained without occlusion augmentation.