Paired video-action demonstrations enable autonomous surgical behavior, but such data is scarce: robots perform roughly 1% of surgeries, while video-only data is abundant. Learning 3D object flow offers an embodiment-agnostic way to utilize video data, but flow alone specifies how an object should move, not where and when the tool should engage it, a distinction that is critical in surgery. We introduce SurgFlow, a framework that learns 3D Object-Centric Contact Flow from stereo surgical video without action labels. For each object point, it predicts a future 3D trajectory and contact scores. We extract targets via 3D tracking and tool-object proximity, train a flow matching generator to predict them, and use predicted contact to trigger grasp and release while optimizing end effector motion from flow. On the da Vinci Research Kit (dVRK), SurgFlow succeeds in 37 of 39 stage evaluations across tissue retraction, bimanual reveal, needle pickup, and handover, outperforming baselines trained on equal data with or without action labels. Zero-shot transfer to a humanoid-based laparoscopic robot achieves 85% and 70% average success under similar and novel camera viewpoints, respectively.
Figures & tables
Fig. 1 : SurgFlow extracts object motion and tool–object contact from stereo video demonstrations without robot action labels. Contact flow directly guides bimanual surgical manipulation tasks, with experiments on two robot platforms.
Data
Learned Representation
Action Interface
New platform
Method
Action labels
Representation
Explicit contact
Motion
Grasping
Requires data
BC/IL [ 1 , 2 , 16 ]
✓
Implicit
✗
Policy
Policy
✓
Video generation [ 9 , 17 ]
✓
Synthetic frames
✗
IDM
IDM
✓
2D flow policy [ 10 , 12 ]
✓
2D point tracks
✗
Policy
Policy
✓
3D flow policy [ 18 , 19 ]
✓
3D keypoint flow
✗
Policy
Policy
✓
Embodiment flow [ 20 ]
✗
3D robot-point flow
✗
Optimization
Learned open/close
✗
TABLE I : Structural comparison of representative methods, as originally proposed. Methods are compared on whether they require action labels, their learned representation and whether it models contact explicitly, how end effector motion and gripper actions are realized, and whether deploying on a new robot platform requires new data.
Fig. 2 : Overview of SurgFlow. (a) Tool–object contact and 3D object flow are extracted from stereo video and combined into an object-centric contact flow representation. (b) Point cloud features are concatenated with query features to condition a flow matching generator that predicts future contact flow. (c) Contact flow supports bimanual manipulation of rigid and deformable objects, from sustained contact during tissue retraction to coordinated grasp and release during needle handover.
Fig. 3 : Bimanual needle handover with SurgFlow. Predicted contact flow (top) guides execution (bottom) through grasping, reorientation, right grasp, and left release. Contact scores remain high at the left grasp during reorientation, then rise at the right before fading on the left to guide the transfer between the two tools. Colors encode contact scores and horizon. Solid arrows show execution; dashed arrows show replanning from updated observations.
Method
Single Reveal
Bimanual reveal
Needle pickup
Needle handover
DP3
9/10
8/10
8/10
4/8
Object Flow(w/o contact)
9/10
4/10
6/10
—
Tool Flow
9/10
7/10
8/10
0/8
SurgFlow+IDM
10/10
10/10
9/10
4/9
SurgFlow
10/10
10/10
9/10
8/9
TABLE II : Task success on the dVRK. Cells report successful trials over attempts from shared randomized initial configurations; handover is evaluated only on trials with successful pickup. Best success rates are shown in bold, including ties; “—” denotes an inapplicable setting.
Grasp point error (mm), median [IQR]
Comparison with SurgFlow
Method
Single Reveal
Bimanual Reveal
Needle Pickup
Needle Handover
All
Δ [95% CI]
pHolm
Random
34.3 [31.1–41.8]
16.6 [14.7–22.4]
15.3 [10.7–22.2]
5.1 [3.8–9.7]
15.9 [9.7–24.3]
13.4 [8.9–18.7]
3.6×10−12
VLM
20.3 [15.9–27.1]
8.2 [6.5–9.7]
15.9 [10.9–18.1]
9.8 [2.3–18.4]
13.7 [7.5–18.4]
9.1 [5.3–14.6]
2.9×10−9
Max Flow
6.7 [4.2–10.4]
10.9 [9.5–14.5]
18.3 [17.4–19.2]
4.6 [1.7–6.2]
10.4 [5.1–16.5]
5.9 [3.1–8.6]
3.5×10−7
Tool Flow
6.8 [4.8–13.0]
5.7 [5.2–6.8]
2.4 [1.7–3.2]
2.2 [1.3–3.1]
4.6 [2.4–6.1]
1.3 [0.5–1.8]
0.007
SurgFlow
6.7 [6.6–7.5]
3.7 [2.8–3.8]
1.3 [0.9–1.6]
2.6 [0.7–3.3]
3.3 [1.4–4.5]
—
—
TABLE III: Grasp point accuracy on 40 decision states. Errors are reported as median [IQR] in millimeters. Δ reports the median paired improvement of SurgFlow over each baseline with a bootstrap 95% confidence interval, and pHolm denotes the Holm corrected paired Wilcoxon p value.
Fig. 4 : Grasp predictions for needle pickup, handover, bimanual reveal, and tissue retraction (columns) from Max Flow, Tool Flow, VLM, and SurgFlow (rows). White circles mark expert annotations; red crosses mark predictions.
Fig. 5 : Experimental platforms: (a) dVRK and (b) LapSurgie, with two endoscope viewpoints for cross platform transfer evaluation. Blue outlines indicate the endoscope cameras.
Task
SurgFlow Similar view
SurgFlow Extreme view
w/o aug. Extreme view
DP3
Bimanual reveal
7/10
5/10
1/10
0/10
Ex-vivo retraction
10/10
9/10
3/10
0/10
TABLE IV : Task success on LapSurgie, reported as successful trials out of ten attempts. Ex-vivo retraction uses one arm; reveal uses both arms. The augmentation ablation uses SurgFlow trained without occlusion augmentation.
World models offer a promising route toward robot planning by enabling agents to imagine and verify the consequences of actions before execution. However, current video-based world models often struggle to capture the physical constraints that govern manipulation, particularly contact. Further, their action conditioning is often constrained to specific embodiments such as parallel grippers. We propose \emph{Contact Flow}, an embodiment-agnostic action representation that encodes manipulation through the trajectory of 3D contact points between an actor and a target object. By discarding actor-specific appearance and kinematics, Contact Flow provides a shared conditioning signal for both human demonstrations and robotic execution. Therefore, we can train a large-scale video generative model on both human and robotic object interaction videos conditioned on Contact Flow, yielding a world model that predicts physically plausible manipulation outcomes. We integrate this model into a propose-imagine-verify-act pipeline, where generated rollouts are assessed by a vision-language model before execution. Experiments on the DROID dataset and real-world tabletop manipulation tasks demonstrate that Contact Flow enables transfer between human demonstrations and different robotic embodiments.
Learning-based surgical robot autonomy requires large-scale demonstrations with synchronized videos and robot actions, but such data are exceedingly rare in clinical or realistic tissue settings because robot kinematics are typically inaccessible outside controlled research systems. In contrast, phantom data collected on research platforms provide accurate action labels but lack the visual diversity of real tissue. We propose SurgVIL, a framework for scaling surgical robot imitation learning using open-source surgical videos. SurgVIL combines kinematically labeled phantom robot demonstrations with surgical videos from open-source datasets and online sources for policy learning. Since these videos lack robot motion labels, we estimate approximate kinematics as weak supervision. We evaluate SurgVIL on two da Vinci robot tasks: needle pick-up and cholecystectomy cutting. Across ACT, π0, and GR00T-H backbones, adding surgical videos substantially improves generalization to real-tissue and out-of-distribution settings, suggesting a scalable path from phantom training toward generalizable surgical robot policies.
Xinhao Chen, JuoTung Chen, Nigel Nelson +4
Laboratory for Computational Sensing and Robotics (LCSR) Johns Hopkins University, USA · Nvidia, USA
Learning reliable surgical manipulation policies is bottlenecked by the scarcity of action-labeled demonstrations: teleoperated surgical robot (e.g., dVRK) trajectories with synchronized kinematics are costly to collect, while surgical tasks demand precise contact handling, long-horizon reasoning, and bimanual coordination. Endoscopic video is comparatively inexpensive and abundant relative to synchronized video--kinematics trajectories, and a natural way to exploit it is to learn world models of surgical scenes. However, existing surgical world models use video primarily for simulation or policy evaluation, and rarely translate the learned dynamics into closed-loop control. This gap raises our central question: under a fixed budget of action-labeled demonstrations, does action-free video pretraining improve closed-loop surgical manipulation? To answer it, we introduce the Surgical World-Action Model (Surgical WAM), a unified generative model built on Cosmos Policy that jointly predicts future endoscopic observations and executable surgical robot action chunks. Surgical WAM first learns surgical visual dynamics from action-free video and is then fine-tuned on the fixed action-labeled budget; at deployment, it acts as a closed-loop, receding-horizon controller that executes a short prefix of each predicted action chunk and replans from the resulting observation. On a suite of four simulated surgical manipulation tasks, video pretraining improves the average success rate from 63.5% to 77.8%, including an absolute gain of 20 percentage points on PegTransfer, with the largest improvements on contact-rich and bimanual tasks. These results demonstrate that action-free video provides transferable visual dynamics priors for learning surgical robot control with limited action supervision, positioning data-efficient video pretraining as a practical path toward scaling up surgical robot learning.