Achieving 3D spatial awareness is crucial for surgical robotic manipulation, where precise and delicate operations are required. Existing methods either explicitly reconstruct the surgical scene prior to manipulation, or enhance multi-view features by adding wrist-mounted cameras to supplement the default stereo endoscopes. However, both paradigms suffer from notable limitations: the former easily leads to error accumulation and prevents end-to-end optimization due to its multi-stage nature, while the latter is rarely adopted in clinical practice since wrist-mounted cameras can interfere with the motion of surgical robot arms. In this work, we introduce the Spatial Surgical Transformer (SST), an end-to-end visuomotor policy that empowers surgical robots with 3D spatial awareness by directly exploring 3D spatial cues embedded in endoscopic images. First, we build Surgical3D, a large-scale photorealistic dataset containing 30K stereo endoscopic image pairs with accurate 3D geometry, addressing the scarcity of 3D data in surgical scenes. Based on Surgical3D, we finetune a powerful geometric transformer to extract robust 3D latent representations from stereo endoscopes images. These representations are then seamlessly aligned with the robot's action space via a lightweight multi-level spatial feature connector (MSFC), all within an endoscope-centric coordinate frame. Extensive real-robot experiments demonstrate that SST achieves state-of-the-art performance and strong spatial generalization on complex surgical tasks such as knot tying and ex-vivo organ dissection, representing a significant step toward practical clinical deployment. The dataset and code will be released.
Figures & tables
Figure 2: Pipeline of Our Method. Top: The geometry transformer is first finetuned on the proposed Surgical3D dataset using a 3D reconstruction objective, enabling the extraction of robust 3D latent embeddings from endoscopic images. Bottom: The geometry transformer is then frozen, while the remaining components are trained to learn surgical manipulation policies with spatial priors from collected demonstrations. A multi-level spatial feature connector (MSFC) is trained to aggregate 3D latent embeddings from multiple geometry transformer blocks and aligns them with the robot’s action space. An endoscope-centric policy decoder generates relative robot actions in the endoscope frame, guided by the learned 3D spatial priors.
Figure 3: Left: Samples from our Surgical3D dataset. We utilize diverse 3D surgical assets to generate highly realistic and varied synthetic surgical scenes. Right: The figure illustrates the reconstruction results from MASt3R under three finetuning configurations. (a) One example of a stereo endoscopic image captured in a real in-vivo surgical scene, used as input. (b) The original MASt3R fails to reconstruct both the patient-side manipulator (PSM) and the organ surface. (c) When finetuned solely on synthetic data, the organ reconstruction remains coarse and the PSM geometry is still incomplete. (d) When finetuned on a combination of synthetic and real data, the model achieves more accurate reconstruction of PSM instruments and better organ completeness.
Figure 4: Visualization of Experimental Settings. Yellow and blue arrows denote the approximate motion directions of the right and left arms, respectively. Top: Peg pickup task. A total of 180 trajectories were collected, with roughly 120 in the green region and 60 in the blue region for training. Test1 and Test2 correspond to evaluations in these respective areas. Middle: Knot tying task. The suture tail was randomly positioned within the blue region during data collection, and evaluations were performed under the same condition. (b1) and (b3) show the grasping and looping actions. Bottom: Ex-vivo gallbladder dissection task. Grasp points on the gallbladder were sampled from varying positions within the blue region during data collection, and all methods were evaluated under the same setup. (c1) and (c3) depict the grasping and cutting actions. Supplementary videos provide detailed views of the task execution.
Figure 5: Qualitative Results of Intermediate 3D Reconstruction. Subscripts at the lower left of each image indicate the manipulation step. Tokens from the finetuned geometry transformer are fed into DPT heads to generate intermediate 3D reconstructions. The results demonstrate that the geometry transformer effectively extracts 3D cues from current endoscopic observations across various test tasks without task-specific training. More details are provided in the supplementary videos .
Peg Pickup
Knot Tying
Gallbladder Dissection
Methods
Setting
Test1
Test2
Grasp
Loop
Whole Task
Grasp
Dissection
Whole Task
SRT [ 17 ]
w/. wrist cams
10/10
6/10
10/10
3/10
2/10
-
-
-
ACT [ 56 ]
w/o. wrist cams
9/10
2/10
4/10
0/10
0/10
0/10
0/10
0/10
DP [ 7 ]
w/o. wrist cams
10/10
1/10
5/10
1/10
1/10
0/10
0/10
0/10
SST (Ours)
w/o. wrist cams
10/10
8/10
10/10
7/10
7/10
10/10
6/10
6/10
Table 1: Success Rates on Three Surgical Tasks Across Various Methods.
Peg Pickup
Knot Tying
Config
test1
test2
Grasp
Loop
Whole Task
w/o ToS
2/10
0/10
0/10
0/10
0/10
w/ ToS (Ours)
10/10
8/10
10/10
7/10
7/10
Table 2: Effectiveness of Finetuning Geometry Transformer on Surgical3D Dataset. “ToS” indicates that the geometry transformer was trained on the Surgical3D dataset.
Figure 6: Alternative Designs of Spatial Connectors. (a) Last-Layer Feature Connector. (b) Multi-Layer Separate Connector
Reconstructing surgical scenes from monocular endoscopic video is critical for advancing robotic-assisted surgery. However, the application of state-of-the-art general-purpose reconstruction models is constrained by two key challenges: the lack of supervised training data and performance degradation over long video sequences. To overcome these limitations, we propose SurgCUT3R, a systematic framework that adapts unified 3D reconstruction models to the surgical domain. Our contributions are threefold. First, we develop a data generation pipeline that exploits public stereo surgical datasets to produce large-scale, metric-scale pseudo-ground-truth depth maps, effectively bridging the data gap. Second, we propose a hybrid supervision strategy that couples our pseudo-ground-truth with geometric self-correction to enhance robustness against inherent data imperfections. Third, we introduce a hierarchical inference framework that employs two specialized models to effectively mitigate accumulated pose drift over long surgical videos: one for global stability and one for local accuracy. Experiments on the SCARED and StereoMIS datasets demonstrate that our method achieves a competitive balance between accuracy and efficiency, delivering near state-of-the-art but substantially faster pose estimation and offering a practical and effective solution for robust reconstruction in surgical environments. Project page: https://chumo-xu.github.io/SurgCUT3R-ICRA26/.
Kaiyuan Xu, Fangzhou Hong, Daniel Elson +1
The Hamlyn Centre for Robotic Surgery, Imperial College London, SW7 2AZ, UK. · S-Lab, College of Computing and Data Science, Nanyang Technological University, Singapore 639798. · Department of Computer Science, University of Liverpool, L69 7ZX, UK.
Recent advances in robot imitation learning have produced powerful visuomotor policies that manipulate diverse objects from visual inputs. However, monocular observations lack depth information, which is critical for precise manipulation in cluttered or geometrically complex scenes. Explicit depth maps and point clouds are often noisy and fragile in real-world manipulation. We introduce StereoPolicy, a visuomotor policy learning framework that directly leverages synchronized stereo image pairs to improve geometric reasoning without constructing explicit 3D representations. StereoPolicy processes each image with pretrained 2D vision encoders and fuses left-right features through a cross-attention-based Stereo Transformer, capturing spatial correspondence and disparity cues implicitly. The framework integrates with diffusion-based and pretrained vision-language-action (VLA) policies, delivering consistent improvements over RGB, RGB-D, point cloud, and multi-view baselines across three simulation benchmarks and seven real-robot tabletop and bimanual mobile manipulation tasks. Our results show that stereo vision bridges 2D pretrained representations and 3D geometric understanding for robotic manipulation.
Evans Han, Yunfan Jiang, Yingke Wang +6
1Stanford University · 2Northwestern University · 3Lambda, Inc
Paired video-action demonstrations enable autonomous surgical behavior, but such data is scarce: robots perform roughly 1% of surgeries, while video-only data is abundant. Learning 3D object flow offers an embodiment-agnostic way to utilize video data, but flow alone specifies how an object should move, not where and when the tool should engage it, a distinction that is critical in surgery. We introduce SurgFlow, a framework that learns 3D Object-Centric Contact Flow from stereo surgical video without action labels. For each object point, it predicts a future 3D trajectory and contact scores. We extract targets via 3D tracking and tool-object proximity, train a flow matching generator to predict them, and use predicted contact to trigger grasp and release while optimizing end effector motion from flow. On the da Vinci Research Kit (dVRK), SurgFlow succeeds in 37 of 39 stage evaluations across tissue retraction, bimanual reveal, needle pickup, and handover, outperforming baselines trained on equal data with or without action labels. Zero-shot transfer to a humanoid-based laparoscopic robot achieves 85% and 70% average success under similar and novel camera viewpoints, respectively.
Changwei Chen, Xiao Liang, Yinuo Yang +6
Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093, USA