Organizations: the T Stone Robotics Institute and Hong Kong Centre for Logistics Robotics, The Chinese University of Hong Kong, Hong Kong SAR, China · the University of Liverpool, Liverpool, U.K. · the School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China · the Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · the Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · the College of Control Science and Engineering, Zhejiang University, Hangzhou, China
Understanding 3D scene dynamics in minimally invasive surgery (MIS) is a fundamental step toward physically intelligent and autonomous surgical systems. However, existing public datasets offer limited scalability and diversity in real-world clinical scenarios. Moreover, annotations typically focus on either tissue or instruments alone, overlooking tool-tissue interactions. This gap limits applications such as augmented reality guidance and autonomous surgery driven by vision-language-action (VLA) models or world models, which demand large-scale, high-quality 3D supervision. In this work, we construct SurgTAP-3D, a large-scale in-vivo pseudo-3D surgical motion tracking dataset and benchmark, comprising 520K frames, 4,549 action clips, and 1.4M 2D/3D point trajectories from diverse robotic and manual procedures under unified action categories. To make annotation scalable, we design a unified framework for surgical 3D point tracking that converts uncalibrated monocular surgical videos into structured 4D annotations through contact-region prompting, active-tool segmentation, dynamic bundle adjustment, and global 4D refinement. This pipeline adapts foundation priors to surgical videos and produces temporally consistent trajectories, pseudo-scale geometry, camera poses, and instrument masks. The dataset captures realistic challenges inherent to surgical environments, including occlusions, smoke, bleeding, and rapid instrument motion. We establish a comprehensive benchmark on SurgPose, SurgT, and STIR, covering both instrument and tissue tracking. We demonstrate instrument displacement estimation, deformable surface reconstruction, and foundation-model fine-tuning. Extensive experiments and analysis demonstrate the effectiveness of our dataset and annotation framework for 3D surgical point tracking, reconstruction, and surgical action understanding. Homepage: https://wrld.github.io/surgtap-3d/
Figures & tables
Dataset
Surgical Domain
Setup
In-vivo
Contact Aware
Annot. Object
Annotation Type
Dataset Scale
2D
3D
Pixel Traj.
Seqs
Trajs
Frames
SurgT [ 1 ]
Multiple Procedures
Manual, Robotic
Partial
✗
Tissue
✓
✓
✗
157
32
24,548
STIR [ 2 ]
Cholecystectomy, Nissen Fundoplication Surgery
Robotic
Partial
✗
Tissue
✓
✓
✓
576
496
13,393
CholecTrack20 [ 3 ]
Cholecystectomy Surgery
Manual
✓
✗
Instrument
✓
✗
✗
20
2,624
35,000
CRCD [ 4 ]
Ex-vivo pseudo-cholecystectomy
Robotic
✗
✗
Instrument
✓
✗
✗
16
-
127,000
SurgPose [ 5 ]
Instrument Motion over Tissue
Robotic
✗
✗
Instrument
✓
✓
✓
30
476
30,030
TABLE I: Comparison with Surgical Tracking Datasets. Our SurgTAP-3D is the first large-scale, in-vivo , contact-aware , 3D pixel-wise trajectory dataset covering both instrument and tissue in real robotic and manual surgeries. Gray row : ours.
Fig. 2: Overview of the Annotation Framework for SurgTAP-3D. We first preprocess the data clips from the public datasets to collect the action clips, select the first frame, and filter the noisy frames. The annotation pipeline begins with the canonical labeling for the first frame of the clip, localizing the contact region and active tool to obtain the tracking prompts, which are input into TAPNext and SAM2 for inference. Next, our proposed pipeline takes frames and optional priors as input, achieving stable and robust 3D reconstruction through surgical-aware geometry priors from Endo3R [ 25 ] and dynamic bundle adjustment for correcting the drift. Finally, we conduct global 4D refinement to optimize the estimated depths and tracking result, enabling temporally smooth and geometrically coherent 3D contact-aware point tracking.
Type
Datasets
Surgery
Clips
Frames
Avg. Frames
Robotic Surgery
GraSP [ 71 ]
Prostatectomy
795
109939
138
SAR-RARP50 [ 72 ]
Prostatectomy
741
62822
85
Manual Surgery
SLAM [ 73 ]
Multiple procedures
203
24609
121
SurgicalActions160 [ 74 ]
Gynecology
26
2194
84
AutoLaparo [ 75 ]
Hysterectomy
43
4684
109
heicoProctocolectomy [ 76 ]
Proctocolectomy
39
4538
116
TABLE II: Description of SurgTAP-3D. The dataset contains the video datasets from both robotic surgery and manual surgery.
Fig. 3: Data Statistics. We analyze the data distribution of action frames, surgery types, surgical modality, and tool categories. We also visualize the clip lengths distribution in different actions, datasets, and the distribution of our labeled contact region.
Fig. 4: Visualization of 2D and 3D Tracking in different actions. We plot the point trajectories in 2D and 3D, with red-yellow trajectories for the active instrument and purple-blue trajectories for the tissues.
Fig. 5: Correlation between Actions and 3D Trajectories. We randomly sample ten clips per action and normalize their 3D instrument trajectories via spatial alignment. The projected mean trajectories form well-separated clusters, revealing a strong correlation between action categories and instrument motion patterns.
Fig. 6: Correlation between Actions and Instrument Types. We plot the heatmap to visualize the distribution of instrument types in different actions.
Fig. 7: Qualitative Comparison of 2D point tracking. We extensively compare with three types of point tracking methods, with blue points as the ground truth trajectory (from SurgPose [ 5 ] , STIR [ 2 ] , and SurgT [ 1 ] ) and orange points as the predicted trajectory. The results exhibit our better robustness to occlusions, illumination variance, and instrument tracking.
Action Class
SR ↑
IMC ↑
TCR ↑
TSR ↑
Dissection
91.89
96.89
98.53
93.70
Tissue Retraction
83.80
97.10
98.07
94.20
Needle Grasping
91.46
93.55
96.90
97.51
Vessel Clipping
88.21
97.03
99.45
92.13
Coagulation
86.71
95.35
98.72
92.88
Average
88.41
95.98
98.33
94.08
TABLE III: Annotation quality analysis across action classes. We report SR, IMC, TCR, and TSR in % as indicators of track survival, instrument-mask consistency, tracking confidence, and temporal smoothness, respectively.
Fig. 8: Challenging-scene analysis of Annotation Quality. Green box shows robust cases, including short-term instrument disappearance ( < 10s), smoke, and bleeding. Red box shows failure-prone cases, including long-term instrument disappearance ( > 10s), large occlusion, and fast instrument motion, where trajectories may drift or become unstable.
2D Tracking
Type
Method
SurgT [ 1 ]
SurgPose [ 5 ]
STIR [ 2 ]
Average
AJ 2D ↑
APD 2D ↑
OA ↑
AJ 2D ↑
APD 2D ↑
OA ↑
AJ 2D ↑
APD 2D ↑
OA ↑
AJ 2D ↑
APD 2D ↑
OA ↑
I
BootsTAPIR [ 86 ]
67.85
73.42
99.10
47.28
57.79
91.65
12.33
16.47
96.74
42.49
49.23
95.83
CoTrackerv3 [ 41 ]
73.32
79.02
98.75
49.40
61.84
83.89
19.34
24.23
96.51
47.35
55.03
93.05
AllTracker [ 87 ]
68.11
75.30
96.59
53.59
64.19
97.93
17.87
22.57
99.48
46.52
54.02
98.00
TAPNext [ 43 ]
74.98
80.52
98.18
60.35
69.78
96.40
18.89
23.46
95.46
51.41
57.92
96.68
TABLE IV: Comprehensive 3D and 2D surgical motion tracking Benchmark. We extensively evaluate the 2D and 3D point tracking for both instrument (SurgPose [ 5 ] ) and tissue surface (SurgT [ 1 ] and STIR [ 2 ] ).
Method
ATE ↓
RPE r ↓
RPE t ↓
Endo-SfM [ 89 ]
0.157
0.252
0.259
AF-SfM [ 90 ]
0.125
0.235
0.241
EndoDAC [ 91 ]
0.124
0.223
0.233
Robust [ 92 ]
0.131
0.241
0.245
Endo3R [ 25 ]
0.112
0.201
0.228
SpatialTrackerV2 [ 28 ]
0.110
0.217
0.239
TABLE V: Pose Comparison on SCARED [ 37 ] .
Method
SCARED [ 37 ]
Hamlyn [ 93 , 94 , 95 ]
Abs Rel ↓
Sq Rel ↓
RMSE ↓
RMSE log ↓
δ<1.25↑
Abs Rel ↓
Sq Rel ↓
RMSE ↓
RMSE log ↓
δ<1.25↑
Monodepth2 [ 96 ]
0.432
3.548
4.704
0.431
0.425
0.379
9.318
20.472
0.403
0.439
Endo-SfM [ 89 ]
0.241
0.865
2.286
0.267
0.585
0.252
4.335
14.430
0.268
0.628
AF-SfM [ 90 ]
0.257
0.960
2.162
0.291
0.573
0.286
5.715
15.895
0.301
0.508
EndoDAC [ 91 ]
0.242
0.934
2.014
0.275
0.584
0.275
5.557
15.669
0.288
0.519
Transfer [ 97 ]
0.297
1.207
2.561
0.319
0.561
0.281
5.790
15.936
0.312
0.504
TABLE VI: Quantitative comparison with state-of-the-art video depth estimation methods .
Table 14
Fig. 9: Real-world Application for Instrument Displacement Estimation. Given the predicted 3D trajectory, the instrument displacement is estimated by computing the centroids of the instrument trajectories at each timestep. To validate the effectiveness, we collect 50 sequences from dVRK with paired video clips and kinematic data. It shows that our framework enables accurate and robust instrument displacement estimation.
Fig. 10: Real-world Application for Deformable Tissue Reconstruction and Rendering. Given the reconstructed results, we train the 4D Gaussian Splatting model for deformable surface reconstruction and novel view synthesis for high-fidelity rendering. Compared with 4DGS with vanilla COLMAP initialization, our reconstructed point clouds, poses, and intrinsics yield significant improvement in reconstruction results.
Method
ATE ↓
RPE t↓
LiteTracker [ 46 ]
0.0035
0.0015
DELTAv2 [ 88 ]
0.0040
0.0022
SpatialTrackerV2 [ 28 ]
0.0064
0.0008
Ours
0.0022
0.0005
TABLE IX: Quantitative results for instrument displacement estimation on the 50 sequences of self-collected paired video–kinematic dVRK dataset (unit: m).
Init. + Sup.
PSNR ↑
SSIM ↑
LPIPS ↓
COLMAP + DAv2
27.82
0.8410
0.2161
COLMAP + Ours
27.51
0.8359
0.2196
Ours
28.16
0.8548
0.1310
TABLE X: Quantitative Comparison for Novel View Synthesis on SurgTAP-3D dataset. “Init.” refers to initialization. “Sup.” refers to depth supervision.
Settings
SurgT [ 1 ]
SurgPose [ 5 ]
AJ 2D↑
APD 2D↑
AJ 2D↑
APD 2D↑
CoTrackerv3 [ 41 ]
73.32
79.02
49.40
61.84
CoTrackerv3 [ 41 ] w/ ft.
74.14
80.25
50.99
62.53
DELTAv2 [ 88 ]
73.75
79.21
51.20
61.97
DELTAv2 [ 88 ] w/ ft.
74.57
80.11
52.75
62.86
TABLE XI: Quantitative Results for Finetuning CoTrackerV3 [ 41 ] and DELTAv2 [ 88 ] on our dataset.
Paired video-action demonstrations enable autonomous surgical behavior, but such data is scarce: robots perform roughly 1% of surgeries, while video-only data is abundant. Learning 3D object flow offers an embodiment-agnostic way to utilize video data, but flow alone specifies how an object should move, not where and when the tool should engage it, a distinction that is critical in surgery. We introduce SurgFlow, a framework that learns 3D Object-Centric Contact Flow from stereo surgical video without action labels. For each object point, it predicts a future 3D trajectory and contact scores. We extract targets via 3D tracking and tool-object proximity, train a flow matching generator to predict them, and use predicted contact to trigger grasp and release while optimizing end effector motion from flow. On the da Vinci Research Kit (dVRK), SurgFlow succeeds in 37 of 39 stage evaluations across tissue retraction, bimanual reveal, needle pickup, and handover, outperforming baselines trained on equal data with or without action labels. Zero-shot transfer to a humanoid-based laparoscopic robot achieves 85% and 70% average success under similar and novel camera viewpoints, respectively.
Changwei Chen, Xiao Liang, Yinuo Yang +6
Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093, USA
Robust point tracking in endoscopic videos is essential for computer-assisted intervention and autonomous robotic surgery, enabling continuous registration between intraoperative video and preoperative imaging despite soft tissue deformation. However, supervised tracking methods depend on large annotated datasets, while surgical conditions make reliable trajectory annotation challenging. We propose a self-supervised Track-Any-Point approach that learns from unlabeled surgical videos by establishing global pixel correspondences and inferring point trajectories through contrastive random walks. Trained without annotations, our method achieves performance comparable to existing semi-supervised approaches while implicitly handling tissue deformation. These findings demonstrate the feasibility of self-supervised point tracking in surgical environments and its potential to reduce reliance on annotated data.
Jiaming Zhang, Zijian Wu, Mehran Armand +1
Department of Computer Science, Johns Hopkins University · Robotics and Control Laboratory, The University of British Columbia · The Institute for Integrative and Innovative Research, University of Arkansas
Reconstructing surgical scenes from monocular endoscopic video is critical for advancing robotic-assisted surgery. However, the application of state-of-the-art general-purpose reconstruction models is constrained by two key challenges: the lack of supervised training data and performance degradation over long video sequences. To overcome these limitations, we propose SurgCUT3R, a systematic framework that adapts unified 3D reconstruction models to the surgical domain. Our contributions are threefold. First, we develop a data generation pipeline that exploits public stereo surgical datasets to produce large-scale, metric-scale pseudo-ground-truth depth maps, effectively bridging the data gap. Second, we propose a hybrid supervision strategy that couples our pseudo-ground-truth with geometric self-correction to enhance robustness against inherent data imperfections. Third, we introduce a hierarchical inference framework that employs two specialized models to effectively mitigate accumulated pose drift over long surgical videos: one for global stability and one for local accuracy. Experiments on the SCARED and StereoMIS datasets demonstrate that our method achieves a competitive balance between accuracy and efficiency, delivering near state-of-the-art but substantially faster pose estimation and offering a practical and effective solution for robust reconstruction in surgical environments. Project page: https://chumo-xu.github.io/SurgCUT3R-ICRA26/.
Kaiyuan Xu, Fangzhou Hong, Daniel Elson +1
The Hamlyn Centre for Robotic Surgery, Imperial College London, SW7 2AZ, UK. · S-Lab, College of Computing and Data Science, Nanyang Technological University, Singapore 639798. · Department of Computer Science, University of Liverpool, L69 7ZX, UK.