SurgTAP-3D: A Large-Scale Cross-Surgery Video Dataset with Contact-Aware 3D Motion Tracking

Jiaxin Guo; Wenzhen Dong; Taoyu Wu; Haomin Kuang; Mengya Xu; Kejian Shi; Ziyi Wang; Tongfan Guan

cs.CVNov 8, 2026

SurgTAP-3D: A Large-Scale Cross-Surgery Video Dataset with Contact-Aware 3D Motion Tracking

Authors: Jiaxin Guo, Wenzhen Dong, Taoyu Wu, Haomin Kuang, Mengya Xu, Kejian Shi, Ziyi Wang, Tongfan Guan,

Organizations: the T Stone Robotics Institute and Hong Kong Centre for Logistics Robotics, The Chinese University of Hong Kong, Hong Kong SAR, China · the University of Liverpool, Liverpool, U.K. · the School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China · the Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China ·

Abstract

Understanding 3D scene dynamics in minimally invasive surgery (MIS) is a fundamental step toward physically intelligent and autonomous surgical systems. However, existing public datasets offer limited scalability and diversity in real-world clinical scenarios. Moreover, annotations typically focus on either tissue or instruments alone, overlooking tool-tissue interactions. This gap limits applications such as augmented reality guidance and autonomous surgery driven by vision-language-action (VLA) models or world models, which demand large-scale, high-quality 3D supervision. In this work, we construct SurgTAP-3D, a large-scale in-vivo pseudo-3D surgical motion tracking dataset and benchmark, comprising 520K frames, 4,549 action clips, and 1.4M 2D/3D point trajectories from diverse robotic and manual procedures under unified action categories. To make annotation scalable, we design a unified framework for surgical 3D point tracking that converts uncalibrated monocular surgical videos into structured 4D annotations through contact-region prompting, active-tool segmentation, dynamic bundle adjustment, and global 4D refinement. This pipeline adapts foundation priors to surgical videos and produces temporally consistent trajectories, pseudo-scale geometry, camera poses, and instrument masks. The dataset captures realistic challenges inherent to surgical environments, including occlusions, smoke, bleeding, and rapid instrument motion. We establish a comprehensive benchmark on SurgPose, SurgT, and STIR, covering both instrument and tissue tracking. We demonstrate instrument displacement estimation, deformable surface reconstruction, and foundation-model fine-tuning. Extensive experiments and analysis demonstrate the effectiveness of our dataset and annotation framework for 3D surgical point tracking, reconstruction, and surgical action understanding. Homepage: https://wrld.github.io/surgtap-3d/

Figures & tables

Dataset	Surgical Domain	Setup	In-vivo	Contact Aware	Annot. Object	Annotation Type			Dataset Scale
Dataset	Surgical Domain	Setup	In-vivo	Contact Aware	Annot. Object	2D	3D	Pixel Traj.	Seqs	Trajs	Frames
SurgT [ 1 ]	Multiple Procedures	Manual, Robotic	Partial	✗	Tissue	✓	✓	✗	157	32	24,548
STIR [ 2 ]	Cholecystectomy, Nissen Fundoplication Surgery	Robotic	Partial	✗	Tissue	✓	✓	✓	576	496	13,393
CholecTrack20 [ 3 ]	Cholecystectomy Surgery	Manual	✓	✗	Instrument	✓	✗	✗	20	2,624	35,000
CRCD [ 4 ]	Ex-vivo pseudo-cholecystectomy	Robotic	✗	✗	Instrument	✓	✗	✗	16	-	127,000
SurgPose [ 5 ]	Instrument Motion over Tissue	Robotic	✗	✗	Instrument	✓	✓	✓	30	476	30,030

TABLE I: Comparison with Surgical Tracking Datasets. Our SurgTAP-3D is the first large-scale, in-vivo , contact-aware , 3D pixel-wise trajectory dataset covering both instrument and tissue in real robotic and manual surgeries. Gray row : ours.

Fig. 2: Overview of the Annotation Framework for SurgTAP-3D. We first preprocess the data clips from the public datasets to collect the action clips, select the first frame, and filter the noisy frames. The annotation pipeline begins with the canonical labeling for the first frame of the clip, localizing the contact region and active tool to obtain the tracking prompts, which are input into TAPNext and SAM2 for inference. Next, our proposed pipeline takes frames and optional priors as input, achieving stable and robust 3D reconstruction through surgical-aware geometry priors from Endo3R [ 25 ] and dynamic bundle adjustment for correcting the drift. Finally, we conduct global 4D refinement to optimize the estimated depths and tracking result, enabling temporally smooth and geometrically coherent 3D contact-aware point tracking.

Type	Datasets	Surgery	Clips	Frames	Avg. Frames
Robotic Surgery	GraSP [ 71 ]	Prostatectomy	795	109939	138
Robotic Surgery	SAR-RARP50 [ 72 ]	Prostatectomy	741	62822	85
Manual Surgery	SLAM [ 73 ]	Multiple procedures	203	24609	121
	SurgicalActions160 [ 74 ]	Gynecology	26	2194	84
	AutoLaparo [ 75 ]	Hysterectomy	43	4684	109
	heicoProctocolectomy [ 76 ]	Proctocolectomy	39	4538	116

TABLE II: Description of SurgTAP-3D. The dataset contains the video datasets from both robotic surgery and manual surgery.

Fig. 3: Data Statistics. We analyze the data distribution of action frames, surgery types, surgical modality, and tool categories. We also visualize the clip lengths distribution in different actions, datasets, and the distribution of our labeled contact region.

Fig. 4: Visualization of 2D and 3D Tracking in different actions. We plot the point trajectories in 2D and 3D, with red-yellow trajectories for the active instrument and purple-blue trajectories for the tissues.

Fig. 5: Correlation between Actions and 3D Trajectories. We randomly sample ten clips per action and normalize their 3D instrument trajectories via spatial alignment. The projected mean trajectories form well-separated clusters, revealing a strong correlation between action categories and instrument motion patterns.

Fig. 6: Correlation between Actions and Instrument Types. We plot the heatmap to visualize the distribution of instrument types in different actions.

Fig. 7: Qualitative Comparison of 2D point tracking. We extensively compare with three types of point tracking methods, with blue points as the ground truth trajectory (from SurgPose [ 5 ] , STIR [ 2 ] , and SurgT [ 1 ] ) and orange points as the predicted trajectory. The results exhibit our better robustness to occlusions, illumination variance, and instrument tracking.

Action Class	SR $\uparrow$	IMC $\uparrow$	TCR $\uparrow$	TSR $\uparrow$
Dissection	91.89	96.89	98.53	93.70
Tissue Retraction	83.80	97.10	98.07	94.20
Needle Grasping	91.46	93.55	96.90	97.51
Vessel Clipping	88.21	97.03	99.45	92.13
Coagulation	86.71	95.35	98.72	92.88
Average	88.41	95.98	98.33	94.08

TABLE III: Annotation quality analysis across action classes. We report SR, IMC, TCR, and TSR in % as indicators of track survival, instrument-mask consistency, tracking confidence, and temporal smoothness, respectively.

Fig. 8: Challenging-scene analysis of Annotation Quality. Green box shows robust cases, including short-term instrument disappearance ( $<$ 10s), smoke, and bleeding. Red box shows failure-prone cases, including long-term instrument disappearance ( $>$ 10s), large occlusion, and fast instrument motion, where trajectories may drift or become unstable.

2D Tracking
Type	Method	SurgT [ 1 ]			SurgPose [ 5 ]			STIR [ 2 ]			Average
Type	Method	AJ 2D $\uparrow$	APD 2D $\uparrow$	OA $\uparrow$	AJ 2D $\uparrow$	APD 2D $\uparrow$	OA $\uparrow$	AJ 2D $\uparrow$	APD 2D $\uparrow$	OA $\uparrow$	AJ 2D $\uparrow$	APD 2D $\uparrow$	OA $\uparrow$
I	BootsTAPIR [ 86 ]	67.85	73.42	99.10	47.28	57.79	91.65	12.33	16.47	96.74	42.49	49.23	95.83
	CoTrackerv3 [ 41 ]	73.32	79.02	98.75	49.40	61.84	83.89	19.34	24.23	96.51	47.35	55.03	93.05
	AllTracker [ 87 ]	68.11	75.30	96.59	53.59	64.19	97.93	17.87	22.57	99.48	46.52	54.02	98.00
	TAPNext [ 43 ]	74.98	80.52	98.18	60.35	69.78	96.40	18.89	23.46	95.46	51.41	57.92	96.68

TABLE IV: Comprehensive 3D and 2D surgical motion tracking Benchmark. We extensively evaluate the 2D and 3D point tracking for both instrument (SurgPose [ 5 ] ) and tissue surface (SurgT [ 1 ] and STIR [ 2 ] ).

Method	ATE $\downarrow$	RPE r $\downarrow$	RPE t $\downarrow$
Endo-SfM [ 89 ]	0.157	0.252	0.259
AF-SfM [ 90 ]	0.125	0.235	0.241
EndoDAC [ 91 ]	0.124	0.223	0.233
Robust [ 92 ]	0.131	0.241	0.245
Endo3R [ 25 ]	0.112	0.201	0.228
SpatialTrackerV2 [ 28 ]	0.110	0.217	0.239

TABLE V: Pose Comparison on SCARED [ 37 ] .

Method	SCARED [ 37 ]					Hamlyn [ 93 , 94 , 95 ]
Method	Abs Rel $\downarrow$	Sq Rel $\downarrow$	RMSE $\downarrow$	RMSE log $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	Sq Rel $\downarrow$	RMSE $\downarrow$	RMSE log $\downarrow$	$\delta<1.25$ $\uparrow$
Monodepth2 [ 96 ]	0.432	3.548	4.704	0.431	0.425	0.379	9.318	20.472	0.403	0.439
Endo-SfM [ 89 ]	0.241	0.865	2.286	0.267	0.585	0.252	4.335	14.430	0.268	0.628
AF-SfM [ 90 ]	0.257	0.960	2.162	0.291	0.573	0.286	5.715	15.895	0.301	0.508
EndoDAC [ 91 ]	0.242	0.934	2.014	0.275	0.584	0.275	5.557	15.669	0.288	0.519
Transfer [ 97 ]	0.297	1.207	2.561	0.319	0.561	0.281	5.790	15.936	0.312	0.504

TABLE VI: Quantitative comparison with state-of-the-art video depth estimation methods .

Table 14

Fig. 9: Real-world Application for Instrument Displacement Estimation. Given the predicted 3D trajectory, the instrument displacement is estimated by computing the centroids of the instrument trajectories at each timestep. To validate the effectiveness, we collect 50 sequences from dVRK with paired video clips and kinematic data. It shows that our framework enables accurate and robust instrument displacement estimation.

Fig. 10: Real-world Application for Deformable Tissue Reconstruction and Rendering. Given the reconstructed results, we train the 4D Gaussian Splatting model for deformable surface reconstruction and novel view synthesis for high-fidelity rendering. Compared with 4DGS with vanilla COLMAP initialization, our reconstructed point clouds, poses, and intrinsics yield significant improvement in reconstruction results.

Method	ATE $\downarrow$	RPE ${}_{t}\downarrow$
LiteTracker [ 46 ]	0.0035	0.0015
DELTAv2 [ 88 ]	0.0040	0.0022
SpatialTrackerV2 [ 28 ]	0.0064	0.0008
Ours	0.0022	0.0005

TABLE IX: Quantitative results for instrument displacement estimation on the 50 sequences of self-collected paired video–kinematic dVRK dataset (unit: m).

Init. + Sup.	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
COLMAP + DAv2	27.82	0.8410	0.2161
COLMAP + Ours	27.51	0.8359	0.2196
Ours	28.16	0.8548	0.1310

TABLE X: Quantitative Comparison for Novel View Synthesis on SurgTAP-3D dataset. “Init.” refers to initialization. “Sup.” refers to depth supervision.

Settings	SurgT [ 1 ]		SurgPose [ 5 ]
Settings	AJ ${}_{2D}\uparrow$	APD ${}_{2D}\uparrow$	AJ ${}_{2D}\uparrow$	APD ${}_{2D}\uparrow$
CoTrackerv3 [ 41 ]	73.32	79.02	49.40	61.84
CoTrackerv3 [ 41 ] w/ ft.	74.14	80.25	50.99	62.53
DELTAv2 [ 88 ]	73.75	79.21	51.20	61.97
DELTAv2 [ 88 ] w/ ft.	74.57	80.11	52.75	62.86

TABLE XI: Quantitative Results for Finetuning CoTrackerV3 [ 41 ] and DELTAv2 [ 88 ] on our dataset.

3D Point Tracking

3D point tracking estimates how scene points move through space and time, even under camera motion, occlusion, and deformation.

See all 33 papers

SurgFlow: 3D Object-Centric Contact Flow for Surgical Robot Manipulation
Sep 27, 2026Changwei Chen, Xiao Liang, Yinuo Yang +6Robotic Manipulation Surgical Robotics
Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093, USA
S3-Tracker: Self-Supervised Surgical Tissue Tracking With Contrastive Random Walks
Sep 13, 2026Jiaming Zhang, Zijian Wu, Mehran Armand +1Point Tracking Surgical Video Understanding
Department of Computer Science, Johns Hopkins University · Robotics and Control Laboratory, The University of British Columbia · The Institute for Integrative and Innovative Research, University of Arkansas
SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation
Mar 7, 2026Kaiyuan Xu, Fangzhou Hong, Daniel Elson +13D Reconstruction Endoscopic 3D Reconstruction
The Hamlyn Centre for Robotic Surgery, Imperial College London, SW7 2AZ, UK. · S-Lab, College of Computing and Data Science, Nanyang Technological University, Singapore 639798. · Department of Computer Science, University of Liverpool, L69 7ZX, UK.

Sep 27, 2026J/K move · Enter open · S save

SurgFlow: 3D Object-Centric Contact Flow for Surgical Robot Manipulation

Changwei Chen, Xiao Liang, Yinuo Yang, Nicole Shen+5

Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093, USA

Robotic Manipulation Surgical Robotics Contact-Rich Robotic Manipulation Zero-Shot Robotic Manipulation

Open paper AskPDF arXiv

Paired video-action demonstrations enable autonomous surgical behavior, but such data is scarce: robots perform roughly 1% of surgeries, while video-only data is abundant. Learning 3D object flow offers an embodiment-agnostic way to utilize video data, but flow alone specifies how an object should move, not where and when the tool should engage it, a distinction that is critical in surgery. We introduce SurgFlow, a framework that learns 3D Object-Centric Contact Flow from stereo surgical video without action labels. For each object point, it predicts a future 3D trajectory and contact scores. We extract targets via 3D tracking and tool-object proximity, train a flow matching generator to predict them, and use predicted contact to trigger grasp and release while optimizing end effector motion from flow. On the da Vinci Research Kit (dVRK), SurgFlow succeeds in 37 of 39 stage evaluations across tissue retraction, bimanual reveal, needle pickup, and handover, outperforming baselines trained on equal data with or without action labels. Zero-shot transfer to a humanoid-based laparoscopic robot achieves 85% and 70% average success under similar and novel camera viewpoints, respectively.

Abstract

3D Point Tracking

Explore similar work

SurgFlow: 3D Object-Centric Contact Flow for Surgical Robot Manipulation

S3-Tracker: Self-Supervised Surgical Tissue Tracking With Contrastive Random Walks

SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation