Organizations: Department of Computer Science, Johns Hopkins University · Robotics and Control Laboratory, The University of British Columbia · The Institute for Integrative and Innovative Research, University of Arkansas
Robust point tracking in endoscopic videos is essential for computer-assisted intervention and autonomous robotic surgery, enabling continuous registration between intraoperative video and preoperative imaging despite soft tissue deformation. However, supervised tracking methods depend on large annotated datasets, while surgical conditions make reliable trajectory annotation challenging. We propose a self-supervised Track-Any-Point approach that learns from unlabeled surgical videos by establishing global pixel correspondences and inferring point trajectories through contrastive random walks. Trained without annotations, our method achieves performance comparable to existing semi-supervised approaches while implicitly handling tissue deformation. These findings demonstrate the feasibility of self-supervised point tracking in surgical environments and its potential to reduce reliance on annotated data.
Figures & tables
Figure 1: Model architecture. For any successive frames, we extract the feature vectors using a ResNet backbone and perform self- and cross-attention over the feature vectors to get the affinity matrix. We calculate the affinity matrices in both forward and backward directions and minimize the consistency loss between them.
Method
δavg↑
EPE (px) ↓
Latency (ms) ↓
Semi-supervised Methods
Endo-TTAP [ 3 ]
0.784
13.32
–
CoTracker
0.611
34.66
–
CoTracker3 [ 1 ]
0.753
16.80
–
Unsupervised Methods
MFT(2024) [ 6 ]
0.776
18.32
417.83
Table 1: Quantitative comparison of semi-supervised and unsupervised methods across four evaluation metrics.
Figure 2: (a) Point tracking results on challenging videos. The tracked points are marked as green circles. (b) The result of the visibility module determines the visibility of a large number of points. 30 × 30 points are initially sampled and are tracked by CoTracker3.