Organizations: Department of Computer Science, Johns Hopkins University · Robotics and Control Laboratory, The University of British Columbia · The Institute for Integrative and Innovative Research, University of Arkansas
Robust point tracking in endoscopic videos is essential for computer-assisted intervention and autonomous robotic surgery, enabling continuous registration between intraoperative video and preoperative imaging despite soft tissue deformation. However, supervised tracking methods depend on large annotated datasets, while surgical conditions make reliable trajectory annotation challenging. We propose a self-supervised Track-Any-Point approach that learns from unlabeled surgical videos by establishing global pixel correspondences and inferring point trajectories through contrastive random walks. Trained without annotations, our method achieves performance comparable to existing semi-supervised approaches while implicitly handling tissue deformation. These findings demonstrate the feasibility of self-supervised point tracking in surgical environments and its potential to reduce reliance on annotated data.
Figures & tables
Figure 1: Model architecture. For any successive frames, we extract the feature vectors using a ResNet backbone and perform self- and cross-attention over the feature vectors to get the affinity matrix. We calculate the affinity matrices in both forward and backward directions and minimize the consistency loss between them.
Method
δavg↑
EPE (px) ↓
Latency (ms) ↓
Semi-supervised Methods
Endo-TTAP [ 3 ]
0.784
13.32
–
CoTracker
0.611
34.66
–
CoTracker3 [ 1 ]
0.753
16.80
–
Unsupervised Methods
MFT(2024) [ 6 ]
0.776
18.32
417.83
Table 1: Quantitative comparison of semi-supervised and unsupervised methods across four evaluation metrics.
Figure 2: (a) Point tracking results on challenging videos. The tracked points are marked as green circles. (b) The result of the visibility module determines the visibility of a large number of points. 30 × 30 points are initially sampled and are tracked by CoTracker3.
Efficient and robust tracking of surgical robotic instruments is important for robot-assisted minimally invasive surgery, yet remains challenging due to the complexity of surgical scenes and the unconventional geometry of surgical instruments. Keypoint-based approaches are efficient, but their performance depends on reliable feature detection. Improving these detectors with real-world supervision is difficult because accurate real-world annotations are costly to obtain at scale. To address this limitation, we introduce a tracker-guided self-training framework that adapts a model pretrained on synthetic images to unlabeled real-world videos. Given measured robot joint states, an uncertainty-aware EKF recursively corrects the instrument pose and the observable joint angles by comparing projected model features with detected keypoints, shaft boundaries, and mask-derived cues. An RTS smoother subsequently refines the resulting trajectory, which is projected into pseudo-labels for fine-tuning the feature detector without laborious pose annotations. Experiments on real-world videos demonstrate consistent improvements from self-training across all evaluated keypoint metrics, and the resulting model outperforms prior approaches in both accuracy and runtime. The code and data will be released upon publication.
Hanyang Hu, Zekai Liang, Florian Richter +1
Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093 USA.
Point tracking in surgery is crucial to enable applications in downstream tasks such as segmentation, 3D reconstruction, virtual tissue landmarking, autonomous probe-based scanning, and subtask autonomy. This paper introduces the 2025 iteration of a point tracking challenge to address this, wherein participants submit their algorithms for quantification. Their algorithms are evaluated using a dataset named surgical tattoos in infrared (STIR), with the challenge named the STIR Challenge 2025 (STIRC2025). The STIR Challenge 2025 comprises two quantitative components: accuracy and efficiency. The accuracy component tests the accuracy of algorithms on in vivo and ex vivo sequences. The efficiency component tests algorithm inference latency. The challenge was conducted as a part of MICCAI EndoVis 2025, and seven teams participated in this challenge. In this paper we summarize the challenge results and participant methods. The challenge dataset is available at: https://zenodo.org/records/20191078, and the code for baseline models and metrics calculation is available here: https://github.com/athaddius/STIRMetrics
Adam Schmidt, Mert Asim Karaoglu, Zijian Wu +32
Intuitive Surgical Inc., Sunnyvale, USA · ImFusion GmbH, Munich, Germany · Technical University of Munich, Munich, Germany +14
Purpose: In this paper, we present a novel approach for online object tracking in laparoscopic cholecystectomy (LC) surgical videos, targeting localisation and tracking of critical anatomical structures and instruments. Our method addresses the challenges of costly pixel-level annotations and label inconsistencies inherent in existing datasets. Methods: Leveraging the inherent object localisation capabilities of pre-trained text-to-image diffusion models, we extract representative features from surgical frames without any training or fine-tuning. Our tracking framework uses these features, along with cross-frame interactions via an affinity matrix inspired by query-key-value attention, to ensure temporal continuity in the tracking process. Results: Through a pilot study, we first demonstrate that diffusion features exhibit superior object localisation and consistent semantics across different decoder levels and temporal frames. Later, we perform extensive experiments to validate the effectiveness of our approach, showcasing its superiority over competitors for the task of temporal object tracking. Specifically, we achieve a per-pixel classification accuracy of 79.19%, mean Jaccard Score of 56.20%, and mean F-Score of 79.48% on the publicly available CholeSeg8K dataset. Conclusion: Our work not only introduces a novel application of text-to-image diffusion models but also contributes to advancing the field of surgical video analysis, offering a promising avenue for accurate and cost-effective temporal object tracking in minimally invasive surgery videos.
Subhadeep Koley, Abdolrahim Kadkhodamohammadi, Santiago Barbarisi +2
Medtronic plc., London, UK. · UCL Hawkes Institute, University College London, London, UK.