Surgical point tracking requires persistent correspondence through tissue deformation and occlusion, yet dense trajectory annotations are costly to obtain. We present STEP, a two-stage framework that learns from surgical video and sparse physical endpoints. First, a frozen offline teacher generates candidate trajectories, and temporal reversibility and stereo consistency select dense supervision for a causal student. Second, a robust physical endpoint loss anchors terminal correspondence, while a round-trip loss constrains temporal consistency. A gradient-free prefix exposes training to accumulated drift, while a final differentiable window bounds activation memory. The trained tracker processes a single RGB stream with fixed weights. In STIR-to-VL-SurgPT transfer, temporal-stereo union selection reduces endpoint error by 7.0% relative to unfiltered trajectories. On a post-hoc diagnostic of 29 VL-SurgPT clips excluded from the endpoint-training manifest, STEP reduces endpoint error from 31.52 to 16.80 pixels relative to dense-label fine-tuning. These results support combining curated trajectories with physical observations to train surgical point trackers.
Figures & tables
Figure 1: Scaling supervision for surgical point tracking. Manual trajectories require repeated annotation. STEP combines curated trajectories and existing physical endpoints to train a causal surgical tracker.
Figure 2: Temporal and stereo consistency selection. STIR candidates accepted by temporal checks only (blue), stereo checks only (red), or both (green). Stage 1 retains their union on stereo sequences.
Figure 3: STEP framework. (1) Temporal and stereo checks select training trajectories. (2) Endpoint-guided training combines physical endpoint and round-trip losses, with gradients limited to a final window.
Backbone
Supervision
δavg↑
EPE ↓ (px)
RAFT ( Teed and Deng, 2020 )
none
33.81
82.16
SEA-RAFT ( Wang et al., 2024 )
none
33.71
80.17
TAPIR ( Doersch et al., 2023 )
none
64.80
15.64
Track-On ( Aydemir et al., 2025 )
none
67.36
11.79
manual trajectories
73.92
11.05
STEP (Ours)
75.89 (+1.97)
10.70 ( − 0.35)
Table 1: VL-SurgPT physical-endpoint comparison. STEP is compared with released models and manual-trajectory fine-tuning across multiple tracking backbones.
Setting
Working res.
Grid
δ60↑
Track-On-R
384×512
none
75.89
Track-On-R
384×512
8×8
76.53
Track-On-R
384×512
16×16
77.54
Track-On-R
480×608
none
77.86
Track-On-R
480×608
8×8
78.10
Track-On-R
480×608
16×16
78.35
Table 2: STIR resolution and support-grid controls. STEP is compared with Track-On-R under matched inference settings.
Table 6
Figure 4: Qualitative endpoint tracking. Predictions and physical references (crosses) on four VL-SurgPT sequences; errors are in pixels. The last three columns use the Track-On-R backbone.
Occluded mid-sequence ( n=145 )
Never occluded ( n=230 )
Visibility
Model
δ↑
EPE ↓
> 100 px ↓
δ↑
EPE ↓
> 100 px ↓
Occ. recall ↑
False occ. ↓
Zero-shot
64.14
28.54
8.97
77.39
6.91
1.30
47.9
2.0
Dense-label FT
64.28 (+0.14)
25.92
9.66
78.78 (+1.39)
7.95
1.30
49.3
1.7
Stage 1 only
64.55 (+0.41)
28.51
8.28
77.74 (+0.35)
6.53
0.87
49.9
1.9
Stage 2 only
66.76 (+2.62)
24.15
7.59
78.17 (+0.78)
7.29
1.30
50.1
1.9
STEP (ours)
66.76 (+2.62)
20.83
7.59
78.70 (+1.31)
6.67
1.30
51.6
2.1
Table 5: Occlusion recovery and visibility. Full-pool endpoints grouped by intermediate visibility. EPE is in pixels; other metrics are percentages.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
VL-SurgPT
STIR
Video streams
754
1,132
Frames
181 k
640 k
Acquisition
Monocular
Stereo
Human-annotated trajectories
1,862
0
Generated pseudo-trajectories
493,893
1,685,211
Pseudo-labeled point-frames
103 M
300 M
Appendix
Table 6: Surgical sources and generated trajectory inventory. STIR counts both views of 566 stereo clips. Pseudo-labels are model-generated; physical labels are existing endpoint observations.
Losses
Endpoint clips
Cycle clips
Max. span
Grad. tail
Endpoint only
2
0
≤192
16
Endpoint + cycle
5
1
≤256
24
Appendix
Table 7: Track-On-R endpoint-guided training recipes. Clips accumulate per optimizer step; all S2 runs use 300 steps. The cycle loss uses a separate 96-frame clip and 12-frame gradient tail.
Prediction
2 mm
4 mm
8 mm
16 mm
32 mm
Avg ↑
Static control (GT start cloud)
15.85
29.55
49.68
71.09
91.22
51.48
STEP
37.37
62.83
81.21
93.13
98.99
74.71
Appendix
Table 8: STIR 3D endpoint accuracy. Official STIR 2024 3D protocol, 59 clips and 495 endpoints, nearest-neighbour matched to the ground-truth end cloud. Higher is better.
Figure 5: Motion-field coherence without ground truth. Track-On-R variants on four VL-SurgPT clips. Red cells mark orientation flips and panels report their rates; compare within columns. Coherent tracks can still drift.
Figure 6: Additional correspondence examples. Predictions at selected frames illustrate tracking through the sequence; numbers give errors in pixels.
Point tracking follows selected or dense image points through video, enabling motion analysis, 3D reconstruction, and precise spatiotemporal understanding.
Robust point tracking in endoscopic videos is essential for computer-assisted intervention and autonomous robotic surgery, enabling continuous registration between intraoperative video and preoperative imaging despite soft tissue deformation. However, supervised tracking methods depend on large annotated datasets, while surgical conditions make reliable trajectory annotation challenging. We propose a self-supervised Track-Any-Point approach that learns from unlabeled surgical videos by establishing global pixel correspondences and inferring point trajectories through contrastive random walks. Trained without annotations, our method achieves performance comparable to existing semi-supervised approaches while implicitly handling tissue deformation. These findings demonstrate the feasibility of self-supervised point tracking in surgical environments and its potential to reduce reliance on annotated data.
Jiaming Zhang, Zijian Wu, Mehran Armand +1
Department of Computer Science, Johns Hopkins University · Robotics and Control Laboratory, The University of British Columbia · The Institute for Integrative and Innovative Research, University of Arkansas
Point tracking in surgery is crucial to enable applications in downstream tasks such as segmentation, 3D reconstruction, virtual tissue landmarking, autonomous probe-based scanning, and subtask autonomy. This paper introduces the 2025 iteration of a point tracking challenge to address this, wherein participants submit their algorithms for quantification. Their algorithms are evaluated using a dataset named surgical tattoos in infrared (STIR), with the challenge named the STIR Challenge 2025 (STIRC2025). The STIR Challenge 2025 comprises two quantitative components: accuracy and efficiency. The accuracy component tests the accuracy of algorithms on in vivo and ex vivo sequences. The efficiency component tests algorithm inference latency. The challenge was conducted as a part of MICCAI EndoVis 2025, and seven teams participated in this challenge. In this paper we summarize the challenge results and participant methods. The challenge dataset is available at: https://zenodo.org/records/20191078, and the code for baseline models and metrics calculation is available here: https://github.com/athaddius/STIRMetrics
Adam Schmidt, Mert Asim Karaoglu, Zijian Wu +32
Intuitive Surgical Inc., Sunnyvale, USA · ImFusion GmbH, Munich, Germany · Technical University of Munich, Munich, Germany +14
Gaussian splatting is the current state-of-the-art for dense, deformable 3D anatomy reconstruction in robot-assisted minimally invasive surgery (RAMIS); however, most pipelines are offline and depend on accurate camera trajectory priors (often from robotic kinematics), limiting applicability when priors are missing or noisy. To address these limitations, we propose Track2Map, an online 3D Gaussian Splatting pipeline that jointly optimizes camera trajectory and 3D deformable scene representation directly from surgical video. Track2Map is therefore capable of robust 3D reconstructions when camera trajectory priors are either absent or noisy, and due to its online nature it effectively works as a Simultaneous Localisation and Mapping (SLAM) method. To stabilize optimization in the presence of tissue motion and ambiguous visual cues, we introduce a track-anchored deformation initialization using dense 2D point tracks. Track statistics are further utilized to disentangle camera motion from scene deformation by detecting static camera periods and reducing drift during incremental mapping. Experiments on StereoMIS show improved reconstruction quality and camera trajectory against competing SLAM methods, as well as compared to non-SLAM methods that utilize camera trajectory priors. The code is available at https://track2map.github.io/.
Tianyi Song, Sierra Bonilla, Xinwei Ju +6
Department of Computer Science and UCL Hawkes Institute, University College London, London WC1E 6BT, UK · Intuitive Surgical, Inc., Sunnyvale, CA 94086, USA