Surgical point tracking requires persistent correspondence through tissue deformation and occlusion, yet dense trajectory annotations are costly to obtain. We present STEP, a two-stage framework that learns from surgical video and sparse physical endpoints. First, a frozen offline teacher generates candidate trajectories, and temporal reversibility and stereo consistency select dense supervision for a causal student. Second, a robust physical endpoint loss anchors terminal correspondence, while a round-trip loss constrains temporal consistency. A gradient-free prefix exposes training to accumulated drift, while a final differentiable window bounds activation memory. The trained tracker processes a single RGB stream with fixed weights. In STIR-to-VL-SurgPT transfer, temporal-stereo union selection reduces endpoint error by 7.0% relative to unfiltered trajectories. On a post-hoc diagnostic of 29 VL-SurgPT clips excluded from the endpoint-training manifest, STEP reduces endpoint error from 31.52 to 16.80 pixels relative to dense-label fine-tuning. These results support combining curated trajectories with physical observations to train surgical point trackers.
Figures & tables
Figure 1: Scaling supervision for surgical point tracking. Manual trajectories require repeated annotation. STEP combines curated trajectories and existing physical endpoints to train a causal surgical tracker.
Figure 2: Temporal and stereo consistency selection. STIR candidates accepted by temporal checks only (blue), stereo checks only (red), or both (green). Stage 1 retains their union on stereo sequences.
Figure 3: STEP framework. (1) Temporal and stereo checks select training trajectories. (2) Endpoint-guided training combines physical endpoint and round-trip losses, with gradients limited to a final window.
Backbone
Supervision
δavg↑
EPE ↓ (px)
RAFT ( Teed and Deng, 2020 )
none
33.81
82.16
SEA-RAFT ( Wang et al., 2024 )
none
33.71
80.17
TAPIR ( Doersch et al., 2023 )
none
64.80
15.64
Track-On ( Aydemir et al., 2025 )
none
67.36
11.79
manual trajectories
73.92
11.05
STEP (Ours)
75.89 (+1.97)
10.70 ( − 0.35)
Table 1: VL-SurgPT physical-endpoint comparison. STEP is compared with released models and manual-trajectory fine-tuning across multiple tracking backbones.
Setting
Working res.
Grid
δ60↑
Track-On-R
384×512
none
75.89
Track-On-R
384×512
8×8
76.53
Track-On-R
384×512
16×16
77.54
Track-On-R
480×608
none
77.86
Track-On-R
480×608
8×8
78.10
Track-On-R
480×608
16×16
78.35
Table 2: STIR resolution and support-grid controls. STEP is compared with Track-On-R under matched inference settings.
Table 6
Figure 4: Qualitative endpoint tracking. Predictions and physical references (crosses) on four VL-SurgPT sequences; errors are in pixels. The last three columns use the Track-On-R backbone.
Occluded mid-sequence ( n=145 )
Never occluded ( n=230 )
Visibility
Model
δ↑
EPE ↓
> 100 px ↓
δ↑
EPE ↓
> 100 px ↓
Occ. recall ↑
False occ. ↓
Zero-shot
64.14
28.54
8.97
77.39
6.91
1.30
47.9
2.0
Dense-label FT
64.28 (+0.14)
25.92
9.66
78.78 (+1.39)
7.95
1.30
49.3
1.7
Stage 1 only
64.55 (+0.41)
28.51
8.28
77.74 (+0.35)
6.53
0.87
49.9
1.9
Stage 2 only
66.76 (+2.62)
24.15
7.59
78.17 (+0.78)
7.29
1.30
50.1
1.9
STEP (ours)
66.76 (+2.62)
20.83
7.59
78.70 (+1.31)
6.67
1.30
51.6
2.1
Table 5: Occlusion recovery and visibility. Full-pool endpoints grouped by intermediate visibility. EPE is in pixels; other metrics are percentages.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
VL-SurgPT
STIR
Video streams
754
1,132
Frames
181 k
640 k
Acquisition
Monocular
Stereo
Human-annotated trajectories
1,862
0
Generated pseudo-trajectories
493,893
1,685,211
Pseudo-labeled point-frames
103 M
300 M
Appendix
Table 6: Surgical sources and generated trajectory inventory. STIR counts both views of 566 stereo clips. Pseudo-labels are model-generated; physical labels are existing endpoint observations.
Losses
Endpoint clips
Cycle clips
Max. span
Grad. tail
Endpoint only
2
0
≤192
16
Endpoint + cycle
5
1
≤256
24
Appendix
Table 7: Track-On-R endpoint-guided training recipes. Clips accumulate per optimizer step; all S2 runs use 300 steps. The cycle loss uses a separate 96-frame clip and 12-frame gradient tail.
Prediction
2 mm
4 mm
8 mm
16 mm
32 mm
Avg ↑
Static control (GT start cloud)
15.85
29.55
49.68
71.09
91.22
51.48
STEP
37.37
62.83
81.21
93.13
98.99
74.71
Appendix
Table 8: STIR 3D endpoint accuracy. Official STIR 2024 3D protocol, 59 clips and 495 endpoints, nearest-neighbour matched to the ground-truth end cloud. Higher is better.
Figure 5: Motion-field coherence without ground truth. Track-On-R variants on four VL-SurgPT clips. Red cells mark orientation flips and panels report their rates; compare within columns. Coherent tracks can still drift.
Figure 6: Additional correspondence examples. Predictions at selected frames illustrate tracking through the sequence; numbers give errors in pixels.
Point tracking follows selected or dense image points through video, enabling motion analysis, 3D reconstruction, and precise spatiotemporal understanding.
Department of Computer Science, Johns Hopkins University · Robotics and Control Laboratory, The University of British Columbia · The Institute for Integrative and Innovative Research, University of Arkansas
Department of Computer Science and UCL Hawkes Institute, University College London, London WC1E 6BT, UK · Intuitive Surgical, Inc., Sunnyvale, CA 94086, USA