Training-free Temporal Object Tracking in Surgical Videos

Subhadeep Koley; Abdolrahim Kadkhodamohammadi; Santiago Barbarisi; Danail Stoyanov; Imanol Luengo

cs.CVMar 8, 2026

Training-free Temporal Object Tracking in Surgical Videos

Authors: Subhadeep Koley, Abdolrahim Kadkhodamohammadi, Santiago Barbarisi, Danail Stoyanov, Imanol Luengo

Organizations: Medtronic plc., London, UK. · UCL Hawkes Institute, University College London, London, UK.

Abstract

Purpose: In this paper, we present a novel approach for online object tracking in laparoscopic cholecystectomy (LC) surgical videos, targeting localisation and tracking of critical anatomical structures and instruments. Our method addresses the challenges of costly pixel-level annotations and label inconsistencies inherent in existing datasets. Methods: Leveraging the inherent object localisation capabilities of pre-trained text-to-image diffusion models, we extract representative features from surgical frames without any training or fine-tuning. Our tracking framework uses these features, along with cross-frame interactions via an affinity matrix inspired by query-key-value attention, to ensure temporal continuity in the tracking process. Results: Through a pilot study, we first demonstrate that diffusion features exhibit superior object localisation and consistent semantics across different decoder levels and temporal frames. Later, we perform extensive experiments to validate the effectiveness of our approach, showcasing its superiority over competitors for the task of temporal object tracking. Specifically, we achieve a per-pixel classification accuracy of 79.19%, mean Jaccard Score of 56.20%, and mean F-Score of 79.48% on the publicly available CholeSeg8K dataset. Conclusion: Our work not only introduces a novel application of text-to-image diffusion models but also contributes to advancing the field of surgical video analysis, offering a promising avenue for accurate and cost-effective temporal object tracking in minimally invasive surgery videos.

Figures & tables

Figure 1: Our method uses pre-trained text-to-image diffusion model Rombach et al. (2022) to extract features from surgical frames (Sec. 4.2 ). Our tracking module (Sec. 4.3 ) uses these features with cross-frame interactions via an affinity matrix, to predict masks in the temporal direction for the entire clip.

Figure 2: PCA rendering of SD’s $\mathcal{U}_{\mathbf{u}}^{n}$ features from different levels ( $n=$ $\{1,2,3,4\}$ ) and time-frames. Notably, same anatomy/instrument is represented by same colour across temporal direction.

Figure 3: Proposed diffusion feature extraction pipeline. Given a surgical frame $\mathbf{x}_{0}$ , it is passed through the encoder $\mathcal{E}(\cdot)$ to generate a latent representation $\mathbf{z}_{0}$ , followed by forward diffusion to produce a noisy latent $\mathbf{z}_{t}$ . The null prompt $\mathcal{T}(\mathbf{c})$ and noisy latent representation $\mathbf{z}_{t}$ are then passed through the UNet $\mathcal{U}(\cdot)$ to extract features from its internal decoders (Sec. 4.2 ).

Methods	Backbone	$\mathcal{P}_{Acc.}$	$\mathcal{J}_{m}$	$\mathcal{F}_{m}$
B-MaskCLIP	ViT-B/16	69.10	39.26	69.43
B-ResNet	ResNet50	54.08	27.83	54.29
B-CLIP	ViT-B/16	64.91	31.68	65.29
B-ViT	ViT-B/16	71.85	41.58	72.14
B-DINO	ViT-S/8	77.08	52.14	77.37
B-DINO	ViT-S/16	74.24	46.32	74.54

Table 1: Quantitative comparison on CholecSeg8K Hong et al. (2020) .

{\dagger}

denotes supervised methods.

Figure 4: Qualitative comparison on CholecSeg8K Hong et al. (2020) . Notably, unlike the baseline competitors, the proposed method precisely tracks both smaller and larger anatomies across the temporal direction.

Figure 5: Qualitative comparison on CholecSeg8K Hong et al. (2020) . Noticeably, the proposed method surpasses competitors, maintaining accuracy even during rapid temporal movements of instruments and anatomis.

Methods	EndoVis-2015 Bodenstedt et al. (2018)		DAVIS-2017 Pont-Tuset et al. (2017)
Methods	$\mathcal{J}_{m}$	$\mathcal{F}_{m}$	$\mathcal{J}_{m}$	$\mathcal{F}_{m}$
B-ResNet	56.61	73.05	40.25	44.92
B-CLIP	63.32	83.34	45.82	61.23
B-DINO	78.36	92.09	63.01	68.17
B-DINOv2	75.16	90.62	57.73	64.03
B-MAE	69.23	88.74	53.49	62.81

Table 2: Quantitative comparison on EndoVis-2015 Bodenstedt et al. (2018) and DAVIS-2017 Pont-Tuset et al. (2017) .

Figure 6: Qualitative comparison on EndoVis-2015 Bodenstedt et al. (2018) . Unlike baselines, the proposed method can track different parts ( e.g. , $\blacksquare$ : Tip, $\blacksquare$ : Shaft) of the same instrument effectively.

Figure 7:

\mathcal{P}_{Acc.}

and

\mathcal{J}_{m}

comparison on CholecSeg8K Hong et al. (2020) for different – (a) diffusion timestep (

t

), (b) UNet Ronneberger et al. (2015) decoder

\mathcal{U}_{\mathbf{u}}^{n}

level, (c) Number of previous frames considered for mask prediction, and (d)

\mathtt{SpatialMask}(\cdot)

window size (

n

SD Versions	$\mathcal{P}_{Acc.}$	$\mathcal{J}_{m}$	$\mathcal{F}_{m}$
v1.4	79.02	55.37	79.32
v1.5	79.06	55.46	79.35
v2.0	79.05	55.78	79.34
Ours (v2.1)	79.19	56.20	79.48

Table 3: Ablating different SD Rombach et al. (2022) versions.

Explore similar work

S3-Tracker: Self-Supervised Surgical Tissue Tracking With Contrastive Random Walks
Sep 13, 2026Jiaming Zhang, Zijian Wu, Mehran Armand +1Point Tracking Surgical Video Understanding
Department of Computer Science, Johns Hopkins University · Robotics and Control Laboratory, The University of British Columbia · The Institute for Integrative and Innovative Research, University of Arkansas
Active Learning for Efficient Annotation of Surgical Videos with Weak Supervision
Jul 14, 2026Manasa Dendukuri, Matjaz Jogan, Daniel A. Hashimoto +1Video Object Segmentation Human-in-the-Loop Annotation
1*GRASP Laboratory, University of Pennsylvania, 3330 Walnut Street, Philadelphia, PA, 19104, USA. · 2PCASO Laboratory, Dept. of Surgery, University of Pennsylvania, 3400 Spruce Street, Philadelphia, PA, 19104, USA. · 3Dept. of Computer and Information Science, University of Pennsylvania, 3330 Walnut Street, Philadelphia, PA, 19104, USA.
Multimodal Optimal Transport for Training-free Temporal Segmentation in Surgical Robotics
Feb 27, 2026Omar Mohamed, Edoardo Fazzari, Ayah Al-Naji +5Temporal Action Segmentation Surgical Video Understanding
Authos are with the Dept. of Robotics, Mohamed bin Zayed University of AI, Abu Dhabi, UAE.

Sep 13, 2026J/K move · Enter open · S save

S3-Tracker: Self-Supervised Surgical Tissue Tracking With Contrastive Random Walks

Jiaming Zhang, Zijian Wu, Mehran Armand, Septimiu Salcudean

Department of Computer Science, Johns Hopkins University · Robotics and Control Laboratory, The University of British Columbia · The Institute for Integrative and Innovative Research, University of Arkansas

Point Tracking Surgical Video Understanding Self-Supervised Learning

Open paper AskPDF arXiv

Robust point tracking in endoscopic videos is essential for computer-assisted intervention and autonomous robotic surgery, enabling continuous registration between intraoperative video and preoperative imaging despite soft tissue deformation. However, supervised tracking methods depend on large annotated datasets, while surgical conditions make reliable trajectory annotation challenging. We propose a self-supervised Track-Any-Point approach that learns from unlabeled surgical videos by establishing global pixel correspondences and inferring point trajectories through contrastive random walks. Trained without annotations, our method achieves performance comparable to existing semi-supervised approaches while implicitly handling tissue deformation. These findings demonstrate the feasibility of self-supervised point tracking in surgical environments and its potential to reduce reliance on annotated data.

Abstract

Explore similar work

S3-Tracker: Self-Supervised Surgical Tissue Tracking With Contrastive Random Walks

Active Learning for Efficient Annotation of Surgical Videos with Weak Supervision

Multimodal Optimal Transport for Training-free Temporal Segmentation in Surgical Robotics