Event-Level Detection of Surgical Instrument Handovers in Videos with Interpretable Vision Models
Authors: Katerina Katsarou, George Zountsas, Karam Tomotaki-Dawoud, Alexander Ehrenhoefer, Paul Chojecki, David Przewozny, Igor Maximilian Sauer, Amira Mouakher, Sebastian Bosse
Organizations: Fraunhofer HHI, Berlin, Germany · Technical University of Berlin, Germany · Universit´e de Perpignan, Perpignan, France
Reliable monitoring of surgical instrument exchanges is essential for maintaining procedural efficiency and patient safety in the operating room. Automatic detection of instrument handovers in intraoperative video remains challenging due to frequent occlusions, background clutter, and the temporally evolving nature of interaction events. We propose a spatiotemporal vision framework for event-level detection and direction classification of surgical instrument handovers in surgical videos. The model combines a Vision Transformer (ViT) backbone for spatial feature extraction with a unidirectional Long Short-Term Memory (LSTM) network for temporal aggregation. A unified multi-task formulation jointly predicts handover occurrence and interaction direction, enabling consistent modeling of transfer dynamics while avoiding error propagation typical of cascaded pipelines. Predicted confidence scores form a temporal signal over the video, from which discrete handover events are identified via peak detection. Experiments on a dataset of kidney transplant procedures demonstrate strong performance, achieving an F1-score of 0.84 for handover detection and a mean F1-score of 0.72 for direction classification, outperforming both a single-task variant and a VideoMamba-based baseline for direction prediction while maintaining comparable detection performance. To improve interpretability, we employ Layer-CAM attribution to visualize spatial regions driving model decisions, highlighting hand-instrument interaction cues.
Figures & tables
Figure 1: Overview of the proposed architecture. Spatial features are extracted independently from sampled video frames using a ViT backbone, projected into a compact embedding space, and temporally aggregated via a unidirectional LSTM. The shared spatiotemporal representation is jointly optimized using task-specific heads for handover detection and direction classification.
Figure 2: Handover detection evaluation approach applied to a video snippet from an actual surgical procedure, which includes two handover events. The smoothed temporal signal is depicted in blue and the grey shaded region indicates the ground truth, defined as the sequences containing at least one handover frame.
Detection Performance
Model
Precision
Recall
F1
Multi-task ViT–LSTM
0.81
0.86
0.84
Single-task ViT–LSTM
0.97
0.66
0.79
VideoMamba
0.82
0.86
0.84
Direction Performance
Model
F1@R
F1@G
Mean
Table 1: Classification metrics for handover detection and direction performance (50 handovers). Detection uses Gaussian smoothing and prominence-based peak detection with a temporal tolerance of two windows. Direction is evaluated independently using Gaussian-weighted aggregation within ground-truth intervals. F1@R and F1@G denote the class-wise F1 scores for the assistant-receives and assistant-gives directions, respectively.
Figure 3: Normalized confusion matrices comparing multi-task ViT–LSTM and VideoMamba.
Model
Params
FLOPs
Latency (ms)
Multi-task ViT–LSTM
304.47M
3.32×1012
232.61
VideoMamba-Middle
74.32M
7.25×109
61.12
Table 2: Computational characteristics of the compared models with an input sequence length of 8. ViT–LSTM (image size of 518×518 , and VideoMamba-Middle ( 512×512 ). The total encoding time for eight images processed by the Vision Transformer (ViT) is 232 milliseconds. However, for real-time applications, the design can be improved by implementing sequential processing and storing the ViT encodings in a queue. As a result, processing each new image would take approximately 30 milliseconds, which is appropriate for real-time performance.
Figure 4: Layer-CAM explanation maps of the multi-task ViT–LSTM model illustrating the contribution of individual frames to the handover detection prediction. Frame 4(a) is extracted from a sequence belonging to the training set, whereas frame 4(b) is taken from a sequence in the test set.
Figure 5: Visualized gradient accumulation of the VideoMamba model demonstrates its ability to recognize the significance of hands during the handover process. The gradient accumulations were collected on a per-pixel basis and subsequently summed by patch to reduce noise.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Quantity
Value
Image Input Size
518 x 518
Feature Projection Dimension
64
Backbone Learning Rate
3×10−6
Backbone Weight Decay
1×10−4
Backbone Layers Frozen
18 of 24
Backbone Ouput Dropout Rate
0.3
Appendix
Table 3: Implementation details of ViT-LSTM model.
Quantity
Value
Backbone Variant
VideoMamba-Middle
Pretraining Dataset
Kinetics-400
Image Input Size
512 × 512
Number of Input Frames
8
Backbone Embedding Dimension
576
Backbone Layers Frozen
12 of 24
Appendix
Table 4: Implementation details of the VideoMamba comparison model.
Figure 6: Layer-CAM explanation maps illustrating the contribution of each frame to the handover detection prediction. The explanation maps are generated using the second-to-last and third-to-last layers of the model.
Real-time hand tracking in trauma surgery is essential for supporting rapid and precise intraoperative decisions. We propose a YOLOv10-based framework that simultaneously localizes hands and classifies their laterality (left or right) in complex surgical scenes. The model is trained on the Trauma THOMPSON Challenge 2025 Task 2 dataset, consisting of first-person surgical videos with annotated hand bounding boxes. Extensive data augmentation and a multi-task detection design improve robustness against motion blur, lighting variations, and diverse hand appearances. Evaluation demonstrates accurate left-hand (67%) and right-hand (71%) classification, while distinguishing hands from the background remains challenging. The model achieves an mAP[0.5:0.95] of 0.33 and maintains real-time inference, highlighting its potential for intraoperative deployment. This work establishes a foundation for advanced hand-instrument interaction analysis in emergency surgical procedures.
Kedi Sun, Le Zhang
School of Engineering, College of Engineering and Physical Sciences, University of Birmingham, Birmingham, UK
Recognizing instruments' interactions with tissues is essential for building context-aware AI assistants in robotic surgery. Vision-language models (VLMs) have opened a new avenue for surgical perception and achieved better generalization on a wide range of tasks compared to conventional task-specific deep learning approaches. However, their performance on instrument--tissue interaction recognition remains limited, largely due to two challenges: (1) many models do not effectively leverage temporal information, and (2) alignment between vision and text often misses fine-grained action details. To address these issues, we propose TrajPred, a framework that encodes instrument trajectories to incorporate temporal motion cues and, conditioned on these trajectories, introduces a predictor module to generate visual semantic embeddings that better capture fine-grained action details. We further incorporate prompt tuning and a verb-rephrasing technique to enable smooth adaptation to the instrument--tissue interaction recognition task. Extensive experiments on the public laparoscopic benchmark, CholecT50, show that our method improves both Average Precision and Top-K accuracy. We also investigate whether visual embeddings of instrument--tissue interaction regions align better with the corresponding text by visualizing the cosine similarity between visual and textual embeddings. The visualization results indicate that the proposed method improves alignment between relevant visual and textual representations.
Video-language foundation models have proven to be highly effective in zero-shot applications across a wide range of tasks. A particularly challenging area is the intraoperative surgical procedure domain, where labeled data is scarce, and precise temporal understanding is often required for complex downstream tasks. To address this challenge, we introduce CliPPER (Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition), a novel video-language pretraining framework trained on surgical lecture videos. Our method is designed for fine-grained temporal video-text recognition and introduces several novel pretraining strategies to improve multimodal alignment in long-form surgical videos. Specifically, we propose Contextual Video-Text Contrastive Learning (VTC_CTX) and Clip Order Prediction (COP) pretraining objectives, both of which leverage temporal and contextual dependencies to enhance local video understanding. In addition, we incorporate a Cycle-Consistency Alignment over video-text matches within the same surgical video to enforce bidirectional consistency and improve overall representation coherence. Moreover, we introduce a more refined alignment loss, Frame-Text Matching (FTM), to improve the alignment between video frames and text. As a result, our model establishes a new state-of-the-art across multiple public surgical benchmarks, including zero-shot recognition of phases, steps, instruments, and triplets. The source code and pretraining captions can be found at https://github.com/CAMMA-public/CliPPER.
Florian Stilz, Vinkle Srivastav, Nassir Navab +1
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France · IHU Strasbourg, France · Technical University of Munich, Germany