cs.CVApr 8, 2026

Event-Level Detection of Surgical Instrument Handovers in Videos with Interpretable Vision Models

Authors: Katerina Katsarou, George Zountsas, Karam Tomotaki-Dawoud, Alexander Ehrenhoefer, Paul Chojecki, David Przewozny, Igor Maximilian Sauer, Amira Mouakher, 

Organizations: Fraunhofer HHI, Berlin, Germany · Technical University of Berlin, Germany · Universit´e de Perpignan, Perpignan, France

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reliable monitoring of surgical instrument exchanges is essential for maintaining procedural efficiency and patient safety in the operating room. Automatic detection of instrument handovers in intraoperative video remains challenging due to frequent occlusions, background clutter, and the temporally evolving nature of interaction events. We propose a spatiotemporal vision framework for event-level detection and direction classification of surgical instrument handovers in surgical videos. The model combines a Vision Transformer (ViT) backbone for spatial feature extraction with a unidirectional Long Short-Term Memory (LSTM) network for temporal aggregation. A unified multi-task formulation jointly predicts handover occurrence and interaction direction, enabling consistent modeling of transfer dynamics while avoiding error propagation typical of cascaded pipelines. Predicted confidence scores form a temporal signal over the video, from which discrete handover events are identified via peak detection. Experiments on a dataset of kidney transplant procedures demonstrate strong performance, achieving an F1-score of 0.84 for handover detection and a mean F1-score of 0.72 for direction classification, outperforming both a single-task variant and a VideoMamba-based baseline for direction prediction while maintaining comparable detection performance. To improve interpretability, we employ Layer-CAM attribution to visualize spatial regions driving model decisions, highlighting hand-instrument interaction cues.

Figures & tables

Figure 1: Overview of the proposed architecture. Spatial features are extracted independently from sampled video frames using a ViT backbone, projected into a compact embedding space, and temporally aggregated via a unidirectional LSTM. The shared spatiotemporal representation is jointly optimized using task-specific heads for handover detection and direction classification.
Figure 2: Handover detection evaluation approach applied to a video snippet from an actual surgical procedure, which includes two handover events. The smoothed temporal signal is depicted in blue and the grey shaded region indicates the ground truth, defined as the sequences containing at least one handover frame.
Detection Performance   
ModelPrecisionRecallF1
Multi-task ViT–LSTM0.810.860.84
Single-task ViT–LSTM0.970.660.79
VideoMamba0.820.860.84
Direction Performance   
ModelF1@RF1@GMean
Table 1: Classification metrics for handover detection and direction performance (50 handovers). Detection uses Gaussian smoothing and prominence-based peak detection with a temporal tolerance of two windows. Direction is evaluated independently using Gaussian-weighted aggregation within ground-truth intervals. F1@R and F1@G denote the class-wise F1 scores for the assistant-receives and assistant-gives directions, respectively.
Figure 3: Normalized confusion matrices comparing multi-task ViT–LSTM and VideoMamba.
ModelParamsFLOPsLatency (ms)
Multi-task ViT–LSTM304.47M3.32×10123.32\times 10^{12}232.61
VideoMamba-Middle74.32M7.25×1097.25\times 10^{9}61.12
Table 2: Computational characteristics of the compared models with an input sequence length of 8. ViT–LSTM (image size of 518×518518\times 518 , and VideoMamba-Middle ( 512×512512\times 512 ). The total encoding time for eight images processed by the Vision Transformer (ViT) is 232 milliseconds. However, for real-time applications, the design can be improved by implementing sequential processing and storing the ViT encodings in a queue. As a result, processing each new image would take approximately 30 milliseconds, which is appropriate for real-time performance.
Figure 4: Layer-CAM explanation maps of the multi-task ViT–LSTM model illustrating the contribution of individual frames to the handover detection prediction. Frame 4(a) is extracted from a sequence belonging to the training set, whereas frame 4(b) is taken from a sequence in the test set.
Figure 5: Visualized gradient accumulation of the VideoMamba model demonstrates its ability to recognize the significance of hands during the handover process. The gradient accumulations were collected on a per-pixel basis and subsequently summed by patch to reduce noise.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

QuantityValue
Image Input Size518 x 518
Feature Projection Dimension64
Backbone Learning Rate3×10−63\times 10^{-6}
Backbone Weight Decay1×10−41\times 10^{-4}
Backbone Layers Frozen18 of 24
Backbone Ouput Dropout Rate0.3
Appendix
Table 3: Implementation details of ViT-LSTM model.
QuantityValue
Backbone VariantVideoMamba-Middle
Pretraining DatasetKinetics-400
Image Input Size512 ×\times 512
Number of Input Frames8
Backbone Embedding Dimension576
Backbone Layers Frozen12 of 24
Appendix
Table 4: Implementation details of the VideoMamba comparison model.
Figure 6: Layer-CAM explanation maps illustrating the contribution of each frame to the handover detection prediction. The explanation maps are generated using the second-to-last and third-to-last layers of the model.
More in this topic

Surgical Video Understanding

Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.

See all 118 papers

Explore similar work

CardsList

  1. YOLOv10-Based Multi-Task Framework for Hand Localization and Laterality Classification in Surgical Videos

    Feb 21, 2026Kedi Sun, Le ZhangSurgical Video UnderstandingVideo Object Detection

    School of Engineering, College of Engineering and Physical Sciences, University of Birmingham, Birmingham, UK

  2. TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models

    Mar 7, 2026Jiajun Cheng, Xiaofan Yu, Subarna Tripathi +2Medical VLMs

  3. CliPPER: Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition

    Mar 25, 2026Florian Stilz, Vinkle Srivastav, Nassir Navab +1Surgical Video UnderstandingVideo-Language Models

    University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France · IHU Strasbourg, France · Technical University of Munich, Germany