Semantic RGB--Depth Based Surgical Skill Assessment in Microscopic Stereo Videos

Jecia Z. Y. Mao; Sue M. Cho; Francis X. Creighton; Deepa Galaiya; Russell H. Taylor; Manish Sahu

cs.CVOct 1, 2026

Semantic RGB--Depth Based Surgical Skill Assessment in Microscopic Stereo Videos

Authors: Jecia Z. Y. Mao, Sue M. Cho, Francis X. Creighton, Deepa Galaiya, Russell H. Taylor, Manish Sahu

Organizations: Laboratory of Computational Sensing and Robotics, Johns Hopkins University, MD, USA · Department of Otolaryngology–Head and Neck Surgery, Johns Hopkins University School of Medicine, MD, USA

Abstract

Objective assessment of microsurgical technical skill is essential for competency-based training and quality assurance, yet existing video-based approaches predominantly rely on RGB images and therefore overlook the 3D spatial relationships that characterize instrument-anatomy interactions. Although stereo operating microscopes provide complementary depth information, conventional stereo matching algorithms can produce sparse and unreliable depth estimates under high-magnification imaging conditions, limiting their use for automated skill assessment. This work presents a semantic RGB-Depth framework for surgical skill assessment from microscopic stereo videos. A regression-based depth fusion method combines sparse metric stereo depth with dense monocular depth estimates to generate a dense geometric representation of the surgical scene. This representation is integrated with semantically decomposed RGB streams corresponding to individual surgical instruments and surrounding anatomy. A hierarchical attention architecture jointly encodes these streams to capture discriminative patterns of instrument use and instrument-anatomy interaction across surgeons at different training levels. The framework was evaluated on 33 ex vivo transoral microlaryngeal procedures performed by six surgeons, comprising attending surgeons and surgical residents, using leave-one-surgeon-out cross-validation. The proposed semantic RGB-Depth model achieved an F1 score of 0.938 for skill-level classification, compared with 0.696 for semantic RGB and 0.929 for semantic depth. These results suggest that geometric information can improve automated surgical skill assessment from microscopic stereo videos. The learned spatial, temporal, and semantic attention patterns also support qualitative examination of the scene regions, video segments, and semantic streams emphasized by the model.

Figures & tables

Figure 1: Architecture of the proposed semantic-stream framework. Each trial is decomposed into grasper, scissors, and anatomy streams, indexed by $k\in\mathcal{K}$ . RGB and depth features are fused within each stream and encoded by an independent dilated TCN to obtain $\mathbf{h}_{t,k}$ . A local score head produces frame-level skill score $s_{t,k}$ , while stream-specific temporal attention $\alpha_{t,k}$ forms the auxiliary prediction $\hat{y}_{k}=\sum_{t=1}^{T}\alpha_{t,k}s_{t,k}$ . For global prediction, frame-wise stream attention $\gamma_{t,k}$ fuses the semantic representations into $\bar{\mathbf{h}}_{t}=\sum_{k\in\mathcal{K}}\gamma_{t,k}\mathbf{h}_{t,k}$ . Global temporal attention $\alpha_{t}$ then identifies informative frames, and the final trial-level prediction is $\hat{y}=\sum_{t=1}^{T}\sum_{k\in\mathcal{K}}\alpha_{t}\gamma_{t,k}s_{t,k}$ . Auxiliary stream-level auxiliary loss encourages each semantic branch to retain independently discriminative skill information.

Figure 2: Overview of the proposed depth-guided semantic feature extraction pipeline. The first frame of each microscopic stereo video is annotated using prompt-based segmentation, and the resulting instrument and foreground masks are propagated throughout the sequence. Stereo depth provides sparse metric measurements, while monocular depth provides dense relative structure. The two estimates are aligned within the foreground region to recover dense metric depth. Semantic RGB and depth features are then extracted separately for the grasper, scissors, and anatomy streams and used for temporal skill assessment.

Figure 3: Spatial attention at stream-specific peak-attention moments for four representative trials, including two expert and two resident trials. For each trial, seven frames are sampled around the peak of the combined temporal and semantic-stream attention response, $\alpha_{t}\gamma_{t,k}$ , independently for the scissors and grasper streams. The first row shows the scissors sequence and the second row the grasper sequence, with each sequence centered on its corresponding peak-attention frame. Spatial attention maps are restricted to the respective encoded semantic-mask region and upsampled to the original image resolution. Warmer colors indicate greater spatial attention.

Experience level	Participants (videos)	Total videos
Expert	E01 (6), E02 (4), E03 (5)	15
Resident	R01 (6), R02 (6), R03 (6)	18
Overall	6 participants	33

Table 1: Participant composition of the laryngeal microsurgery dataset.

Trial	Initialization Dice	Sequence Dice
E01	0.9706	0.9792
E02	0.9594	0.9899
E03	0.9728	0.9513
R01	0.9588	0.9742
R02	0.9343	0.9683
R03	0.9533	0.9759

Table 2: Quantitative evaluation of segmentation initialization and propagation for one video trial from each participant.

Figure 4: Representative segmentation results from the six videos used for quantitative evaluation. One frame is shown from each participant, including three expert and three resident cases, illustrating propagated tool segmentation across different users and video sequences.

Individual representation baselines
Model	MAE $\downarrow$	$\rho$ $\uparrow$	Acc. $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$
RGB	0.267	0.863	0.788	1.000	0.533	0.696
Depth (ASPP)	0.487	0.147	0.576	0.519	0.933	0.667
Depth (SPP)	0.381	0.748	0.939	1.000	0.867	0.929
Proposed model
Depth (SPP) + RGB	0.227	0.863	0.939	0.882	1.000	0.938

Table 3: Evaluation results of leave-one-surgeon-out cross-validation. Best values are shown in bold. NB: ∗ USSA (RGB) [ 12 ] predicted all trials as resident, resulting in zero precision, recall, and F1 for the expert class.

Held-out	Class	$n$	Semantic RGB	Depth (ASPP)	Depth (SPP)	Depth (SPP) + RGB	w/o semantic	w/o attention	w/o auxiliary	USSA (RGB)
E01	Expert	6	0.286	0.667	0.909	1.000	0.000	1.000	0.667	0.000
E02	Expert	4	1.000	0.800	0.857	1.000	1.000	1.000	1.000	0.000
E03	Expert	5	0.750	0.667	1.000	1.000	0.571	1.000	1.000	0.000
R01	Resident	6	1.000	0.800	1.000	0.800	0.000	0.000	0.000	1.000
R02	Resident	6	1.000	0.571	1.000	1.000	0.800	0.000	0.000	1.000
R03	Resident	6	1.000	0.571	1.000	1.000	0.000	0.500	0.000	1.000

Table 4: Class-specific F1 scores for each held-out surgeon under participant-disjoint cross-validation.

n

denotes the number of validation trials, and each model column reports F1 for the corresponding expertise class. ∗ USSA (RGB) [ 12 ] predicted all trials as resident; consequently, F1 is zero for each held-out expert surgeon.

Run	Grasper	Scissors	Anatomy
Semantic RGB	0.46 $\pm$ 0.42	0.27 $\pm$ 0.32	0.26 $\pm$ 0.31
Semantic Depth (SPP)	0.44 $\pm$ 0.47	0.17 $\pm$ 0.33	0.39 $\pm$ 0.45
Semantic Depth (ASPP)	0.36 $\pm$ 0.43	0.28 $\pm$ 0.42	0.36 $\pm$ 0.43
Semantic Depth (SPP) + RGB	0.24 $\pm$ 0.37	0.28 $\pm$ 0.36	0.47 $\pm$ 0.38

Table 5: Average per-frame semantic-component attention. Values are mean

\pm

standard deviation across validation trials; each trial contributes its within-trial mean equally.

Run	Expertise	Peak Location
Semantic RGB	Expert	0.47 $\pm$ 0.42
	Resident	0.50 $\pm$ 0.29
Semantic Depth (SPP)	Expert	0.49 $\pm$ 0.26
	Resident	0.99 $\pm$ 0.01
Semantic Depth (ASPP)	Expert	0.46 $\pm$ 0.40
	Resident	0.44 $\pm$ 0.46

Table 6: Normalized global temporal-attention peak location. Values are mean

\pm

standard deviation across validation trials, with 0 and 1 denoting the first and final analyzed frames, respectively.

Figure 5: Stream-weighted local scores aligned to the peak temporal-attention frame. The horizontal axis represents time relative to the frame receiving the maximum global temporal-attention weight. Solid lines show the mean across trials, and shaded regions indicate the standard deviation.

Explore similar work

SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment
Aug 31, 2026Chaohui Dang, Zheheng Jiang, James Glasbey +3Surgery Multimodal Evaluation
Department of Electronic, Electrical and Systems Engineering, School of Engineering, University of Birmingham, Birmingham, UK · School of Computing and Mathematical Science, University of Leicester, Leicester, UK · Department of Applied Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK +1
OSS: Open Suturing Skills Vision-Based Assessment Challenge 2024-2025
May 21, 2026Hanna Hoffmann, Setareh Bady, Claas de Boer +54Surgery Computer Vision
StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation
Sep 14, 2026Yanan Zhou, Zhaoyan Qian, James Zhao +1Robotic Perception Visuomotor Policy
School of Computer Science, The University of Sydney, Australia. · Australian Centre for Robotics, The University of Sydney, Australia.

Aug 31, 2026, cs.CVJ/K move · Enter open · S save

SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment

Chaohui Dang, Zheheng Jiang, James Glasbey, David Luke+2

Department of Electronic, Electrical and Systems Engineering, School of Engineering, University of Birmingham, Birmingham, UK · School of Computing and Mathematical Science, University of Leicester, Leicester, UK · Department of Applied Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK+1

Surgery Multimodal Evaluation Assessment

Open paper AskPDF arXiv

Objective assessment of surgical technical skill is important for surgical training and structured feedback, but current workflows remain dependent on labor-intensive expert review. Existing automated approaches primarily focus on visual inputs and provide limited support for jointly studying operative performance, structured skill scores, and evaluator feedback. We introduce SurgSkill-Bench, an initial video-score-text benchmark-style dataset containing 214 surgical training simulation videos, six-dimensional OSATS scores, and expert free-text comments. We define two evaluation settings: video-only OSATS prediction for automated assessment and post hoc expert-comment-assisted prediction, where evaluator comments are available as auxiliary information. We provide controlled baseline experiments using representative frozen visual backbones, content-adaptive key-frame sampling, and a simple video-text co-attention fusion module. Under internal video-level validation, content-adaptive sampling improves video-only performance in this dataset, while evaluator comments provide additional score-related signal in the assisted setting. The best mean AUROC reaches 0.88 under dataset-specific median dichotomization. We further discuss evaluation constraints related to dataset scale, metadata completeness, and the interpretation of comment-assisted prediction. Code will be released publicly at a later date.

Abstract

Explore similar work

SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment

OSS: Open Suturing Skills Vision-Based Assessment Challenge 2024-2025

StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation