SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessment
Authors: Runlong He, Freweini M. Tesfai, Matthew W. E. Boal, Nazir Sirajudeen, Dimitrios Anastasiou, Jialang Xu, Mobarak I. Hoque, Philip J. Edwards, John D. Kelly, Ashwin Sridhar, Abdolrahim Kadkhodamohammadi, Dhivya Chandrasekaran, Matthew J. Clarkson, Danail Stoyanov, Nader Francis, Evangelos B. Mazomenos
Organizations: the UCL Hawkes Institute and the Department of Medical Physics & Biomedical Engineering, UCL, UK. · the Griffin Institute, Northwick Park and St Mark’s Hospital, UK. · the UCL Hawkes Institute and the Division of Informatics, Imaging & Data Sciences, The University of Manchester, UK · the UCL Hawkes Institute and the Department of Computer Science, UCL, UK. · the UCL Hospitals NHS Foundation Trust, UK. · Medtronic Digital Surgery, London, UK
Robotic-assisted surgery (RAS) is established in clinical practice, and automated surgical skill assessment utilizing multimodal data offers transformative potential for surgical analytics and education. However, developing effective multimodal methods remains challenging due to the task complexity, limited annotated datasets and insufficient techniques for cross-modal information fusion. Existing state-of-the-art relies exclusively on RGB video and only applies on dry-lab settings, failing to address the significant domain gap between controlled simulation and real clinical cases, where the surgical environment together with camera and tissue motion introduce substantial complexities. This work introduces SurgFusion-Net and Divergence Regulated Attention (DRA), an innovative fusion strategy for multimodal surgical skill assessment. We contribute two first-of-their-kind clinical datasets: the RAH-skill dataset containing 279,691 RGB frames from 37 videos of Robot-assisted Hysterectomy (RAH), and the RARP-skill dataset containing 70,661 RGB frames from 33 videos of Robot-Assisted Radical Prostatectomy (RARP). Both datasets include M-GEARS skill annotations, corresponding optical flow and tool segmentation masks. DRA incorporates adaptive dual attention and diversity-promoting multi-head attention to fuse multimodal information, from three modalities, based on surgical context, enhancing assessment accuracy and reliability. Validated on the JIGSAWS benchmark, RAH-skill, and RARP-skill datasets, our approach outperforms recent baselines with SCC improvements of 0.02 in LOSO, 0.04 in LOUO across JIGSAWS tasks, and 0.0538 and 0.0493 gains on RAH-skill and RARP-skill, respectively.
Figures & tables
Fig. 1: Examples from the multimodal RAH-skill and RARP-skill datasets, along with the enhanced JIGSAWS dataset. JIGSAWS comprises three tasks: Suturing, Knot Tying, and Needle Passing. All datasets contain RGB frames, optical flow maps, and segmentation masks. In optical flow maps, black background indicates no motion, brightness represents motion speed, and colors indicate motion directions. RAH-skill and RARP-skill include tool segmentation masks, while JIGSAWS contains both tool and reference object masks.
Fig. 2: SurgFusion-Net: the network consists of three unimodal branches and a multimodal fusion branch. The architecture progressively fuses unimodal features to construct comprehensive multimodal representations for robotic surgical skill assessment.
Fig. 3: Computation flow of Divergence Regulated Attention. Qh , Kh , Vh are query, key, value representations for each head. ⊗ : matrix multiplication; ⊕ : element-wise addition; Proj: projection; Sim: similarity.
JIGSAWS
Input Modality
Method
KT
NP
SU
Across Tasks
LOSO
LOUO
4-Fold
LOSO
LOUO
4-Fold
LOSO
LOUO
4-Fold
LOSO
LOUO
4-Fold
Spearman’s Correlation Coefficient (SSC)
K
SMT-DCT-DFT [ 20 ]
0.70
0.73
—
0.38
0.23
—
0.64
0.10
—
0.59
0.40
—
DCT-DFT-ApEn [ 20 ]
0.63
0.60
—
0.46
0.25
—
0.75
0.37
—
0.63
0.42
—
ReCAP [ 6 ]
0.88
—
—
0.85
—
—
0.83
—
—
0.85
—
—
TABLE I: Comparative results on the JIGSAWS dataset. K for kinematic, V for video, F for optical flow, M for segmentation mask, and SD for standard deviation.
Input Modality
Method
RAH-skill
RARP-skill
SCC
MAE
SCC
MAE
V
CoRe + GART [ 36 ]
0.5893
2.9434
0.5656
2.0240
ViSA [ 10 ]
0.5856
2.6852
0.5997
2.1041
Contra-Sformer [ 9 ]
0.5695
2.5428
0.6083
1.6345
CoRe + PECoP [ 37 ]
0.6658
2.8975
0.7103
2.2799
V+F+M
MAQA [ 12 ]
0.7756
1.9645
0.7993
1.3588
TABLE II: Comparative results on RAH-skill and RARP-skill datasets.
Branch (Block)
Modality
JIGSAWS (LOUO)
RAH-skill
RARP-skill
KT
NP
SU
RGB
Flow
Mask
SCC
MAE
SCC
MAE
SCC
MAE
SCC
MAE
SCC
MAE
Single Modality
✓
0.8422
1.7873
0.8624
2.5825
0.7748
1.9907
0.6960
2.3041
0.7368
2.0024
✓
0.8307
2.0367
0.8431
2.8785
0.7539
1.8390
0.6282
2.8250
0.7254
1.8059
✓
0.8367
1.5216
0.8315
2.1611
0.7663
1.7851
0.7056
2.1335
0.7459
1.7138
Multimodal Fusion (CSFB)
✓
✓
0.8783
1.4713
0.8746
2.0201
0.8044
1.7529
0.7441
2.0762
0.7891
1.4823
TABLE III: Evaluation of the fusion branches on JIGSAWS, RAH-skill and RARP-skill datasets.
Fusion Strategy
SU (LOUO)
RAH-skill
RARP-skill
SCC
MAE
SCC
MAE
SCC
MAE
AVG
0.8081
1.5491
0.7529
1.9340
0.7379
1.3910
CAT
0.8110
1.9093
0.7633
1.9981
0.7460
1.4304
Vanilla CA [ 31 ]
0.8154
1.8795
0.7676
1.9031
0.7853
1.3102
Negative CA [ 12 ]
0.8210
1.6883
0.7756
1.9645
0.7993
1.3588
DRA (Ours)
0.8355
1.4866
0.8294
1.8182
0.8486
1.2974
TABLE IV: Evaluation of the fusion Strategy on JIGSAWS (SU), RAH-skill and RARP-skill datasets. CA for Cross-Attention.
# Heads
SU (LOUO)
RAH-skill
RARP-skill
SCC
MAE
SCC
MAE
SCC
MAE
1
0.8223
1.5195
0.7859
1.8433
0.8055
1.3298
2
0.8355
1.4866
0.8294
1.8182
0.8486
1.2974
4
0.8279
1.4954
0.7892
1.8875
0.8246
1.3375
8
0.8221
1.7481
0.7927
1.9112
0.8270
1.3717
TABLE V: Evaluation of the number of heads on JIGSAWS (SU), RAH-skill and RARP-skill datasets.
Fig. 4: Visualization of attention weights of the cross-stage fusion block (CSFB) in the stage two on RAH-skill dataset. The top left figure displays the attention weights across three modalities on feature sequences, with four purple vertical dashed lines marking the time windows of specific features. Figures (a)-(d) show RGB frames, optical flows, and segmentation masks (from top to bottom in each column) corresponding to time windows T1 to T4. In stage two, each horizontal interval corresponds to about 14 seconds of video.
Objective assessment of surgical technical skill is important for surgical training and structured feedback, but current workflows remain dependent on labor-intensive expert review. Existing automated approaches primarily focus on visual inputs and provide limited support for jointly studying operative performance, structured skill scores, and evaluator feedback. We introduce SurgSkill-Bench, an initial video-score-text benchmark-style dataset containing 214 surgical training simulation videos, six-dimensional OSATS scores, and expert free-text comments. We define two evaluation settings: video-only OSATS prediction for automated assessment and post hoc expert-comment-assisted prediction, where evaluator comments are available as auxiliary information. We provide controlled baseline experiments using representative frozen visual backbones, content-adaptive key-frame sampling, and a simple video-text co-attention fusion module. Under internal video-level validation, content-adaptive sampling improves video-only performance in this dataset, while evaluator comments provide additional score-related signal in the assisted setting. The best mean AUROC reaches 0.88 under dataset-specific median dichotomization. We further discuss evaluation constraints related to dataset scale, metadata completeness, and the interpretation of comment-assisted prediction. Code will be released publicly at a later date.
Chaohui Dang, Zheheng Jiang, James Glasbey +3
Department of Electronic, Electrical and Systems Engineering, School of Engineering, University of Birmingham, Birmingham, UK · School of Computing and Mathematical Science, University of Leicester, Leicester, UK · Department of Applied Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK +1
The assessment of surgical skills in robotic surgery training has traditionally relied on subjective evaluations by human experts, creating bottlenecks in surgical education and limiting scalability. We present a novel foundation model that automatically analyzes robotic surgical videos to provide comprehensive skill assessment and gesture understanding. Combining a visual encoder with a transformer architecture, the model performs three key tasks simultaneously: 1) surgical task classification (knot tying, needle passing, suturing), 2) skill level assessment on a standardized scale, and 3) natural language generation of surgical gesture descriptions. The model processes video sequences from the da Vinci surgical robot and generates human-readable text descriptions of surgical actions, similar to how language models generate text from prompts, using a multi-task learning framework with adaptive uncertainty weighting to optimize performance across all objectives. Our transformer-based architecture incorporates positional encoding for temporal understanding and attention mechanisms for spatiotemporal feature integration. Evaluated on the da Vinci Skills Simulator dataset, the proposed framework demonstrates state-of-the-art performance in surgical task recognition and provides interpretable skill assessments that correlate with expert evaluations, with the model's ability to generate natural language descriptions of surgical gestures opening new possibilities for automated surgical education, real-time feedback systems, and objective skill assessment in surgical training and certification programs.
Parham M. Kebria, Saeid Nahavandi, Mehrdad Saif +1
Electrical and Computer Engineering Department, North Carolina A&T State University, Greensboro, NC, USA · Swinburne University of Technology, Hawthorn, VIC, Australia · Harvard Paulson School of Engineering and Applied Sciences, Harvard University, Allston, MA, USA +3
Robot-assisted minimally invasive surgery improves surgical precision but introduces complexity, making technical error detection essential for ensuring patient safety. Current executional error detection methods using video data often overlook fine-grained contextual descriptions of activities and error types within the hierarchical structure of surgical procedures. They also under-utilize complementary multimodal information. We propose a unified framework for executional error detection that leverages multimodal input, including video, kinematics, and descriptive textual prompts. Through activity prompting, we integrate descriptive language in gesture-level activities, instrument-object interactions, and error definitions. We also introduce activity-aware visual embeddings derived from vision encoders pretrained on surgical activity labels to compare the effectiveness of contrastive language-image embeddings with traditional image-based embeddings for error detection. By seamlessly integrating kinematic data with video and textual modalities, our framework significantly improves error detection performance. Achieving up to 5% and 16.6% F1 score improvements over state-of-the-art baselines on the JIGSAWS and SAR-RARP50 datasets, respectively, we demonstrate the value of combining curated textual prompts with multimodal data for accurate error detection.
Electrical and Computer Engineering Department, School of Engineering and Applied Science, University of Virginia, Charlottesville, VA, United States of America