SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessment

Runlong He; Freweini M. Tesfai; Matthew W. E. Boal; Nazir Sirajudeen; Dimitrios Anastasiou; Jialang Xu; Mobarak I. Hoque; Philip J. Edwards

cs.ROFeb 18, 2026

SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessment

Authors: Runlong He, Freweini M. Tesfai, Matthew W. E. Boal, Nazir Sirajudeen, Dimitrios Anastasiou, Jialang Xu, Mobarak I. Hoque, Philip J. Edwards,

Organizations: the UCL Hawkes Institute and the Department of Medical Physics & Biomedical Engineering, UCL, UK. · the Griffin Institute, Northwick Park and St Mark’s Hospital, UK. · the UCL Hawkes Institute and the Division of Informatics, Imaging & Data Sciences, The University of Manchester, UK · the UCL Hawkes Institute and the Department of Computer Science, UCL, UK. ·

Abstract

Robotic-assisted surgery (RAS) is established in clinical practice, and automated surgical skill assessment utilizing multimodal data offers transformative potential for surgical analytics and education. However, developing effective multimodal methods remains challenging due to the task complexity, limited annotated datasets and insufficient techniques for cross-modal information fusion. Existing state-of-the-art relies exclusively on RGB video and only applies on dry-lab settings, failing to address the significant domain gap between controlled simulation and real clinical cases, where the surgical environment together with camera and tissue motion introduce substantial complexities. This work introduces SurgFusion-Net and Divergence Regulated Attention (DRA), an innovative fusion strategy for multimodal surgical skill assessment. We contribute two first-of-their-kind clinical datasets: the RAH-skill dataset containing 279,691 RGB frames from 37 videos of Robot-assisted Hysterectomy (RAH), and the RARP-skill dataset containing 70,661 RGB frames from 33 videos of Robot-Assisted Radical Prostatectomy (RARP). Both datasets include M-GEARS skill annotations, corresponding optical flow and tool segmentation masks. DRA incorporates adaptive dual attention and diversity-promoting multi-head attention to fuse multimodal information, from three modalities, based on surgical context, enhancing assessment accuracy and reliability. Validated on the JIGSAWS benchmark, RAH-skill, and RARP-skill datasets, our approach outperforms recent baselines with SCC improvements of 0.02 in LOSO, 0.04 in LOUO across JIGSAWS tasks, and 0.0538 and 0.0493 gains on RAH-skill and RARP-skill, respectively.

Figures & tables

Fig. 1: Examples from the multimodal RAH-skill and RARP-skill datasets, along with the enhanced JIGSAWS dataset. JIGSAWS comprises three tasks: Suturing, Knot Tying, and Needle Passing. All datasets contain RGB frames, optical flow maps, and segmentation masks. In optical flow maps, black background indicates no motion, brightness represents motion speed, and colors indicate motion directions. RAH-skill and RARP-skill include tool segmentation masks, while JIGSAWS contains both tool and reference object masks.

Fig. 2: SurgFusion-Net: the network consists of three unimodal branches and a multimodal fusion branch. The architecture progressively fuses unimodal features to construct comprehensive multimodal representations for robotic surgical skill assessment.

Fig. 3: Computation flow of Divergence Regulated Attention.

Q_{h}

K_{h}

V_{h}

are query, key, value representations for each head.

\otimes

: matrix multiplication;

\oplus

: element-wise addition; Proj: projection; Sim: similarity.

JIGSAWS
Input Modality	Method	KT			NP			SU			Across Tasks
Input Modality	Method	LOSO	LOUO	4-Fold	LOSO	LOUO	4-Fold	LOSO	LOUO	4-Fold	LOSO	LOUO	4-Fold
Spearman’s Correlation Coefficient (SSC)
K	SMT-DCT-DFT [ 20 ]	0.70	0.73	—	0.38	0.23	—	0.64	0.10	—	0.59	0.40	—
	DCT-DFT-ApEn [ 20 ]	0.63	0.60	—	0.46	0.25	—	0.75	0.37	—	0.63	0.42	—
	ReCAP [ 6 ]	0.88	—	—	0.85	—	—	0.83	—	—	0.85	—	—

TABLE I: Comparative results on the JIGSAWS dataset. K for kinematic, V for video, F for optical flow, M for segmentation mask, and SD for standard deviation.

Input Modality	Method	RAH-skill		RARP-skill
Input Modality	Method	SCC	MAE	SCC	MAE
V	CoRe + GART [ 36 ]	0.5893	2.9434	0.5656	2.0240
	ViSA [ 10 ]	0.5856	2.6852	0.5997	2.1041
	Contra-Sformer [ 9 ]	0.5695	2.5428	0.6083	1.6345
	CoRe + PECoP [ 37 ]	0.6658	2.8975	0.7103	2.2799
V+F+M	MAQA [ 12 ]	0.7756	1.9645	0.7993	1.3588

TABLE II: Comparative results on RAH-skill and RARP-skill datasets.

Branch (Block)	Modality			JIGSAWS (LOUO)						RAH-skill		RARP-skill
	Modality			KT		NP		SU		RAH-skill		RARP-skill
	RGB	Flow	Mask	SCC	MAE	SCC	MAE	SCC	MAE	SCC	MAE	SCC	MAE
Single Modality	✓			0.8422	1.7873	0.8624	2.5825	0.7748	1.9907	0.6960	2.3041	0.7368	2.0024
		✓		0.8307	2.0367	0.8431	2.8785	0.7539	1.8390	0.6282	2.8250	0.7254	1.8059
			✓	0.8367	1.5216	0.8315	2.1611	0.7663	1.7851	0.7056	2.1335	0.7459	1.7138
Multimodal Fusion (CSFB)	✓	✓		0.8783	1.4713	0.8746	2.0201	0.8044	1.7529	0.7441	2.0762	0.7891	1.4823

TABLE III: Evaluation of the fusion branches on JIGSAWS, RAH-skill and RARP-skill datasets.

Fusion Strategy	SU (LOUO)		RAH-skill		RARP-skill
Fusion Strategy	SCC	MAE	SCC	MAE	SCC	MAE
AVG	0.8081	1.5491	0.7529	1.9340	0.7379	1.3910
CAT	0.8110	1.9093	0.7633	1.9981	0.7460	1.4304
Vanilla CA [ 31 ]	0.8154	1.8795	0.7676	1.9031	0.7853	1.3102
Negative CA [ 12 ]	0.8210	1.6883	0.7756	1.9645	0.7993	1.3588
DRA (Ours)	0.8355	1.4866	0.8294	1.8182	0.8486	1.2974

TABLE IV: Evaluation of the fusion Strategy on JIGSAWS (SU), RAH-skill and RARP-skill datasets. CA for Cross-Attention.

# Heads	SU (LOUO)		RAH-skill		RARP-skill
# Heads	SCC	MAE	SCC	MAE	SCC	MAE
1	0.8223	1.5195	0.7859	1.8433	0.8055	1.3298
2	0.8355	1.4866	0.8294	1.8182	0.8486	1.2974
4	0.8279	1.4954	0.7892	1.8875	0.8246	1.3375
8	0.8221	1.7481	0.7927	1.9112	0.8270	1.3717

TABLE V: Evaluation of the number of heads on JIGSAWS (SU), RAH-skill and RARP-skill datasets.

Fig. 4: Visualization of attention weights of the cross-stage fusion block (CSFB) in the stage two on RAH-skill dataset. The top left figure displays the attention weights across three modalities on feature sequences, with four purple vertical dashed lines marking the time windows of specific features. Figures (a)-(d) show RGB frames, optical flows, and segmentation masks (from top to bottom in each column) corresponding to time windows T1 to T4. In stage two, each horizontal interval corresponds to about 14 seconds of video.

Surgical Video Understanding

Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.

See all 118 papers

SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment
Aug 31, 2026Chaohui Dang, Zheheng Jiang, James Glasbey +3Surgical Video Understanding Multimodal Model Evaluation
Department of Electronic, Electrical and Systems Engineering, School of Engineering, University of Birmingham, Birmingham, UK · School of Computing and Mathematical Science, University of Leicester, Leicester, UK · Department of Applied Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK +1
SurGPT: A Multi-Task Foundation Model for Robotic Surgical Skills Assessment and Gesture Recognition
Oct 18, 2026Parham M. Kebria, Saeid Nahavandi, Mehrdad Saif +1Surgical Video Understanding Surgical Robotics
Electrical and Computer Engineering Department, North Carolina A&T State University, Greensboro, NC, USA · Swinburne University of Technology, Hawthorn, VIC, Australia · Harvard Paulson School of Engineering and Applied Sciences, Harvard University, Allston, MA, USA +3
Real-Time Multimodal Activity-Aware Error Detection in Robot-Assisted Surgery
Jun 22, 2026Seyed Hamid Reza Roodabeh, Zongyu Li, Homa AlemzadehSurgical Robotics Multimodal Sensor Fusion
Electrical and Computer Engineering Department, School of Engineering and Applied Science, University of Virginia, Charlottesville, VA, United States of America

Aug 31, 2026J/K move · Enter open · S save

SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment

Chaohui Dang, Zheheng Jiang, James Glasbey, David Luke+2

Department of Electronic, Electrical and Systems Engineering, School of Engineering, University of Birmingham, Birmingham, UK · School of Computing and Mathematical Science, University of Leicester, Leicester, UK · Department of Applied Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK+1

Surgical Video Understanding Multimodal Model Evaluation

Open paper AskPDF arXiv

Objective assessment of surgical technical skill is important for surgical training and structured feedback, but current workflows remain dependent on labor-intensive expert review. Existing automated approaches primarily focus on visual inputs and provide limited support for jointly studying operative performance, structured skill scores, and evaluator feedback. We introduce SurgSkill-Bench, an initial video-score-text benchmark-style dataset containing 214 surgical training simulation videos, six-dimensional OSATS scores, and expert free-text comments. We define two evaluation settings: video-only OSATS prediction for automated assessment and post hoc expert-comment-assisted prediction, where evaluator comments are available as auxiliary information. We provide controlled baseline experiments using representative frozen visual backbones, content-adaptive key-frame sampling, and a simple video-text co-attention fusion module. Under internal video-level validation, content-adaptive sampling improves video-only performance in this dataset, while evaluator comments provide additional score-related signal in the assisted setting. The best mean AUROC reaches 0.88 under dataset-specific median dichotomization. We further discuss evaluation constraints related to dataset scale, metadata completeness, and the interpretation of comment-assisted prediction. Code will be released publicly at a later date.

Abstract

Surgical Video Understanding

Explore similar work

SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment

SurGPT: A Multi-Task Foundation Model for Robotic Surgical Skills Assessment and Gesture Recognition

Real-Time Multimodal Activity-Aware Error Detection in Robot-Assisted Surgery