cs.ROOct 18, 2026

SurGPT: A Multi-Task Foundation Model for Robotic Surgical Skills Assessment and Gesture Recognition

Authors: Parham M. Kebria, Saeid Nahavandi, Mehrdad Saif, Fernando Bello

Organizations: Electrical and Computer Engineering Department, North Carolina A&T State University, Greensboro, NC, USA · Swinburne University of Technology, Hawthorn, VIC, Australia · Harvard Paulson School of Engineering and Applied Sciences, Harvard University, Allston, MA, USA · Department of Electrical and Computer Engineering, University of Windsor, Windsor, Ont. Canada · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The assessment of surgical skills in robotic surgery training has traditionally relied on subjective evaluations by human experts, creating bottlenecks in surgical education and limiting scalability. We present a novel foundation model that automatically analyzes robotic surgical videos to provide comprehensive skill assessment and gesture understanding. Combining a visual encoder with a transformer architecture, the model performs three key tasks simultaneously: 1) surgical task classification (knot tying, needle passing, suturing), 2) skill level assessment on a standardized scale, and 3) natural language generation of surgical gesture descriptions. The model processes video sequences from the da Vinci surgical robot and generates human-readable text descriptions of surgical actions, similar to how language models generate text from prompts, using a multi-task learning framework with adaptive uncertainty weighting to optimize performance across all objectives. Our transformer-based architecture incorporates positional encoding for temporal understanding and attention mechanisms for spatiotemporal feature integration. Evaluated on the da Vinci Skills Simulator dataset, the proposed framework demonstrates state-of-the-art performance in surgical task recognition and provides interpretable skill assessments that correlate with expert evaluations, with the model's ability to generate natural language descriptions of surgical gestures opening new possibilities for automated surgical education, real-time feedback systems, and objective skill assessment in surgical training and certification programs.

Figures & tables

Fig. 1: Example scene frames of the three surgical tasks of JIGSAWS and their corresponding task orders and flow of gestures, from Suturing (left) to Knot Tying (right).
Fig. 2: Multi-task transformer architecture for surgical skill assessment; a CNN backbone, temporal modeling, and specialized prediction heads.
TaskMetricValue
Task ClassificationAccuracy100.0%
F1-macro100.0%
F1-micro100.0%
Skill AssessmentMAE0.314
RMSE0.840
Pearson R²0.660
TABLE I: Overall Performance Summary of the Surgical Skills Foundation Model
MetricValue
Accuracy100.0%
F1-macro100.0%
F1-micro100.0%
F1-weighted100.0%
Precision (macro)100.0%
Recall (macro)100.0%
TABLE II: Detailed Task Classification Results
MetricValue
Classification Accuracy91.7%
F1-macro93.8%
F1-weighted91.3%
Regression Metrics:
Mean Absolute Error (MAE)0.314
Root Mean Square Error (RMSE)0.840
TABLE III: Skill Assessment Evaluation Results
MetricValue
BLEU Scores:
BLEU-10.906
BLEU-20.881
BLEU-30.863
BLEU-40.847
ROUGE Scores:
TABLE IV: Text Generation Quality Assessment
Fig. 3: Sample multi-task performance of the model in a test scenario. The specific sample is presented to show the real evaluation of the model, where the generated text does not perfectly match the ground truth description.
More in this topic

Surgical Video Understanding

Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.

See all 118 papers

Explore similar work

CardsList
Feb 5, 2026

SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos

While foundation models have advanced surgical video analysis, current approaches rely predominantly on pixel-level reconstruction objectives that waste model capacity on low-level visual details, such as smoke, specular reflections, and fluid motion, rather than semantic structures essential for surgical understanding. We present SurgMotion, a video-native foundation model that shifts the learning paradigm from pixel-level reconstruction to latent motion prediction. Built on the Video Joint Embedding Predictive Architecture (V-JEPA), SurgMotion introduces three key technical innovations tailored to surgical videos: (1) motion-guided latent masked prediction to prioritize semantically meaningful regions, (2) spatiotemporal affinity self-distillation to enforce relational consistency, and (3) spatiotemporal feature diversity regularization (SFDR) to prevent representation collapse in texture-sparse surgical scenes. To enable large-scale pretraining, we curate SurgMotion-15M, the largest surgical video dataset to date, comprising 3,658 hours of video from 50 sources across 13 anatomical regions. Extensive experiments across 17 benchmarks demonstrate that SurgMotion significantly outperforms state-of-the-art methods on surgical workflow recognition, achieving 14.6 percent improvement in F1 score on EgoSurgery and 10.3 percent on PitVis; on action triplet recognition with 39.54 percent mAP-IVT on CholecT50; as well as on skill assessment, polyp segmentation, and depth estimation. These results establish SurgMotion as a new standard for universal, motion-oriented surgical video understanding.

Jinlin Wu, Felix Holm, Chuxi Chen +18

Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany +7

Joint-Embedding Predictive ArchitectureSurgical Video UnderstandingVideo Representation Learning
AskPDF
Aug 31, 2026

SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment

Objective assessment of surgical technical skill is important for surgical training and structured feedback, but current workflows remain dependent on labor-intensive expert review. Existing automated approaches primarily focus on visual inputs and provide limited support for jointly studying operative performance, structured skill scores, and evaluator feedback. We introduce SurgSkill-Bench, an initial video-score-text benchmark-style dataset containing 214 surgical training simulation videos, six-dimensional OSATS scores, and expert free-text comments. We define two evaluation settings: video-only OSATS prediction for automated assessment and post hoc expert-comment-assisted prediction, where evaluator comments are available as auxiliary information. We provide controlled baseline experiments using representative frozen visual backbones, content-adaptive key-frame sampling, and a simple video-text co-attention fusion module. Under internal video-level validation, content-adaptive sampling improves video-only performance in this dataset, while evaluator comments provide additional score-related signal in the assisted setting. The best mean AUROC reaches 0.88 under dataset-specific median dichotomization. We further discuss evaluation constraints related to dataset scale, metadata completeness, and the interpretation of comment-assisted prediction. Code will be released publicly at a later date.

Chaohui Dang, Zheheng Jiang, James Glasbey +3

Department of Electronic, Electrical and Systems Engineering, School of Engineering, University of Birmingham, Birmingham, UK · School of Computing and Mathematical Science, University of Leicester, Leicester, UK · Department of Applied Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK +1

Surgical Video UnderstandingMultimodal Model Evaluation
AskPDF
Mar 13, 2026

Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planning

Artificial intelligence, imaging, and large language models have the potential to transform surgical practice, training, and automation. Understanding and modeling of basic surgical actions (BSA), the fundamental unit of operation in any surgery, is important to drive the evolution of this field. In this paper, we present a BSA dataset comprising 10 basic actions across 6 surgical specialties with over 11,000 video clips, which is the largest to date. Based on the BSA dataset, we developed a new foundation model that conducts general-purpose recognition of basic actions. Our approach demonstrates robust cross-specialist performance in experiments validated on datasets from different procedural types and various body parts. Furthermore, we demonstrate downstream applications enabled by the BAS foundation model through surgical skill assessment in prostatectomy using domain-specific knowledge, and action planning in cholecystectomy and nephrectomy using large vision-language models. Multinational surgeons' evaluation of the language model's output of the action planning explainable texts demonstrated clinical relevance. These findings indicate that basic surgical actions can be robustly recognized across scenarios, and an accurate BSA understanding model can essentially facilitate complex applications and speed up the realization of surgical superintelligence.

Mengya Xu, Daiyun Shen, Jie Zhang +19

Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Biomedical Engineering, National University of Singapore, Singapore · Department of Surgery, The Chinese University of Hong Kong, Hong Kong, China +13

Surgical Video UnderstandingVideo Action Recognition
AskPDF