Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planning

Mengya Xu; Daiyun Shen; Jie Zhang; Hon Chi Yip; Yujia Gao; Cheng Chen; Dillan Imans; Yonghao Long

cs.CVMar 13, 2026

Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planning

Authors: Mengya Xu, Daiyun Shen, Jie Zhang, Hon Chi Yip, Yujia Gao, Cheng Chen, Dillan Imans, Yonghao Long,

Organizations: Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Biomedical Engineering, National University of Singapore, Singapore · Department of Surgery, The Chinese University of Hong Kong, Hong Kong, China · Division of Hepatobiliary and Pancreatic Surgery, Department of Surgery, National University Hospital, Singapore · iHealthTech, National University Singapore, Singapore · Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong, China · Department of Urology, Peking University Third Hospital, Beijing, China · Department of Hepatobiliary and Pancreatic Surgery, Guangxi Medical University Cancer Hospital, Guangxi Medical University, Nanning, China · Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, China · Global College, Shanghai Jiao Tong University, Shanghai, China · Department of Thoracic Surgery, Shenzhen People’s Hospital (The Second Clinical Medical College, Jinan University; The First Affiliated Hospital, Southern University of Science and Technology), Shenzhen, Guangdong, China · Division of Urology, Department of Surgery, The Chinese University of Hong Kong, Hong Kong, China · Division of General Surgery (Endocrine and Thyroid Surgery), Department of Surgery, National University Hospital, Singapore · Johns Hopkins University, Baltimore, United States · NVIDIA Corporation · Department of Electrical and Computer Engineering, National University of Singapore, Singapore

Abstract

Artificial intelligence, imaging, and large language models have the potential to transform surgical practice, training, and automation. Understanding and modeling of basic surgical actions (BSA), the fundamental unit of operation in any surgery, is important to drive the evolution of this field. In this paper, we present a BSA dataset comprising 10 basic actions across 6 surgical specialties with over 11,000 video clips, which is the largest to date. Based on the BSA dataset, we developed a new foundation model that conducts general-purpose recognition of basic actions. Our approach demonstrates robust cross-specialist performance in experiments validated on datasets from different procedural types and various body parts. Furthermore, we demonstrate downstream applications enabled by the BAS foundation model through surgical skill assessment in prostatectomy using domain-specific knowledge, and action planning in cholecystectomy and nephrectomy using large vision-language models. Multinational surgeons' evaluation of the language model's output of the action planning explainable texts demonstrated clinical relevance. These findings indicate that basic surgical actions can be robustly recognized across scenarios, and an accurate BSA understanding model can essentially facilitate complex applications and speed up the realization of surgical superintelligence.

Figures & tables

Figure 1: Illustration of our BSA-10 dataset. a Dataset generation pipeline. The process involves four key steps: BSA criteria establishment, reuse of existing annotations, annotation of new BSA clips, and overall dataset review; b Our dataset is collected from $6$ body parts (i.e., gallbladder, stomach, kidney, intestine, prostate gland, and uterus) and sourced from $15$ public datasets and our SurgYT collection (surgical videos from credentialed YouTube channels); c Description of basic surgical actions; d Statistical analysis of the dataset across different basic surgical actions and procedure types.

Figure 2: Analysis results of 10-fold cross-validation on the developed dataset. a The receiver operating characteristic (ROC) curve of ten action classes; b Results of the model on ten action classes based on the Youden Index which are presented as 95% confidence interval; c The confusion matrix across ten action classes by aggregating the individual confusion matrices from each of the ten folds; d Surgery-wise performance metrics, displayed with 95% confidence intervals; e Representative frames from ten basic surgical actions across eight surgery types, highlighting intra-class variability.

Figure 3: Action barcode visualization of BSA distributions across expertise levels . The skill analysis for the three surgical procedures conducted by the a experienced consultant, b the early consultant, and c the junior registrar in RARP50. The action barcode delineates three BSAs, namely needle grasping, needle puncture, and suture pulling, represented by distinct colors: purple, blue, and orange, respectively. d Quantitative comparison of multiple-attempt frequency and idle-state proportion across three surgical expertise levels.

Figure 4: BSA for surgical action planning with our AI agent. a Our AI Agent integrates surgical context knowledge, historical data, and current observations as inputs, generating a response upon receiving a user prompt. b Our AI Agent’s reasoning process through two representative examples from the C-CVS scenario. c Next action prediction results. We report the local and global accuracy under both strict and relaxed conditions for the C-CVS and N-RCS scenarios.

Figure 5: Multi-national surgeon evaluation of AI-generated surgical reasoning and action recommendations. a Comprehensive scoring evaluation process by surgeons across five clinical criteria, including terminology correctness, description correctness, description completeness, safety considerations, and next action reasonableness. b Quantitative surgeon scoring results from Hong Kong and Singapore surgeons across 98 samples, demonstrating international consensus on AI output quality. c Representative examples of detailed surgeon scoring with specific feedback highlighting clinical judgment variations and the importance of flexible evaluation metrics in surgical AI systems.

Figure 6: BSA foundation model. a Each frame from the action clip is divided into patches, which are subsequently transformed through linear embedding. These flattened patches are then fed into the video transformer encoder, followed by processing through a dual-head architecture. The dual head comprises an MLP head and a newly proposed imbalance compensation head. The latter is designed to refine class predictions by applying weights to the output probabilities generated by the MLP, thereby addressing disparities in class distributions. The video self-attention blocks involve sequentially applied temporal and spatial attention layers that perform self-attention over a defined spatiotemporal region of frame-level patches. We employ residual connections to integrate information across different attention layers within each block. At the end of each block, a single-hidden-layer MLP is applied. The complete model is built by stacking these blocks sequentially. b Visualization of the space-time self-attention mechanism applied to a video example.

Surgery type	Dataset Name
Cholecystectomy	CholecT50 [ 33 ] , Cholec80 [ 50 ] , M2cai16-workflow [ 9 ] , Heichole [ 54 ] , SurgYT,
Nephrectomy	Endovis17 [ 3 ] , Endovis18 [ 3 ] , SurgT [ 10 ] , Nephrec9 [ 36 ] , SurgYT,
Gastrectomy	MultiBypass140 [ 38 ] , SurgYT
Hysterectomy	Surgicalactions160 [ 42 ] , Autolaparo [ 56 ] , SurgYT,
Prostatectomy	MesadReal [ 6 ] , SAR-RARP50 2022 [ 37 ] , GraSP [ 5 ] , SurgYT
Intestinal Resection	HeiCo [ 29 ] , SurgYT

Table S1 : Surgery types and the corresponding public datasets and YouTube video sources (SurgYT).

Action	Cholecystectomy	Gastrectomy	Hysterectomy	Intestinal Resection	Nephrectomy	Prostatectomy	Total
Aspiration	182	310	86	141	23	55	797
Clipping	668	27	12	10	103	132	952
Coagulation	205	280	26	69	23	32	635
Dissection	1562	2956	274	141	39	301	5273
Knot-tying	24	107	54	38	26	106	355
Needle Grasping	7	43	20	5	53	647	775

Table S2 : Surgery-action distribution summary across six surgery types.

	fold1	fold2	fold3	fold4	fold5	fold6	fold7	fold8	fold9	fold10	TOTAL
Aspiration	102	78	70	84	55	79	102	81	64	82	797
Clipping	116	128	83	97	84	96	76	80	100	92	952
Coagulation	81	98	38	126	30	90	41	41	57	33	635
Dissection	389	689	473	599	498	625	650	502	446	402	5273
Knot-tying	23	44	47	24	22	22	79	21	27	46	355
Needle Grasping	58	66	73	102	97	82	50	85	88	74	775

Table S3 : Distribution of BSA classes across ten-fold cross-validation splits for the BSA-10 dataset.

Surgery Type	fold1	fold2	fold3	fold4	fold5	fold6	fold7	fold8	fold9	fold10	TOTAL
Cholecystectomy	273	277	297	420	414	373	285	275	407	381	3402
Gastrectomy	410	482	432	568	290	492	571	429	411	365	4450
Hysterectomy	12	20	36	37	108	65	146	115	19	45	603
Intestinal Resection	66	88	25	29	19	137	174	18	5	15	576
Nephrectomy	83	52	21	16	16	27	44	90	60	58	467
Prostatectomy	223	520	255	381	226	190	114	142	191	175	2417

Table S4 : Per-fold sample distribution across surgery types.

Action	AUROC	Sensitivity	Specificity
Aspiration	90.62 (87.94, 93.31)	82.63 (75.75, 89.52)	90.45 (83.86, 97.04)
Clipping	89.63 (85.32, 93.93)	85.45 (75.19, 95.71)	88.06 (82.74, 93.39)
Coagulation	79.11 (75.14, 83.09)	80.00 (68.00, 92.00)	72.68 (62.01, 83.35)
Dissection	88.29 (85.62, 90.97)	85.13 (81.47, 88.78)	79.09 (75.63, 82.55)
Knot-tying	91.72 (87.12, 96.32)	85.00 (75.08, 94.92)	92.33 (87.73, 96.93)
Needle Grasping	92.96 (89.48, 96.43)	88.33 (80.29, 96.38)	94.00 (90.38, 97.62)

Table S5 : In lobectomy procedures, the model’s performance across ten basic surgical action classes is quantified by the Youden Index (95% CI).

Class	AUROC (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)
Aspiration	88.30 (87.29, 89.32)	74.32 (70.65, 78.00)	91.04 (87.47, 94.61)
Clipping	97.05 (96.24, 97.86)	96.77 (93.90, 99.65)	89.39 (86.35, 92.42)
Coagulation	95.07 (93.35, 96.79)	91.54 (88.41, 94.66)	93.86 (90.22, 97.50)
Dissection	94.68 (94.00, 95.36)	87.56 (83.98, 91.14)	91.64 (88.74, 94.54)
Knot-tying	95.41 (92.21, 98.61)	92.86 (85.63, 100.00)	91.97 (87.72, 96.23)
Needle Grasping	79.58 (73.90, 85.25)	57.50 (42.78, 72.22)	92.43 (85.98, 98.88)

Table S6 : In hepatectomy procedures, the model’s performance across ten basic surgical action classes is quantified by the Youden Index (95% CI).

Figure S1 : AI Agent’s comprehensive surgical reasoning example 1 from C-CVS scenario.

Figure S2 : AI Agent’s comprehensive surgical reasoning example 2 from C-CVS scenario.

Surgical Video Understanding

Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.

See all 118 papers

Scaling Video Pretraining for Surgical Foundation Models
Mar 31, 2026Sicheng Lu, Zikai Xiao, Jianhui Wei +7Surgical Video Understanding Video Representation Learning
The Johns Hopkins University, United States · Zhejiang University, China · Zhejiang University-University of Illinois Urbana-Champaign Institute, China +4
SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery
Jun 24, 2026Filippos Bellos, Andre S. Gala-Garza, Miaowei Wang +8Medical VQA Surgical Video Understanding
University of Michigan Ann Arbor, Michigan, USA · University of Edinburgh Edinburgh, United Kingdom · Vanderbilt University Medical Center Nashville, Tennessee, USA +3
SurgVLA-Bench: Towards Evaluating Vision-Language-Action Models for Laparoscopic Surgical Robotics
Jun 28, 2026Jiashuo Sun, Yue He, Wenxuan Liu +4Vision-Language-Action Models Robot Manipulation Benchmarks
the National Engineering Research Center of Robot Visual Perception and Control Technology, China · the School of Artificial Intelligence and Robotics, Hunan University, Hunan, China · the national graduate college for elite engineers, Hunan University, Hunan, China

Mar 31, 2026J/K move · Enter open · S save

Scaling Video Pretraining for Surgical Foundation Models

Sicheng Lu, Zikai Xiao, Jianhui Wei, Danyu Sun+6

The Johns Hopkins University, United States · Zhejiang University, China · Zhejiang University-University of Illinois Urbana-Champaign Institute, China+4

Surgical Video Understanding Video Representation Learning

Open paper AskPDF arXiv

Surgical video understanding is essential for computer-assisted interventions, yet existing surgical foundation models remain constrained by limited data scale, procedural diversity, and inconsistent evaluation, often lacking a reproducible training pipeline. We propose SurgRec, a scalable and reproducible pretraining recipe for surgical video understanding, instantiated with two variants: SurgRec-MAE and SurgRec-JEPA. We curate a large multi-source corpus of 10,535 videos and 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery. Building on this corpus, we develop a unified pretraining pipeline with balanced sampling and standardize a reproducible benchmark across 16 downstream datasets and four clinical domains with consistent data splits. Across extensive comparisons against SSL baselines and vision-language models, SurgRec consistently achieves superior performance across downstream datasets. In contrast, VLMs prove unreliable for fine-grained temporal recognition, exhibiting both performance gaps and sensitivity to prompt phrasing. Our work provides a reproducible, scalable foundation for the community to build more general surgical video models. All code, models, and data will be publicly released.

Abstract

Surgical Video Understanding

Explore similar work

Scaling Video Pretraining for Surgical Foundation Models

SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery

SurgVLA-Bench: Towards Evaluating Vision-Language-Action Models for Laparoscopic Surgical Robotics