SHANDS: A Multi-View Dataset and Benchmark for Surgical Hand-Gesture and Error Recognition Toward Medical Training

Le Ma; Thiago Freitas dos Santos; Nadia Magnenat-Thalmann; Katarzyna Wac

cs.CVMar 27, 2026

SHANDS: A Multi-View Dataset and Benchmark for Surgical Hand-Gesture and Error Recognition Toward Medical Training

Authors: Le Ma, Thiago Freitas dos Santos, Nadia Magnenat-Thalmann, Katarzyna Wac

Organizations: 1MIRALab · 2Quality of Life Technologies Lab (QoL Lab), University of Geneva

Abstract

In surgical training for medical students, proficiency development relies on expert-led skill assessment, which is costly, time-limited, difficult to scale, and its expertise remains confined to institutions with available specialists. Automated AI-based assessment offers a viable alternative, but progress is constrained by the lack of datasets containing realistic trainee errors and the multi-view variability needed to train robust computer vision approaches. To address this gap, we present Surgical-Hands (SHands), a large-scale multi-view video dataset for surgical hand-gesture and error recognition for medical training. SHands captures linear incision and suturing using five RGB cameras from complementary viewpoints, performed by 52 participants (20 experts and 32 trainees), each completing three standardized trials per procedure. The videos are annotated at the frame level with 15 gesture primitives and include a validated taxonomy of 8 trainee error types, enabling both gesture recognition and error detection. We further define standardized evaluation protocols for single-view, multi-view, and cross-view generalization, and benchmark state-of-the-art deep learning models on the dataset. SHands is publicly released to support the development of robust and scalable AI systems for surgical training grounded in clinically curated domain knowledge.

Figures & tables

Figure 1: The SHands multi-view dataset. A five-camera RGB setup (C1–C5) records synchronized multi-view videos of incision and suturing tasks on ex vivo tissue. The top row shows a gesture-annotated timeline with fine-grained labels (I1–I5 for incision, S1–S10 for suturing) and transition boundaries. The bottom rows show aligned frames from all views for both incision and suturing, highlighting the complementary spatial information captured across cameras.

Dataset	Hours	Participants	Views	Classes	Errors	Avg. V. Length (s)	Domain
Surgical Datasets
JIGSAWS [ 8 ]	3.5	8	1	15	–	120	Robotic bench-top
DESK [ 15 ]	5.2	11	1	7	–	-	Robotic bench-top
Cholec80 [ 9 ]	50	13	1	Phases only	–	2300	Endoscopic OR
M2CAI16 [ 10 ]	16	-	1	Phases only	–	-	Endoscopic OR
CATARACTS [ 16 ]	9	50	1	Steps/Tools	–	656	Microsurgery

Table 1: Comparison with existing surgical and multi-view action datasets. Our dataset is the first to provide synchronized multi-view open hand surgery videos with comprehensive gesture and error annotations.

Task	ID	Description
Incision	I1	Grasping scalpel and forceps
	I2	Positioning toward tissue site
	I3	Stabilizing surrounding tissue
	I4	Executing incision with blade
	I5	Retracting incised tissue
Suturing	S1	Grasping forceps and needle holder

Table 2: Surgical gesture taxonomy for incision and suturing. Each gesture represents a clinically meaningful primitive motion used in open-surgery training.

Task	ID	Error Description
Incision	II1	Improper instrument grip
	II2	Incorrect angle
	II3	Repetitive cutting
Suturing	IS1	Incorrect needle handling
	IS2	Excessive force application
	IS3	Faulty knot technique

Table 3: Common error taxonomy curated by surgical educators. Each category denotes a clinically relevant technical deviation observed during incision or suturing.

Figure 2: Overview of dataset composition and annotation coverage. The pie chart on the left illustrates the distribution of total recording time contributed by medical trainees (90.9%) and surgeons (9.1%). The middle plots report the proportion of annotated footage, showing 92% labeled coverage for surgeons and 55% for trainees. The bar plots on the right present gesture distribution for surgeons (top) and trainees (bottom), illustrating variability across incision (I1–I5) and suturing (S1–S10) categories, as well as error types (II1–II3, IS1–IS5).

Method	Pretrain Dataset	Top-1 (%)
CNN-based architectures:
R3D [ 62 ]	K400	$52.3\pm 0.8$
X3D-M [ 63 ]	K400	$53.9\pm 0.9$
SlowFast [ 39 ]	K400	$\textbf{55.7}\pm 1.2$
Transformer-based architectures:
TimeSformer [ 14 ]	K400	$58.4\pm 1.1$

Table 4: Single-view gesture classification on SHands. Models trained and evaluated on a single-view camera. K400 = Kinetics-400; Im21K = ImageNet-21K.

Method	Pretrain Dataset	Top-1 (%)	Macro-F1
MVAction [ 65 ]	NTU RGB 120	$70.4\pm 0.9$	0.704
ViewClr [ 66 ]	NTU RGB 120	$71.1\pm 0.7$	0.717
ViewCon [ 67 ]	NTU RGB 120	$72.5\pm 0.6$	0.723
DVANet [ 47 ]	NTU RGB 120	$\textbf{73.9}\pm 0.6$	0.736

Table 5: Multi-view gesture classification on SHands. Models trained and evaluated with 5 synchronized cameras following a cross-subject protocol.

Per-class accuracy of DAVNet (5 views):
Method	Camera Views	Top-1
VideoMAE-B	single-view	60.4
ViewCon [ 67 ]	All views	66.2
DVANet [ 47 ]	All views	68.5
II1 Improper instrument grip		77.3
II2 Incorrect angle		69.8

Table 6: Error detection on SHands student subset. The accuracy classification over eight error types.

Train: C1 + C2 + C3, Test: Same views
Configuration	Method	Top-1 (%)	Macro-F1
	ViewCon	68.4	0.662
	DVANet	72.8	0.706
Train: C1 + C2 + C3, Test: C4 + C5 (unseen)
	ViewCon	63.2	0.613
	DVANet	68.5	0.651

Table 7: Cross-view generalization on SHands. Models trained on cameras C1–C3 and evaluated on seen views (same cameras) versus unseen views (C4–C5). Retention ratio measures the preservation of performance across novel viewpoints.

Figure 3: Annotation quality analysis. Gesture probability distributions predicted by DVANet around two annotated boundary frames (61 and 109). Sharp probability transitions and narrow uncertainty regions indicate high temporal precision and consistency of the manual annotations, with strong classification confidence on either side of each gesture change.

Surgical Video Understanding

Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.

See all 118 papers

SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment
Aug 31, 2026Chaohui Dang, Zheheng Jiang, James Glasbey +3Surgical Video Understanding Multimodal Model Evaluation
Department of Electronic, Electrical and Systems Engineering, School of Engineering, University of Birmingham, Birmingham, UK · School of Computing and Mathematical Science, University of Leicester, Leicester, UK · Department of Applied Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK +1
Scaling Video Pretraining for Surgical Foundation Models
Mar 31, 2026Sicheng Lu, Zikai Xiao, Jianhui Wei +7Surgical Video Understanding Video Representation Learning
The Johns Hopkins University, United States · Zhejiang University, China · Zhejiang University-University of Illinois Urbana-Champaign Institute, China +4
ExpOS: Explainable Open-Surgery Skills Assessment Using 3D Hand Reconstruction
May 22, 2026Roi Papo, Idan Smoller, Shlomi LauferExplainable Artificial Intelligence Surgical Video Understanding
Faculty of Data and Decision Sciences, Technion - Israel Institute of Technology, Haifa, 3200003, Israel

Aug 31, 2026J/K move · Enter open · S save

SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment

Chaohui Dang, Zheheng Jiang, James Glasbey, David Luke+2

Department of Electronic, Electrical and Systems Engineering, School of Engineering, University of Birmingham, Birmingham, UK · School of Computing and Mathematical Science, University of Leicester, Leicester, UK · Department of Applied Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK+1

Surgical Video Understanding Multimodal Model Evaluation

Open paper AskPDF arXiv

Objective assessment of surgical technical skill is important for surgical training and structured feedback, but current workflows remain dependent on labor-intensive expert review. Existing automated approaches primarily focus on visual inputs and provide limited support for jointly studying operative performance, structured skill scores, and evaluator feedback. We introduce SurgSkill-Bench, an initial video-score-text benchmark-style dataset containing 214 surgical training simulation videos, six-dimensional OSATS scores, and expert free-text comments. We define two evaluation settings: video-only OSATS prediction for automated assessment and post hoc expert-comment-assisted prediction, where evaluator comments are available as auxiliary information. We provide controlled baseline experiments using representative frozen visual backbones, content-adaptive key-frame sampling, and a simple video-text co-attention fusion module. Under internal video-level validation, content-adaptive sampling improves video-only performance in this dataset, while evaluator comments provide additional score-related signal in the assisted setting. The best mean AUROC reaches 0.88 under dataset-specific median dichotomization. We further discuss evaluation constraints related to dataset scale, metadata completeness, and the interpretation of comment-assisted prediction. Code will be released publicly at a later date.

Abstract

Surgical Video Understanding

Explore similar work

SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment

Scaling Video Pretraining for Surgical Foundation Models

ExpOS: Explainable Open-Surgery Skills Assessment Using 3D Hand Reconstruction