Organizations: State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · Institute of Medical Artificial Intelligence, South China Hospital, Medical School, Shenzhen University, Shenzhen, Guangdong, China · Department of Urology, Peking University Third Hospital
Surgical procedures are inherently partially observable, requiring surgeons to continuously infer evolving anatomical dynamics and future surgical states beyond the current observation. However, existing self-supervised learning methods primarily learn from observed surgical videos through masked reconstruction or predicting representations within the camera's current field of view, lacking the capability to explicitly model the out-of-sight surgical dynamics. Inspired by latent predictive world models, we introduce SurgWorld, a video-native, self-supervised framework for surgical representation learning, which treats a specified spatial or temporal shift as a pseudo-action and learns to predict the target latent representation from a partial observation. Specifically, we propose Out-of-Sight Dynamics Modeling (OSDM) as the pretraining objective for predicting representations conditioned on relative spatio-temporal shifts. We further regularize the learned latent dynamics from OSDM with a Geometric Visual Similarity (GVS) objective by aligning patch-level relational structures between predicted and target representations, preserving spatial geometry and local visual coherence for dense perception. These two objectives are integrated through an agglomerative training framework that consolidates complementary supervisory signals from multiple teachers to learn a unified latent representation of surgical dynamics and visual geometry. Extensive transfer learning experiments on 17 surgical benchmarks show that SurgWorld consistently outperforms existing vision foundation models across surgical workflow recognition, fine-grained action understanding, and dense visual perception. We further conduct tailored qualitative and quantitative analyses, demonstrating that SurgWorld effectively captures surgical dynamics beyond the directly observed context through latent prediction.
Figures & tables
Fig. 1: (a) Motivation: From partial surgical observations, surgeons leverage anatomical knowledge, procedural experience, and mental simulation to infer out-of-sight surgical states across space and time, supporting surgical understanding and decision-making. (b) SurgWorld first predicts unobserved temporal or spatial surgical states from partial observations, and then constructs complete representations for temporal understanding and segmentation, whereas conventional surgical foundation models rely on fully observed inputs. (c) SurgWorld achieves state-of-the-art performance across all evaluated surgical benchmarks.
Fig. 2: Overview of the SurgWorld framework. Given a context clip x and a target clip y sampled from the same surgical video, SurgWorld encodes the context and predicts the target latent state conditioned on their relative shift Δx→y . Training is driven by two complementary objectives: Out-of-Sight Dynamics Modeling (OSDM) and Geometric Visual Similarity (GVS) loss. M denotes the masking strategy. For clarity, the framework is presented in a simplified form to highlight its key methodological innovations.
Dataset
Surgical Procedure
Data Statistics
Task
Train
Val+Test
All
Cholec80 ∗ [ 24 ]
Laparoscopic Cholecystectomy
40 (86,344)
35 (88,459)
75 (174,803)
Surgical Workflow Recognition
M2CAI16-Workflow ∗ [ 25 ]
Laparoscopic Cholecystectomy
27 (67,578)
14 (26,957)
41 (94,535)
Surgical Workflow Recognition
AutoLaparo ∗ [ 26 ]
Laparoscopic Hysterectomy
10 (40,211)
11 (39,479)
21 (79,690)
Surgical Workflow Recognition
PMLR50 [ 27 ]
Laparoscopic Liver Resection
35 (17,378)
15 (7,659)
50 (25,037)
Surgical Workflow Recognition
CATARACTS [ 28 ]
Cataract Surgery
25 (16,953)
25 (15,861)
50 (32,814)
Surgical Workflow Recognition
TABLE I: Overview and statistics of the surgical datasets used for SurgWorld pre-training and downstream evaluation. Video datasets are reported as the number of clips, with frame counts in parentheses, whereas image datasets are reported by the number of images. Datasets marked with * contribute their training splits to SurgWorld pre-training.
Models
Pretrain. data
Backbone
Cholec80 ∗
AutoLaparo ∗
M2CAI16 ∗
PMLR50 ∗
Cataract ∗
Cataract21 ∗
Cataract101 ∗
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
DINOv3
LVD-1689M
ViT-L
85.16
77.56
76.28
67.43
83.19
70.18
67.67
53.72
75.84
70.18
79.91
75.11
87.59
83.43
ViT-H
84.95
76.75
76.85
67.55
82.52
69.86
72.32
54.99
72.04
67.47
74.81
68.15
85.29
80.43
VideoMAEv2
7500h
ViT-L
80.30
72.84
69.88
58.73
75.27
62.24
75.29
56.62
71.26
63.65
83.44
73.73
85.44
79.44
ViT-G
76.69
68.20
61.38
52.18
71.18
59.11
72.88
52.24
73.85
64.81
60.46
53.38
78.94
71.13
SurgMotion
3658h † (1fps)
ViT-L
80.55
73.40
70.44
62.06
77.88
68.21
71.88
55.14
76.14
68.51
72.21
57.64
81.81
73.44
TABLE II: Performance on seven downstream surgical workflow recognition benchmarks using attentive probing. Results are reported in accuracy and F1 score. The best two results are bold-faced and underlined, respectively. ∗ denotes significant improvements over the strongest baseline on both metrics of the marked dataset under two-sided Wilcoxon signed-rank tests ( p<0.05 ). † denotes the inclusion of private data in pre-training.
Models
Backbone
Cholec80CVS ∗
EndoscapesCVS
SurgicalActions160
CholecT50 ∗
Acc
F1
Acc
F1
Acc
F1
AP i
AP v
AP t
AP iv
AP it
AP ivt
DINOv3
ViT-L
51.67
27.09
58.20
38.47
35.42
32.58
86.51
56.25
45.69
36.94
38.62
33.85
ViT-H
46.67
16.33
57.36
34.62
20.83
13.19
86.26
56.27
45.09
37.44
39.31
35.07
VideoMAEv2
ViT-L
45.00
29.87
52.81
29.55
58.33
57.44
80.90
50.39
42.77
32.11
32.19
28.46
ViT-G
49.17
35.38
54.69
25.85
27.08
23.01
69.68
44.94
41.73
27.38
28.86
27.07
SurgMotion
ViT-L
45.00
23.32
54.64
34.18
47.92
47.18
71.58
45.48
43.33
30.10
36.05
33.08
TABLE III: Performance on four surgical fine-grained action understanding benchmarks using attentive probing. mAP is reported for CholecT50 and Acc / F1 is reported for the rest of the datasets. The best two results are bold-faced and underlined, respectively.
Models
In-Distribution
Out-of-Distribution
ClinicDB
Kvasir
CVC-300
ColonDB
ETIS
CholecSeg8k
DINOv3
78.95
82.04
79.47
60.11
65.22
75.29
VideoMAE
71.18
74.94
71.06
50.83
51.80
69.49
SurgMotion
63.40
67.15
53.52
41.25
41.37
68.66
EndoViT
67.09
71.36
69.07
54.58
58.68
72.59
GSViT
34.85
46.80
20.70
24.74
19.51
42.90
TABLE IV: Performance on six surgical segmentation benchmarks (ID and OOD settings for polyp segmentation). Results are reported in Dice score.
Fig. 3: Qualitative comparison of semantic segmentation results on CholecSeg8k.
Fig. 4: Fine-tuning with 5%, 10%, 20%, 50% and 100% training data on Cholec80.
Fig. 5: UMAP visualizations of learned representations. Fine-tuning with only 10% of the labeled data yields seven clearly separated phase clusters.
Fig. 6: t-SNE visualization of frozen video representations colored by surgical procedure categories (left), and PCA projections of dense feature maps from representative surgical scenes (right).
Fig. 7: Future latent prediction under varying temporal shifts.
Fig. 8: Dense segmentation under varying spatial masking ratios.
Method
Recall@1 (%)
Recall@3 (%)
Recall@5 (%)
Random
1.56
4.68
7.81
w/o Shift Cond.
30.60 (29.38, 31.84)
52.23 (50.88, 53.46)
61.77 (60.66, 62.97)
SurgWorld
50.63 (49.21, 52.02)
73.36 (71.92, 74.60)
82.40 (81.22, 83.56)
TABLE V: Results of the pseudo-action sensitivity analysis. Recall values are reported in percentage, with 95% confidence intervals shown in parentheses.
Fig. 9: Visualization of SurgWorld’s prediction.
Pre-training Components
Cataract101
Cataract
Cataract21
M2CAI16
AutoLaparo
Cholec80
PMLR50
CholecT50
CholecSeg8k
OSDM_s
OSDM_t
GVS
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
AP ivt
Dice
83.25
77.83
70.06
64.02
73.21
67.71
82.31
72.50
77.13
66.64
85.98
78.78
72.53
55.95
35.67
66.65
✓
90.15
86.72
78.71
71.31
78.72
75.20
82.60
73.31
79.63
68.16
86.37
79.33
70.05
53.20
40.29
72.49
✓
✓
91.99
88.52
81.39
76.43
87.92
85.84
84.15
75.86
82.63
70.12
88.03
80.80
76.67
61.22
42.53
78.15
✓
✓
✓
92.68
90.98
83.02
78.20
91.94
90.13
84.49
77.06
83.04
72.62
89.42
81.64
82.31
67.68
44.32
80.36
TABLE VI: Ablation study on the contributions of individual pre-training components.
Pretraining stage
Temporal shift
Cataract
Cataract21
M2CAI16
AutoLaparo
Cataract101
Cholec80
PMLR50
CholecT50
CholecSeg8k
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
AP ivt
Dice
Stage 1
0–8
82.27
76.51
90.66
89.83
83.91
75.71
81.72
70.99
90.45
89.84
89.24
81.41
81.27
66.93
42.27
78.77
0–16
78.15
71.98
87.16
86.06
82.17
72.08
77.96
67.46
90.23
88.33
85.37
78.16
74.09
56.55
36.82
76.21
Stage 2
0–8
82.21
77.17
90.78
88.33
83.22
76.38
82.68
71.23
91.67
89.17
88.61
81.51
81.48
65.46
42.16
78.23
0–16
83.02
78.20
91.94
90.13
84.49
77.06
83.04
72.62
92.68
90.98
89.42
81.64
82.31
67.68
44.32
80.36
0–32
82.05
77.09
90.80
88.13
83.88
74.67
82.36
71.96
92.10
89.99
87.58
80.40
80.61
65.14
42.96
77.97
TABLE VII: Ablation study on the effects of temporal shift ranges across the two pre-training stages. Temporal shifts are measured in frames.
GVS Teacher
Cataract
Cataract21
M2CAI16
AutoLaparo
Cataract101
Cholec80
PMLR50
CholecT50
CholecSeg8k
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
AP ivt
Dice
VideoMAE
82.48
77.58
91.93
90.77
86.03
77.08
82.68
70.96
92.94
91.06
89.01
81.79
84.04
68.08
42.72
78.83
DINOv3
83.02
78.20
91.94
90.13
84.49
77.06
83.04
72.62
92.68
90.98
89.42
81.64
82.31
67.68
44.32
80.36
TABLE VIII: Ablation study on the effect of different GVS teachers
Extra GVS
GVS weight
Cataract
Cataract21
M2CAI16
AutoLaparo
Cataract101
Cholec80
PMLR50
CholecT50
CholecSeg8k
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
AP ivt
Dice
No
2.0
78.87
74.56
90.63
88.91
83.77
74.32
80.97
71.25
92.64
90.56
88.08
81.06
73.40
57.38
41.06
77.19
Yes
0.2
79.88
74.61
90.18
88.53
83.64
75.52
80.61
69.76
91.91
89.51
87.75
80.36
77.67
59.47
40.54
77.70
1.0
81.22
76.37
91.16
89.77
85.32
75.94
82.31
73.63
93.63
91.97
89.44
82.52
79.45
63.66
40.27
81.51
2.0
83.02
78.20
91.94
90.13
84.49
77.06
83.04
72.62
92.68
90.98
89.42
81.64
82.31
67.68
44.32
80.36
5.0
81.20
75.28
90.78
88.90
86.07
76.76
83.39
73.25
93.38
91.28
89.56
82.31
78.73
64.76
40.36
80.59
TABLE IX: Ablation study on GVS supervision configurations, including additional supervision on cross-view prediction branches and sensitivity to the GVS loss weight