Organizations: State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · Institute of Medical Artificial Intelligence, South China Hospital, Medical School, Shenzhen University, Shenzhen, Guangdong, China · Department of Urology, Peking University Third Hospital
Surgical procedures are inherently partially observable, requiring surgeons to continuously infer evolving anatomical dynamics and future surgical states beyond the current observation. However, existing self-supervised learning methods primarily learn from observed surgical videos through masked reconstruction or predicting representations within the camera's current field of view, lacking the capability to explicitly model the out-of-sight surgical dynamics. Inspired by latent predictive world models, we introduce SurgWorld, a video-native, self-supervised framework for surgical representation learning, which treats a specified spatial or temporal shift as a pseudo-action and learns to predict the target latent representation from a partial observation. Specifically, we propose Out-of-Sight Dynamics Modeling (OSDM) as the pretraining objective for predicting representations conditioned on relative spatio-temporal shifts. We further regularize the learned latent dynamics from OSDM with a Geometric Visual Similarity (GVS) objective by aligning patch-level relational structures between predicted and target representations, preserving spatial geometry and local visual coherence for dense perception. These two objectives are integrated through an agglomerative training framework that consolidates complementary supervisory signals from multiple teachers to learn a unified latent representation of surgical dynamics and visual geometry. Extensive transfer learning experiments on 17 surgical benchmarks show that SurgWorld consistently outperforms existing vision foundation models across surgical workflow recognition, fine-grained action understanding, and dense visual perception. We further conduct tailored qualitative and quantitative analyses, demonstrating that SurgWorld effectively captures surgical dynamics beyond the directly observed context through latent prediction.
Figures & tables
Fig. 1: (a) Motivation: From partial surgical observations, surgeons leverage anatomical knowledge, procedural experience, and mental simulation to infer out-of-sight surgical states across space and time, supporting surgical understanding and decision-making. (b) SurgWorld first predicts unobserved temporal or spatial surgical states from partial observations, and then constructs complete representations for temporal understanding and segmentation, whereas conventional surgical foundation models rely on fully observed inputs. (c) SurgWorld achieves state-of-the-art performance across all evaluated surgical benchmarks.
Fig. 2: Overview of the SurgWorld framework. Given a context clip x and a target clip y sampled from the same surgical video, SurgWorld encodes the context and predicts the target latent state conditioned on their relative shift Δx→y . Training is driven by two complementary objectives: Out-of-Sight Dynamics Modeling (OSDM) and Geometric Visual Similarity (GVS) loss. M denotes the masking strategy. For clarity, the framework is presented in a simplified form to highlight its key methodological innovations.
Dataset
Surgical Procedure
Data Statistics
Task
Train
Val+Test
All
Cholec80 ∗ [ 24 ]
Laparoscopic Cholecystectomy
40 (86,344)
35 (88,459)
75 (174,803)
Surgical Workflow Recognition
M2CAI16-Workflow ∗ [ 25 ]
Laparoscopic Cholecystectomy
27 (67,578)
14 (26,957)
41 (94,535)
Surgical Workflow Recognition
AutoLaparo ∗ [ 26 ]
Laparoscopic Hysterectomy
10 (40,211)
11 (39,479)
21 (79,690)
Surgical Workflow Recognition
PMLR50 [ 27 ]
Laparoscopic Liver Resection
35 (17,378)
15 (7,659)
50 (25,037)
Surgical Workflow Recognition
CATARACTS [ 28 ]
Cataract Surgery
25 (16,953)
25 (15,861)
50 (32,814)
Surgical Workflow Recognition
TABLE I: Overview and statistics of the surgical datasets used for SurgWorld pre-training and downstream evaluation. Video datasets are reported as the number of clips, with frame counts in parentheses, whereas image datasets are reported by the number of images. Datasets marked with * contribute their training splits to SurgWorld pre-training.
Models
Pretrain. data
Backbone
Cholec80 ∗
AutoLaparo ∗
M2CAI16 ∗
PMLR50 ∗
Cataract ∗
Cataract21 ∗
Cataract101 ∗
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
DINOv3
LVD-1689M
ViT-L
85.16
77.56
76.28
67.43
83.19
70.18
67.67
53.72
75.84
70.18
79.91
75.11
87.59
83.43
ViT-H
84.95
76.75
76.85
67.55
82.52
69.86
72.32
54.99
72.04
67.47
74.81
68.15
85.29
80.43
VideoMAEv2
7500h
ViT-L
80.30
72.84
69.88
58.73
75.27
62.24
75.29
56.62
71.26
63.65
83.44
73.73
85.44
79.44
ViT-G
76.69
68.20
61.38
52.18
71.18
59.11
72.88
52.24
73.85
64.81
60.46
53.38
78.94
71.13
SurgMotion
3658h † (1fps)
ViT-L
80.55
73.40
70.44
62.06
77.88
68.21
71.88
55.14
76.14
68.51
72.21
57.64
81.81
73.44
TABLE II: Performance on seven downstream surgical workflow recognition benchmarks using attentive probing. Results are reported in accuracy and F1 score. The best two results are bold-faced and underlined, respectively. ∗ denotes significant improvements over the strongest baseline on both metrics of the marked dataset under two-sided Wilcoxon signed-rank tests ( p<0.05 ). † denotes the inclusion of private data in pre-training.
Models
Backbone
Cholec80CVS ∗
EndoscapesCVS
SurgicalActions160
CholecT50 ∗
Acc
F1
Acc
F1
Acc
F1
AP i
AP v
AP t
AP iv
AP it
AP ivt
DINOv3
ViT-L
51.67
27.09
58.20
38.47
35.42
32.58
86.51
56.25
45.69
36.94
38.62
33.85
ViT-H
46.67
16.33
57.36
34.62
20.83
13.19
86.26
56.27
45.09
37.44
39.31
35.07
VideoMAEv2
ViT-L
45.00
29.87
52.81
29.55
58.33
57.44
80.90
50.39
42.77
32.11
32.19
28.46
ViT-G
49.17
35.38
54.69
25.85
27.08
23.01
69.68
44.94
41.73
27.38
28.86
27.07
SurgMotion
ViT-L
45.00
23.32
54.64
34.18
47.92
47.18
71.58
45.48
43.33
30.10
36.05
33.08
TABLE III: Performance on four surgical fine-grained action understanding benchmarks using attentive probing. mAP is reported for CholecT50 and Acc / F1 is reported for the rest of the datasets. The best two results are bold-faced and underlined, respectively.
Models
In-Distribution
Out-of-Distribution
ClinicDB
Kvasir
CVC-300
ColonDB
ETIS
CholecSeg8k
DINOv3
78.95
82.04
79.47
60.11
65.22
75.29
VideoMAE
71.18
74.94
71.06
50.83
51.80
69.49
SurgMotion
63.40
67.15
53.52
41.25
41.37
68.66
EndoViT
67.09
71.36
69.07
54.58
58.68
72.59
GSViT
34.85
46.80
20.70
24.74
19.51
42.90
TABLE IV: Performance on six surgical segmentation benchmarks (ID and OOD settings for polyp segmentation). Results are reported in Dice score.
Fig. 3: Qualitative comparison of semantic segmentation results on CholecSeg8k.
Fig. 4: Fine-tuning with 5%, 10%, 20%, 50% and 100% training data on Cholec80.
Fig. 5: UMAP visualizations of learned representations. Fine-tuning with only 10% of the labeled data yields seven clearly separated phase clusters.
Fig. 6: t-SNE visualization of frozen video representations colored by surgical procedure categories (left), and PCA projections of dense feature maps from representative surgical scenes (right).
Fig. 7: Future latent prediction under varying temporal shifts.
Fig. 8: Dense segmentation under varying spatial masking ratios.
Method
Recall@1 (%)
Recall@3 (%)
Recall@5 (%)
Random
1.56
4.68
7.81
w/o Shift Cond.
30.60 (29.38, 31.84)
52.23 (50.88, 53.46)
61.77 (60.66, 62.97)
SurgWorld
50.63 (49.21, 52.02)
73.36 (71.92, 74.60)
82.40 (81.22, 83.56)
TABLE V: Results of the pseudo-action sensitivity analysis. Recall values are reported in percentage, with 95% confidence intervals shown in parentheses.
Fig. 9: Visualization of SurgWorld’s prediction.
Pre-training Components
Cataract101
Cataract
Cataract21
M2CAI16
AutoLaparo
Cholec80
PMLR50
CholecT50
CholecSeg8k
OSDM_s
OSDM_t
GVS
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
AP ivt
Dice
83.25
77.83
70.06
64.02
73.21
67.71
82.31
72.50
77.13
66.64
85.98
78.78
72.53
55.95
35.67
66.65
✓
90.15
86.72
78.71
71.31
78.72
75.20
82.60
73.31
79.63
68.16
86.37
79.33
70.05
53.20
40.29
72.49
✓
✓
91.99
88.52
81.39
76.43
87.92
85.84
84.15
75.86
82.63
70.12
88.03
80.80
76.67
61.22
42.53
78.15
✓
✓
✓
92.68
90.98
83.02
78.20
91.94
90.13
84.49
77.06
83.04
72.62
89.42
81.64
82.31
67.68
44.32
80.36
TABLE VI: Ablation study on the contributions of individual pre-training components.
Pretraining stage
Temporal shift
Cataract
Cataract21
M2CAI16
AutoLaparo
Cataract101
Cholec80
PMLR50
CholecT50
CholecSeg8k
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
AP ivt
Dice
Stage 1
0–8
82.27
76.51
90.66
89.83
83.91
75.71
81.72
70.99
90.45
89.84
89.24
81.41
81.27
66.93
42.27
78.77
0–16
78.15
71.98
87.16
86.06
82.17
72.08
77.96
67.46
90.23
88.33
85.37
78.16
74.09
56.55
36.82
76.21
Stage 2
0–8
82.21
77.17
90.78
88.33
83.22
76.38
82.68
71.23
91.67
89.17
88.61
81.51
81.48
65.46
42.16
78.23
0–16
83.02
78.20
91.94
90.13
84.49
77.06
83.04
72.62
92.68
90.98
89.42
81.64
82.31
67.68
44.32
80.36
0–32
82.05
77.09
90.80
88.13
83.88
74.67
82.36
71.96
92.10
89.99
87.58
80.40
80.61
65.14
42.96
77.97
TABLE VII: Ablation study on the effects of temporal shift ranges across the two pre-training stages. Temporal shifts are measured in frames.
GVS Teacher
Cataract
Cataract21
M2CAI16
AutoLaparo
Cataract101
Cholec80
PMLR50
CholecT50
CholecSeg8k
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
AP ivt
Dice
VideoMAE
82.48
77.58
91.93
90.77
86.03
77.08
82.68
70.96
92.94
91.06
89.01
81.79
84.04
68.08
42.72
78.83
DINOv3
83.02
78.20
91.94
90.13
84.49
77.06
83.04
72.62
92.68
90.98
89.42
81.64
82.31
67.68
44.32
80.36
TABLE VIII: Ablation study on the effect of different GVS teachers
Extra GVS
GVS weight
Cataract
Cataract21
M2CAI16
AutoLaparo
Cataract101
Cholec80
PMLR50
CholecT50
CholecSeg8k
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
AP ivt
Dice
No
2.0
78.87
74.56
90.63
88.91
83.77
74.32
80.97
71.25
92.64
90.56
88.08
81.06
73.40
57.38
41.06
77.19
Yes
0.2
79.88
74.61
90.18
88.53
83.64
75.52
80.61
69.76
91.91
89.51
87.75
80.36
77.67
59.47
40.54
77.70
1.0
81.22
76.37
91.16
89.77
85.32
75.94
82.31
73.63
93.63
91.97
89.44
82.52
79.45
63.66
40.27
81.51
2.0
83.02
78.20
91.94
90.13
84.49
77.06
83.04
72.62
92.68
90.98
89.42
81.64
82.31
67.68
44.32
80.36
5.0
81.20
75.28
90.78
88.90
86.07
76.76
83.39
73.25
93.38
91.28
89.56
82.31
78.73
64.76
40.36
80.59
TABLE IX: Ablation study on GVS supervision configurations, including additional supervision on cross-view prediction branches and sensitivity to the GVS loss weight
Scaling robot policy learning for autonomous surgery is challenging, as expert demonstrations are expensive and in vivo exploration poses substantial safety risks. Surgical world models address this by generating realistic, action-conditioned future frames from an initial observation, but existing methods exhibit two persistent failure modes: spatial interaction incoherence, where visible instrument contact fails to induce spatially consistent tissue deformation, and temporal fidelity collapse, where prediction errors compound across autoregressive rollouts and progressively corrupt visual quality. We present SurgVista, a surgical world model that mitigates both failures through two training recipes. Deformation Consistency Regularization extracts scene-point trajectories from training videos and enforces cross-frame coherence through latent contrastive learning, strengthening physically consistent instrument-tissue dynamics. Drift Adaptation Training mitigates long-horizon drift by perturbing conditioning frames with online prediction residuals and photometric augmentations calibrated to long-horizon drift statistics, sustaining visual fidelity over extended rollouts. To enable rigorous evaluation, we further introduce SurgWorld-Bench, featuring diverse procedure types, long-range rollouts, and decoupled metrics for instrument-motion accuracy and tissue-response fidelity. Extensive experiments show that SurgVista consistently outperforms state-of-the-art methods across visual quality, temporal consistency, and interaction fidelity, with gains widening as the prediction horizon grows.
Wentao Pan, Wuyang Li, Shengyuan Liu +3
The Chinese University of Hong Kong · EPFL · Imperial College London
Surgical world models have the potential to provide intra-operative guidance, accelerate risk-free training and facilitate transfer of expert surgical maneuvers. However, the use of world models for transferring surgical expertise has been largely unexplored. Moreover, recent approaches in this domain have mainly relied on ground truth kinematics actions or expert annotations for specific tasks, limiting their applicability beyond the narrow range of tasks with ground-truth annotations. In contrast, existing works in the natural domain have shown promising results in learning latent actions solely from unlabeled videos and transferring the learned actions across contexts. We observe that this paradigm is short-sighted for the surgical domain, failing to capture the hierarchical and goal-directed structure of surgical procedures. Thus, we propose a novel multi-step action learning framework to learn context-invariant latent actions that are easily transferable across various surgical contexts. We perform extensive experiments on the publicly available datasets, SurgToolLoc-2022 and PitVis-2023, verifying our approach with both quantitative metrics and qualitative evaluation by surgical experts. We have published our code and the implementation details at https://github.com/bhattarailab/Surgical-Vision-World-Model.
Saurabh Koju, Love Panta, Deepak Thapa +6
NAAMII · University of Aberdeen · Toshiba Europe +3
Learning reliable surgical manipulation policies is bottlenecked by the scarcity of action-labeled demonstrations: teleoperated surgical robot (e.g., dVRK) trajectories with synchronized kinematics are costly to collect, while surgical tasks demand precise contact handling, long-horizon reasoning, and bimanual coordination. Endoscopic video is comparatively inexpensive and abundant relative to synchronized video--kinematics trajectories, and a natural way to exploit it is to learn world models of surgical scenes. However, existing surgical world models use video primarily for simulation or policy evaluation, and rarely translate the learned dynamics into closed-loop control. This gap raises our central question: under a fixed budget of action-labeled demonstrations, does action-free video pretraining improve closed-loop surgical manipulation? To answer it, we introduce the Surgical World-Action Model (Surgical WAM), a unified generative model built on Cosmos Policy that jointly predicts future endoscopic observations and executable surgical robot action chunks. Surgical WAM first learns surgical visual dynamics from action-free video and is then fine-tuned on the fixed action-labeled budget; at deployment, it acts as a closed-loop, receding-horizon controller that executes a short prefix of each predicted action chunk and replans from the resulting observation. On a suite of four simulated surgical manipulation tasks, video pretraining improves the average success rate from 63.5% to 77.8%, including an absolute gain of 20 percentage points on PegTransfer, with the largest improvements on contact-rich and bimanual tasks. These results demonstrate that action-free video provides transferable visual dynamics priors for learning surgical robot control with limited action supervision, positioning data-efficient video pretraining as a practical path toward scaling up surgical robot learning.