cs.CVMar 13, 2026

SAW: Toward a Surgical Action World Model via Controllable and Scalable Video Generation

Authors: Sampath Rapuri, Lalithkumar Seenivasan, Dominik Schneider, Roger Soberanis-Mukul, Yufan He, Hao Ding, Jiru Xu, Chenhao Yu, 

Organizations: Johns Hopkins University, Baltimore MD, USA · NVIDIA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A surgical world model capable of generating realistic surgical action videos with precise control over tool-tissue interactions can address fundamental challenges in surgical AI and simulation -- from data scarcity and rare event synthesis to bridging the sim-to-real gap for surgical automation. However, current video generation methods, the very core of such surgical world models, require expensive annotations or complex structured intermediates as conditioning signals at inference, limiting their scalability. Other approaches exhibit limited temporal consistency across complex laparoscopic scenes and do not possess sufficient realism. We propose Surgical Action World (SAW) -- a step toward surgical action world modeling through video diffusion conditioned on four lightweight signals: language prompts encoding tool-action context, a reference surgical scene, tissue affordance mask, and 2D tool-tip trajectories. We design a conditional video diffusion approach that reformulates video-to-video diffusion into trajectory-conditioned surgical action synthesis. The backbone diffusion model is fine-tuned on a custom-curated dataset of 12,044 laparoscopic clips with lightweight spatiotemporal conditioning signals, leveraging a depth consistency loss to enforce geometric plausibility without requiring depth at inference. SAW achieves state-of-the-art temporal consistency (CD-FVD: 199.19 vs. 546.82) and strong visual quality on held-out test data. Furthermore, we demonstrate its downstream utility for (a) surgical AI, where augmenting rare actions with SAW-generated videos improves action recognition (clipping F1-score: 20.93% to 43.14%; cutting: 0.00% to 8.33%) on real test data, and (b) surgical simulation, where rendering tool-tissue interaction videos from simulator-derived trajectory points toward a visually faithful simulation engine.

Figures & tables

Figure 1: a) At inference, SAW leverages a user-inputted 2D tool trajectory, language prompt, tissue affordance, and a background surgical scene, enabling realistic and controllable video diffusion. b) During training, normalized depth and RGB videos are encoded using a frozen VAE, while the RGB video embedding is then flattened and noised into a sequence of noisy tokens. c) A depth consistency loss ( LDC\mathcal{L}_{\text{DC}} ) is computed during training from the predicted masked depth video tokens solely using the denoised RGB video tokens.
ModelFVD ↓\downarrowCD-FVD ↓\downarrowSSIM ↑\uparrowPSNR ↑\uparrowLPIPS ↓\downarrow
WAN [ 21 ]439.60429.670.575015.820.4480
LTX b [ 7 ]319.37504.310.563015.070.4920
SurgSora [ 3 ]541.61546.820.416317.100.3825
Ours (SAW)224.28199.190.594817.360.4100
Table 1: Quantitative evaluation of our model performance in video generation against state-of-the-art models on a held-out test set. WAN [ 21 ] and LTX b [ 7 ] are used off-the-shelf whereas SurgSora [ 3 ] is fine-tuned on our training dataset.
Figure 2: Qualitative examples of generated surgical videos for (a) grasping and (b) dissecting actions. Red overlays indicate where a generated video lacks realism or becomes visually distorted in comparison to the ground truth video at the corresponding frame.
Ablated ComponentFVD ↓\downarrowCD-FVD ↓\downarrowLPIPS ↓\downarrowSSIM ↑\uparrowPSNR ↑\uparrow
SAW224.28199.190.41000.594817.36
Ablation on Conditioning signals
w/o Affordance223.78200.100.41010.594517.33
w/o First Frame1096.21338.750.62170.484812.50
w/o Language219.23204.890.40520.597717.47
w/o Trajectory278.26344.790.43250.581816.45
Table 2: Ablation study across conditioning modalities and loss components. Each row removes a single conditioning signal or loss term from the full model. Best results per metric are in bold .
ActionsSpatiotemporal CNN [ 17 ]ViT [ 4 ]
w/o Augwith Augw/o Augwith Aug
Clipping20.9343.1484.6283.64
Cutting0.008.3330.7747.06
Grasping58.5255.8774.1973.80
Dissection83.8784.2586.2788.26
Table 3: Per-class action recognition F1 score before and after video augmentation. Best results per action label are in bold .
Figure 3: a) Simulator-derived tool segmentations, tissue affordance, tool tip trajectory. b) Simulator-derived tools are overlayed on a background surgical scene and used as conditional inputs to our video diffusion model, allowing us to model both tool kinematics and background tissue deformation.
More in this topic

Trajectory-Guided Video Generation

See all 13 papers

Explore similar work

CardsList
Aug 26, 2026

Surgical Video Generation From Diffusion to World Models: A Survey

Surgical video data provides the primary training resource for models of intraoperative perception, surgical workflow understanding, and robotic decision-making. However, clinical data acquisition remains constrained by privacy, cost, and class imbalance. Surgical video generation has emerged as a transformative approach to addressing data scarcity and as a foundation for surgical simulation, training, and robotic policy learning. The field has developed rapidly without a clear conceptual framework. This survey organizes the 2024-2026 literature into three categories: unconditional generation, conditional generation, and world modeling generation, revealing a fundamental shift in how the task is defined from synthesizing visually plausible frames to modeling the causal dynamics of surgical scenes. We examine the persistent gap between pixel-level fidelity and clinical plausibility, and identify generalization, physical realism, controllability, and interpretability as bottlenecks. We further summarize experimental results of representative methods on public datasets to provide a quantitative reference for the field. This survey provides a structured overview of the current state and open challenges, offering a reference for researchers working at the intersection of intelligent perception, multi-modal fusion, generative AI, and surgical data science.

Fuxiang Huang, Chenxu Zhang, Liang Han +1

School of Data Science Lingnan University Hong Kong, China · School of Microelectronics and Communication Engineering Chongqing University Chongqing, China

Video GenerationVideo World ModelsConditional Generative ModelingPhysical Consistency in Video Generation
AskPDF
Aug 7, 2026

Surg-UniWorld: A Unified Surgical World Model with Multimodal Control Experts

Controllable surgical world models can provide a generative foundation for surgical artificial intelligence and simulation by synthesizing realistic instrument--tissue interactions. However, existing methods lack a unified multimodal control paradigm, while direct fusion of heterogeneous visual conditions often causes anatomical distortion, instrument appearance drift, and temporally inconsistent interactions. In this work, we propose {Surg-UniWorld}, a unified surgical world model with multimodal control experts. Surg-UniWorld first constructs a {Hierarchical Surgical Anchor} from first-frame appearance and hierarchical semantic masks to preserve persistent scene identity, anatomical organization, and interaction boundaries. {Anchor-Relative Modality Experts} then interpret edge, depth, and optical-flow evidence relative to the shared anchor, capturing complementary boundary, geometric, and motion information. A {Multimodal Control Expert} further performs contribution-preserving stage-wise composition of the activated modality increments and generates control hints for the Wan2.2 video diffusion backbone. To support multimodal surgical world modeling, we further construct Cholec80-SurgWAM, a benchmark for controllable surgical video generation. Extensive experiments demonstrate that Surg-UniWorld consistently outperforms existing controllable video generation methods and surgical world-model baselines in generation quality, temporal consistency, and multimodal controllability.

Rulin Zhou, Wanhao Liu, Guoheng Ma +8

Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR 999077, China · Shenzhen Loop Area Institute, Shen Zhen 518055, China · School of Electrical and Computer Engineering, Faculty of Engineering, the University of Sydney, Sydney, NSW 2006, Australia

Controllable Video GenerationTemporal Consistency in Video GenerationVideo World ModelsMultimodal Diffusion Models
AskPDF
Oct 7, 2026

Beyond Masks and Trajectories: Flow-Guided Latent Action Injection for Stable Surgical Video Generation

Surgical video generation holds substantial potential for surgical education, simulation, and data augmentation, yet generating surgical videos with realistic and clinically plausible motion remains challenging. Most existing methods rely on auxiliary conditions, such as masks, trajectories, depth, or reference videos, to achieve visually plausible synthesis. Yet, these auxiliary conditions typically require additional manual annotation or specialized acquisition, making it difficult to scale such methods beyond small, curated datasets. This motivates the need for a reference-free architecture capable of generating high-quality surgical video without requiring auxiliary visual conditions at inference time. We propose FLAIR, a Flow-guided LatentAction Injection framework for Reference-free surgical video generation. FLAIR learns action priors from optical flow of real surgical videos, dynamically predicts corresponding latent action representation from an input prompt, and injects it into a frozen base model to generate surgical videos with improved action consistency. We further construct SurgActionClip-30K, the first large-scale surgical vision dataset comprising action-centric segmented clips and structured caption labels, addressing the persistent lack of fine-grained, action-centric surgical datasets. Lastly, we introduce SurgMetrics, the first surgical domain-specific evaluation metrics for quantifying the quality of generated surgical videos, addressing the persistent absence of clinically grounded evaluation standards in this domain. Extensive experiments demonstrate that FLAIR enables generating high-quality surgical videos using text-only inference without auxiliary conditions, and validation in SurgMetrics demonstrates its strength in alignment with human perception compared to traditional metrics.

Tsz-Yui Qin, Siyu Zhou, Chi-Keung Tang +2

The Hong Kong University of Science and Technology

Video GenerationAction-Conditioned Video GenerationControllable Video GenerationVideo Generation Evaluation
AskPDF