Organizations: Interdisciplinary Program in Medical Informatics, Seoul National University College of Medicine, Seoul, Republic of Korea · Department of Transdisciplinary Medicine, Seoul National University Hospital, Seoul, Republic of Korea · Department of Surgery, Seoul National University Hospital, Seoul, Republic of Korea · YONSEI WITH-YOU Surgery Clinic, Seoul, Republic of Korea · Institute of Convergence Medicine with Innovative Technology, Seoul National University Hospital, Seoul, Republic of Korea · Department of Medicine, Seoul National University College of Medicine, Seoul, Republic of Korea · Seoul National University College of Medicine, Seoul, Republic of Korea
High-fidelity surgical video generation is increasingly important for surgical education, simulation, and downstream AI training, as conventional simulation pipelines are limited by cost, realism, and scenario diversity. Adapting generic text-to-video diffusion models to surgical procedures remains difficult, however, due to complex anatomical structures, intricate tool-tissue interactions, the inherently hierarchical nature of surgical workflows, and the scarcity of curated training data outside a few well-studied procedures. We propose a surgical video generation framework that combines hierarchical procedural annotation with curriculum transfer learning. Surgical procedures are decomposed into three levels (phase, task, and action) and a pretrained text-to-video diffusion model is progressively adapted from coarse to fine-grained surgical context, supported by a dynamic sampling strategy that handles variable-resolution and variable-length surgical clips. We instantiate the framework on Bilateral Axillo-Breast Approach robotic thyroidectomy, a procedure for which no curated benchmark currently exists. Relative to the preceding task-level model, the final action-level model reduces FVD by 20.1% and FID by 15.7%, and a matched-budget ablation indicates that these gains arise from the curriculum design rather than from additional training. A user study with six clinicians rates visual realism highly while highlighting remaining gaps in anatomical accuracy and clinical reliability. The proposed framework offers a step toward AI-driven surgical simulation in procedures where curated data are scarce.
Figures & tables
Figure 1: Schematic and example of the hierarchical annotation system for surgical video. BABA robotic thyroidectomy is divided into six Phases (MID, LAT, INF, BER, SUP, END), each further decomposed into Tasks and Actions.
Figure 2: Architecture of the surgical video generation model. The pipeline integrates a T5-XXL text encoder, the Open-Sora 3D VAE, and the Spatial-Temporal Diffusion Transformer (STDiT3).
Figure 3: Dynamic bucket sampling over the resolution–frame lattice. A clip Vi is routed to the highest-resolution valid bucket and accepted with probability keep_probB ; otherwise the next lower resolution is tried. The accepted clip is downsampled into a shape-aligned batch. Per-stage keep_prob (Fig. 4 ) acts as the curriculum control. Only a subset of buckets is shown. Based on the bucket mechanism of [ 52 ] .
Figure 4: Average bucket keep probability by resolution across the three curriculum stages. Stage 1 (Phase) heavily downsamples high-resolution buckets to emphasize global context, while Stages 2 (Task) and 3 (Action) retain high-resolution clips to capture fine visual detail.
Figure 5: Three-stage curriculum transfer learning pipeline. The text encoder and 3D VAE are frozen, and STDiT is fine-tuned progressively from Phase to Task to Action levels.
labeled at each level; “Train”: cases used after the test split.
Table 1: Dataset split and per-stage curriculum composition. “Annotated”: cases labeled at each level; “Train”: cases used after the 4-case test split. Clips, steps, and time refer to training.
Setting
Value
Backbone
Open-Sora v1.2 (STDiT3-XL/2)
Scheduler
rectified flow (rflow)
Batch size
1 per GPU
Precision
bf16
Distributed
DeepSpeed ZeRO Stage 2
Gradient checkpointing
yes
Table 2: Implementation details and training/evaluation environment. Optimization hyperparameters (learning rate, ϵ , warmup, EMA decay) are given in the text.
Model Stage
Steps
SSIM ↑
PSNR ↑
LPIPS ↓
FloLPIPS ↓
FVD ↓
FID ↓
Stage 2 (Task)
3k
0.307
9.77
0.717
0.726
4242.5
244.9
7k
0.285
9.64
0.744
0.739
4652.7
249.1
11k
0.264
10.18
0.680
0.709
5097.4
213.5
15k
0.285
10.19
0.719
0.722
4105.4
241.2
Stage 3 (Action)
0.4k
0.299
10.41
0.708
0.734
4665.6
233.6
0.8k
0.282
10.53
0.677
0.739
4608.1
198.5
Table 3: Per-stage quantitative evaluation of the curriculum transfer learning framework.
Schedule
SSIM ↑
PSNR ↑
LPIPS ↓
FloLPIPS ↓
FVD ↓
FID ↓
Stage 1→2→2 (matched budget)
0.263
9.83
0.696
0.721
3997.2
200.9
Stage 1→2→3 (Ours)
0.288
10.77
0.708
0.729
3278.4
203.4
Table 4: Ablation study on the effect of curriculum transfer learning. Both models share Stage 1 (55k) and Stage 2 (15k) and differ only in the final 1.2k steps.
Figure 6: Qualitative results across Stage 1 training steps (15k, 35k, 55k). Physical inconsistencies (e.g., extra robotic arms) progressively disappear and tissue textures become realistic.
Figure 7: Qualitative comparison between Stage 2 and Stage 3 on suturing and irrigation tasks. Stage 3 reproduces fine instruments (thread, needle) and fluid dynamics more faithfully.
Figure 8: User study results from six clinicians. The model performs best on imaging-related criteria but is more limited on anatomical accuracy and clinical reliability.
School of Data Science Lingnan University Hong Kong, China · School of Microelectronics and Communication Engineering Chongqing University Chongqing, China