Purpose: Curating large-scale datasets of operating room (OR) workflow, encompassing rare, safety-critical, or atypical events, remains operationally and ethically challenging. This data bottleneck complicates the development of ambient intelligence for detecting, understanding, and mitigating rare or safety-critical events in the OR. Methods: This work presents an OR video diffusion framework that enables controlled synthesis of rare and safety-critical events. The framework integrates a geometric abstraction module, a conditioning module, and a fine-tuned diffusion model to first transform OR scenes into abstract geometric representations, then condition the synthesis process, and finally generate realistic OR event videos. Using this framework, we also curate a synthetic dataset to train and validate AI models for detecting near-misses of sterile-field violations. Results: In synthesizing routine OR events, our method outperforms off-the-shelf video diffusion baselines, achieving lower FVD/LPIPS and higher SSIM/PSNR in both in- and out-of-domain datasets. Through qualitative results, we illustrate its ability for controlled video synthesis of counterfactual events. An AI model trained and validated on the generated synthetic data achieved a RECALL of 70.13% in detecting near safety-critical events. Finally, we conduct an ablation study to quantify performance gains from key design choices. Conclusion: Our solution enables controlled synthesis of routine and rare OR events from abstract geometric representations. Beyond demonstrating its capability to generate rare and safety-critical scenarios, we show its potential to support the development of ambient intelligence models.
Figures & tables
Figure 1: abstract, geometry conditioned OR diffusion framework consists of three main modules: (i) Geometric Abstraction Module converts the initial OR scene into an abstract geometric scene representation using ellipsoids. (ii) Conditioning Module generates temporal sequences of abstract geometric scenes through two pathways: from routine OR events (blue dash path), or from incorporating user-defined trajectories (dashdotted green path). (iii) Diffusion Module synthesizes videos of OR events conditioned on the initial scene and the geometric sequences.
Figure 2: Geometric abstraction module pipeline: Given an initial scene and segmentation point prompts, SAM2 Ravi et al. (2024) propagates instance segmentation masks across video. Depth information is estimated using Video Depth Anything Chen et al. (2025a) . Each segmented instance is then approximated by an ellipsoid parameterized by its centroid position and spatial spread (height, width, rotation angle). The resulting Abstract Geometric Scene Representation encodes class information in the red and green channels, combined with normalized relative depth in the blue channel intensity.
Figure 3: Interactive conditioning module for counterfactual event generation. Given an input OR video sequence, the Abstraction module converts the scene into an abstract geometric representation. A graphical user interface enables direct manipulation of these ellipsoids through drag-and-drop operations to sketch desired trajectories. The Conditioning Module transforms the original geometric sequence into a counterfactual event by incorporating the user-modified trajectories.
Method
FVD ↓
SSIM ↑
PSNR ↑
LPIPS ↓
FVD ↓
SSIM ↑
PSNR ↑
LPIPS ↓
MMOR (In-Domain)
4DOR (Out-of-Domain)
WAN Wan et al. (2025)
1190.57
0.78
18.95
0.20
699.78
0.86
21.72
0.13
SVD Blattmann et al. (2023)
5021.19
0.68
17.91
0.24
3790.73
0.74
19.86
0.16
LTX b HaCohen et al. (2024)
2439.33
0.47
12.88
0.58
1135.26
0.46
13.10
0.58
Ours
689.88
0.86
23.21
0.13
265.25
0.90
25.87
0.07
Table 1: Comparison of our framework against out-of-the-box baseline models on in- and out-of-domain testsets. WAN Wan et al. (2025) & LTX-base (LTX b ) HaCohen et al. (2024) : Text-conditioned generation using VLM descriptions of the groundtruth scene. SVD Blattmann et al. (2023) : Image-to-video generation with low dynamic motion setting. Ours : Our proposed framework.
Figure 4: Qualitative comparison of video synthesis methods on out-of-domain (4DOR) dataset. Groundtruth : Original video frames to reconstruct. WAN Wan et al. (2025) & LTX-Base : Text-conditioned generation using VLM descriptions of the groundtruth scene. SVD : Image-to-video generation with low dynamic motion setting. Ours : Our proposed video synthesis using abstract geometric representation.
Figure 5: Controllable synthesis of safety-critical, interactions, and alternate OR events. Each column pair shows a routine OR event (left) with its abstract geometric representation (top), and a counterfactual event (right) generated by providing a trajectory for geometric conditioning. Left pair (safety-critical event): A non-sterile assistant approaches the sterile instrument table. Middle pair (Interaction): Personnel walking toward and reaching for interaction with the table. Right pair (Alternate event): Modified trajectory where personnel walks directly toward the patient bed instead of the original path around the room.
Table 7
Figure 6: Left: abstract geometric conditioning. Right: synthesized video frames. Positive (red) and negative (green) training samples for near safety-critical event (near misses in sterile-field violation) detection from counterfactual synthetic data generated using our framework. Two scenes demonstrate near-miss progressions: Scene 1 shows non-sterile personnel approaching then retreating from the instrument table. Scene 2 shows personnel passing close to the instrument table.
Method
FVD ↓
SSIM ↑
PSNR ↑
LPIPS ↓
BB IoU ↑
Seg IoU ↑
Base
Seg
E
D
Lg
✓
✓
347.88
0.88
25.34
0.09
0.96
0.95
✓
✓
518.50
0.86
23.65
0.12
0.93
0.90
✓
✓
✓
532.05
0.86
23.74
0.12
0.93
0.90
✓
✓
✓
✓
487.20
0.88
24.71
0.11
0.93
0.91
Table 4: Ablation study of our framework with and without base (diffusion backbone and baseline fine-tuning), Seg (segmentation mask-based conditioning), E (ellipsoids-based conditioning), D (depth encoded ellpsoids) and Lg (PatchGAN loss integrated for finetuning).
Generating high-fidelity and controllable synthetic data is critical for advancing end-to-end autonomous driving, particularly for addressing the long tail of rare safety-critical scenarios. Existing occupancy-guided methods typically rely on shallow conditioning mechanisms and reference-frame-dependent video synthesis, which limits fine-grained controllability from arbitrary BEV layouts and restricts their applicability for scalable simulation. In this paper, we propose AnyScene, a unified occupancy-centric framework for driving scene generation. AnyScene generates semantic occupancy sequences from BEV layouts through a Spatial-Temporal Occupancy Diffusion Transformer that jointly tokenizes BEV and occupancy features in an autoregressive manner. This design enables precise controllability from cross-dataset and user-defined BEV inputs while naturally supporting long-horizon generation. Building upon the generated occupancy, a Geometry-Grounded View Expansion module treats occupancy as the canonical spatial representation and synthesizes temporally consistent multi-view driving videos in a reference-free and autoregressive fashion, supporting flexible camera configurations at inference time. Extensive experiments demonstrate that AnyScene achieves state-of-the-art performance in both occupancy and video generation. It exhibits strong generalization to unseen and customized layouts, and provides measurable benefits for downstream tasks such as sparse-view 3D reconstruction.
Haiming Zhang, Junfei Zhou, Feng Jiang +6
Li Auto · Southwest Jiaotong University · Tsinghua University
Surgical video data provides the primary training resource for models of intraoperative perception, surgical workflow understanding, and robotic decision-making. However, clinical data acquisition remains constrained by privacy, cost, and class imbalance. Surgical video generation has emerged as a transformative approach to addressing data scarcity and as a foundation for surgical simulation, training, and robotic policy learning. The field has developed rapidly without a clear conceptual framework. This survey organizes the 2024-2026 literature into three categories: unconditional generation, conditional generation, and world modeling generation, revealing a fundamental shift in how the task is defined from synthesizing visually plausible frames to modeling the causal dynamics of surgical scenes. We examine the persistent gap between pixel-level fidelity and clinical plausibility, and identify generalization, physical realism, controllability, and interpretability as bottlenecks. We further summarize experimental results of representative methods on public datasets to provide a quantitative reference for the field. This survey provides a structured overview of the current state and open challenges, offering a reference for researchers working at the intersection of intelligent perception, multi-modal fusion, generative AI, and surgical data science.
Fuxiang Huang, Chenxu Zhang, Liang Han +1
School of Data Science Lingnan University Hong Kong, China · School of Microelectronics and Communication Engineering Chongqing University Chongqing, China
Text-to-video retrieval in operating rooms (OR) is an enabling technology for OR safety, as it allows stakeholders to retrieve and inspect recordings of specific events. However, because the most safety-critical events may not follow the common structure, to unlock its full potential text-to-video retrieval must be able to handle implicit queries that require reasoning to identify the right video (e.g., the step right before clipping). However, existing methods rely on global embeddings that cannot reason over such queries. We propose OR3, a text-to-video retrieval method that converts clips into action-driven digital twins (ActDTs), grouping concurrent subject-action-object triplets under non-overlapping temporal intervals. Moreover, rather than cross-modal matching through paired encoders, OR3 performs imagination-based retrieval where an LLM generates hypothetical ActDTs from queries. This enables intra-modal matching via a single encoder trained with ActDT-tailored hard negatives. Finally, evidence-grounded refinement revises imagined ActDTs based on discrepancies with top candidates to capture procedure-specific patterns. We construct a benchmark from MM-OR with 276 implicit queries across four reasoning categories over 386 clips from robotic knee procedures. OR3 achieves 57.6 R@1 and 77.3 R@5, outperforming the strongest baseline. These results demonstrate that OR3 enables fine-grained discrimination between visually similar OR video clips through temporal action reasoning.