Towards Controllable Video Synthesis of Routine and Rare OR Events

Dominik Schneider; Lalithkumar Seenivasan; Sampath Rapuri; Vishalroshan Anil; Aiza Maksutova; Yiqing Shen; Jan Emily Mangulabnan; Hao Ding

cs.CVFeb 24, 2026

Towards Controllable Video Synthesis of Routine and Rare OR Events

Authors: Dominik Schneider, Lalithkumar Seenivasan, Sampath Rapuri, Vishalroshan Anil, Aiza Maksutova, Yiqing Shen, Jan Emily Mangulabnan, Hao Ding,

Organizations: Johns Hopkins University, Baltimore, 21211, MD, USA. · Technical University Munich, Munich, 80333, BY, Germany. · Johns Hopkins Medical Institutions, Baltimore, 21287, MD, USA.

Abstract

Purpose: Curating large-scale datasets of operating room (OR) workflow, encompassing rare, safety-critical, or atypical events, remains operationally and ethically challenging. This data bottleneck complicates the development of ambient intelligence for detecting, understanding, and mitigating rare or safety-critical events in the OR. Methods: This work presents an OR video diffusion framework that enables controlled synthesis of rare and safety-critical events. The framework integrates a geometric abstraction module, a conditioning module, and a fine-tuned diffusion model to first transform OR scenes into abstract geometric representations, then condition the synthesis process, and finally generate realistic OR event videos. Using this framework, we also curate a synthetic dataset to train and validate AI models for detecting near-misses of sterile-field violations. Results: In synthesizing routine OR events, our method outperforms off-the-shelf video diffusion baselines, achieving lower FVD/LPIPS and higher SSIM/PSNR in both in- and out-of-domain datasets. Through qualitative results, we illustrate its ability for controlled video synthesis of counterfactual events. An AI model trained and validated on the generated synthetic data achieved a RECALL of 70.13% in detecting near safety-critical events. Finally, we conduct an ablation study to quantify performance gains from key design choices. Conclusion: Our solution enables controlled synthesis of routine and rare OR events from abstract geometric representations. Beyond demonstrating its capability to generate rare and safety-critical scenarios, we show its potential to support the development of ambient intelligence models.

Figures & tables

Figure 1: abstract, geometry conditioned OR diffusion framework consists of three main modules: (i) Geometric Abstraction Module converts the initial OR scene into an abstract geometric scene representation using ellipsoids. (ii) Conditioning Module generates temporal sequences of abstract geometric scenes through two pathways: from routine OR events (blue dash path), or from incorporating user-defined trajectories (dashdotted green path). (iii) Diffusion Module synthesizes videos of OR events conditioned on the initial scene and the geometric sequences.

Figure 2: Geometric abstraction module pipeline: Given an initial scene and segmentation point prompts, SAM2 Ravi et al. (2024) propagates instance segmentation masks across video. Depth information is estimated using Video Depth Anything Chen et al. (2025a) . Each segmented instance is then approximated by an ellipsoid parameterized by its centroid position and spatial spread (height, width, rotation angle). The resulting Abstract Geometric Scene Representation encodes class information in the red and green channels, combined with normalized relative depth in the blue channel intensity.

Figure 3: Interactive conditioning module for counterfactual event generation. Given an input OR video sequence, the Abstraction module converts the scene into an abstract geometric representation. A graphical user interface enables direct manipulation of these ellipsoids through drag-and-drop operations to sketch desired trajectories. The Conditioning Module transforms the original geometric sequence into a counterfactual event by incorporating the user-modified trajectories.

Method	FVD $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
MMOR (In-Domain)					4DOR (Out-of-Domain)
WAN Wan et al. (2025)	1190.57	0.78	18.95	0.20	699.78	0.86	21.72	0.13
SVD Blattmann et al. (2023)	5021.19	0.68	17.91	0.24	3790.73	0.74	19.86	0.16
LTX b HaCohen et al. (2024)	2439.33	0.47	12.88	0.58	1135.26	0.46	13.10	0.58
Ours	689.88	0.86	23.21	0.13	265.25	0.90	25.87	0.07

Table 1: Comparison of our framework against out-of-the-box baseline models on in- and out-of-domain testsets. WAN Wan et al. (2025) & LTX-base (LTX b ) HaCohen et al. (2024) : Text-conditioned generation using VLM descriptions of the groundtruth scene. SVD Blattmann et al. (2023) : Image-to-video generation with low dynamic motion setting. Ours : Our proposed framework.

Figure 4: Qualitative comparison of video synthesis methods on out-of-domain (4DOR) dataset. Groundtruth : Original video frames to reconstruct. WAN Wan et al. (2025) & LTX-Base : Text-conditioned generation using VLM descriptions of the groundtruth scene. SVD : Image-to-video generation with low dynamic motion setting. Ours : Our proposed video synthesis using abstract geometric representation.

Figure 5: Controllable synthesis of safety-critical, interactions, and alternate OR events. Each column pair shows a routine OR event (left) with its abstract geometric representation (top), and a counterfactual event (right) generated by providing a trajectory for geometric conditioning. Left pair (safety-critical event): A non-sterile assistant approaches the sterile instrument table. Middle pair (Interaction): Personnel walking toward and reaching for interaction with the table. Right pair (Alternate event): Modified trajectory where personnel walks directly toward the patient bed instead of the original path around the room.

Table 7

Figure 6: Left: abstract geometric conditioning. Right: synthesized video frames. Positive (red) and negative (green) training samples for near safety-critical event (near misses in sterile-field violation) detection from counterfactual synthetic data generated using our framework. Two scenes demonstrate near-miss progressions: Scene 1 shows non-sterile personnel approaching then retreating from the instrument table. Scene 2 shows personnel passing close to the instrument table.

Method					FVD $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	BB IoU $\uparrow$	Seg IoU $\uparrow$
Base	Seg	E	D	$L_{g}$	FVD $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	BB IoU $\uparrow$	Seg IoU $\uparrow$
✓	✓				347.88	0.88	25.34	0.09	0.96	0.95
✓		✓			518.50	0.86	23.65	0.12	0.93	0.90
✓		✓	✓		532.05	0.86	23.74	0.12	0.93	0.90
✓		✓	✓	✓	487.20	0.88	24.71	0.11	0.93	0.91

Table 4: Ablation study of our framework with and without base (diffusion backbone and baseline fine-tuning), Seg (segmentation mask-based conditioning), E (ellipsoids-based conditioning), D (depth encoded ellpsoids) and

L_{g}

(PatchGAN loss integrated for finetuning).

Controllable Video Generation

Controllable video generation steers how videos look, move, and change over time using user-provided instructions and structured controls.

See all 321 papers

AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond
May 25, 2026Haiming Zhang, Junfei Zhou, Feng Jiang +6Long-Horizon Video Generation 3D Scene Generation
Li Auto · Southwest Jiaotong University · Tsinghua University
Surgical Video Generation From Diffusion to World Models: A Survey
Aug 26, 2026Fuxiang Huang, Chenxu Zhang, Liang Han +1Video Generation Video World Models
School of Data Science Lingnan University Hong Kong, China · School of Microelectronics and Communication Engineering Chongqing University Chongqing, China
Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins
Jun 15, 2026Yiqing Shen, Hao Ding, Mathias UnberathSurgical Video Understanding Video Representation Learning
Johns Hopkins University, Baltimore, MD, USA

May 25, 2026J/K move · Enter open · S save

AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond

Haiming Zhang, Junfei Zhou, Feng Jiang, Jingzhong Li+5

Li Auto · Southwest Jiaotong University · Tsinghua University

Long-Horizon Video Generation 3D Scene Generation Controllable Video Generation Autonomous Driving Simulation Autonomous Driving Scenario Generation

Open paper AskPDF arXiv

Generating high-fidelity and controllable synthetic data is critical for advancing end-to-end autonomous driving, particularly for addressing the long tail of rare safety-critical scenarios. Existing occupancy-guided methods typically rely on shallow conditioning mechanisms and reference-frame-dependent video synthesis, which limits fine-grained controllability from arbitrary BEV layouts and restricts their applicability for scalable simulation. In this paper, we propose AnyScene, a unified occupancy-centric framework for driving scene generation. AnyScene generates semantic occupancy sequences from BEV layouts through a Spatial-Temporal Occupancy Diffusion Transformer that jointly tokenizes BEV and occupancy features in an autoregressive manner. This design enables precise controllability from cross-dataset and user-defined BEV inputs while naturally supporting long-horizon generation. Building upon the generated occupancy, a Geometry-Grounded View Expansion module treats occupancy as the canonical spatial representation and synthesizes temporally consistent multi-view driving videos in a reference-free and autoregressive fashion, supporting flexible camera configurations at inference time. Extensive experiments demonstrate that AnyScene achieves state-of-the-art performance in both occupancy and video generation. It exhibits strong generalization to unseen and customized layouts, and provides measurable benefits for downstream tasks such as sparse-view 3D reconstruction.

Abstract

Controllable Video Generation

Explore similar work

AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond

Surgical Video Generation From Diffusion to World Models: A Survey

Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins