Beyond Masks and Trajectories: Flow-Guided Latent Action Injection for Stable Surgical Video Generation

Tsz-Yui Qin; Siyu Zhou; Chi-Keung Tang; Yuxiang Nie; Shu Yang

cs.CVOct 7, 2026

Beyond Masks and Trajectories: Flow-Guided Latent Action Injection for Stable Surgical Video Generation

Authors: Tsz-Yui Qin, Siyu Zhou, Chi-Keung Tang, Yuxiang Nie, Shu Yang

Organizations: The Hong Kong University of Science and Technology

Abstract

Surgical video generation holds substantial potential for surgical education, simulation, and data augmentation, yet generating surgical videos with realistic and clinically plausible motion remains challenging. Most existing methods rely on auxiliary conditions, such as masks, trajectories, depth, or reference videos, to achieve visually plausible synthesis. Yet, these auxiliary conditions typically require additional manual annotation or specialized acquisition, making it difficult to scale such methods beyond small, curated datasets. This motivates the need for a reference-free architecture capable of generating high-quality surgical video without requiring auxiliary visual conditions at inference time. We propose FLAIR, a Flow-guided LatentAction Injection framework for Reference-free surgical video generation. FLAIR learns action priors from optical flow of real surgical videos, dynamically predicts corresponding latent action representation from an input prompt, and injects it into a frozen base model to generate surgical videos with improved action consistency. We further construct SurgActionClip-30K, the first large-scale surgical vision dataset comprising action-centric segmented clips and structured caption labels, addressing the persistent lack of fine-grained, action-centric surgical datasets. Lastly, we introduce SurgMetrics, the first surgical domain-specific evaluation metrics for quantifying the quality of generated surgical videos, addressing the persistent absence of clinically grounded evaluation standards in this domain. Extensive experiments demonstrate that FLAIR enables generating high-quality surgical videos using text-only inference without auxiliary conditions, and validation in SurgMetrics demonstrates its strength in alignment with human perception compared to traditional metrics.

Figures & tables

Figure 1: Overview of FLAIR. During training, the FAE extracts flow-derived latent action sequences $Z$ , which supervise text-to-action prediction and adapter optimization. At inference, the FAE is removed, and the surgical prompt provides both the predicted motion prior $\widehat{Z}$ and the backbone-native text condition, enabling reference-free surgical video generation.

Figure 2: Overview of SurgMetrics. During inference, SurgMetrics extracts SurgViSTA representation and Farneback statistics from input videos to predict six artifact scores: “semantic implausibility, appearance artifact, drift, instrument deformation, tissue deformation, and tracklet instability”. Then, SurgMetrics compute the Domain Distance of input video independently by comparing its SurgViSTA features with a real surgical reference distribution to compute how much a video belongs to the surgical domain. SurgMetrics incurs low computational overhead at inference time.

Method	General Video Metrics					SurgMetrics
	V.CLIP $\uparrow$	FID $\downarrow$	FVD $\downarrow$	TI $\uparrow$	AF $\downarrow$	Sem $\downarrow$	App $\downarrow$	Drift $\downarrow$	Inst $\downarrow$	Tissue $\downarrow$	Track $\downarrow$	Dom $\downarrow$
Cosmos-H-Surgical	–	–	–	0.780	8.52	–	–	–	–	–	–	–
KVLR	–	–	–	0.893	4.48	–	–	–	–	–	–	–
Wan2.1 Base	8.80	246.8	3466	0.814	1.50	–	–	–	–	–	–	4.228
Wan2.1 LoRA	8.82	126.3	1340	0.877	0.11	0.097	0.132	0.027	0.059	0.085	0.100	0.931
Wan2.1 LoRA-FLAIR (ours)	9.08	117.2	1148	0.913	0.03	0.029	0.175	-0.005	0.027	-0.023	-0.001	1.165

Table 1: Comparison of LoRA-FLAIR on three base models with baselines and related works on general video metrics and SurgMetrics. LoRA has already achieved a greater level of TI and AF owing to the great preparation of our training datasets. LoRA-FLAIR further surpasses the result of LoRA on both general metrics and SurgMetrics. “ – ” denotes data not comparable or not available. “V.CLIP” denotes VideoCLIP [ 43 ] .

Figure 3: Qualitative temporal comparison between Wan2.1 LoRA and Wan2.1 LoRA-FLAIR motion conditioning, with key changes highlighted by bounding boxes. Under identical generation settings, Wan2.1 LoRA-FLAIR produces a more coherent motion trajectory and more sustained instrument–tissue interaction across the generated surgical video. Seven synchronized frames are shown from frame 0 to frame 80.

Latent Condition	Next-Frame MSE $\downarrow$
No latent	0.002521 (reference)
Zero latent	0.002525 $(+0.15\%)$
Shuffled FAE latent	0.002601 $(+3.17\%)$
FLAIR FAE latent	0.002356 $\mathbf{(-6.55\%)}$

Table 2: Next-frame prediction error under different FAE latent conditions. Percentages denote relative MSE changes compared with the no-latent baseline.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Overview of the SurgActionClip-30K construction pipeline. We first aggregate eleven surgical source datasets spanning laparoscopic, ophthalmic, robotic, and endoscopic domains. Action-centric clips are extracted using available temporal annotations, instrument-aware refinement, and duration-based filtering. Each clip is then paired with a structured action-aware caption generated by a medical VLM using a shared prompt template covering surgical context, instruments, actions, anatomical targets, tissue response, and imaging conditions. Finally, quality control and consistency checks are applied to assemble the unified SurgActionClip-30K corpus.

Source	Procedure or domain	Boundary supervision	Retained clips
CholecT50	Laparoscopic cholecystectomy	Instrument–verb–target triplets	3,773
M2CAI16	Laparoscopic cholecystectomy	Phase labels; detector refinement	3,086
AutoLaparo	Laparoscopic hysterectomy	Phase labels; detector refinement	3,416
SLAM	Laparoscopic cholecystectomy	Native action groups	221
PhaKIR	Laparoscopic surgery	Joint phase and instrument labels	1,268
SurgicalActions160	Laparoscopic action demonstrations	Native action classes	75

Appendix

Table 3: Compact summary of the eleven source datasets, their temporal supervision, and the number of clips retained in SurgActionClip-30K.

ID	Procedure	Count	Design
000–029	Cholecystectomy	30	10 actions $\times$ 3 variants
030–059	Gastrectomy	30	10 actions $\times$ 3 variants
060–089	Hysterectomy	30	10 actions $\times$ 3 variants
090–119	Intestinal Resection	30	10 actions $\times$ 3 variants
120–149	Nephrectomy	30	10 actions $\times$ 3 variants
150–179	Prostatectomy	30	10 actions $\times$ 3 variants

Appendix

Table 4: Distribution of the 200-prompt inference benchmark across procedures. The first 180 prompts (IDs 000–179) follow a strict 6-procedure

\times

10-action

\times

3-variant factorial design; the last 20 prompts (IDs 180–199) extend coverage to two external-validation procedures.

Figure 5: Qualitative temporal comparison between HunyuanVideo-1.5 LoRA and HunyuanVideo-1.5 LoRA-FLAIR, with key changes highlighted by bounding boxes. Under identical generation settings, HunyuanVideo-1.5 LoRA-FLAIR produces visually and anatomically more plausible surgical scenes with a consistently visible, stable instrument throughout the sequence, whereas the LoRA-only baseline exhibits drifting instrument motion and localized structural artifacts in later frames. Seven synchronized frames are shown from frame 0 to frame 80.

Figure 6: Qualitative temporal comparison between CogVideoX-2B LoRA and CogVideoX-2B LoRA-FLAIR, with key changes highlighted by bounding boxes. CogVideoX-2B LoRA-FLAIR maintains a grounded, physically anchored instrument with consistent visibility and a more stable camera view throughout the sequence, while the LoRA-only baseline shows the instrument intermittently disappearing, floating detached from tissue, and increasing viewpoint instability as generation progresses. Seven synchronized frames are shown from frame 0 to frame 80.

Figure 7: Qualitative visualization of the 21 controlled degradation families applied to a representative clean laparoscopic clip, grouped by their primary supervision axis (Drift, Appearance, Semantic, Instrument, and Tissue). Each tile shows three temporally ordered frames rendered at a fixed within-family severity of $s=0.75$ . Localized instrument and tissue degradations are shown with magnified regions of interest, while semantic action discontinuity uses a color-matched donor segment. Tracklet is not shown as a separate group because it receives auxiliary supervision from the instrument and tissue families rather than an exclusive degradation family.

Evaluation setting	$\rho_{\Delta}$ $\uparrow$	AUC $\uparrow$
Held-out cases (multi)	$0.566{\pm}0.004$	$0.914{\pm}0.004$
External (single)	$0.731{\pm}0.000$	$0.924{\pm}0.001$
External (multi)	$0.336{\pm}0.002$	$0.743{\pm}0.006$

Appendix

Table 5: Artifact sensitivity and external generalization of SurgMetrics. Values report mean

\pm

std over three seeds. External evaluation pools SLAM and PhaKIR; single and multi denote one and multiple simultaneous degradation families.

Dataset	Mean robust $z$ $\downarrow$	AUROC vs. in-domain $\uparrow$
In-domain unseen test	0.5614	0.5000
PhaKIR	1.8538	0.7646
SLAM	1.9552	0.7933
OphNet	3.6019	0.9547
SurgVU24	4.9917	0.9923
UCF101	5.8199	0.9957

Appendix

Table 6: Domain-distance reference values across datasets spanning in-domain, semi-in-domain, and out-of-domain video, using 100 clips per dataset. Higher values indicate greater deviation from the training laparoscopic distribution.

				Held-out		External
Configuration	Params	Test Acc $\uparrow$	Internal $\rho$ $\uparrow$	multi $\rho$ $\uparrow$	AUC $\uparrow$	single $\rho$ $\uparrow$	single AUC $\uparrow$
Full	354,502	0.8250	0.5436	0.5661	0.9144	0.7307	0.9243
SurgViSTA-only	345,478	0.7917	0.5144	0.5278	0.9029	0.5934	0.8556
Optical-only	92,102	0.6569	0.2779	0.2452	0.6376	0.5752	0.7729

Appendix

Table 7: Ablation over SurgMetrics input branches across adapter test, internal, held-out, and external evaluation cohorts. All values are averaged over seeds 42/43/44.

Configuration	Adapter test	Internal
	Drift Acc	Drift $\rho$
Full	0.9500	0.6106
SurgViSTA-only	0.7250	0.4375
Optical-only	0.5333	0.5235

Appendix

Table 8: Drift-axis performance under each ablation configuration, the axis most affected by the optical-flow leg.

Backbone	Params	Training budget	Resolution	Selected step	Adapter scale
Wan2.1-T2V-1.3B	1,904,896	1k / 1 GPU	480 $\times$ 832	500	0.30
HunyuanVideo 1.5	2,430,208	1k / 4 GPU (DDP)	320 $\times$ 576	1,000	0.50
CogVideoX-2B	1,576,576	1k / 8 GPU (DDP)	480 $\times$ 832	400	0.50 (LoRA 1/32)

Appendix

Table 9: Summary of adapter configurations and deployment settings across the three video backbones.

Parameter	Wan2.1-1.3B	HunyuanVideo 1.5	CogVideoX-2B
Resolution (W $\times$ H)	832 $\times$ 480	576 $\times$ 320	832 $\times$ 480
Frames	81	81	81
FPS	16	24	16
Denoising steps	20	30	50
Guidance scale	5.0	6.0	6.0
Flow shift	5.0	2.0	–

Appendix

Table 10: Inference configuration for the 200-prompt evaluation set across the three video backbones. Prompt set, frame count, and seed are shared; all other settings follow each backbone’s validated native configuration.

Degradation	Description	Head
A. Drift: temporal and global-motion degradations
local temporal rewind	Reverses or loops a local temporal segment (8%–38% of the clip), producing action rewind or repetition.	Drift
temporal shuffle	Swaps adjacent short temporal blocks while preserving intra-block motion order, disrupting local event ordering.	Drift
frame freeze drop	Replaces random frames with the preceding frame, producing freezing, duplication, and discontinuous jumps after drops.	Drift
motion jerk	Applies stride-and-hold sampling within a short segment, producing sudden acceleration, pauses, and jerks.	Drift
speed distortion	Applies non-linear temporal resampling across the clip, causing gradual speed-up or slow-down instead of uniform playback.	Drift

Appendix

Table 11: Definitions of the 21 controlled degradation families used to construct SurgMetrics training supervision, grouped by primary target axis.

Family	Sem.	App.	Drift	Instr.	Tissue	Track.
local temporal rewind	0.35	–	1.00	–	–	–
temporal shuffle	–	–	0.75	–	–	–
frame freeze drop	–	–	0.80	–	–	–
motion jerk	–	–	0.80	–	–	–
speed distortion	–	–	0.80	–	–	–
global jitter	–	–	0.90	–	–	–

Appendix

Table 12: Routing-weight matrix from the 21 degradation families to the six SurgMetrics artifact heads. Weights denote label-routing strength used in the noisy-OR aggregation (Eq. 12 in the main paper).

Explore similar work

May 9, 2026cs.CV

From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation

Action-conditioned surgical video generation is a critical yet highly challenging problem for robotic surgery. The core difficulty is that low-dimensional control vectors must precisely govern complex image-space evolution. In this work, we propose a kinematic-to-visual lifting paradigm that converts articulated kinematics into a unified set of five image-aligned control modalities. Building on this representation, we introduce a hierarchically routed visual control framework that selectively activates the most relevant control modalities and motion scales. Instead of uniformly applying all control signals, our model performs hierarchical routing to dynamically allocate conditioning capacity. We further design kinematic-prior-guided routing loss functions to ensure physically meaningful, temporally stable, and efficient expert utilization. To improve efficiency, we propose a budgeted training and inference scheme that leverages routing-induced sparsity. By selectively discarding low-significance control pathways during training and execution, our approach enables adaptive computation that is complementary to standard distillation. We additionally construct a new benchmark with curated articulated annotations, obtained through human-in-the-loop semantic labeling and differentiable pose tracking, providing realistic supervision for action-conditioned surgical video generation. Extensive experiments demonstrate that our method consistently improves action faithfulness, visual fidelity, and cross-domain generalization over diverse baselines. Moreover, our efficient variant achieves substantial reductions in latency while maintaining strong control accuracy.

Bohan Li, Shuojue Yang, Baorui Peng +10

1SJTU · 2NUS · 4EIT +5

Surgical Videos Camera-Conditioned Video Generation Articulated Kinematics Surgical Robots Visuomotor Control

Ask PDF

May 11, 2026cs.CV

Hi-GaTA: Hierarchical Gated Temporal Aggregation Adapter for Surgical Video Report Generation

Automated, clinician-grade assessment reports for surgical procedures could reduce documentation burden and provide objective feedback, yet remain challenging due to the difficulty of aligning dense spatio-temporal video representations with language-based reasoning and the scarcity of high-quality, privacy-preserving datasets. To address this gap, we establish a benchmark comprising 214 high-quality simulated surgical videos paired with surgeon-authored evaluation reports. Building on this resource, we propose a Perception-Alignment-Reasoning framework for surgical video report generation, featuring Hi-GaTA, a novel lightweight temporal adapter that efficiently compresses long video sequences into compact, LLM-compatible visual prefix tokens through short-to-long-range temporal aggregation. For robust visual perception, we pretrain Sur40k, a surgical-specific ViViT-style video encoder on 40,000 minutes of public surgical videos to capture fine-grained spatio-temporal procedural priors. Hi-GaTA employs a temporal pyramid with text-conditioned dual cross-attention, and improves multi-scale consistency through cross-level gated fusion and an increasing-depth strategy. Finally, we fine-tune the LLM backbone using LoRA to enable coherent and stylistically consistent surgical report generation under limited supervision. Experiments show our approach achieves the best overall performance, with consistent gains over strong Multimodal Large Language Model (MLLM) baselines. Ablation studies further validate the effectiveness of each proposed component.

Kedi Sun, Chaohui Dang, Yue Feng +3

School of Engineering, College of Engineering and Physical Sciences, University of Birmingham, Birmingham, UK · School of Computer Science, University of Birmingham, Birmingham, UK · Department of Applied Health Sciences, University of Birmingham, Birmingham, UK

Surgical Videos Video Temporal Grounding Long Visual-Token Sequences Spatial Priors Global Average Pooling

Ask PDF

May 8, 2026cs.CV

OphEdit: Training-Free Text-Guided Editing of Ophthalmic Surgical Videos

High-fidelity surgical video generation can greatly improve medical training and the development of AI, adapting these generative models for precise video editing remains a formidable challenge. Modifying surgical attributes, such as instrument tissue interactions or procedural phases is challenging due to the strict anatomical and temporal constraints. In this paper, we propose OphEdit, a novel training-free framework for the text-guided editing of ophthalmic surgical videos. Our approach leverages a deterministic second-order ODE inversion pipeline to capture Attention Value (V) tensors from the original video. By selectively injecting these stored tensors into the conditional Classifier-Free Guidance (CFG) branch during the denoising phase, OphEdit rigorously preserves the intricate anatomical geometry of the eye while seamlessly mapping text-driven semantic modifications onto the video stream. Clinical evaluations demonstrates that OphEdit effectively handles complex surgical transformations, such as instrument swaps and procedural variations, with superior structural fidelity and temporal consistency compared to natural-domain video editors. Our work represents the first application of training-free video editing in the ophthalmic surgical domain, offering a scalable solution for generating diverse, annotated medical datasets without the need for exhaustive manual recording or costly model fine-tuning. The code and prompts can be accessed at https://github.com/ophedit/OphEdit

Ritul Jangir, Arkya Jyoti Bagchi, Aiman Farooq +3

Indian Institute of Technology Jodhpur · All India Institute of Medical Sciences Delhi

Video Editing Surgical Videos Ophthalmology Novel Modular Learning Framework

Ask PDF