Shifted Window Temporal Transformer for YOLO-Guided Phase Recognition in Laparoscopic Cholecystectomy Surgery

Surgical video phase recognition is a fundamental task in computer-assisted intervention, supporting workflow understanding, intraoperative guidance, and surgical quality assessment. Although recent visual-temporal models have achieved promising progress, accurate and temporally coherent phase recognition remains challenging due to local visual ambiguity, transient prediction noise, and insufficient use of procedural semantics. To address these challenges, we propose HTT-Net, a Hierarchical Text-guided Transition modeling Network for surgical video phase recognition. The key idea is to introduce structured surgical semantic knowledge into phase-aware segment construction and semantic refinement. Specifically, we construct a hierarchical surgical semantic memory with intra-phase descriptions, inter-phase transition descriptions, and fine-grained semantic units. Based on this memory, the proposed Transition-Aware Segment Construction (TAS-Con) organizes frame-level evidence into coherent segment representations and handles boundary clips with inter-phase transition descriptions. Furthermore, we introduce Transition-Aware Segment Calibration (TAS-Calib), which calibrates phase-aware segment representations through hierarchical surgical semantics and improves discrimination under visual ambiguity without dense frame-level vision-language fusion. Experiments on Cholec80 and LCRS-100 demonstrate the effectiveness of HTT-Net for robust surgical video phase recognition.

Reliable workflow transition detection is important for context-aware surgical assistance and downstream decision support. However, online surgical phase recognizers primarily focus on frame-wise accuracy and temporal consistency, rather than the reliability of workflow transition events. Directly converting phase changes into events is unreliable: temporal jitter and workflow-illegal switches produce false or duplicate events, while persistent, workflow-consistent candidates may remain incorrect. To address this limitation, we formulate reliable workflow transition detection as a distinct event-level task operating on outputs of a frozen online phase recognizer. We propose the Two-Stage Transition Detection Framework (TTDF), a causal framework that progressively filters transition candidates. Transition Candidate Extraction (TCE) first applies a minimum-duration requirement and a workflow-graph constraint to remove false candidates caused by temporal jitter and phase transitions not allowed by the workflow graph. Specifically, a candidate is retained only if the predicted target phase persists for a minimum duration and the ordered phase pair belongs to the workflow graph's allowed transition set. TCE thereby produces a high-recall candidate set without additional training. Transition Candidate Verification (TCV) suppresses remaining false candidates using phase-posterior shifts and visual-change cues from frozen DINOv2 features. Events are assessed using a phase-pair-aware one-to-one matching protocol. Experiments on Cholec80 show that TTDF reduces false transition emissions while preserving recall and controlling decision delay.

Minimally invasive colorectal surgeries (MIS-CRS) are characterised by significant variability and inconsistent outcomes. ColoWorkflow, a tool for the video-based assessment (VBA) of MIS-CRS workflow, was recently validated. However, manual VBA is time-consuming, limiting implementation. This study presents AI-ColoWorkflow, a deep learning model for automated surgical workflow analysis across MIS-CRS. Operative videos of MIS-CRS were collected from 4 centres and a publicly available dataset. Phases and steps were manually annotated according to ColoWorkflow. A deep learning model combining a fine-tuned DINOv3 vision transformer for per-frame visual feature extraction with a hierarchical multi-stage temporal convolutional network was jointly optimized for phase and step recognition. The model trained on pooled multicentric data, namely AI-ColoWorkflow was compared against centre-specific and procedure-specific models on a held-out test set. The following metrics were used for evaluation: macro F1 score, balanced accuracy, precision, and recall. AI-ColoWorkflow achieved a macro F1 of 73.01% $\pm$ 10.27 (balanced accuracy 73.43%) for phase recognition and 39.82% $\pm$ 7.06 (balanced accuracy 38.65%) for step recognition. The global model outperformed centre- and procedure-specific models in most experiments except procedure-specific step recognition. In the generalization analysis, mean F1 was 48.42% for phase recognition. AI-ColoWorkflow can reliably recognize MIS-CRS phases. A single model trained on pooled, multicentric, multi-procedural data generalises at least as well as and often better than centre- or procedure-specific models for phase recognition in MIS-CRS, while procedure-specific step models retain advantages for certain procedure types, motivating hybrid training strategies for future surgical AI development.