Artificial Intelligence for Workflow Analysis in Colorectal Surgery: A Multicentric, Cross-Procedural Development and Generalization Study
Authors: Pietro Mascagni, Julia Alekseenko, Pooja P Jain, Marta Goglia, Andrea Balla, Ludovica Baldari, Gianfranco Silecchia, Claudio Fiorillo, Vincenzo Tondolo, Salvador Morales-Conde, Luigi Boni, Sergio Alfieri, Nicolas Padoy
Organizations: IHU Strasbourg, Strasbourg, France · Surgical Video Analysis, Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy · Università Cattolica del Sacro Cuore, Rome, Italy · University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France · Department of Medical-Surgical Sciences and Translational Medicine, Faculty of Medicine and Psychology, Sapienza University of Rome, Italy · Department of General & Minimally Invasive Surgery, Fondazione IRCCS - Ca’ Granda - Ospedale Maggiore Policlinico di Milano, Milan, Italy · University Hospital Virgen Macarena, University of Sevilla, Seville, Spain
Minimally invasive colorectal surgeries (MIS-CRS) are characterised by significant variability and inconsistent outcomes. ColoWorkflow, a tool for the video-based assessment (VBA) of MIS-CRS workflow, was recently validated. However, manual VBA is time-consuming, limiting implementation. This study presents AI-ColoWorkflow, a deep learning model for automated surgical workflow analysis across MIS-CRS. Operative videos of MIS-CRS were collected from 4 centres and a publicly available dataset. Phases and steps were manually annotated according to ColoWorkflow. A deep learning model combining a fine-tuned DINOv3 vision transformer for per-frame visual feature extraction with a hierarchical multi-stage temporal convolutional network was jointly optimized for phase and step recognition. The model trained on pooled multicentric data, namely AI-ColoWorkflow was compared against centre-specific and procedure-specific models on a held-out test set. The following metrics were used for evaluation: macro F1 score, balanced accuracy, precision, and recall. AI-ColoWorkflow achieved a macro F1 of 73.01% ± 10.27 (balanced accuracy 73.43%) for phase recognition and 39.82% ± 7.06 (balanced accuracy 38.65%) for step recognition. The global model outperformed centre- and procedure-specific models in most experiments except procedure-specific step recognition. In the generalization analysis, mean F1 was 48.42% for phase recognition. AI-ColoWorkflow can reliably recognize MIS-CRS phases. A single model trained on pooled, multicentric, multi-procedural data generalises at least as well as and often better than centre- or procedure-specific models for phase recognition in MIS-CRS, while procedure-specific step models retain advantages for certain procedure types, motivating hybrid training strategies for future surgical AI development.
Figures & tables
Fig. 1 : AI-ColoWorkflow system. Two sequential stages: (1) spatial feature extraction with a fine-tuned DINOv3 vision transformer; (2) temporal modelling with the SAHC hierarchical multi-stage TCN, jointly trained for phase and step recognition. d: feature dimension; SAHC: Segment-Attentive Hierarchical Consistency network; T: time; TCN: Temporal Convolutional Network
Centre
Surgery type
Train
Val
Test
Subtotal
Centre 1
LH
2
1
2
5
RH
1
0
0
1
Sub
3
1
2
6
Centre 2
LH
1
1
0
2
RH
3
1
1
5
RR
1
0
0
1
Table 1: Dataset splits by centre and procedure type. LH: Left hemicolectomy; RH: Right hemicolectomy; RR: Rectal resection; SR: Sigmoid resection; TPC: Total proctocolectomy.
Surgical workflow item
Prec. (%)
Rec. (%)
F1 (%)
Phase recognition
P1. Port placement and abdomen exploration
76.96 ± 16.02
69.36 ± 15.10
72.96 ± 10.68
P2. Vascular dissection and ligation, mesocolon/mesorectum dissection, additional lymphadenectomy
80.82 ± 13.39
72.77 ± 16.48
76.58 ± 12.87
P3. Colon and/or rectum mobilisation
69.83 ± 9.62
75.55 ± 16.64
72.58 ± 10.64
P4. Colorectal transection
64.07 ± 30.09
64.16 ± 20.42
64.11 ± 19.07
P5. Anastomosis
69.50 ± 27.02
82.08 ± 27.81
75.27 ± 23.85
Table 2: AI-ColoWorkflow performance across phases and steps. All values shown as global ± std, where global is computed on concatenated test frames and ± is the standard deviation of per-video scores across individual test videos. P7 (Preplanned additional procedures) and S25 (Ileoanal pouch preparation - intraabdominal part) were not evaluated since these were not represented in the test set.
Fig. 2 : Comparison of AI-ColoWorkflow performance versus procedure and centre-specific models. Numbers and *** in green suggest superiority of AI-ColoWorkflow while those in red suggest superiority of specific models. LH: Left hemicolectomy; RH: Right hemicolectomy; RR: Rectal resection; SR: Sigmoid resection; TPC: Total proctocolectomy.
Phase F1 (%)
Step F1 (%)
Held-out
Held-out
Seen
Held-out
Seen
HeiCo
43.83
65.10
16.74
40.44
Centre 1
43.46
60.63
18.57
39.28
Centre 2
37.49
60.05
23.38
42.24
Centre 3
51.56
60.62
18.11
36.67
Centre 4
65.78
61.55
32.91
44.42
Table 3: Leave-one-centre-out (LOCO) cross-validation results. Macro F1 (%) on the held-out centre and the remaining seen centres per fold, compared with the globally trained AI-ColoWorkflow model. All values are macro averaged.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Specification
Hardware
GPU
NVIDIA V100 (32 GB)
Spatial encoder
Architecture
DINOv3 vision transformer (fine-tuned)
Input resolution
224x224
Temporal model
Appendix
Table S1 : Hardware and training configuration.
Fig. S1 : Frame-level dataset composition by centre and surgery type within each of the training, validation, and test splits. (A) Donut chart showing overall train/validation/test frame proportions (462k total frames in total). (B) Stacked horizontal bars showing the percentage of frames contributed by each centre within each split. (C) Stacked bars showing the percentage of frames contributed by each surgery type within each split. Fr: frames; LH: Left hemicolectomy; RH: Right hemicolectomy; RR: Rectal resection; SR: Sigmoid resection; TPC: Total proctocolectomy.
Fig. S2 : Per-video duration variability by procedure type, centre, and split at 1 frame per second. Strip plots showing individual video frame counts grouped by (A) surgery type, (B) centre, and (C) split. Horizontal bars mark group means. LH: Left hemicolectomy; RH: Right hemicolectomy; RR: Rectal resection; SR: Sigmoid resection; TPC: Total proctocolectomy.
Fig. S3 : Normalised frame-level class distributions for phases and steps across splits. (A) Grouped bar chart of phase frame counts normalised by split total (% of split frames) for P1–P9; P7 (Preplanned additional procedures) is absent from the test set and was excluded from evaluation. (B) Grouped bar chart of normalised step frame counts (% of split frames) for all 33 evaluated steps (S25 is an unused gap), illustrating the long-tail imbalance that motivates macro-averaged F1 as the primary metric. Complete phases (P) and steps (S) definitions can be found in Table 2 .
Fig. S4 : Typical frames by centre and phase. Each cell shows a representative frame for the (centre, phase) pair. Phase 9 (Extracorporeal procedures) is blurred to comply with institutional privacy requirements; grey cells indicate no available frames.
Fig. S5 : Single-task versus joint training. Joint training improved macro F1 by +2.8% for phase (70.2% to 73.0%) and +1.7% for step (38.1% to 39.8%) relative to separately trained single-task models, with gains consistent across nearly all subgroups. (A) Phase recognition; (B) step recognition. Each row is a cohort (overall, 5 centres, 5 procedure types). Circles = single-task model F1; diamonds = joint model (AI-ColoWorkflow) F1. Green connectors indicate joint training improved performance; red indicate regression. Annotations show Δ F1 (joint − single-task).
Fig. S6 : Predictions for a representative video. (A) Ground-truth phase timeline; (B) predicted phase timeline, with model confidence (softmax maximum probability, black line); (C) ground-truth step timeline; (D) predicted step timeline with model confidence, grey regions indicate frames with no step ground-truth annotation; (E) per-video F1, Jaccard, balanced accuracy, precision, and recall.
Fig. S7 : Confusion matrices for phase (A) and step (B) recognition. Row-normalised matrices: each cell shows the fraction of true-class frames predicted as each target class; diagonal = correct prediction. The phase confusion matrix is computed from model outputs; the step confusion matrix is approximated from per-class recall and support (i.e., the number of samples). P8 (Unplanned procedures) shows the most diffuse phase confusion. Step classes with near-zero recall (S20, S23, S29, S32) appear as near-empty rows.
Fig. S8 : Phase and step F1 score summary. (A) Phase classes (n = 8 evaluated); (B) step classes (n = 33 evaluated), sorted by F1 in descending order. Bars are coloured by F1 tier (green ≥ 70%, blue 50–70%, orange 30–50%, red <30%). The dashed vertical line marks the macro mean.
Fig. S9 : AI-ColoWorkflow performance across evaluation cohorts. (A) Single-task (i.e., phase or step recognition) training: scatter of phase against step macro F1 score. (B) Joint (i.e., phase and step recognition) training: scatter of phase against step macro F1 score. For A and B, circles represent centres (sized by frame count) while squares represent procedure types, star represents overall. (C) Phase macro F1 versus log-transformed frame count per centre. (D) Step macro F1 versus log-transformed frame count per centre. For C and D, circles represent single-task training while diamonds represent joint training; areas are proportional to frame count.
Phase F1 (%)
Step F1 (%)
Cohort
Type
Global
Spec.
Δ
Global
Spec.
Δ
HeiCo
Centre
73.53
73.18
+0.4
33.96
35.42
− 1.5
Centre 1
Centre
59.19
50.11
+9.1
24.72
15.50
+9.2
Centre 2
Centre
72.78
53.86
+18.9
31.16
15.21
+15.9
Centre 3
Centre
79.34
80.31
− 1.0
38.86
22.82
+16.0
Centre 4
Centre
70.61
56.34
+14.3
44.81
21.44
+23.4
Appendix
Table S2 : AI-ColoWorkflow (global) versus procedure- and centre-specific models. Macro-averaged F1 score (%) on the test set. Δ = global - specialised. LH: Left hemicolectomy; RH: Right hemicolectomy; RR: Rectal resection; SR: Sigmoid resection; TPC: Total proctocolectomy.
Held-out centre
Evaluation on
Task
Balanced accuracy (%)
F1 (%)
Precision (%)
Recall (%)
HeiCo
Held-out
Phase
48.01
43.83
45.51
48.01
Step
23.42
16.74
20.88
18.45
Seen centres
Phase
66.12
65.10
64.54
66.12
Step
40.37
40.44
51.15
40.37
Centre 1
Held-out
Phase
45.32
43.46
47.50
45.32
Step
21.99
18.57
29.44
21.31
Appendix
Table S3 : Full leave-one-centre-out (LOCO) cross-validation results. For each fold (held-out centre), metrics are reported separately on the held-out centre and on the four seen centres. All values are macro-averaged (%).
Fig. S10 : Leave-one-centre-out (LOCO) performance heatmaps for phase (A) and step (B) recognition. Three conditions (rows): LOCO held-out, LOCO seen centres, global model. Five centres (columns). Colour: macro F1 (%), green = high, red = low.
Early screening via colonoscopy is critical for colon cancer prevention, yet developing robust AI systems for this domain is hindered by the lack of densely annotated, long-sequence video datasets. Existing datasets predominantly focus on single-class polyp detection and lack the rich spatial, temporal, and linguistic annotations required to evaluate modern Multimodal Large Language Models (MLLMs). To address this critical gap, we introduce Colon-Bench, generated via a novel multi-stage agentic workflow. Our pipeline seamlessly integrates temporal proposals, bounding-box tracking, AI-driven visual confirmation, and human-in-the-loop review to scalably annotate full-procedure videos. The resulting verified benchmark is unprecedented in scope, encompassing 528 videos, 14 distinct lesion categories (including polyps, ulcers, and bleeding), over 300,000 bounding boxes, 213,000 segmentation masks, and 133,000 words of clinical descriptions. We utilize Colon-Bench to rigorously evaluate state-of-the-art MLLMs across lesion classification, Open-Vocabulary Video Object Segmentation (OV-VOS), and video Visual Question Answering (VQA). The MLLM results demonstrate surprisingly high localization performance in medical domains compared to SAM-3. Finally, we analyze common VQA errors from MLLMs to introduce a novel "colon-skill" prompting strategy, improving zero-shot MLLM performance by up to 9.7% across most MLLMs. The dataset and the code are available at https://abdullahamdi.com/colon-bench .
Abdullah Hamdi, Changchun Yang, Xin Gao
King Abdullah University of Science and Technology (KAUST)
Intelligent operating rooms require accurate awareness of surgical workflow for better intra-operative decisions, teamwork, and surgical documentation. Therefore, accurately recognizing surgical phase during surgery is crucial. However, previous artificial intelligence-based methods provide fluctuating predictions for the transitions, restricting their clinical use. To address this challenge, we propose a clinically meaningful framework that integrates a Swin Transformer and Temporal Transformer Encoder (TTE) for effective surgical phase recognition and a parallel YOLOv11s network for real-time surgical instrument localization. To avoid phase flickering and to produce stable workflow predictions, a causal temporal smoothing is applied. The proposed framework is tested on the Cholec80 dataset for the simultaneous recognition of the surgical phases and instruments. The overall phase recognition accuracy of this model was 92%, and the macro average F1-score was 0.86. This performance surpasses CNN based methods such as EndoNet (81.7%) and SV-RCNet (85.3%), as well as temporal convolutional and transformer-based approaches including TeCNO (88.6%), Trans-SVNet (90.9%), and LoViT (91.5%). Besides, the module of instrument detection achieved [email protected] = 0.863. The reliability of the results is confirmed, obtaining a high recognition rate of the workflows, and a high accuracy of the localization of the instruments. The proposed framework can serve as a foundation for intelligent surgical assistance, enriching situation awareness, and potential robotic surgery applications. Code will be made publicly available at: https://github.com/ac005sheekar/LapChol-Surgery-Swin-Cholec80
Sheekar Banerjee, Md. Shahabuddin, Md. Mahbub Hasan Akash +2
Department of Computer Science and Engineering, IUBAT—International University of Business Agriculture and Technology, 4 Embankment Drive Road, Sector-10, Uttara Model Town, Dhaka, 1230, Bangladesh. · Boston Children’s Hospital, Harvard Medical School, 401 Park Dr, Boston, USA.
Colorectal cancer (CRC) remains one of the leading causes of cancer-related mortality worldwide, predominantly arising from precancerous polyps. Accurate detection, segmentation, and endoscopic and histological classification of colorectal polyps are crucial for timely clinical intervention. In this study, we present PolypVision, a three-stage hierarchical deep learning framework that sequentially performs: (Stage 1) binary classification of polyps as adenomatous or hyperplastic, with simultaneous Paris and JNet classification, using EfficientNetV2-M with Focal Loss; (Stage 2) polyp segmentation with recommended resection method using a UNet++ decoder with the Stage 1 backbone as encoder, optimized with Dice and BCE losses; and (Stage 3) adenoma subtype classification (tubular, tubulovillous, villous) using EfficientNetV2-M with transfer learning from Stage 2. Evaluated on three public datasets -- PolypGen, Kvasir-SEG, and CVC-ClinicDB -- PolypVision achieves an AUC of approximately 0.99 for frame classification and a detection mAP@50 of 94.4% on Kvasir-SEG, outperforming or matching state-of-the-art methods. Gradient-weighted Class Activation Maps (Grad-CAM) confirm that the model attends to clinically relevant lesion features. The framework is device-independent, operating across diverse endoscopic imaging systems without hardware-specific adaptation. These results demonstrate that a hierarchical, transfer-learning-driven pipeline with task-specific loss functions offers a robust, device-independent, and clinically meaningful approach to automated colorectal polyp analysis. PolypVision is freely available as a web application at https://polypvision.com, a DataBioX initiative, with a free usage tier open to all users.
Hamidreza Bolhasani, Hamidreza Rastad, Amir Mohammad Akbari +4
DataBioX Research, Tehran, Iran · Iran University of Science and Technology, Tehran, Iran · Tehran University of Medical Sciences, Tehran, Iran +1