Artificial Intelligence for Workflow Analysis in Colorectal Surgery: A Multicentric, Cross-Procedural Development and Generalization Study

Pietro Mascagni; Julia Alekseenko; Pooja P Jain; Marta Goglia; Andrea Balla; Ludovica Baldari; Gianfranco Silecchia; Claudio Fiorillo

cs.CVAug 20, 2026

Artificial Intelligence for Workflow Analysis in Colorectal Surgery: A Multicentric, Cross-Procedural Development and Generalization Study

Authors: Pietro Mascagni, Julia Alekseenko, Pooja P Jain, Marta Goglia, Andrea Balla, Ludovica Baldari, Gianfranco Silecchia, Claudio Fiorillo,

Organizations: IHU Strasbourg, Strasbourg, France · Surgical Video Analysis, Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy · Università Cattolica del Sacro Cuore, Rome, Italy · University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France ·

Abstract

Minimally invasive colorectal surgeries (MIS-CRS) are characterised by significant variability and inconsistent outcomes. ColoWorkflow, a tool for the video-based assessment (VBA) of MIS-CRS workflow, was recently validated. However, manual VBA is time-consuming, limiting implementation. This study presents AI-ColoWorkflow, a deep learning model for automated surgical workflow analysis across MIS-CRS. Operative videos of MIS-CRS were collected from 4 centres and a publicly available dataset. Phases and steps were manually annotated according to ColoWorkflow. A deep learning model combining a fine-tuned DINOv3 vision transformer for per-frame visual feature extraction with a hierarchical multi-stage temporal convolutional network was jointly optimized for phase and step recognition. The model trained on pooled multicentric data, namely AI-ColoWorkflow was compared against centre-specific and procedure-specific models on a held-out test set. The following metrics were used for evaluation: macro F1 score, balanced accuracy, precision, and recall. AI-ColoWorkflow achieved a macro F1 of 73.01% $\pm$ 10.27 (balanced accuracy 73.43%) for phase recognition and 39.82% $\pm$ 7.06 (balanced accuracy 38.65%) for step recognition. The global model outperformed centre- and procedure-specific models in most experiments except procedure-specific step recognition. In the generalization analysis, mean F1 was 48.42% for phase recognition. AI-ColoWorkflow can reliably recognize MIS-CRS phases. A single model trained on pooled, multicentric, multi-procedural data generalises at least as well as and often better than centre- or procedure-specific models for phase recognition in MIS-CRS, while procedure-specific step models retain advantages for certain procedure types, motivating hybrid training strategies for future surgical AI development.

Figures & tables

Fig. 1 : AI-ColoWorkflow system. Two sequential stages: (1) spatial feature extraction with a fine-tuned DINOv3 vision transformer; (2) temporal modelling with the SAHC hierarchical multi-stage TCN, jointly trained for phase and step recognition. d: feature dimension; SAHC: Segment-Attentive Hierarchical Consistency network; T: time; TCN: Temporal Convolutional Network

Centre	Surgery type	Train	Val	Test	Subtotal
Centre 1	LH	2	1	2	5
	RH	1	0	0	1
	Sub	3	1	2	6
Centre 2	LH	1	1	0	2
	RH	3	1	1	5
	RR	1	0	0	1

Table 1: Dataset splits by centre and procedure type. LH: Left hemicolectomy; RH: Right hemicolectomy; RR: Rectal resection; SR: Sigmoid resection; TPC: Total proctocolectomy.

Surgical workflow item	Prec. (%)	Rec. (%)	F1 (%)
Phase recognition
P1. Port placement and abdomen exploration	76.96 $\pm$ 16.02	69.36 $\pm$ 15.10	72.96 $\pm$ 10.68
P2. Vascular dissection and ligation, mesocolon/mesorectum dissection, additional lymphadenectomy	80.82 $\pm$ 13.39	72.77 $\pm$ 16.48	76.58 $\pm$ 12.87
P3. Colon and/or rectum mobilisation	69.83 $\pm$ 9.62	75.55 $\pm$ 16.64	72.58 $\pm$ 10.64
P4. Colorectal transection	64.07 $\pm$ 30.09	64.16 $\pm$ 20.42	64.11 $\pm$ 19.07
P5. Anastomosis	69.50 $\pm$ 27.02	82.08 $\pm$ 27.81	75.27 $\pm$ 23.85

Table 2: AI-ColoWorkflow performance across phases and steps. All values shown as global

\pm

std, where global is computed on concatenated test frames and

\pm

is the standard deviation of per-video scores across individual test videos. P7 (Preplanned additional procedures) and S25 (Ileoanal pouch preparation - intraabdominal part) were not evaluated since these were not represented in the test set.

Fig. 2 : Comparison of AI-ColoWorkflow performance versus procedure and centre-specific models. Numbers and *** in green suggest superiority of AI-ColoWorkflow while those in red suggest superiority of specific models. LH: Left hemicolectomy; RH: Right hemicolectomy; RR: Rectal resection; SR: Sigmoid resection; TPC: Total proctocolectomy.

	Phase F1 (%)		Step F1 (%)
Held-out	Held-out	Seen	Held-out	Seen
HeiCo	43.83	65.10	16.74	40.44
Centre 1	43.46	60.63	18.57	39.28
Centre 2	37.49	60.05	23.38	42.24
Centre 3	51.56	60.62	18.11	36.67
Centre 4	65.78	61.55	32.91	44.42

Table 3: Leave-one-centre-out (LOCO) cross-validation results. Macro F1 (%) on the held-out centre and the remaining seen centres per fold, compared with the globally trained AI-ColoWorkflow model. All values are macro averaged.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Specification
Hardware
GPU	NVIDIA V100 (32 GB)
Spatial encoder
Architecture	DINOv3 vision transformer (fine-tuned)
Input resolution	224x224
Temporal model

Appendix

Table S1 : Hardware and training configuration.

Fig. S1 : Frame-level dataset composition by centre and surgery type within each of the training, validation, and test splits. (A) Donut chart showing overall train/validation/test frame proportions (462k total frames in total). (B) Stacked horizontal bars showing the percentage of frames contributed by each centre within each split. (C) Stacked bars showing the percentage of frames contributed by each surgery type within each split. Fr: frames; LH: Left hemicolectomy; RH: Right hemicolectomy; RR: Rectal resection; SR: Sigmoid resection; TPC: Total proctocolectomy.

Fig. S2 : Per-video duration variability by procedure type, centre, and split at 1 frame per second. Strip plots showing individual video frame counts grouped by (A) surgery type, (B) centre, and (C) split. Horizontal bars mark group means. LH: Left hemicolectomy; RH: Right hemicolectomy; RR: Rectal resection; SR: Sigmoid resection; TPC: Total proctocolectomy.

Fig. S3 : Normalised frame-level class distributions for phases and steps across splits. (A) Grouped bar chart of phase frame counts normalised by split total (% of split frames) for P1–P9; P7 (Preplanned additional procedures) is absent from the test set and was excluded from evaluation. (B) Grouped bar chart of normalised step frame counts (% of split frames) for all 33 evaluated steps (S25 is an unused gap), illustrating the long-tail imbalance that motivates macro-averaged F1 as the primary metric. Complete phases (P) and steps (S) definitions can be found in Table 2 .

Fig. S4 : Typical frames by centre and phase. Each cell shows a representative frame for the (centre, phase) pair. Phase 9 (Extracorporeal procedures) is blurred to comply with institutional privacy requirements; grey cells indicate no available frames.

Fig. S5 : Single-task versus joint training. Joint training improved macro F1 by +2.8% for phase (70.2% to 73.0%) and +1.7% for step (38.1% to 39.8%) relative to separately trained single-task models, with gains consistent across nearly all subgroups. (A) Phase recognition; (B) step recognition. Each row is a cohort (overall, 5 centres, 5 procedure types). Circles = single-task model F1; diamonds = joint model (AI-ColoWorkflow) F1. Green connectors indicate joint training improved performance; red indicate regression. Annotations show $\Delta$ F1 (joint $-$ single-task).

Fig. S6 : Predictions for a representative video. (A) Ground-truth phase timeline; (B) predicted phase timeline, with model confidence (softmax maximum probability, black line); (C) ground-truth step timeline; (D) predicted step timeline with model confidence, grey regions indicate frames with no step ground-truth annotation; (E) per-video F1, Jaccard, balanced accuracy, precision, and recall.

Fig. S7 : Confusion matrices for phase (A) and step (B) recognition. Row-normalised matrices: each cell shows the fraction of true-class frames predicted as each target class; diagonal = correct prediction. The phase confusion matrix is computed from model outputs; the step confusion matrix is approximated from per-class recall and support (i.e., the number of samples). P8 (Unplanned procedures) shows the most diffuse phase confusion. Step classes with near-zero recall (S20, S23, S29, S32) appear as near-empty rows.

Fig. S8 : Phase and step F1 score summary. (A) Phase classes (n = 8 evaluated); (B) step classes (n = 33 evaluated), sorted by F1 in descending order. Bars are coloured by F1 tier (green $\geq$ 70%, blue 50–70%, orange 30–50%, red <30%). The dashed vertical line marks the macro mean.

Fig. S9 : AI-ColoWorkflow performance across evaluation cohorts. (A) Single-task (i.e., phase or step recognition) training: scatter of phase against step macro F1 score. (B) Joint (i.e., phase and step recognition) training: scatter of phase against step macro F1 score. For A and B, circles represent centres (sized by frame count) while squares represent procedure types, star represents overall. (C) Phase macro F1 versus log-transformed frame count per centre. (D) Step macro F1 versus log-transformed frame count per centre. For C and D, circles represent single-task training while diamonds represent joint training; areas are proportional to frame count.

		Phase F1 (%)			Step F1 (%)
Cohort	Type	Global	Spec.	$\Delta$	Global	Spec.	$\Delta$
HeiCo	Centre	73.53	73.18	+0.4	33.96	35.42	$-$ 1.5
Centre 1	Centre	59.19	50.11	+9.1	24.72	15.50	+9.2
Centre 2	Centre	72.78	53.86	+18.9	31.16	15.21	+15.9
Centre 3	Centre	79.34	80.31	$-$ 1.0	38.86	22.82	+16.0
Centre 4	Centre	70.61	56.34	+14.3	44.81	21.44	+23.4

Appendix

Table S2 : AI-ColoWorkflow (global) versus procedure- and centre-specific models. Macro-averaged F1 score (%) on the test set.

\Delta

= global - specialised. LH: Left hemicolectomy; RH: Right hemicolectomy; RR: Rectal resection; SR: Sigmoid resection; TPC: Total proctocolectomy.

Held-out centre	Evaluation on	Task	Balanced accuracy (%)	F1 (%)	Precision (%)	Recall (%)
HeiCo	Held-out	Phase	48.01	43.83	45.51	48.01
	Held-out	Step	23.42	16.74	20.88	18.45
	Seen centres	Phase	66.12	65.10	64.54	66.12
	Seen centres	Step	40.37	40.44	51.15	40.37
Centre 1	Held-out	Phase	45.32	43.46	47.50	45.32
	Held-out	Step	21.99	18.57	29.44	21.31

Appendix

Table S3 : Full leave-one-centre-out (LOCO) cross-validation results. For each fold (held-out centre), metrics are reported separately on the held-out centre and on the four seen centres. All values are macro-averaged (%).

Fig. S10 : Leave-one-centre-out (LOCO) performance heatmaps for phase (A) and step (B) recognition. Three conditions (rows): LOCO held-out, LOCO seen centres, global model. Five centres (columns). Colour: macro F1 (%), green = high, red = low.

Surgical Video Understanding

Surgical video understanding uses computer vision and multimodal AI to interpret procedures, instruments, actions, and events over time.

See all 118 papers

Colon-Bench: An Agentic Workflow for Scalable Dense Lesion Annotation in Full-Procedure Colonoscopy Videos
Mar 26, 2026Abdullah Hamdi, Changchun Yang, Xin GaoVideo Object Segmentation Medical Image Analysis
King Abdullah University of Science and Technology (KAUST)
Shifted Window Temporal Transformer for YOLO-Guided Phase Recognition in Laparoscopic Cholecystectomy Surgery
Oct 26, 2026Sheekar Banerjee, Md. Shahabuddin, Md. Mahbub Hasan Akash +2Surgical Video Understanding Laparoscopic Surgery
Department of Computer Science and Engineering, IUBAT—International University of Business Agriculture and Technology, 4 Embankment Drive Road, Sector-10, Uttara Model Town, Dhaka, 1230, Bangladesh. · Boston Children’s Hospital, Harvard Medical School, 401 Park Dr, Boston, USA.
PolypVision: A Three-Stage Hierarchical Deep Learning Framework for Classification and Segmentation of Colorectal Polyps
Aug 11, 2026Hamidreza Bolhasani, Hamidreza Rastad, Amir Mohammad Akbari +4Medical Image Classification Polyp Segmentation
DataBioX Research, Tehran, Iran · Iran University of Science and Technology, Tehran, Iran · Tehran University of Medical Sciences, Tehran, Iran +1

Mar 26, 2026J/K move · Enter open · S save

Colon-Bench: An Agentic Workflow for Scalable Dense Lesion Annotation in Full-Procedure Colonoscopy Videos

Abdullah Hamdi, Changchun Yang, Xin Gao

King Abdullah University of Science and Technology (KAUST)

Video Object Segmentation Medical Image Analysis Multimodal Model Evaluation Medical Image Benchmarks

Open paper AskPDF arXiv

Early screening via colonoscopy is critical for colon cancer prevention, yet developing robust AI systems for this domain is hindered by the lack of densely annotated, long-sequence video datasets. Existing datasets predominantly focus on single-class polyp detection and lack the rich spatial, temporal, and linguistic annotations required to evaluate modern Multimodal Large Language Models (MLLMs). To address this critical gap, we introduce Colon-Bench, generated via a novel multi-stage agentic workflow. Our pipeline seamlessly integrates temporal proposals, bounding-box tracking, AI-driven visual confirmation, and human-in-the-loop review to scalably annotate full-procedure videos. The resulting verified benchmark is unprecedented in scope, encompassing 528 videos, 14 distinct lesion categories (including polyps, ulcers, and bleeding), over 300,000 bounding boxes, 213,000 segmentation masks, and 133,000 words of clinical descriptions. We utilize Colon-Bench to rigorously evaluate state-of-the-art MLLMs across lesion classification, Open-Vocabulary Video Object Segmentation (OV-VOS), and video Visual Question Answering (VQA). The MLLM results demonstrate surprisingly high localization performance in medical domains compared to SAM-3. Finally, we analyze common VQA errors from MLLMs to introduce a novel "colon-skill" prompting strategy, improving zero-shot MLLM performance by up to 9.7% across most MLLMs. The dataset and the code are available at https://abdullahamdi.com/colon-bench .

Abstract

Surgical Video Understanding

Explore similar work

Colon-Bench: An Agentic Workflow for Scalable Dense Lesion Annotation in Full-Procedure Colonoscopy Videos

Shifted Window Temporal Transformer for YOLO-Guided Phase Recognition in Laparoscopic Cholecystectomy Surgery

PolypVision: A Three-Stage Hierarchical Deep Learning Framework for Classification and Segmentation of Colorectal Polyps