Automated recognition of surgical phases and steps is a fundamental capability for intraoperative decision support, workflow automation, and skill assessment in robotic-assisted surgery. Existing approaches either depend on large-scale annotated surgical datasets or require expensive domain-specific pretraining on thousands of labeled videos, limiting their practical deployability across diverse robotic platforms and clinical environments. In this work, we propose TASOT (Text-Augmented Action Segmentation Optimal Transport), an annotation-free framework for surgical temporal segmentation that requires no task-specific annotations or surgical-domain pretraining. TASOT extends the Action Segmentation Optimal Transport (ASOT) formulation by incorporating temporally aligned textual descriptions generated directly from the input video, fusing visual and semantic cues within a unified unbalanced Gromov-Wasserstein optimal transport objective. Visual representations are extracted using DINOv3, while temporal captions produced by a vision-language model are encoded via CLIP and temporally aligned to individual frames, providing complementary semantic structure to the transport cost. We evaluate TASOT on three public surgical datasets and four benchmark settings spanning laparoscopic and robotic procedures, showing substantial improvements over the strongest zero-shot baselines: +18.9 F1 on Cholec80, +33.7 on AutoLaparo, +23.7 on StrasByPass70, and +4.5 on BernByPass70. These results suggest that fine-grained surgical workflow understanding in robotic settings can be achieved without manual training annotations or surgical-specific pretraining pipelines, offering a promising alternative for real-world robotic surgical systems.
Figures & tables
Fig. 1: Overview of TASOT. Surgical videos are divided into temporal windows and processed by a vision-language model to generate structured temporal captions. Visual features extracted with DINOv3 and temporally aligned textual features encoded with CLIP are integrated within TASOT through a weighted multimodal transport cost, enabling annotation-free surgical temporal segmentation.
Method
Cholec80
AutoLaparo
BernByPass70
StrasByPass70
Phase F1
Phase F1
Phase F1
Step F1
Phase F1
Step F1
Zero-shot Methods
MIL-NCE [ 5 ]
7.3
7.9
2.1
—
3.1
—
CLIP-SVL [ 8 ]
19.6
16.2
7.1
—
8.6
—
SurgVLP [ 7 ]
24.4
16.6
7.2
—
6.9
—
HecVL [ 17 ]
26.3
18.9
13.6
—
18.3
—
TABLE I: Phase and step recognition results on Cholec80 [ 3 ] , AutoLaparo [ 11 ] , and MultiBypass140 [ 12 ] . TASOT is compared with zero-shot surgical video-language models and the annotation-free ASOT baseline. Best results are shown in bold . Step-level F1 is not reported for zero-shot methods because their pretraining protocols do not use step-level supervision.
Model
BernByPass70
StrasByPass70
Phase F1
Step F1
Phase F1
Step F1
Video-only (DINOv3 + ASOT)
27.9
20.3
51.3
27.9
Text-only
18.2
7.98
26.3
9.39
Feature concatenation
27.4
14.3
50.4
21.2
TASOT (multimodal cost)
27.1
23.0
52.3
30.7
TABLE II: Component ablation of TASOT on MultiBypass140. Video-only uses DINOv3 features within ASOT, Text-only uses aligned caption features only, and Feature concatenation replaces cost-level fusion with feature-space fusion. TASOT (multimodal cost) denotes the full model. Best results are shown in bold .
β
BernByPass70
StrasByPass70
Phase F1
Step F1
Phase F1
Step F1
1.0 (visual-only)
27.9
20.3
51.3
27.9
0.8
27.1
23.0
52.3
30.7
0.5
25.6
12.93
42.0
16.17
0.0 (text-only)
18.2
7.98
26.3
9.39
TABLE III: Sensitivity to the multimodal weight β on MultiBypass140. β=1.0 is visual-only, β=0.0 is text-only, and β=0.8 is used as the default setting in the remaining experiments. Best results are shown in bold .
Visual Encoder
Text Encoder
BernByPass70
StrasByPass70
Phase F1
Step F1
Phase F1
Step F1
DINOv3
CLIP
27.1
23.0
52.3
30.7
DINOv3
Gemma
26.7
22.4
49.2
29.9
DINOv3
Gemma ∥ CLIP
26.6
22.1
50.3
30.6
TABLE IV: Text encoder analysis on MultiBypass140 using DINOv3 visual features within TASOT. We compare CLIP, Gemma, and their feature-space concatenation for the textual branch. ∥ denotes feature concatenation. Best results are shown in bold .
Method
BernByPass70
StrasByPass70
Phase F1
Step F1
Phase F1
Step F1
Annotation-free Methods
TASOT
27.1
23.0
52.3
30.7
TASOT ( k -specific, oracle)
49.7
48.8
56.9
52.4
Supervised Methods
TeCNO [ 12 ]
59.2
47.5
80.7
58.1
TABLE V: Effect of video-specific cluster counts on MultiBypass140. The k -specific setting is an oracle analysis that uses the number of active classes per video and is not available in practical deployment. Best annotation-free results are shown in bold .
Fig. 2: Qualitative results. Ground-truth (GT) and TASOT predictions on MultiByPass140. Phase-level segmentations are temporally coherent with well-aligned boundaries. Step-level segmentation is more challenging due to finer temporal granularity and a fixed cluster count K that may exceed the number of active steps in a given video.
Department of Automation, Tsinghua University, Beijing, China · Surgical Planning Laboratory, Brigham and Women’s Hospital, Harvard Medical School, United States
Center for Humanoid Research, Korea Institute of Science and Technology, Seoul, Republic of Korea · School of Electrical and Electronic Engineering, Yonsei University, Seoul, Republic of Korea