Data-Efficient Surgical Phase Segmentation in Small-Incision Cataract Surgery: A Controlled Study of Vision Foundation Models

Lincoln Spencer; Song Wang; Chen Chen

cs.CVApr 12, 2026

Data-Efficient Surgical Phase Segmentation in Small-Incision Cataract Surgery: A Controlled Study of Vision Foundation Models

Authors: Lincoln Spencer, Song Wang, Chen Chen

Organizations: Institute of Artificial Intelligence, University of Central Florida

Abstract

Surgical phase segmentation is central to computer-assisted surgery, yet robust models remain difficult to develop when labeled surgical videos are scarce. We study data-efficient phase segmentation for manual small-incision cataract surgery (SICS) through a controlled comparison of visual representations. To isolate representation quality, we pair each visual encoder with the same temporal model (MS-TCN++) under identical training and evaluation settings on SICS-155 (19 phases). We compare supervised encoders (ResNet-50, I3D) against large self-supervised foundation models (DINOv3, V-JEPA2), and use a cached-feature pipeline that decouples expensive visual encoding from lightweight temporal learning. Foundation-model features improve segmentation performance in this setup, with DINOv3 ViT-7B achieving the best overall results (83.4% accuracy, 87.0 edit score). We further examine cataract-domain transfer using unlabeled videos and lightweight adaptation, and analyze when it helps or hurts. Overall, the study indicates strong transferability of modern vision foundation models to surgical workflow understanding and provides practical guidance for low-label medical video settings. The project website is available at: https://sl2005.github.io/DataEfficient-sics-phase-seg/

Figures & tables

Figure 1: Controlled study pipeline. Encoders are frozen after optional cataract-domain SSL continuation and LoRA adaptation, features are cached once, and the same MS-TCN++ head is trained for all encoders.

Figure 2: SICS-155 phase distribution at frame level. The long-tail imbalance motivates representation learning approaches that remain robust under limited labels.

Encoder	Pretraining / Source	Modality	Input	Clip	Feat dim	Params
ResNet-50 (ImageNet)	supervised (ImageNet-1K)	2D image	$224$	–	$2048$	$25.6$ M
I3D-R50 (Kinetics)	supervised (Kinetics-400)	3D video	$224$	$16$	$2048$	$28.0$ M
DINOv3 ViT-B/16	self-supervised (LVD-1689M)	2D image	$224$	–	$768$	$85.7$ M
DINOv3 ViT-L/16	self-supervised (LVD-1689M)	2D image	$224$	–	$1024$	$300$ M
DINOv3 ViT-7B/16	self-supervised (LVD-1689M)	2D image	$224$	–	$4096$	$6.716$ B
V-JEPA2 ViT-L	self-supervised video (Meta)	3D video	$256$	$64$	$1024$	$326.0$ M

Table 1: Feature extractor backbones used in our pipeline. “Feat dim” is the per-frame feature dimensionality consumed by MS-TCN++. Parameter counts are for the encoder backbone only; V-JEPA2/ResNet/I3D/DINOv3-B are computed from instantiated architectures (without loading weights), while DINOv3-L and DINOv3-7B are from model releases. V-JEPA2 uses stride-4 clip extraction with linear interpolation to full frame rate.

Model	Accuracy	F1 Score (macro)	Edit Score	PR-AUC	F1@10	F1@25	F1@50	mIoU (Jaccard)
ResNet-50 (ImageNet)	75.7 $\pm$ 1.1	67.7 $\pm$ 1.4	79.3 $\pm$ 1.5	71.8 $\pm$ 1.2	79.8 $\pm$ 1.1	75.3 $\pm$ 2.0	62.2 $\pm$ 2.5	53.5 $\pm$ 1.6
I3D	79.8 $\pm$ 0.9	71.7 $\pm$ 1.4	82.6 $\pm$ 2.5	75.7 $\pm$ 1.1	83.2 $\pm$ 2.5	79.2 $\pm$ 3.2	68.9 $\pm$ 2.6	58.3 $\pm$ 1.8
DINOv3 ViT-B/16	78.4 $\pm$ 0.7	71.2 $\pm$ 0.9	82.5 $\pm$ 1.7	75.0 $\pm$ 1.2	83.8 $\pm$ 1.1	79.9 $\pm$ 1.8	68.2 $\pm$ 2.0	57.0 $\pm$ 2.0
DINOv3 ViT-L/16	82.2 $\pm$ 1.3	75.8 $\pm$ 2.1	85.7 $\pm$ 1.7	78.5 $\pm$ 1.3	87.2 $\pm$ 1.8	84.4 $\pm$ 2.5	73.7 $\pm$ 4.4	62.4 $\pm$ 3.6
DINOv3 ViT-7B/16	83.4 $\pm$ 1.4	76.5 $\pm$ 2.0	87.0 $\pm$ 1.5	79.4 $\pm$ 1.7	88.0 $\pm$ 2.2	84.5 $\pm$ 2.6	75.0 $\pm$ 1.8	64.1 $\pm$ 3.2
V-JEPA2 ViT-L	77.9 $\pm$ 1.0	69.9 $\pm$ 0.5	83.2 $\pm$ 1.0	74.2 $\pm$ 0.6	83.0 $\pm$ 0.9	78.2 $\pm$ 1.5	66.6 $\pm$ 1.7	55.6 $\pm$ 2.3

Table 2: 5-Fold Cross Validation Results on SICS-155 (mean

\pm

std, %). Best per metric is bolded and lightly shaded.

Figure 3: Qualitative phase segmentation results for BL_0172 and BN_0036 (colors defined by the legend). For each video, rows show ground truth, the supervised baseline (ResNet-50 features), a representative foundation-model alternative (V-JEPA2 ViT-L features), and the best-performing foundation model (DINOv3 ViT-7B features), all with the same temporal head. The bottom strip in each column shows aligned keyframes with sampled timepoints and corresponding ground-truth/predicted phases. The callout highlights a choppy baseline region that is reduced with stronger representations.

Encoder	Acc (FT)	$\Delta$ Acc	$\Delta$ Macro-F1	$\Delta$ Edit
DINOv3 ViT-B/16	81.27	+2.86	+2.87	+1.66
DINOv3 ViT-L/16	81.26	-0.93	-1.82	-0.69
V-JEPA2 ViT-L	71.53	-6.37	-8.08	-3.31

Table 3: Preliminary CataractFT transfer to SICS-155. Deltas are measured on SICS-155 (5-fold CV) relative to the corresponding base encoder; Acc (FT) is the post-fine-tuning mean accuracy on SICS-155.

Encoder	Variant	Accuracy	Macro-F1	Edit	PR-AUC	F1@50	mIoU
DINOv3 ViT-B/16	Base	78.41 $\pm$ 0.71	71.23 $\pm$ 0.87	82.55 $\pm$ 1.71	75.04 $\pm$ 1.23	68.20 $\pm$ 1.96	57.01 $\pm$ 1.96
	SSL-only (no LoRA)	78.26 $\pm$ 1.86	70.45 $\pm$ 2.19	85.50 $\pm$ 1.61	74.74 $\pm$ 1.66	68.83 $\pm$ 3.47	57.31 $\pm$ 2.20
	CataractFT + LoRA	81.27 $\pm$ 1.84	74.10 $\pm$ 2.23	84.21 $\pm$ 2.58	77.03 $\pm$ 2.02	70.58 $\pm$ 3.02	61.46 $\pm$ 2.61
DINOv3 ViT-L/16	Base	82.19 $\pm$ 1.28	75.80 $\pm$ 2.07	85.71 $\pm$ 1.74	78.50 $\pm$ 1.31	73.74 $\pm$ 4.44	62.40 $\pm$ 3.61
	SSL-only (no LoRA)	80.80 $\pm$ 1.31	72.56 $\pm$ 1.33	86.82 $\pm$ 0.56	77.08 $\pm$ 0.84	70.74 $\pm$ 1.51	59.58 $\pm$ 3.20
	CataractFT + LoRA	81.26 $\pm$ 0.91	73.98 $\pm$ 1.02	85.02 $\pm$ 2.04	77.93 $\pm$ 0.96	71.84 $\pm$ 3.22	61.23 $\pm$ 2.55

Table 4: Side-by-side CataractFT ablation for encoders with all three variants available on SICS-155 (5-fold CV, mean

\pm

std, %). Base uses the encoder as a standard frozen feature extractor, SSL-only continues from the same cataract-domain checkpoint without LoRA, and CataractFT + LoRA applies the full adaptation pipeline. Best within each encoder block is bolded.

Figure 4: Per-phase change from CataractFT + LoRA relative to SSL-only continuation from the same cataract-domain checkpoint (no LoRA). Heatmap entries report delta points for classwise F1 and IoU for DINOv3 ViT-B and ViT-L. Positive values indicate phases that benefit from LoRA adaptation; negative values indicate phases where the checkpoint-only variant remains better.

Explore similar work

Large-Small Model Collaboration for Zero-Shot Surgical Phase Recognition
Aug 24, 2026Yiyi Zhang, Ying Zheng, Wenxin Fan +5Surgical Video Understanding Self-Training
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China · Department of Surgery, The Chinese University of Hong Kong, Hong Kong, China +1
SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos
Feb 5, 2026Jinlin Wu, Felix Holm, Chuxi Chen +18Joint-Embedding Predictive Architecture Surgical Video Understanding
Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany +7
Scaling Video Pretraining for Surgical Foundation Models
Mar 31, 2026Sicheng Lu, Zikai Xiao, Jianhui Wei +7Surgical Video Understanding Video Representation Learning
The Johns Hopkins University, United States · Zhejiang University, China · Zhejiang University-University of Illinois Urbana-Champaign Institute, China +4

Aug 24, 2026J/K move · Enter open · S save

Large-Small Model Collaboration for Zero-Shot Surgical Phase Recognition

Yiyi Zhang, Ying Zheng, Wenxin Fan, Yu Zhu+4

Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China · Department of Surgery, The Chinese University of Hong Kong, Hong Kong, China+1

Surgical Video Understanding Self-Training Temporal Action Detection Medical Imaging Foundation Models

Open paper AskPDF arXiv

Task-specific lightweight models for surgical phase recognition excel at capturing temporal dynamics but generalize poorly under domain shift. Conversely, surgical foundation models (FMs) offer superior transferability via large-scale pretraining, yet their lack of explicit temporal modeling often yields temporally inconsistent predictions, leading to degraded performance. To exploit the complementary strengths of both paradigms, we propose Large-Small Temporal adaptation (LaST), a novel large-small collaborative framework that enables zero-shot adaptation to unseen clinical domains. In LaST, the FM initiates the pipeline by generating frame-level phase priors that serve as initial weak supervision. To effectively utilize these noisy phase priors, we introduce an iterative temporal refinement scheme that integrates dynamic quality control to filter reliable predictions and dual-model cross-learning to mitigate confirmation bias. Simultaneously, the lightweight model leverages its intrinsic temporal modeling ability to progressively correct inconsistent predictions and enhance overall accuracy across iterations. At the end, a cycle replay strategy is employed to close the loop: the refined, more accurate predictions are utilized as upgraded supervision signals for the subsequent iterations, fostering a self-reinforcing evolution of both label quality and model capability. Extensive experiments demonstrate that LaST achieves robust adaptation to unseen domains for zero-shot surgical phase recognition, outperforming the baseline (PeskaVLP) by 24.85%-43.17% in accuracy and even surpassing fully supervised linear probing and several state-of-the-art few-shot approaches. Codes will be released at https://github.com/YIYIZH/LaST.

Abstract

Explore similar work

Large-Small Model Collaboration for Zero-Shot Surgical Phase Recognition

SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos

Scaling Video Pretraining for Surgical Foundation Models